2024-04-21T08:21:50.082 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-21T08:21:50.083 DEBUG:teuthology.run:Teuthology command: teuthology --name leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi --description fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iozone}} --verbose --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512 -- /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512/orig.config.yaml 2024-04-21T08:21:50.144 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T08:21:50.229 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512 branch: wip-lusov-quiescer description: fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iozone}} email: null first_in_suite: false job_id: '7666512' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 4f6bc5273643aaf5660909e85400647a8951c10b owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7599 sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e sleep_before_teardown: 0 subset: 111/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa suite_relpath: qa suite_repo: https://github.com/leonid-s-usov/ceph-ceph suite_sha1: 4f6bc5273643aaf5660909e85400647a8951c10b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNIFBuKaEXxjOb8QzHl0379u67RH2hUXtlQSyz+IhlA2fUAl9brqO3uP4Db7DxWwQxcl0KQ23aNOT2nUfs9Zs3U= smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlL+im8GNPnIN8QL3FOltbdTy6qnnD72uwVmDgWrJ/hzNgIck0YuGjsYJDDbmih146ixgWH//ZAH7V+oDaFMDY= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB7GJmLqkmdcWAPzZ4FhfC/FlnOrpfoB1ofRRCSGeX0eL47SV5maah1HDYDCVJiX6TnNmpa9w8JLe11slWsC+3g= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_08:12:55 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-21T08:21:50.229 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa; will attempt to use it 2024-04-21T08:21:50.230 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks 2024-04-21T08:21:50.230 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-21T08:21:50.232 INFO:teuthology.task.internal:Checking packages... 2024-04-21T08:21:50.257 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a630765c1f93c6e61d9b40c4b76437611d5ef29e' 2024-04-21T08:21:50.257 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-21T08:21:50.257 INFO:teuthology.packaging:ref: None 2024-04-21T08:21:50.257 INFO:teuthology.packaging:tag: None 2024-04-21T08:21:50.258 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-21T08:21:50.258 INFO:teuthology.packaging:sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:21:50.258 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-21T08:21:50.495 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3154.ga630765c 2024-04-21T08:21:50.496 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-21T08:21:50.513 INFO:teuthology.task.internal:no buildpackages task found 2024-04-21T08:21:50.513 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-21T08:21:50.558 INFO:teuthology.task.internal:Saving configuration 2024-04-21T08:21:50.580 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-21T08:21:50.589 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T08:21:50.615 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:16:23.096186', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNIFBuKaEXxjOb8QzHl0379u67RH2hUXtlQSyz+IhlA2fUAl9brqO3uP4Db7DxWwQxcl0KQ23aNOT2nUfs9Zs3U='} 2024-04-21T08:21:50.634 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:16:23.094181', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlL+im8GNPnIN8QL3FOltbdTy6qnnD72uwVmDgWrJ/hzNgIck0YuGjsYJDDbmih146ixgWH//ZAH7V+oDaFMDY='} 2024-04-21T08:21:50.658 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:16:23.097246', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB7GJmLqkmdcWAPzZ4FhfC/FlnOrpfoB1ofRRCSGeX0eL47SV5maah1HDYDCVJiX6TnNmpa9w8JLe11slWsC+3g='} 2024-04-21T08:21:50.659 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-21T08:21:50.666 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-21T08:21:50.666 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-21T08:21:50.666 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-21T08:21:50.666 INFO:teuthology.run_tasks:Running task console_log... 2024-04-21T08:21:50.780 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faebc6510d0>, signals=[15]) 2024-04-21T08:21:50.781 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-21T08:21:50.787 INFO:teuthology.task.internal:Opening connections... 2024-04-21T08:21:50.787 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:21:50.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:21:50.872 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-04-21T08:21:50.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:21:50.963 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-04-21T08:21:50.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:21:51.040 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-21T08:21:51.046 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-04-21T08:21:51.065 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-04-21T08:21:51.065 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-04-21T08:21:51.122 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-04-21T08:21:51.122 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-04-21T08:21:51.122 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-04-21T08:21:51.122 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:21:51.122 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-04-21T08:21:51.122 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:21:51.122 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:21:51.122 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-04-21T08:21:51.122 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:21:51.123 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:21:51.123 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:21:51.123 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:21:51.123 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:21:51.123 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:21:51.123 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-04-21T08:21:51.146 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-04-21T08:21:51.163 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-04-21T08:21:51.164 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-04-21T08:21:51.221 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-04-21T08:21:51.221 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9" 2024-04-21T08:21:51.221 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-04-21T08:21:51.221 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:21:51.221 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9" 2024-04-21T08:21:51.221 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:21:51.221 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:21:51.222 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-04-21T08:21:51.222 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:21:51.222 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:21:51.222 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:21:51.222 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:21:51.222 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:21:51.222 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:21:51.222 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-04-21T08:21:51.246 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-04-21T08:21:51.264 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-04-21T08:21:51.264 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-04-21T08:21:51.321 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-04-21T08:21:51.321 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:21:51.322 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:21:51.323 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-04-21T08:21:51.337 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-21T08:21:51.348 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-21T08:21:51.372 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-21T08:21:51.373 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:21:51.375 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:21:51.377 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:21:51.393 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-21T08:21:51.399 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-21T08:21:51.399 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:21:51.432 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:21:51.435 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:21:51.622 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-21T08:21:51.700 INFO:teuthology.run_tasks:Running task kernel... 2024-04-21T08:21:51.714 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-21T08:21:51.715 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-21T08:21:51.715 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-21T08:21:51.715 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:21:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:21:51.733 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-04-21T08:21:51.790 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-437.el9.x86_64 2024-04-21T08:21:51.791 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.14.0-437.el9.x86_64 2024-04-21T08:21:51.791 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-04-21T08:21:54.477 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 7.6 MB/s | 8.0 MB 00:01 2024-04-21T08:21:56.514 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-04-21T08:22:00.646 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-21T08:22:02.555 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-04-21T08:22:03.857 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-21T08:22:08.907 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-21T08:22:10.813 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-21T08:22:10.813 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-21T08:22:10.883 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-21T08:22:10.884 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-21T08:22:10.885 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-21T08:22:10.990 DEBUG:teuthology.orchestra.run.smithi094:> echo no | sudo yum reinstall kernel || true 2024-04-21T08:22:11.484 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:03 ago on Sun 21 Apr 2024 08:22:08 AM UTC. 2024-04-21T08:22:11.601 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-21T08:22:11.611 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-21T08:22:11.611 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-04-21T08:22:11.611 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-21T08:22:11.611 INFO:teuthology.orchestra.run.smithi094.stdout:Reinstalling: 2024-04-21T08:22:11.611 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T08:22:11.611 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:22:11.611 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-21T08:22:11.611 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-21T08:22:11.612 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:22:11.612 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 255 k 2024-04-21T08:22:11.612 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 0 2024-04-21T08:22:11.612 INFO:teuthology.orchestra.run.smithi094.stderr:Operation aborted. 2024-04-21T08:22:11.672 INFO:teuthology.orchestra.run.smithi094.stdout:Is this ok [y/N]: 2024-04-21T08:22:11.673 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum reinstall -y kernel || true 2024-04-21T08:22:12.162 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:04 ago on Sun 21 Apr 2024 08:22:08 AM UTC. 2024-04-21T08:22:12.277 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-21T08:22:12.277 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-21T08:22:12.277 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-04-21T08:22:12.278 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-21T08:22:12.278 INFO:teuthology.orchestra.run.smithi094.stdout:Reinstalling: 2024-04-21T08:22:12.278 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T08:22:12.278 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:22:12.278 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-21T08:22:12.278 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-04-21T08:22:12.278 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:22:12.278 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 255 k 2024-04-21T08:22:12.278 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 0 2024-04-21T08:22:12.278 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-21T08:22:12.701 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.7 MB/s | 255 kB 00:00 2024-04-21T08:22:12.701 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:22:12.701 INFO:teuthology.orchestra.run.smithi094.stdout:Total 602 kB/s | 255 kB 00:00 2024-04-21T08:22:12.717 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-21T08:22:12.874 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-21T08:22:12.874 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-21T08:22:12.878 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-21T08:22:12.879 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-21T08:22:13.002 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-21T08:22:13.085 INFO:teuthology.orchestra.run.smithi094.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T08:22:13.086 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:22:13.776 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:22:13.777 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T08:22:14.469 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:22:14.469 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:22:14.469 INFO:teuthology.orchestra.run.smithi094.stdout:Reinstalled: 2024-04-21T08:22:14.469 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-21T08:22:14.470 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:22:14.470 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-21T08:22:14.657 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-04-21T08:22:14.703 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-21T08:22:14.704 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-21T08:22:14.704 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-21T08:22:14.704 DEBUG:teuthology.parallel:result is None 2024-04-21T08:22:14.704 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-21T08:22:14.712 INFO:teuthology.task.internal:Creating test directory... 2024-04-21T08:22:14.712 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:22:14.714 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:22:14.717 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:22:14.736 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-21T08:22:14.745 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-21T08:22:14.812 INFO:teuthology.task.internal:Creating archive directory... 2024-04-21T08:22:14.812 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:22:14.814 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:22:14.818 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:22:14.937 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-21T08:22:14.944 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-21T08:22:14.944 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:22:14.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:22:14.961 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:22:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:22:14.977 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:22:14.979 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:22:15.003 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:22:15.022 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:22:15.039 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:22:15.041 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:22:15.061 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:22:15.081 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:22:15.082 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:22:15.083 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-21T08:22:15.094 INFO:teuthology.task.internal:Configuring sudo... 2024-04-21T08:22:15.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:22:15.096 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:22:15.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:22:15.153 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-21T08:22:15.163 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-21T08:22:15.163 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:22:15.166 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:22:15.192 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:22:15.210 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:22:15.266 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:22:15.310 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:22:15.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:22:15.430 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:22:15.430 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:22:15.495 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:22:15.537 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:22:15.587 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:22:15.673 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:22:15.713 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:22:15.713 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:22:15.779 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:22:15.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:22:15.879 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:22:15.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:22:16.004 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:22:16.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:22:16.070 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-04-21T08:22:16.073 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-04-21T08:22:16.075 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-04-21T08:22:16.108 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:22:16.113 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:22:16.149 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:22:16.550 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-21T08:22:16.558 INFO:teuthology.task.internal:Starting timer... 2024-04-21T08:22:16.558 INFO:teuthology.run_tasks:Running task pcp... 2024-04-21T08:22:16.606 INFO:teuthology.run_tasks:Running task selinux... 2024-04-21T08:22:16.631 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-21T08:22:16.631 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-04-21T08:22:16.697 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-04-21T08:22:16.699 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-04-21T08:22:16.751 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-04-21T08:22:16.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-04-21T08:22:16.815 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-04-21T08:22:16.817 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-21T08:22:16.817 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-04-21T08:22:16.844 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-04-21T08:22:16.844 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-04-21T08:22:16.871 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-04-21T08:22:16.871 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-04-21T08:22:16.907 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-04-21T08:22:16.907 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-04-21T08:22:16.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:22:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:22:16.934 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:22:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:22:16.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:22:16.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:22:16.989 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-21T08:22:16.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:22:17.013 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:22:17.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:22:17.075 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-21T08:22:17.085 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-21T08:22:17.101 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-21T08:22:17.129 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com,smithi111.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-21T08:26:39.427 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-04-21T08:26:39.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-21T08:26:39.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:26:39.546 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-04-21T08:26:39.579 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-21T08:26:39.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-21T08:26:39.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:26:39.657 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-04-21T08:26:39.737 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-21T08:26:39.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-21T08:26:39.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:26:39.819 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-04-21T08:26:40.065 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-21T08:26:40.066 INFO:teuthology.run_tasks:Running task clock... 2024-04-21T08:26:40.086 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-21T08:26:40.087 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:26:40.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:26:40.090 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:26:40.090 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:26:40.092 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:26:40.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:26:40.119 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:26:40.120 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:26:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:26:40.134 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:26:40.137 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:26:40.152 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:26:40.157 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-04-21T08:26:40.158 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-04-21T08:26:40.168 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-04-21T08:26:40.168 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-04-21T08:26:40.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-04-21T08:26:40.181 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:26:40.182 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:26:40.187 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-04-21T08:26:40.194 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:26:40.195 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:26:40.201 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:26:40.214 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:26:40.248 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:26:40.249 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:26:40.251 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:26:40.251 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-04-21T08:26:40.251 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.252 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.252 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.252 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.253 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:26:40.253 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-21T08:26:40.253 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.253 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.254 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.254 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.273 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:26:40.276 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:26:40.276 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-04-21T08:26:40.276 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.277 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.277 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.277 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:26:40.278 INFO:teuthology.run_tasks:Running task install... 2024-04-21T08:26:40.286 DEBUG:teuthology.task.install:project ceph 2024-04-21T08:26:40.286 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e'}} 2024-04-21T08:26:40.286 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e'} 2024-04-21T08:26:40.286 INFO:teuthology.task.install:Using flavor: default 2024-04-21T08:26:40.293 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-21T08:26:40.293 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-21T08:26:40.293 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:26:40.294 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:26:40.296 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:26:40.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:26:40.297 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:26:40.297 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:26:40.556 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:26:40.557 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:26:40.563 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:26:40.563 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:26:40.569 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:26:40.569 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:26:40.675 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:26:40.676 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:26:40.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:26:40.701 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:26:40.701 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:26:40.728 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:26:40.729 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:26:40.729 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:26:40.739 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:26:40.739 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:26:40.739 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:26:40.755 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:26:40.755 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:26:40.763 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:26:40.763 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:26:40.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:26:40.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:26:40.828 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:26:40.839 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:26:40.895 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:26:40.899 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:26:40.904 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-04-21T08:26:40.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-04-21T08:26:40.958 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-04-21T08:26:40.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-04-21T08:26:40.962 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-04-21T08:26:40.964 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-04-21T08:26:41.133 INFO:teuthology.orchestra.run.smithi173.stdout:48 files removed 2024-04-21T08:26:41.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:26:41.184 INFO:teuthology.orchestra.run.smithi094.stdout:48 files removed 2024-04-21T08:26:41.187 INFO:teuthology.orchestra.run.smithi111.stdout:48 files removed 2024-04-21T08:26:41.211 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:26:41.214 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:26:41.836 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-04-21T08:26:41.857 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 217 kB/s | 83 kB 00:00 2024-04-21T08:26:41.886 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-04-21T08:26:42.203 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-04-21T08:26:42.232 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-21T08:26:42.313 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-21T08:26:42.562 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-21T08:26:42.633 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-04-21T08:26:42.721 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-04-21T08:26:43.910 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.0 MB 00:01 2024-04-21T08:26:44.387 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-04-21T08:26:45.817 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-21T08:26:46.071 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-04-21T08:26:46.954 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-04-21T08:26:49.014 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-21T08:26:49.810 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-21T08:26:50.288 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.2 MB 00:00 2024-04-21T08:26:51.428 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-21T08:26:51.748 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-04-21T08:26:53.057 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-21T08:26:54.558 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 21 MB 00:02 2024-04-21T08:26:58.055 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-21T08:26:59.507 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-21T08:27:00.197 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:27:00.197 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:27:00.208 INFO:teuthology.orchestra.run.smithi111.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:27:00.208 INFO:teuthology.orchestra.run.smithi111.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:27:00.209 INFO:teuthology.orchestra.run.smithi111.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:27:00.210 INFO:teuthology.orchestra.run.smithi111.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:27:00.211 INFO:teuthology.orchestra.run.smithi111.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:27:00.214 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:27:00.214 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:27:00.273 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-21T08:27:00.294 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-21T08:27:00.294 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-21T08:27:00.294 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-21T08:27:00.294 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-21T08:27:00.294 INFO:teuthology.orchestra.run.smithi111.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:27:00.294 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:27:00.294 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:27:00.294 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:27:00.295 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:27:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:27:00.297 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:27:00.298 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-21T08:27:00.298 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:27:00.298 INFO:teuthology.orchestra.run.smithi111.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:27:00.298 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:27:00.298 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:27:00.298 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:27:00.298 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:27:00.298 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:27:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:27:00.300 INFO:teuthology.orchestra.run.smithi111.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:27:00.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:27:00.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:27:00.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:27:00.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:27:00.305 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-21T08:27:00.306 INFO:teuthology.orchestra.run.smithi111.stdout:Install 148 Packages 2024-04-21T08:27:00.307 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-04-21T08:27:00.307 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:00.307 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 218 M 2024-04-21T08:27:00.308 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-21T08:27:00.606 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 791 kB/s | 6.2 MB 00:07 2024-04-21T08:27:01.080 INFO:teuthology.orchestra.run.smithi111.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-04-21T08:27:01.422 INFO:teuthology.orchestra.run.smithi111.stdout:(2/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 9.3 MB/s | 5.4 MB 00:00 2024-04-21T08:27:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 2.8 MB/s | 145 kB 00:00 2024-04-21T08:27:01.506 INFO:teuthology.orchestra.run.smithi111.stdout:(4/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 2.0 MB/s | 890 kB 00:00 2024-04-21T08:27:01.666 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:27:01.667 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:27:01.673 INFO:teuthology.orchestra.run.smithi111.stdout:(5/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 12 MB/s | 2.4 MB 00:00 2024-04-21T08:27:01.678 INFO:teuthology.orchestra.run.smithi173.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:27:01.679 INFO:teuthology.orchestra.run.smithi173.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:27:01.680 INFO:teuthology.orchestra.run.smithi173.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:27:01.680 INFO:teuthology.orchestra.run.smithi173.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:27:01.682 INFO:teuthology.orchestra.run.smithi173.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:27:01.684 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:27:01.685 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:27:01.746 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-04-21T08:27:01.764 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-21T08:27:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-04-21T08:27:01.764 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:27:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:27:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:27:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:27:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:27:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:27:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:27:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:27:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:27:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:27:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:27:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:27:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:27:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:27:01.776 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-04-21T08:27:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:27:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:01.776 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-04-21T08:27:01.776 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-21T08:27:01.776 INFO:teuthology.orchestra.run.smithi173.stdout:Install 148 Packages 2024-04-21T08:27:01.776 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-04-21T08:27:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:01.777 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 218 M 2024-04-21T08:27:01.777 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-04-21T08:27:01.890 INFO:teuthology.orchestra.run.smithi111.stdout:(6/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 3.9 MB/s | 1.5 MB 00:00 2024-04-21T08:27:02.040 INFO:teuthology.orchestra.run.smithi111.stdout:(7/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 13 MB/s | 4.7 MB 00:00 2024-04-21T08:27:02.401 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-04-21T08:27:02.692 INFO:teuthology.orchestra.run.smithi173.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-21T08:27:02.865 INFO:teuthology.orchestra.run.smithi111.stdout:(8/150): ceph-radosgw-19.0.0-3154.ga630765c.el9 13 MB/s | 11 MB 00:00 2024-04-21T08:27:02.949 INFO:teuthology.orchestra.run.smithi111.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 303 kB/s | 25 kB 00:00 2024-04-21T08:27:03.392 INFO:teuthology.orchestra.run.smithi173.stdout:(2/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 1.2 MB/s | 890 kB 00:00 2024-04-21T08:27:03.559 INFO:teuthology.orchestra.run.smithi173.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 870 kB/s | 145 kB 00:00 2024-04-21T08:27:03.724 INFO:teuthology.orchestra.run.smithi111.stdout:(10/150): ceph-common-19.0.0-3154.ga630765c.el9 7.3 MB/s | 21 MB 00:02 2024-04-21T08:27:03.833 INFO:teuthology.orchestra.run.smithi111.stdout:(11/150): cephfs-mirror-19.0.0-3154.ga630765c.e 2.0 MB/s | 225 kB 00:00 2024-04-21T08:27:03.941 INFO:teuthology.orchestra.run.smithi111.stdout:(12/150): cephfs-shell-19.0.0-3154.ga630765c.el 250 kB/s | 27 kB 00:00 2024-04-21T08:27:03.951 INFO:teuthology.orchestra.run.smithi173.stdout:(4/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 3.3 MB/s | 5.4 MB 00:01 2024-04-21T08:27:04.009 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-21T08:27:04.033 INFO:teuthology.orchestra.run.smithi111.stdout:(13/150): libcephfs-devel-19.0.0-3154.ga630765c 347 kB/s | 32 kB 00:00 2024-04-21T08:27:04.333 INFO:teuthology.orchestra.run.smithi111.stdout:(14/150): libcephfs2-19.0.0-3154.ga630765c.el9. 2.4 MB/s | 746 kB 00:00 2024-04-21T08:27:04.384 INFO:teuthology.orchestra.run.smithi173.stdout:(5/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 3.5 MB/s | 1.5 MB 00:00 2024-04-21T08:27:04.425 INFO:teuthology.orchestra.run.smithi111.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 1.7 MB/s | 163 kB 00:00 2024-04-21T08:27:04.525 INFO:teuthology.orchestra.run.smithi111.stdout:(16/150): librados-devel-19.0.0-3154.ga630765c. 1.2 MB/s | 127 kB 00:00 2024-04-21T08:27:04.701 INFO:teuthology.orchestra.run.smithi111.stdout:(17/150): libradosstriper1-19.0.0-3154.ga630765 2.8 MB/s | 501 kB 00:00 2024-04-21T08:27:05.343 INFO:teuthology.orchestra.run.smithi173.stdout:(6/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 1.3 MB/s | 2.4 MB 00:01 2024-04-21T08:27:05.535 INFO:teuthology.orchestra.run.smithi173.stdout:(7/150): ceph-common-19.0.0-3154.ga630765c.el9. 6.6 MB/s | 21 MB 00:03 2024-04-21T08:27:05.634 INFO:teuthology.orchestra.run.smithi111.stdout:(18/150): ceph-osd-19.0.0-3154.ga630765c.el9.x8 4.5 MB/s | 17 MB 00:03 2024-04-21T08:27:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:(8/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 3.6 MB/s | 4.7 MB 00:01 2024-04-21T08:27:05.718 INFO:teuthology.orchestra.run.smithi111.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 542 kB/s | 45 kB 00:00 2024-04-21T08:27:05.768 INFO:teuthology.orchestra.run.smithi173.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 336 kB/s | 25 kB 00:00 2024-04-21T08:27:05.818 INFO:teuthology.orchestra.run.smithi111.stdout:(20/150): python3-ceph-common-19.0.0-3154.ga630 1.3 MB/s | 135 kB 00:00 2024-04-21T08:27:05.935 INFO:teuthology.orchestra.run.smithi111.stdout:(21/150): librgw2-19.0.0-3154.ga630765c.el9.x86 4.4 MB/s | 5.4 MB 00:01 2024-04-21T08:27:05.961 INFO:teuthology.orchestra.run.smithi111.stdout:(22/150): python3-cephfs-19.0.0-3154.ga630765c. 1.1 MB/s | 162 kB 00:00 2024-04-21T08:27:06.036 INFO:teuthology.orchestra.run.smithi111.stdout:(23/150): python3-rados-19.0.0-3154.ga630765c.e 3.1 MB/s | 321 kB 00:00 2024-04-21T08:27:06.069 INFO:teuthology.orchestra.run.smithi111.stdout:(24/150): python3-rbd-19.0.0-3154.ga630765c.el9 2.7 MB/s | 298 kB 00:00 2024-04-21T08:27:06.128 INFO:teuthology.orchestra.run.smithi111.stdout:(25/150): python3-rgw-19.0.0-3154.ga630765c.el9 1.1 MB/s | 100 kB 00:00 2024-04-21T08:27:06.170 INFO:teuthology.orchestra.run.smithi111.stdout:(26/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 850 kB/s | 85 kB 00:00 2024-04-21T08:27:06.279 INFO:teuthology.orchestra.run.smithi111.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 1.5 MB/s | 171 kB 00:00 2024-04-21T08:27:06.370 INFO:teuthology.orchestra.run.smithi111.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 373 kB/s | 34 kB 00:00 2024-04-21T08:27:06.479 INFO:teuthology.orchestra.run.smithi111.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 1.3 MB/s | 142 kB 00:00 2024-04-21T08:27:06.602 INFO:teuthology.orchestra.run.smithi173.stdout:(10/150): ceph-radosgw-19.0.0-3154.ga630765c.el 9.9 MB/s | 11 MB 00:01 2024-04-21T08:27:06.677 INFO:teuthology.orchestra.run.smithi173.stdout:(11/150): cephfs-mirror-19.0.0-3154.ga630765c.e 2.9 MB/s | 225 kB 00:00 2024-04-21T08:27:06.744 INFO:teuthology.orchestra.run.smithi173.stdout:(12/150): cephfs-shell-19.0.0-3154.ga630765c.el 405 kB/s | 27 kB 00:00 2024-04-21T08:27:06.763 INFO:teuthology.orchestra.run.smithi111.stdout:(30/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 4.9 MB/s | 3.1 MB 00:00 2024-04-21T08:27:06.811 INFO:teuthology.orchestra.run.smithi173.stdout:(13/150): libcephfs-devel-19.0.0-3154.ga630765c 475 kB/s | 32 kB 00:00 2024-04-21T08:27:06.969 INFO:teuthology.orchestra.run.smithi173.stdout:(14/150): libcephfs2-19.0.0-3154.ga630765c.el9. 4.6 MB/s | 746 kB 00:00 2024-04-21T08:27:07.053 INFO:teuthology.orchestra.run.smithi173.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 1.9 MB/s | 163 kB 00:00 2024-04-21T08:27:07.137 INFO:teuthology.orchestra.run.smithi173.stdout:(16/150): librados-devel-19.0.0-3154.ga630765c. 1.5 MB/s | 127 kB 00:00 2024-04-21T08:27:07.229 INFO:teuthology.orchestra.run.smithi173.stdout:(17/150): libradosstriper1-19.0.0-3154.ga630765 5.3 MB/s | 501 kB 00:00 2024-04-21T08:27:07.246 INFO:teuthology.orchestra.run.smithi111.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 4.6 MB/s | 3.5 MB 00:00 2024-04-21T08:27:07.364 INFO:teuthology.orchestra.run.smithi111.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3154.ga6 2.1 MB/s | 249 kB 00:00 2024-04-21T08:27:07.472 INFO:teuthology.orchestra.run.smithi111.stdout:(33/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 456 kB/s | 49 kB 00:00 2024-04-21T08:27:07.581 INFO:teuthology.orchestra.run.smithi111.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3154.ga 152 kB/s | 16 kB 00:00 2024-04-21T08:27:07.706 INFO:teuthology.orchestra.run.smithi111.stdout:(35/150): ceph-volume-19.0.0-3154.ga630765c.el9 2.1 MB/s | 273 kB 00:00 2024-04-21T08:27:07.829 INFO:teuthology.orchestra.run.smithi173.stdout:(18/150): librgw2-19.0.0-3154.ga630765c.el9.x86 9.0 MB/s | 5.4 MB 00:00 2024-04-21T08:27:07.930 INFO:teuthology.orchestra.run.smithi173.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 446 kB/s | 45 kB 00:00 2024-04-21T08:27:07.948 INFO:teuthology.orchestra.run.smithi111.stdout:(36/150): cephadm-19.0.0-3154.ga630765c.el9.noa 3.1 MB/s | 762 kB 00:00 2024-04-21T08:27:07.997 INFO:teuthology.orchestra.run.smithi173.stdout:(20/150): python3-ceph-common-19.0.0-3154.ga630 2.0 MB/s | 135 kB 00:00 2024-04-21T08:27:08.056 INFO:teuthology.orchestra.run.smithi173.stdout:(21/150): python3-cephfs-19.0.0-3154.ga630765c. 2.7 MB/s | 162 kB 00:00 2024-04-21T08:27:08.090 INFO:teuthology.orchestra.run.smithi111.stdout:(37/150): ceph-mgr-diskprediction-local-19.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-04-21T08:27:08.106 INFO:teuthology.orchestra.run.smithi173.stdout:(22/150): python3-rados-19.0.0-3154.ga630765c.e 6.3 MB/s | 321 kB 00:00 2024-04-21T08:27:08.115 INFO:teuthology.orchestra.run.smithi111.stdout:(38/150): cephfs-top-19.0.0-3154.ga630765c.el9. 109 kB/s | 18 kB 00:00 2024-04-21T08:27:08.164 INFO:teuthology.orchestra.run.smithi173.stdout:(23/150): python3-rbd-19.0.0-3154.ga630765c.el9 5.0 MB/s | 298 kB 00:00 2024-04-21T08:27:08.182 INFO:teuthology.orchestra.run.smithi111.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 878 kB/s | 80 kB 00:00 2024-04-21T08:27:08.223 INFO:teuthology.orchestra.run.smithi173.stdout:(24/150): python3-rgw-19.0.0-3154.ga630765c.el9 1.7 MB/s | 100 kB 00:00 2024-04-21T08:27:08.274 INFO:teuthology.orchestra.run.smithi111.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-21T08:27:08.282 INFO:teuthology.orchestra.run.smithi173.stdout:(25/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-21T08:27:08.325 INFO:teuthology.orchestra.run.smithi111.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-04-21T08:27:08.375 INFO:teuthology.orchestra.run.smithi111.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-04-21T08:27:08.442 INFO:teuthology.orchestra.run.smithi111.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-21T08:27:08.517 INFO:teuthology.orchestra.run.smithi111.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-04-21T08:27:08.559 INFO:teuthology.orchestra.run.smithi111.stdout:(45/150): python3-cryptography-36.0.1-4.el9.x86 5.2 MB/s | 1.2 MB 00:00 2024-04-21T08:27:08.582 INFO:teuthology.orchestra.run.smithi173.stdout:(26/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 10 MB/s | 3.1 MB 00:00 2024-04-21T08:27:08.618 INFO:teuthology.orchestra.run.smithi111.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-04-21T08:27:08.641 INFO:teuthology.orchestra.run.smithi173.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 2.9 MB/s | 171 kB 00:00 2024-04-21T08:27:08.660 INFO:teuthology.orchestra.run.smithi111.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 2.1 MB/s | 215 kB 00:00 2024-04-21T08:27:08.691 INFO:teuthology.orchestra.run.smithi173.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 687 kB/s | 34 kB 00:00 2024-04-21T08:27:08.749 INFO:teuthology.orchestra.run.smithi173.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 2.4 MB/s | 142 kB 00:00 2024-04-21T08:27:08.769 INFO:teuthology.orchestra.run.smithi111.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 4.5 MB/s | 689 kB 00:00 2024-04-21T08:27:09.008 INFO:teuthology.orchestra.run.smithi173.stdout:(30/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 14 MB/s | 3.5 MB 00:00 2024-04-21T08:27:09.034 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-21T08:27:09.211 INFO:teuthology.orchestra.run.smithi111.stdout:(49/150): ceph-test-19.0.0-3154.ga630765c.el9.x 7.8 MB/s | 49 MB 00:06 2024-04-21T08:27:09.261 INFO:teuthology.orchestra.run.smithi111.stdout:(50/150): bison-3.7.4-5.el9.x86_64.rpm 1.5 MB/s | 943 kB 00:00 2024-04-21T08:27:09.311 INFO:teuthology.orchestra.run.smithi111.stdout:(51/150): boost-program-options-1.75.0-8.el9.x8 197 kB/s | 107 kB 00:00 2024-04-21T08:27:09.336 INFO:teuthology.orchestra.run.smithi111.stdout:(52/150): elfutils-libelf-devel-0.190-2.el9.x86 172 kB/s | 21 kB 00:00 2024-04-21T08:27:09.387 INFO:teuthology.orchestra.run.smithi111.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 434 kB/s | 33 kB 00:00 2024-04-21T08:27:09.412 INFO:teuthology.orchestra.run.smithi111.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 2.0 MB/s | 313 kB 00:00 2024-04-21T08:27:09.462 INFO:teuthology.orchestra.run.smithi111.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 242 kB/s | 18 kB 00:00 2024-04-21T08:27:09.479 INFO:teuthology.orchestra.run.smithi111.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 167 kB/s | 11 kB 00:00 2024-04-21T08:27:09.500 INFO:teuthology.orchestra.run.smithi173.stdout:(31/150): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-04-21T08:27:09.538 INFO:teuthology.orchestra.run.smithi111.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 774 kB/s | 45 kB 00:00 2024-04-21T08:27:09.563 INFO:teuthology.orchestra.run.smithi111.stdout:(58/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-04-21T08:27:09.567 INFO:teuthology.orchestra.run.smithi173.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3154.ga6 3.7 MB/s | 249 kB 00:00 2024-04-21T08:27:09.634 INFO:teuthology.orchestra.run.smithi173.stdout:(33/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 740 kB/s | 49 kB 00:00 2024-04-21T08:27:09.709 INFO:teuthology.orchestra.run.smithi173.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3154.ga 219 kB/s | 16 kB 00:00 2024-04-21T08:27:09.722 INFO:teuthology.orchestra.run.smithi111.stdout:(59/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.8 MB/s | 3.0 MB 00:00 2024-04-21T08:27:09.747 INFO:teuthology.orchestra.run.smithi111.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-04-21T08:27:09.780 INFO:teuthology.orchestra.run.smithi111.stdout:(61/150): libtool-2.4.6-45.el9.x86_64.rpm 2.6 MB/s | 582 kB 00:00 2024-04-21T08:27:09.793 INFO:teuthology.orchestra.run.smithi173.stdout:(35/150): ceph-volume-19.0.0-3154.ga630765c.el9 3.2 MB/s | 273 kB 00:00 2024-04-21T08:27:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:(62/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 304 kB/s | 25 kB 00:00 2024-04-21T08:27:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-04-21T08:27:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:(64/150): openblas-0.3.21-2.el9.x86_64.rpm 605 kB/s | 35 kB 00:00 2024-04-21T08:27:09.916 INFO:teuthology.orchestra.run.smithi111.stdout:(65/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-04-21T08:27:09.940 INFO:teuthology.orchestra.run.smithi111.stdout:(66/150): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-04-21T08:27:10.126 INFO:teuthology.orchestra.run.smithi173.stdout:(36/150): cephadm-19.0.0-3154.ga630765c.el9.noa 2.2 MB/s | 762 kB 00:00 2024-04-21T08:27:10.149 INFO:teuthology.orchestra.run.smithi111.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-04-21T08:27:10.201 INFO:teuthology.orchestra.run.smithi173.stdout:(37/150): cephfs-top-19.0.0-3154.ga630765c.el9. 243 kB/s | 18 kB 00:00 2024-04-21T08:27:10.299 INFO:teuthology.orchestra.run.smithi111.stdout:(68/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2024-04-21T08:27:10.310 INFO:teuthology.orchestra.run.smithi173.stdout:(38/150): fuse-2.9.9-15.el9.x86_64.rpm 743 kB/s | 80 kB 00:00 2024-04-21T08:27:10.366 INFO:teuthology.orchestra.run.smithi111.stdout:(69/150): openblas-openmp-0.3.21-2.el9.x86_64.r 10 MB/s | 4.7 MB 00:00 2024-04-21T08:27:10.400 INFO:teuthology.orchestra.run.smithi111.stdout:(70/150): postgresql-contrib-13.14-1.el9.x86_64 3.6 MB/s | 911 kB 00:00 2024-04-21T08:27:10.410 INFO:teuthology.orchestra.run.smithi173.stdout:(39/150): libgfortran-11.4.1-3.el9.x86_64.rpm 7.8 MB/s | 803 kB 00:00 2024-04-21T08:27:10.441 INFO:teuthology.orchestra.run.smithi111.stdout:(71/150): postgresql-private-libs-13.14-1.el9.x 938 kB/s | 133 kB 00:00 2024-04-21T08:27:10.469 INFO:teuthology.orchestra.run.smithi173.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2024-04-21T08:27:10.511 INFO:teuthology.orchestra.run.smithi173.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.9 MB/s | 253 kB 00:00 2024-04-21T08:27:10.627 INFO:teuthology.orchestra.run.smithi111.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 956 kB/s | 87 kB 00:00 2024-04-21T08:27:10.627 INFO:teuthology.orchestra.run.smithi111.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-04-21T08:27:10.628 INFO:teuthology.orchestra.run.smithi173.stdout:(42/150): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-21T08:27:10.628 INFO:teuthology.orchestra.run.smithi173.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-21T08:27:10.662 INFO:teuthology.orchestra.run.smithi173.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-21T08:27:10.704 INFO:teuthology.orchestra.run.smithi173.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-21T08:27:10.709 INFO:teuthology.orchestra.run.smithi111.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-04-21T08:27:10.748 INFO:teuthology.orchestra.run.smithi173.stdout:(46/150): python3-urllib3-1.26.5-5.el9.noarch.r 4.8 MB/s | 215 kB 00:00 2024-04-21T08:27:10.809 INFO:teuthology.orchestra.run.smithi111.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-21T08:27:10.893 INFO:teuthology.orchestra.run.smithi111.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 308 kB/s | 26 kB 00:00 2024-04-21T08:27:10.968 INFO:teuthology.orchestra.run.smithi111.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 256 kB/s | 19 kB 00:00 2024-04-21T08:27:11.035 INFO:teuthology.orchestra.run.smithi111.stdout:(78/150): postgresql-server-13.14-1.el9.x86_64. 8.6 MB/s | 5.8 MB 00:00 2024-04-21T08:27:11.111 INFO:teuthology.orchestra.run.smithi111.stdout:(79/150): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-04-21T08:27:11.152 INFO:teuthology.orchestra.run.smithi111.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 297 kB/s | 35 kB 00:00 2024-04-21T08:27:11.156 INFO:teuthology.orchestra.run.smithi173.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 689 kB 00:00 2024-04-21T08:27:11.178 INFO:teuthology.orchestra.run.smithi111.stdout:(81/150): python3-mako-1.1.4-6.el9.noarch.rpm 823 kB/s | 172 kB 00:00 2024-04-21T08:27:11.219 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:27:11.219 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:27:11.230 INFO:teuthology.orchestra.run.smithi094.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:27:11.231 INFO:teuthology.orchestra.run.smithi094.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:27:11.232 INFO:teuthology.orchestra.run.smithi094.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:27:11.232 INFO:teuthology.orchestra.run.smithi094.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:27:11.234 INFO:teuthology.orchestra.run.smithi094.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:27:11.236 INFO:teuthology.orchestra.run.smithi094.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:27:11.237 INFO:teuthology.orchestra.run.smithi094.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:27:11.253 INFO:teuthology.orchestra.run.smithi111.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-04-21T08:27:11.278 INFO:teuthology.orchestra.run.smithi111.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-21T08:27:11.296 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-21T08:27:11.305 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-04-21T08:27:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-21T08:27:11.305 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-04-21T08:27:11.305 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-21T08:27:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:27:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:27:11.305 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:27:11.306 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:27:11.307 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:27:11.308 INFO:teuthology.orchestra.run.smithi094.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:27:11.309 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:27:11.310 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:27:11.311 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:27:11.312 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:27:11.313 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:27:11.314 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:27:11.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:27:11.316 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:27:11.317 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-21T08:27:11.317 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-04-21T08:27:11.317 INFO:teuthology.orchestra.run.smithi094.stdout:Install 148 Packages 2024-04-21T08:27:11.317 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2024-04-21T08:27:11.317 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:27:11.317 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 218 M 2024-04-21T08:27:11.317 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-21T08:27:11.345 INFO:teuthology.orchestra.run.smithi111.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 635 kB/s | 42 kB 00:00 2024-04-21T08:27:11.370 INFO:teuthology.orchestra.run.smithi111.stdout:(85/150): python3-packaging-20.9-5.el9.noarch.r 658 kB/s | 77 kB 00:00 2024-04-21T08:27:11.381 INFO:teuthology.orchestra.run.smithi173.stdout:(48/150): bison-3.7.4-5.el9.x86_64.rpm 4.1 MB/s | 943 kB 00:00 2024-04-21T08:27:11.432 INFO:teuthology.orchestra.run.smithi173.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-04-21T08:27:11.445 INFO:teuthology.orchestra.run.smithi111.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 2.6 MB/s | 269 kB 00:00 2024-04-21T08:27:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-04-21T08:27:11.482 INFO:teuthology.orchestra.run.smithi173.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 431 kB/s | 21 kB 00:00 2024-04-21T08:27:11.574 INFO:teuthology.orchestra.run.smithi173.stdout:(51/150): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-04-21T08:27:11.580 INFO:teuthology.orchestra.run.smithi111.stdout:(88/150): python3-pytz-2021.1-5.el9.noarch.rpm 469 kB/s | 51 kB 00:00 2024-04-21T08:27:11.624 INFO:teuthology.orchestra.run.smithi173.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 658 kB/s | 33 kB 00:00 2024-04-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi111.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 1.4 MB/s | 279 kB 00:00 2024-04-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi111.stdout:(90/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.3 MB/s | 5.1 MB 00:00 2024-04-21T08:27:11.747 INFO:teuthology.orchestra.run.smithi111.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 321 kB/s | 54 kB 00:00 2024-04-21T08:27:11.797 INFO:teuthology.orchestra.run.smithi111.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-04-21T08:27:11.823 INFO:teuthology.orchestra.run.smithi111.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 586 kB/s | 44 kB 00:00 2024-04-21T08:27:11.873 INFO:teuthology.orchestra.run.smithi111.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 722 kB/s | 36 kB 00:00 2024-04-21T08:27:11.898 INFO:teuthology.orchestra.run.smithi111.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-04-21T08:27:11.957 INFO:teuthology.orchestra.run.smithi111.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-04-21T08:27:11.983 INFO:teuthology.orchestra.run.smithi173.stdout:(53/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.3 MB/s | 3.0 MB 00:00 2024-04-21T08:27:12.015 INFO:teuthology.orchestra.run.smithi111.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 2.5 MB/s | 305 kB 00:00 2024-04-21T08:27:12.032 INFO:teuthology.orchestra.run.smithi111.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 769 kB/s | 58 kB 00:00 2024-04-21T08:27:12.033 INFO:teuthology.orchestra.run.smithi173.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 366 kB/s | 18 kB 00:00 2024-04-21T08:27:12.083 INFO:teuthology.orchestra.run.smithi173.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-21T08:27:12.099 INFO:teuthology.orchestra.run.smithi111.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-21T08:27:12.119 INFO:teuthology.orchestra.run.smithi094.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-21T08:27:12.124 INFO:teuthology.orchestra.run.smithi111.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 694 kB/s | 64 kB 00:00 2024-04-21T08:27:12.133 INFO:teuthology.orchestra.run.smithi173.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-21T08:27:12.184 INFO:teuthology.orchestra.run.smithi173.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-04-21T08:27:12.200 INFO:teuthology.orchestra.run.smithi111.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 447 kB/s | 45 kB 00:00 2024-04-21T08:27:12.225 INFO:teuthology.orchestra.run.smithi111.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-04-21T08:27:12.284 INFO:teuthology.orchestra.run.smithi173.stdout:(58/150): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-04-21T08:27:12.359 INFO:teuthology.orchestra.run.smithi173.stdout:(59/150): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 582 kB 00:00 2024-04-21T08:27:12.500 INFO:teuthology.orchestra.run.smithi111.stdout:(103/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-04-21T08:27:12.535 INFO:teuthology.orchestra.run.smithi173.stdout:(60/150): ceph-osd-19.0.0-3154.ga630765c.el9.x8 2.3 MB/s | 17 MB 00:07 2024-04-21T08:27:12.552 INFO:teuthology.orchestra.run.smithi173.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 133 kB/s | 25 kB 00:00 2024-04-21T08:27:12.592 INFO:teuthology.orchestra.run.smithi111.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-04-21T08:27:12.602 INFO:teuthology.orchestra.run.smithi173.stdout:(62/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-21T08:27:12.636 INFO:teuthology.orchestra.run.smithi173.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 473 kB/s | 47 kB 00:00 2024-04-21T08:27:12.661 INFO:teuthology.orchestra.run.smithi173.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-21T08:27:12.694 INFO:teuthology.orchestra.run.smithi173.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-04-21T08:27:12.726 INFO:teuthology.orchestra.run.smithi111.stdout:(105/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-21T08:27:12.751 INFO:teuthology.orchestra.run.smithi111.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 122 kB/s | 19 kB 00:00 2024-04-21T08:27:12.818 INFO:teuthology.orchestra.run.smithi111.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-04-21T08:27:12.843 INFO:teuthology.orchestra.run.smithi111.stdout:(108/150): gperftools-libs-2.9.1-2.el9.x86_64.r 901 kB/s | 309 kB 00:00 2024-04-21T08:27:12.877 INFO:teuthology.orchestra.run.smithi111.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 829 kB/s | 49 kB 00:00 2024-04-21T08:27:12.936 INFO:teuthology.orchestra.run.smithi173.stdout:(66/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 17 MB/s | 4.1 MB 00:00 2024-04-21T08:27:12.969 INFO:teuthology.orchestra.run.smithi111.stdout:(110/150): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-21T08:27:12.986 INFO:teuthology.orchestra.run.smithi111.stdout:(111/150): libunwind-1.6.2-1.el9.x86_64.rpm 473 kB/s | 67 kB 00:00 2024-04-21T08:27:13.020 INFO:teuthology.orchestra.run.smithi173.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-04-21T08:27:13.061 INFO:teuthology.orchestra.run.smithi111.stdout:(112/150): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-21T08:27:13.087 INFO:teuthology.orchestra.run.smithi173.stdout:(68/150): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2024-04-21T08:27:13.129 INFO:teuthology.orchestra.run.smithi173.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 8.2 MB/s | 911 kB 00:00 2024-04-21T08:27:13.136 INFO:teuthology.orchestra.run.smithi111.stdout:(113/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-04-21T08:27:13.162 INFO:teuthology.orchestra.run.smithi173.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-04-21T08:27:13.237 INFO:teuthology.orchestra.run.smithi111.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 248 kB/s | 43 kB 00:00 2024-04-21T08:27:13.262 INFO:teuthology.orchestra.run.smithi111.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 257 kB/s | 32 kB 00:00 2024-04-21T08:27:13.296 INFO:teuthology.orchestra.run.smithi111.stdout:(116/150): python3-asyncssh-2.13.2-5.el9.noarch 1.7 MB/s | 548 kB 00:00 2024-04-21T08:27:13.305 INFO:teuthology.orchestra.run.smithi173.stdout:(71/150): postgresql-server-13.14-1.el9.x86_64. 33 MB/s | 5.8 MB 00:00 2024-04-21T08:27:13.321 INFO:teuthology.orchestra.run.smithi111.stdout:(117/150): python3-certifi-2021.10.8-2.el9.noar 179 kB/s | 15 kB 00:00 2024-04-21T08:27:13.338 INFO:teuthology.orchestra.run.smithi173.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 5.7 MB/s | 1.0 MB 00:00 2024-04-21T08:27:13.354 INFO:teuthology.orchestra.run.smithi111.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-04-21T08:27:13.363 INFO:teuthology.orchestra.run.smithi173.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-21T08:27:13.396 INFO:teuthology.orchestra.run.smithi111.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-21T08:27:13.414 INFO:teuthology.orchestra.run.smithi173.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-04-21T08:27:13.438 INFO:teuthology.orchestra.run.smithi111.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-04-21T08:27:13.464 INFO:teuthology.orchestra.run.smithi173.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-04-21T08:27:13.472 INFO:teuthology.orchestra.run.smithi111.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-04-21T08:27:13.515 INFO:teuthology.orchestra.run.smithi173.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 511 kB/s | 26 kB 00:00 2024-04-21T08:27:13.530 INFO:teuthology.orchestra.run.smithi111.stdout:(122/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.4 MB/s | 303 kB 00:00 2024-04-21T08:27:13.564 INFO:teuthology.orchestra.run.smithi111.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-21T08:27:13.564 INFO:teuthology.orchestra.run.smithi173.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-04-21T08:27:13.603 INFO:teuthology.orchestra.run.smithi094.stdout:(2/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 600 kB/s | 890 kB 00:01 2024-04-21T08:27:13.615 INFO:teuthology.orchestra.run.smithi173.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-21T08:27:13.639 INFO:teuthology.orchestra.run.smithi111.stdout:(124/150): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-04-21T08:27:13.665 INFO:teuthology.orchestra.run.smithi111.stdout:(125/150): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-04-21T08:27:13.665 INFO:teuthology.orchestra.run.smithi173.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 692 kB/s | 35 kB 00:00 2024-04-21T08:27:13.690 INFO:teuthology.orchestra.run.smithi111.stdout:(126/150): python3-jaraco-classes-3.2.1-5.el9.n 141 kB/s | 18 kB 00:00 2024-04-21T08:27:13.740 INFO:teuthology.orchestra.run.smithi111.stdout:(127/150): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-04-21T08:27:13.766 INFO:teuthology.orchestra.run.smithi111.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-04-21T08:27:13.766 INFO:teuthology.orchestra.run.smithi173.stdout:(80/150): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-04-21T08:27:13.791 INFO:teuthology.orchestra.run.smithi111.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-21T08:27:13.824 INFO:teuthology.orchestra.run.smithi173.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-21T08:27:13.833 INFO:teuthology.orchestra.run.smithi111.stdout:(130/150): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-21T08:27:13.858 INFO:teuthology.orchestra.run.smithi111.stdout:(131/150): python3-logutils-0.3.5-21.el9.noarch 503 kB/s | 46 kB 00:00 2024-04-21T08:27:13.883 INFO:teuthology.orchestra.run.smithi111.stdout:(132/150): python3-more-itertools-8.12.0-2.el9. 856 kB/s | 79 kB 00:00 2024-04-21T08:27:13.891 INFO:teuthology.orchestra.run.smithi173.stdout:(82/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-04-21T08:27:13.916 INFO:teuthology.orchestra.run.smithi173.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-21T08:27:13.920 INFO:teuthology.orchestra.run.smithi094.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 458 kB/s | 145 kB 00:00 2024-04-21T08:27:13.925 INFO:teuthology.orchestra.run.smithi111.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 626 kB/s | 58 kB 00:00 2024-04-21T08:27:13.941 INFO:teuthology.orchestra.run.smithi173.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-21T08:27:13.951 INFO:teuthology.orchestra.run.smithi111.stdout:(134/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-21T08:27:13.967 INFO:teuthology.orchestra.run.smithi173.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 839 kB/s | 42 kB 00:00 2024-04-21T08:27:13.978 INFO:teuthology.orchestra.run.smithi111.stdout:(135/150): python3-portend-3.1.0-2.el9.noarch.r 174 kB/s | 16 kB 00:00 2024-04-21T08:27:13.992 INFO:teuthology.orchestra.run.smithi173.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-21T08:27:14.011 INFO:teuthology.orchestra.run.smithi111.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-04-21T08:27:14.017 INFO:teuthology.orchestra.run.smithi173.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-21T08:27:14.037 INFO:teuthology.orchestra.run.smithi111.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 329 kB/s | 28 kB 00:00 2024-04-21T08:27:14.043 INFO:teuthology.orchestra.run.smithi173.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-21T08:27:14.062 INFO:teuthology.orchestra.run.smithi111.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 366 kB/s | 31 kB 00:00 2024-04-21T08:27:14.068 INFO:teuthology.orchestra.run.smithi173.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-21T08:27:14.087 INFO:teuthology.orchestra.run.smithi111.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-21T08:27:14.093 INFO:teuthology.orchestra.run.smithi173.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-21T08:27:14.112 INFO:teuthology.orchestra.run.smithi111.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-04-21T08:27:14.138 INFO:teuthology.orchestra.run.smithi111.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-04-21T08:27:14.143 INFO:teuthology.orchestra.run.smithi173.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-04-21T08:27:14.163 INFO:teuthology.orchestra.run.smithi111.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-04-21T08:27:14.194 INFO:teuthology.orchestra.run.smithi173.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 882 kB/s | 44 kB 00:00 2024-04-21T08:27:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-21T08:27:14.230 INFO:teuthology.orchestra.run.smithi111.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 973 kB/s | 90 kB 00:00 2024-04-21T08:27:14.244 INFO:teuthology.orchestra.run.smithi173.stdout:(93/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-21T08:27:14.272 INFO:teuthology.orchestra.run.smithi111.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-21T08:27:14.289 INFO:teuthology.orchestra.run.smithi111.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-04-21T08:27:14.294 INFO:teuthology.orchestra.run.smithi173.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-04-21T08:27:14.345 INFO:teuthology.orchestra.run.smithi173.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-21T08:27:14.364 INFO:teuthology.orchestra.run.smithi111.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-04-21T08:27:14.395 INFO:teuthology.orchestra.run.smithi173.stdout:(96/150): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-21T08:27:14.423 INFO:teuthology.orchestra.run.smithi111.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-21T08:27:14.445 INFO:teuthology.orchestra.run.smithi173.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-21T08:27:14.496 INFO:teuthology.orchestra.run.smithi173.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-21T08:27:14.538 INFO:teuthology.orchestra.run.smithi173.stdout:(99/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-21T08:27:14.579 INFO:teuthology.orchestra.run.smithi173.stdout:(100/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-21T08:27:14.630 INFO:teuthology.orchestra.run.smithi173.stdout:(101/150): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-04-21T08:27:14.789 INFO:teuthology.orchestra.run.smithi173.stdout:(102/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-04-21T08:27:14.872 INFO:teuthology.orchestra.run.smithi173.stdout:(103/150): protobuf-compiler-3.14.0-13.el9.x86_ 3.5 MB/s | 863 kB 00:00 2024-04-21T08:27:15.097 INFO:teuthology.orchestra.run.smithi173.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-04-21T08:27:15.156 INFO:teuthology.orchestra.run.smithi173.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-04-21T08:27:15.181 INFO:teuthology.orchestra.run.smithi173.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-04-21T08:27:15.240 INFO:teuthology.orchestra.run.smithi173.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-04-21T08:27:15.299 INFO:teuthology.orchestra.run.smithi173.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 832 kB/s | 49 kB 00:00 2024-04-21T08:27:15.349 INFO:teuthology.orchestra.run.smithi173.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-21T08:27:15.407 INFO:teuthology.orchestra.run.smithi173.stdout:(110/150): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-21T08:27:15.491 INFO:teuthology.orchestra.run.smithi173.stdout:(111/150): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-21T08:27:15.550 INFO:teuthology.orchestra.run.smithi173.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-04-21T08:27:15.556 INFO:teuthology.orchestra.run.smithi111.stdout:(149/150): librbd1-19.0.0-3154.ga630765c.el9.x8 2.6 MB/s | 3.1 MB 00:01 2024-04-21T08:27:15.575 INFO:teuthology.orchestra.run.smithi173.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-04-21T08:27:15.608 INFO:teuthology.orchestra.run.smithi173.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 744 kB/s | 43 kB 00:00 2024-04-21T08:27:15.634 INFO:teuthology.orchestra.run.smithi173.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-04-21T08:27:15.667 INFO:teuthology.orchestra.run.smithi173.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-04-21T08:27:15.693 INFO:teuthology.orchestra.run.smithi173.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-21T08:27:15.734 INFO:teuthology.orchestra.run.smithi173.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-04-21T08:27:15.760 INFO:teuthology.orchestra.run.smithi173.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-04-21T08:27:15.794 INFO:teuthology.orchestra.run.smithi173.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 602 kB/s | 35 kB 00:00 2024-04-21T08:27:15.815 INFO:teuthology.orchestra.run.smithi111.stdout:(150/150): librados2-19.0.0-3154.ga630765c.el9. 2.2 MB/s | 3.4 MB 00:01 2024-04-21T08:27:15.818 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:27:15.818 INFO:teuthology.orchestra.run.smithi111.stdout:Total 14 MB/s | 218 MB 00:15 2024-04-21T08:27:15.819 INFO:teuthology.orchestra.run.smithi173.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-04-21T08:27:15.886 INFO:teuthology.orchestra.run.smithi173.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-21T08:27:15.952 INFO:teuthology.orchestra.run.smithi173.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-04-21T08:27:15.994 INFO:teuthology.orchestra.run.smithi173.stdout:(124/150): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-04-21T08:27:16.011 INFO:teuthology.orchestra.run.smithi173.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-04-21T08:27:16.045 INFO:teuthology.orchestra.run.smithi173.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-21T08:27:16.062 INFO:teuthology.orchestra.run.smithi173.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-21T08:27:16.095 INFO:teuthology.orchestra.run.smithi173.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-21T08:27:16.162 INFO:teuthology.orchestra.run.smithi173.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-04-21T08:27:16.179 INFO:teuthology.orchestra.run.smithi173.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 555 kB/s | 46 kB 00:00 2024-04-21T08:27:16.221 INFO:teuthology.orchestra.run.smithi173.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-21T08:27:16.254 INFO:teuthology.orchestra.run.smithi173.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 767 kB/s | 58 kB 00:00 2024-04-21T08:27:16.280 INFO:teuthology.orchestra.run.smithi173.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-04-21T08:27:16.305 INFO:teuthology.orchestra.run.smithi173.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-04-21T08:27:16.330 INFO:teuthology.orchestra.run.smithi173.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-04-21T08:27:16.356 INFO:teuthology.orchestra.run.smithi173.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 560 kB/s | 28 kB 00:00 2024-04-21T08:27:16.389 INFO:teuthology.orchestra.run.smithi173.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 522 kB/s | 31 kB 00:00 2024-04-21T08:27:16.423 INFO:teuthology.orchestra.run.smithi173.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 2.7 MB/s | 188 kB 00:00 2024-04-21T08:27:16.448 INFO:teuthology.orchestra.run.smithi173.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-21T08:27:16.482 INFO:teuthology.orchestra.run.smithi173.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 610 kB/s | 36 kB 00:00 2024-04-21T08:27:16.507 INFO:teuthology.orchestra.run.smithi173.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 870 kB/s | 51 kB 00:00 2024-04-21T08:27:16.541 INFO:teuthology.orchestra.run.smithi173.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-04-21T08:27:16.566 INFO:teuthology.orchestra.run.smithi173.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-21T08:27:16.600 INFO:teuthology.orchestra.run.smithi173.stdout:(144/150): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-04-21T08:27:16.603 INFO:teuthology.orchestra.run.smithi094.stdout:(4/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 1.1 MB/s | 5.4 MB 00:04 2024-04-21T08:27:16.625 INFO:teuthology.orchestra.run.smithi173.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-04-21T08:27:16.659 INFO:teuthology.orchestra.run.smithi173.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-21T08:27:16.872 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-21T08:27:16.998 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-21T08:27:16.998 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-21T08:27:17.101 INFO:teuthology.orchestra.run.smithi173.stdout:(147/150): ceph-test-19.0.0-3154.ga630765c.el9. 4.3 MB/s | 49 MB 00:11 2024-04-21T08:27:17.143 INFO:teuthology.orchestra.run.smithi173.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-04-21T08:27:17.496 INFO:teuthology.orchestra.run.smithi094.stdout:(5/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 1.7 MB/s | 1.5 MB 00:00 2024-04-21T08:27:17.876 INFO:teuthology.orchestra.run.smithi173.stdout:(149/150): librbd1-19.0.0-3154.ga630765c.el9.x8 4.1 MB/s | 3.1 MB 00:00 2024-04-21T08:27:18.013 INFO:teuthology.orchestra.run.smithi094.stdout:(6/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 590 kB/s | 2.4 MB 00:04 2024-04-21T08:27:18.218 INFO:teuthology.orchestra.run.smithi173.stdout:(150/150): librados2-19.0.0-3154.ga630765c.el9. 2.2 MB/s | 3.4 MB 00:01 2024-04-21T08:27:18.223 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:27:18.223 INFO:teuthology.orchestra.run.smithi173.stdout:Total 13 MB/s | 218 MB 00:16 2024-04-21T08:27:18.480 INFO:teuthology.orchestra.run.smithi094.stdout:(7/150): ceph-common-19.0.0-3154.ga630765c.el9. 3.1 MB/s | 21 MB 00:06 2024-04-21T08:27:18.719 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-21T08:27:18.720 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-21T08:27:19.290 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-04-21T08:27:19.415 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-04-21T08:27:19.415 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-04-21T08:27:19.505 INFO:teuthology.orchestra.run.smithi094.stdout:(8/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-04-21T08:27:19.555 INFO:teuthology.orchestra.run.smithi094.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 504 kB/s | 25 kB 00:00 2024-04-21T08:27:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:(10/150): ceph-radosgw-19.0.0-3154.ga630765c.el 5.7 MB/s | 11 MB 00:01 2024-04-21T08:27:20.406 INFO:teuthology.orchestra.run.smithi094.stdout:(11/150): cephfs-mirror-19.0.0-3154.ga630765c.e 3.8 MB/s | 225 kB 00:00 2024-04-21T08:27:20.464 INFO:teuthology.orchestra.run.smithi094.stdout:(12/150): cephfs-shell-19.0.0-3154.ga630765c.el 463 kB/s | 27 kB 00:00 2024-04-21T08:27:20.523 INFO:teuthology.orchestra.run.smithi094.stdout:(13/150): libcephfs-devel-19.0.0-3154.ga630765c 544 kB/s | 32 kB 00:00 2024-04-21T08:27:20.673 INFO:teuthology.orchestra.run.smithi094.stdout:(14/150): libcephfs2-19.0.0-3154.ga630765c.el9. 4.9 MB/s | 746 kB 00:00 2024-04-21T08:27:20.732 INFO:teuthology.orchestra.run.smithi094.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 2.7 MB/s | 163 kB 00:00 2024-04-21T08:27:20.792 INFO:teuthology.orchestra.run.smithi094.stdout:(16/150): librados-devel-19.0.0-3154.ga630765c. 2.1 MB/s | 127 kB 00:00 2024-04-21T08:27:20.883 INFO:teuthology.orchestra.run.smithi094.stdout:(17/150): libradosstriper1-19.0.0-3154.ga630765 5.3 MB/s | 501 kB 00:00 2024-04-21T08:27:21.142 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-04-21T08:27:21.143 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-04-21T08:27:21.609 INFO:teuthology.orchestra.run.smithi094.stdout:(18/150): librgw2-19.0.0-3154.ga630765c.el9.x86 7.4 MB/s | 5.4 MB 00:00 2024-04-21T08:27:21.651 INFO:teuthology.orchestra.run.smithi094.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 1.1 MB/s | 45 kB 00:00 2024-04-21T08:27:21.710 INFO:teuthology.orchestra.run.smithi094.stdout:(20/150): python3-ceph-common-19.0.0-3154.ga630 2.3 MB/s | 135 kB 00:00 2024-04-21T08:27:21.760 INFO:teuthology.orchestra.run.smithi094.stdout:(21/150): python3-cephfs-19.0.0-3154.ga630765c. 3.2 MB/s | 162 kB 00:00 2024-04-21T08:27:21.827 INFO:teuthology.orchestra.run.smithi094.stdout:(22/150): python3-rados-19.0.0-3154.ga630765c.e 4.7 MB/s | 321 kB 00:00 2024-04-21T08:27:21.894 INFO:teuthology.orchestra.run.smithi094.stdout:(23/150): python3-rbd-19.0.0-3154.ga630765c.el9 4.4 MB/s | 298 kB 00:00 2024-04-21T08:27:21.944 INFO:teuthology.orchestra.run.smithi094.stdout:(24/150): python3-rgw-19.0.0-3154.ga630765c.el9 2.0 MB/s | 100 kB 00:00 2024-04-21T08:27:22.178 INFO:teuthology.orchestra.run.smithi094.stdout:(25/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 365 kB/s | 85 kB 00:00 2024-04-21T08:27:22.537 INFO:teuthology.orchestra.run.smithi094.stdout:(26/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 8.6 MB/s | 3.1 MB 00:00 2024-04-21T08:27:22.587 INFO:teuthology.orchestra.run.smithi094.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 3.4 MB/s | 171 kB 00:00 2024-04-21T08:27:22.637 INFO:teuthology.orchestra.run.smithi094.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 688 kB/s | 34 kB 00:00 2024-04-21T08:27:22.696 INFO:teuthology.orchestra.run.smithi094.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 2.4 MB/s | 142 kB 00:00 2024-04-21T08:27:23.097 INFO:teuthology.orchestra.run.smithi094.stdout:(30/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 8.9 MB/s | 3.5 MB 00:00 2024-04-21T08:27:23.871 INFO:teuthology.orchestra.run.smithi094.stdout:(31/150): ceph-mgr-diskprediction-local-19.0.0- 9.6 MB/s | 7.4 MB 00:00 2024-04-21T08:27:23.930 INFO:teuthology.orchestra.run.smithi094.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3154.ga6 4.2 MB/s | 249 kB 00:00 2024-04-21T08:27:23.980 INFO:teuthology.orchestra.run.smithi094.stdout:(33/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 987 kB/s | 49 kB 00:00 2024-04-21T08:27:24.031 INFO:teuthology.orchestra.run.smithi094.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3154.ga 328 kB/s | 16 kB 00:00 2024-04-21T08:27:24.098 INFO:teuthology.orchestra.run.smithi094.stdout:(35/150): ceph-volume-19.0.0-3154.ga630765c.el9 4.0 MB/s | 273 kB 00:00 2024-04-21T08:27:24.223 INFO:teuthology.orchestra.run.smithi094.stdout:(36/150): cephadm-19.0.0-3154.ga630765c.el9.noa 5.9 MB/s | 762 kB 00:00 2024-04-21T08:27:24.274 INFO:teuthology.orchestra.run.smithi094.stdout:(37/150): cephfs-top-19.0.0-3154.ga630765c.el9. 366 kB/s | 18 kB 00:00 2024-04-21T08:27:24.486 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-21T08:27:24.568 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:27:24.685 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:27:24.855 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:27:24.991 INFO:teuthology.orchestra.run.smithi094.stdout:(38/150): fuse-2.9.9-15.el9.x86_64.rpm 112 kB/s | 80 kB 00:00 2024-04-21T08:27:25.193 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:27:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:27:25.366 INFO:teuthology.orchestra.run.smithi094.stdout:(39/150): ceph-osd-19.0.0-3154.ga630765c.el9.x8 2.3 MB/s | 17 MB 00:07 2024-04-21T08:27:25.424 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:27:25.453 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:27:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:27:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:27:25.599 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:27:25.625 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:27:25.781 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:27:25.839 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:27:25.866 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:27:25.971 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:27:26.001 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:27:26.072 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:27:26.094 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:27:26.116 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:27:26.293 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:27:26.330 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:27:26.434 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:27:26.449 INFO:teuthology.orchestra.run.smithi094.stdout:(40/150): ceph-test-19.0.0-3154.ga630765c.el9.x 7.1 MB/s | 49 MB 00:06 2024-04-21T08:27:26.475 INFO:teuthology.orchestra.run.smithi094.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 180 kB/s | 195 kB 00:01 2024-04-21T08:27:26.533 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:27:26.568 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:27:26.603 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:27:26.676 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:27:26.719 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:27:26.762 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:27:26.794 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:27:26.822 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:27:26.875 INFO:teuthology.orchestra.run.smithi094.stdout:(42/150): libgfortran-11.4.1-3.el9.x86_64.rpm 426 kB/s | 803 kB 00:01 2024-04-21T08:27:26.893 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:27:27.047 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:27:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-04-21T08:27:27.133 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:27:27.169 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:27:27.178 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:27:27.204 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:27:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:27:27.254 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:27:27.279 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:27:27.322 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:27:27.343 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:27:27.416 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:27:27.425 INFO:teuthology.orchestra.run.smithi094.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 260 kB/s | 253 kB 00:00 2024-04-21T08:27:27.457 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:27:27.534 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:27:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:27:27.643 INFO:teuthology.orchestra.run.smithi094.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 139 kB/s | 106 kB 00:00 2024-04-21T08:27:27.672 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:27:27.705 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:27:27.728 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:27:27.818 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:27:27.839 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:27:27.914 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:27:27.956 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:27:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:27:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:27:28.162 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:27:28.209 INFO:teuthology.orchestra.run.smithi094.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 173 kB/s | 135 kB 00:00 2024-04-21T08:27:28.216 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:27:28.244 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:27:28.268 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:27:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:27:28.349 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:27:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:27:28.377 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:27:28.410 INFO:teuthology.orchestra.run.smithi094.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 163 kB/s | 125 kB 00:00 2024-04-21T08:27:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:27:28.433 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:27:28.454 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:27:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:27:28.472 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:27:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:27:28.535 INFO:teuthology.orchestra.run.smithi094.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 5.4 MB/s | 689 kB 00:00 2024-04-21T08:27:28.547 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:27:28.621 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:27:28.627 INFO:teuthology.orchestra.run.smithi094.stdout:(48/150): python3-cryptography-36.0.1-4.el9.x86 583 kB/s | 1.2 MB 00:02 2024-04-21T08:27:28.637 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:27:28.658 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:27:28.671 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:27:28.677 INFO:teuthology.orchestra.run.smithi094.stdout:(49/150): bison-3.7.4-5.el9.x86_64.rpm 6.5 MB/s | 943 kB 00:00 2024-04-21T08:27:28.695 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:27:28.719 INFO:teuthology.orchestra.run.smithi094.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-04-21T08:27:28.736 INFO:teuthology.orchestra.run.smithi094.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 365 kB/s | 21 kB 00:00 2024-04-21T08:27:28.743 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:27:28.777 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:27:28.778 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:27:28.787 INFO:teuthology.orchestra.run.smithi094.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 656 kB/s | 33 kB 00:00 2024-04-21T08:27:28.820 INFO:teuthology.orchestra.run.smithi094.stdout:(53/150): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 313 kB 00:00 2024-04-21T08:27:28.821 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:27:28.863 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:27:28.870 INFO:teuthology.orchestra.run.smithi094.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-04-21T08:27:28.873 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:27:28.903 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:27:28.921 INFO:teuthology.orchestra.run.smithi094.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-21T08:27:28.946 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:27:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:27:29.010 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:27:29.030 INFO:teuthology.orchestra.run.smithi094.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-04-21T08:27:29.046 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:27:29.051 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:27:29.073 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:27:29.088 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:27:29.096 INFO:teuthology.orchestra.run.smithi094.stdout:(57/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.7 MB/s | 3.0 MB 00:00 2024-04-21T08:27:29.115 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:27:29.121 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:27:29.122 INFO:teuthology.orchestra.run.smithi094.stdout:(58/150): python3-urllib3-1.26.5-5.el9.noarch.r 235 kB/s | 215 kB 00:00 2024-04-21T08:27:29.147 INFO:teuthology.orchestra.run.smithi094.stdout:(59/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 386 kB/s | 45 kB 00:00 2024-04-21T08:27:29.149 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:27:29.197 INFO:teuthology.orchestra.run.smithi094.stdout:(60/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 510 kB/s | 25 kB 00:00 2024-04-21T08:27:29.212 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:27:29.218 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:27:29.223 INFO:teuthology.orchestra.run.smithi094.stdout:(61/150): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-04-21T08:27:29.248 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:27:29.256 INFO:teuthology.orchestra.run.smithi094.stdout:(62/150): libtool-2.4.6-45.el9.x86_64.rpm 4.3 MB/s | 582 kB 00:00 2024-04-21T08:27:29.272 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:27:29.282 INFO:teuthology.orchestra.run.smithi094.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-04-21T08:27:29.288 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:27:29.302 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:27:29.313 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:27:29.332 INFO:teuthology.orchestra.run.smithi094.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-21T08:27:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:27:29.338 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:27:29.358 INFO:teuthology.orchestra.run.smithi094.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 467 kB/s | 35 kB 00:00 2024-04-21T08:27:29.383 INFO:teuthology.orchestra.run.smithi094.stdout:(66/150): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-04-21T08:27:29.414 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:27:29.414 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:27:29.448 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:27:29.491 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:27:29.520 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:27:29.533 INFO:teuthology.orchestra.run.smithi094.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-21T08:27:29.543 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:27:29.579 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:27:29.591 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:27:29.598 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:27:29.633 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:27:29.642 INFO:teuthology.orchestra.run.smithi094.stdout:(68/150): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-04-21T08:27:29.643 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:27:29.677 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:27:29.701 INFO:teuthology.orchestra.run.smithi094.stdout:(69/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 12 MB/s | 4.1 MB 00:00 2024-04-21T08:27:29.715 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:27:29.718 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:27:29.743 INFO:teuthology.orchestra.run.smithi094.stdout:(70/150): postgresql-contrib-13.14-1.el9.x86_64 4.3 MB/s | 911 kB 00:00 2024-04-21T08:27:29.755 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:27:29.785 INFO:teuthology.orchestra.run.smithi094.stdout:(71/150): postgresql-private-libs-13.14-1.el9.x 939 kB/s | 133 kB 00:00 2024-04-21T08:27:29.785 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:27:29.819 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:27:29.843 INFO:teuthology.orchestra.run.smithi094.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-21T08:27:29.852 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:27:29.853 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:27:29.877 INFO:teuthology.orchestra.run.smithi094.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 7.5 MB/s | 1.0 MB 00:00 2024-04-21T08:27:29.894 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:27:29.905 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:27:29.928 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:27:29.952 INFO:teuthology.orchestra.run.smithi094.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 3.2 MB/s | 244 kB 00:00 2024-04-21T08:27:30.018 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:27:30.036 INFO:teuthology.orchestra.run.smithi094.stdout:(75/150): postgresql-server-13.14-1.el9.x86_64. 17 MB/s | 5.8 MB 00:00 2024-04-21T08:27:30.067 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:27:30.087 INFO:teuthology.orchestra.run.smithi094.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 515 kB/s | 26 kB 00:00 2024-04-21T08:27:30.098 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:27:30.128 INFO:teuthology.orchestra.run.smithi094.stdout:(77/150): python3-jinja2-2.11.3-5.el9.noarch.rp 1.4 MB/s | 248 kB 00:00 2024-04-21T08:27:30.145 INFO:teuthology.orchestra.run.smithi094.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 327 kB/s | 19 kB 00:00 2024-04-21T08:27:30.146 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:27:30.187 INFO:teuthology.orchestra.run.smithi094.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 833 kB/s | 35 kB 00:00 2024-04-21T08:27:30.213 INFO:teuthology.orchestra.run.smithi094.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-21T08:27:30.244 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:27:30.296 INFO:teuthology.orchestra.run.smithi094.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-04-21T08:27:30.300 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:27:30.363 INFO:teuthology.orchestra.run.smithi094.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-04-21T08:27:30.397 INFO:teuthology.orchestra.run.smithi094.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-21T08:27:30.444 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:27:30.447 INFO:teuthology.orchestra.run.smithi094.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-04-21T08:27:30.473 INFO:teuthology.orchestra.run.smithi094.stdout:(85/150): python3-packaging-20.9-5.el9.noarch.r 709 kB/s | 77 kB 00:00 2024-04-21T08:27:30.525 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:27:30.553 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:27:30.581 INFO:teuthology.orchestra.run.smithi094.stdout:(86/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-04-21T08:27:30.596 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:27:30.606 INFO:teuthology.orchestra.run.smithi094.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-21T08:27:30.631 INFO:teuthology.orchestra.run.smithi094.stdout:(88/150): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-04-21T08:27:30.634 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:27:30.667 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:27:30.682 INFO:teuthology.orchestra.run.smithi094.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-04-21T08:27:30.716 INFO:teuthology.orchestra.run.smithi094.stdout:(90/150): python3-pyasn1-modules-0.4.8-6.el9.no 2.0 MB/s | 279 kB 00:00 2024-04-21T08:27:30.723 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:27:30.741 INFO:teuthology.orchestra.run.smithi094.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 491 kB/s | 54 kB 00:00 2024-04-21T08:27:30.792 INFO:teuthology.orchestra.run.smithi094.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-04-21T08:27:30.796 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:27:30.817 INFO:teuthology.orchestra.run.smithi094.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 585 kB/s | 44 kB 00:00 2024-04-21T08:27:30.826 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:27:30.863 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:27:30.884 INFO:teuthology.orchestra.run.smithi094.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 538 kB/s | 36 kB 00:00 2024-04-21T08:27:30.910 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:27:30.918 INFO:teuthology.orchestra.run.smithi094.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-04-21T08:27:30.943 INFO:teuthology.orchestra.run.smithi094.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-04-21T08:27:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:27:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:27:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:27:31.048 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:27:31.085 INFO:teuthology.orchestra.run.smithi094.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 408 kB/s | 58 kB 00:00 2024-04-21T08:27:31.097 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:27:31.127 INFO:teuthology.orchestra.run.smithi094.stdout:(98/150): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 305 kB 00:00 2024-04-21T08:27:31.131 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:27:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:27:31.186 INFO:teuthology.orchestra.run.smithi094.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 934 kB/s | 94 kB 00:00 2024-04-21T08:27:31.211 INFO:teuthology.orchestra.run.smithi094.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 761 kB/s | 64 kB 00:00 2024-04-21T08:27:31.220 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:27:31.234 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:27:31.250 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:27:31.286 INFO:teuthology.orchestra.run.smithi094.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 447 kB/s | 45 kB 00:00 2024-04-21T08:27:31.289 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:27:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:27:31.312 INFO:teuthology.orchestra.run.smithi094.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-04-21T08:27:31.379 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:27:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:27:31.416 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:27:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:27:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:27:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:27:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:27:31.554 INFO:teuthology.orchestra.run.smithi094.stdout:(103/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-04-21T08:27:31.594 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:27:31.616 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:27:31.622 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:27:31.701 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:27:31.704 INFO:teuthology.orchestra.run.smithi094.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-04-21T08:27:31.744 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:27:31.789 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:27:31.813 INFO:teuthology.orchestra.run.smithi094.stdout:(105/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-21T08:27:31.817 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:27:31.851 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:27:31.863 INFO:teuthology.orchestra.run.smithi094.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-04-21T08:27:31.895 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:27:31.897 INFO:teuthology.orchestra.run.smithi094.stdout:(107/150): gperftools-libs-2.9.1-2.el9.x86_64.r 902 kB/s | 309 kB 00:00 2024-04-21T08:27:31.933 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:27:31.938 INFO:teuthology.orchestra.run.smithi094.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-04-21T08:27:31.964 INFO:teuthology.orchestra.run.smithi094.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 724 kB/s | 49 kB 00:00 2024-04-21T08:27:31.973 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:27:32.003 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:27:32.037 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:27:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:27:32.089 INFO:teuthology.orchestra.run.smithi094.stdout:(110/150): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-21T08:27:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:27:32.115 INFO:teuthology.orchestra.run.smithi094.stdout:(111/150): libunwind-1.6.2-1.el9.x86_64.rpm 383 kB/s | 67 kB 00:00 2024-04-21T08:27:32.140 INFO:teuthology.orchestra.run.smithi094.stdout:(112/150): luarocks-3.9.2-1.el9.noarch.rpm 861 kB/s | 151 kB 00:00 2024-04-21T08:27:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:27:32.174 INFO:teuthology.orchestra.run.smithi094.stdout:(113/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-04-21T08:27:32.234 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:27:32.266 INFO:teuthology.orchestra.run.smithi094.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 347 kB/s | 43 kB 00:00 2024-04-21T08:27:32.283 INFO:teuthology.orchestra.run.smithi094.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 296 kB/s | 32 kB 00:00 2024-04-21T08:27:32.285 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:27:32.341 INFO:teuthology.orchestra.run.smithi094.stdout:(116/150): python3-asyncssh-2.13.2-5.el9.noarch 2.4 MB/s | 548 kB 00:00 2024-04-21T08:27:32.362 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:27:32.367 INFO:teuthology.orchestra.run.smithi094.stdout:(117/150): python3-certifi-2021.10.8-2.el9.noar 148 kB/s | 15 kB 00:00 2024-04-21T08:27:32.384 INFO:teuthology.orchestra.run.smithi094.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-21T08:27:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:27:32.443 INFO:teuthology.orchestra.run.smithi094.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-21T08:27:32.468 INFO:teuthology.orchestra.run.smithi094.stdout:(120/150): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-21T08:27:32.481 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:27:32.484 INFO:teuthology.orchestra.run.smithi094.stdout:(121/150): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-04-21T08:27:32.518 INFO:teuthology.orchestra.run.smithi094.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-04-21T08:27:32.560 INFO:teuthology.orchestra.run.smithi094.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-21T08:27:32.586 INFO:teuthology.orchestra.run.smithi094.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 159 kB/s | 11 kB 00:00 2024-04-21T08:27:32.591 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:27:32.636 INFO:teuthology.orchestra.run.smithi094.stdout:(125/150): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-21T08:27:32.661 INFO:teuthology.orchestra.run.smithi094.stdout:(126/150): python3-jaraco-classes-3.2.1-5.el9.n 175 kB/s | 18 kB 00:00 2024-04-21T08:27:32.687 INFO:teuthology.orchestra.run.smithi094.stdout:(127/150): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-04-21T08:27:32.702 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:27:32.712 INFO:teuthology.orchestra.run.smithi094.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-21T08:27:32.752 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:27:32.754 INFO:teuthology.orchestra.run.smithi094.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 295 kB/s | 20 kB 00:00 2024-04-21T08:27:32.787 INFO:teuthology.orchestra.run.smithi094.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 614 kB/s | 46 kB 00:00 2024-04-21T08:27:32.838 INFO:teuthology.orchestra.run.smithi094.stdout:(131/150): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-04-21T08:27:32.863 INFO:teuthology.orchestra.run.smithi094.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 765 kB/s | 58 kB 00:00 2024-04-21T08:27:32.888 INFO:teuthology.orchestra.run.smithi094.stdout:(133/150): python3-more-itertools-8.12.0-2.el9. 587 kB/s | 79 kB 00:00 2024-04-21T08:27:32.914 INFO:teuthology.orchestra.run.smithi094.stdout:(134/150): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-21T08:27:32.939 INFO:teuthology.orchestra.run.smithi094.stdout:(135/150): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-04-21T08:27:32.964 INFO:teuthology.orchestra.run.smithi094.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-21T08:27:32.990 INFO:teuthology.orchestra.run.smithi094.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 373 kB/s | 28 kB 00:00 2024-04-21T08:27:33.015 INFO:teuthology.orchestra.run.smithi094.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 406 kB/s | 31 kB 00:00 2024-04-21T08:27:33.040 INFO:teuthology.orchestra.run.smithi094.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-21T08:27:33.066 INFO:teuthology.orchestra.run.smithi094.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 782 kB/s | 59 kB 00:00 2024-04-21T08:27:33.091 INFO:teuthology.orchestra.run.smithi094.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 472 kB/s | 36 kB 00:00 2024-04-21T08:27:33.116 INFO:teuthology.orchestra.run.smithi094.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-04-21T08:27:33.142 INFO:teuthology.orchestra.run.smithi094.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-21T08:27:33.167 INFO:teuthology.orchestra.run.smithi094.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-21T08:27:33.209 INFO:teuthology.orchestra.run.smithi094.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 298 kB/s | 20 kB 00:00 2024-04-21T08:27:33.235 INFO:teuthology.orchestra.run.smithi094.stdout:(146/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-04-21T08:27:33.260 INFO:teuthology.orchestra.run.smithi094.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-21T08:27:33.310 INFO:teuthology.orchestra.run.smithi094.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-21T08:27:33.488 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:27:33.540 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:27:33.574 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:27:33.675 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:27:33.732 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:27:33.752 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:27:33.820 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:27:33.860 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:27:33.867 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:27:33.887 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:27:33.920 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:27:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:27:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:27:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:27:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:27:34.292 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:27:34.382 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:27:34.413 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:27:34.449 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:27:34.816 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:27:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:27:34.926 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:27:34.961 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:27:35.015 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:27:35.038 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:27:35.685 INFO:teuthology.orchestra.run.smithi094.stdout:(149/150): librbd1-19.0.0-3154.ga630765c.el9.x8 1.3 MB/s | 3.1 MB 00:02 2024-04-21T08:27:36.160 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:27:36.243 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:27:36.271 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:27:36.304 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:27:36.609 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:27:36.635 INFO:teuthology.orchestra.run.smithi094.stdout:(150/150): librados2-19.0.0-3154.ga630765c.el9. 1.0 MB/s | 3.4 MB 00:03 2024-04-21T08:27:36.639 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:27:36.639 INFO:teuthology.orchestra.run.smithi094.stdout:Total 8.6 MB/s | 218 MB 00:25 2024-04-21T08:27:36.650 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:27:36.749 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:27:36.780 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:27:36.818 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:27:37.290 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:27:37.293 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:27:37.320 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:27:37.326 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:27:37.391 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:27:37.413 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:27:37.682 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-21T08:27:37.809 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-21T08:27:37.809 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-21T08:27:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:27:37.896 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:27:37.989 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:27:37.990 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:27:37.990 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:27:37.990 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:38.022 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:27:39.523 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-21T08:27:39.524 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-21T08:27:39.766 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:27:39.802 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:27:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:27:40.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:27:40.481 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:27:40.481 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:27:40.481 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:27:40.481 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:40.532 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:27:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-21T08:27:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:27:45.539 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:27:45.581 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:27:45.911 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:27:46.032 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:27:46.161 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:27:46.183 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:27:46.274 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:27:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:27:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:27:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:27:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:27:46.578 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:27:46.606 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:27:46.710 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:27:46.732 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:27:46.795 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:27:46.817 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:27:46.838 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:27:47.034 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:27:47.067 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:27:47.173 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:27:47.272 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:27:47.317 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:27:47.352 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:27:47.426 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:27:47.468 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:27:47.513 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:27:47.561 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:27:47.590 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:27:47.661 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:27:47.721 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:27:47.751 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:27:47.795 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:27:47.872 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:27:47.914 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:27:47.957 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:27:47.986 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:27:48.058 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:27:48.109 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:27:48.185 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:27:48.327 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:27:48.379 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:27:48.905 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:27:48.992 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:27:49.021 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:27:49.064 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:27:49.085 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:27:49.085 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-04-21T08:27:49.085 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-04-21T08:27:49.085 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-04-21T08:27:49.085 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-04-21T08:27:49.085 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-04-21T08:27:49.085 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /root 2024-04-21T08:27:49.085 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-04-21T08:27:49.085 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:49.106 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:27:49.143 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:27:49.199 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:27:49.260 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:27:49.271 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:27:49.274 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:27:49.275 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:49.275 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:27:49.275 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:49.303 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:27:49.340 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:27:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:27:49.423 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:27:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:27:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:27:49.645 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:27:49.689 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:27:49.719 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:27:49.761 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:27:49.856 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:27:49.903 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:27:49.943 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:27:49.968 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:27:49.993 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:27:50.068 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:27:50.196 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:27:50.235 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:27:50.254 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:27:50.289 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:27:50.333 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:27:50.371 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:27:50.402 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:27:50.433 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:27:50.467 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:27:50.517 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:27:50.559 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:27:50.593 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:27:50.682 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:27:50.724 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:27:50.747 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:27:50.757 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:27:50.772 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:27:50.772 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:50.772 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:27:50.772 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:27:50.773 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:27:50.773 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:50.802 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:27:50.901 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:27:50.966 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:27:50.997 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:27:51.020 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:27:51.020 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:51.020 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:27:51.020 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:27:51.020 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:27:51.020 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:51.473 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:27:51.495 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:27:51.496 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:51.496 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:27:51.496 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:27:51.496 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:27:51.496 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:51.595 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:27:51.652 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:27:51.700 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:27:51.707 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:27:51.747 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:27:51.780 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:27:51.881 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:27:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:27:51.939 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:27:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:27:52.027 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:27:52.027 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:27:52.058 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:27:52.116 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:27:52.201 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:27:52.201 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-04-21T08:27:52.201 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-04-21T08:27:52.201 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-04-21T08:27:52.201 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-04-21T08:27:52.202 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-04-21T08:27:52.202 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-04-21T08:27:52.202 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-04-21T08:27:52.202 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:27:52.264 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:27:52.265 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:27:52.307 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:27:52.339 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:27:52.342 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:27:52.367 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:27:52.376 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:27:52.392 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:27:52.392 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:52.392 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:27:52.392 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:52.794 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:27:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:27:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:27:52.918 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:27:52.947 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:27:53.015 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:27:53.123 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:27:53.869 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:27:53.895 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:27:53.896 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:53.896 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:27:53.896 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:27:53.896 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:27:53.896 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:53.991 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:27:54.024 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:27:54.128 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:27:54.151 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:27:54.151 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:54.151 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:27:54.151 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:27:54.151 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:27:54.151 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:54.369 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:27:54.464 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:27:54.492 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:27:54.533 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:27:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:27:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:27:54.637 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:54.637 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:27:54.637 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:27:54.637 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:27:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:27:54.800 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:27:54.849 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:27:54.855 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:27:54.888 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:27:54.978 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:27:54.994 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:27:55.010 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:27:55.024 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:27:55.042 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:27:55.046 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:27:55.089 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:27:55.127 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:27:55.166 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:27:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:27:55.297 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:27:55.326 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:27:55.349 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:27:55.349 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:55.349 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:27:55.349 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:27:55.349 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:27:55.349 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:55.385 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:27:55.393 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:27:55.404 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:27:55.431 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:27:55.463 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:27:55.491 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:27:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:27:55.526 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:27:55.550 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:27:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:27:55.604 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:27:55.635 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:27:55.918 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:27:55.939 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:27:55.971 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:27:56.033 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:27:56.070 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:27:56.509 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:27:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:27:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:27:56.569 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:56.569 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:27:56.569 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:27:56.569 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:27:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:56.613 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:27:56.635 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:27:56.635 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:56.636 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:27:56.636 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:56.681 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:27:56.703 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:27:56.703 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:56.703 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:27:56.703 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:27:56.703 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:27:56.703 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:57.015 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:27:57.037 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:27:57.037 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:57.037 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:27:57.037 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:27:57.038 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:27:57.038 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:27:57.099 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:27:57.139 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:27:57.946 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:27:57.991 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:27:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:27:58.139 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:27:58.241 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:27:58.411 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:27:58.441 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:27:58.464 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:27:58.465 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:58.465 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:27:58.465 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:27:58.465 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:27:58.465 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:58.508 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:27:58.520 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:27:58.536 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:27:58.579 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:27:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:27:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:27:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:27:58.672 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:27:58.673 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:27:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:27:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:27:59.586 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:27:59.627 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:27:59.650 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:27:59.650 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:59.650 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:27:59.651 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:27:59.651 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:27:59.651 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:59.695 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:27:59.717 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:27:59.717 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:59.717 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:27:59.717 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:27:59.762 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:27:59.787 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:27:59.787 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:27:59.787 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:27:59.787 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:27:59.787 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:27:59.788 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:28:00.087 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:28:00.110 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:28:00.110 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:28:00.110 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:28:00.110 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:28:00.110 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:28:00.110 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:28:01.602 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:28:01.643 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:28:01.684 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:28:01.716 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:28:01.743 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:28:01.929 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:28:01.982 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:28:02.011 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:28:02.635 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:28:02.676 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:28:02.697 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:28:03.481 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:28:03.563 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:28:03.604 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:28:03.687 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:28:03.718 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:28:03.718 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:28:03.785 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:28:03.785 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:28:04.699 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:28:04.749 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:28:04.790 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:28:04.822 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:28:04.849 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:28:05.031 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:28:05.070 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:28:05.100 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:28:05.657 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:28:05.689 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:28:05.709 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:28:06.494 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:28:06.585 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:28:06.626 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:28:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:28:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:28:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:28:06.823 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:28:06.857 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:28:09.922 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:28:09.922 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:28:09.922 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:28:09.922 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:28:09.922 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:28:09.922 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:28:09.922 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:28:09.922 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:28:09.922 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:28:09.922 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:28:09.923 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:28:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:28:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:28:09.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:28:09.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:28:09.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:28:09.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:28:09.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:28:09.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:28:09.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:28:09.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:28:09.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:28:09.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:28:09.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:28:09.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:28:10.177 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:28:10.178 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-04-21T08:28:10.178 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-04-21T08:28:10.178 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-04-21T08:28:10.178 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-04-21T08:28:10.178 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-04-21T08:28:10.178 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-04-21T08:28:10.178 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-04-21T08:28:10.178 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:10.367 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:28:10.382 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:28:10.383 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:28:10.383 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:28:10.383 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:11.862 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:28:11.888 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:28:11.888 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:28:11.888 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:28:11.888 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:28:11.889 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:28:11.889 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:11.987 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:28:11.987 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:28:11.987 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-04-21T08:28:11.987 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.987 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.987 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-21T08:28:11.987 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:28:11.987 INFO:teuthology.orchestra.run.smithi111.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:11.988 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:11.989 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:28:11.990 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.991 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:28:11.992 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:28:11.993 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:28:11.993 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:28:11.993 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:28:11.993 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:28:11.993 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:28:11.993 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:28:11.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:28:11.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:28:11.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:28:11.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:28:11.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:28:11.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:28:11.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:28:11.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:28:11.998 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.999 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.999 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:11.999 INFO:teuthology.orchestra.run.smithi111.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:28:11.999 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:28:11.999 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:28:11.999 INFO:teuthology.orchestra.run.smithi111.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:28:11.999 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:28:11.999 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:28:12.000 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:28:12.000 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T08:28:12.000 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-21T08:28:12.122 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:28:12.143 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:28:12.144 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:28:12.144 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:28:12.144 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:28:12.144 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:28:12.144 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:12.352 DEBUG:teuthology.parallel:result is None 2024-04-21T08:28:12.581 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:28:12.603 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:28:12.603 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:28:12.603 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:28:12.603 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:28:12.603 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:28:12.603 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:28:12.771 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:28:12.771 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:28:12.772 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:28:12.773 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:28:12.779 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:28:12.786 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:28:12.786 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:28:12.786 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:28:12.786 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:28:12.786 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:28:12.786 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:28:12.787 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:28:12.788 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:28:12.789 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:28:12.790 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:28:12.791 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:28:12.792 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:28:12.793 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:28:12.794 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:28:12.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:28:12.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:28:12.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:28:12.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:28:12.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:28:12.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:28:12.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:28:12.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:28:12.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:28:12.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:28:12.831 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:28:13.017 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:28:13.066 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:28:13.169 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:28:13.257 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:28:13.388 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:28:13.425 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:28:13.465 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:28:13.493 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:28:13.945 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:28:13.976 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:28:14.008 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:28:14.069 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:28:14.108 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:28:14.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.831 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.832 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:28:14.833 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:28:14.834 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:28:14.835 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:28:14.836 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:28:14.837 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:28:14.838 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:28:14.839 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:28:14.840 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:28:14.841 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:28:14.841 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.841 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.841 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:14.841 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:28:14.841 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:28:14.841 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:28:14.841 INFO:teuthology.orchestra.run.smithi173.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:28:14.841 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:28:14.841 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:28:14.842 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:28:14.842 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:28:14.842 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-04-21T08:28:15.137 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:28:15.177 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:28:15.199 DEBUG:teuthology.parallel:result is None 2024-04-21T08:28:16.134 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:28:16.169 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:28:16.271 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:28:16.598 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:28:16.630 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:28:16.652 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:28:16.652 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:28:16.652 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:28:16.652 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:28:16.653 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:28:16.653 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:16.698 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:28:16.709 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:28:16.810 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:28:16.862 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:28:17.781 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:28:17.809 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:28:17.832 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:28:17.832 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:28:17.832 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:28:17.832 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:28:17.832 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:28:17.832 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:17.877 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:28:17.898 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:28:17.898 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:28:17.898 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:28:17.899 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:28:17.966 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:28:17.966 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:28:17.966 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:28:17.966 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:28:17.966 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:28:17.966 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:18.261 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:28:18.283 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:28:18.284 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:28:18.284 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:28:18.284 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:28:18.284 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:28:18.284 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:22.858 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:28:22.907 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:28:22.951 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:28:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:28:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:28:23.204 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:28:23.247 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:28:23.277 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:28:23.869 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:28:23.916 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:28:23.935 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:28:24.707 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:28:24.795 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:28:24.836 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:28:24.915 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:28:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:28:24.966 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:28:25.049 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:28:25.050 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:28:31.115 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:28:31.115 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:28:31.115 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:28:31.115 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:28:31.115 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:28:31.115 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:28:31.116 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:28:31.117 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:28:31.117 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:28:31.117 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:28:31.117 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:28:31.117 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:28:31.117 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:28:31.117 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:28:31.117 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:28:31.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:28:31.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:28:31.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:28:31.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:28:31.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:28:31.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:28:31.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:28:31.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:28:31.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:28:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:28:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:28:31.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:28:31.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:28:31.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:28:31.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:28:31.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:28:31.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:28:31.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:28:31.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:28:31.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:28:31.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:28:31.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:28:31.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:28:31.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:28:31.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:28:31.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:28:33.167 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:28:33.167 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:33.167 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-04-21T08:28:33.167 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.168 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.168 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-21T08:28:33.168 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:28:33.168 INFO:teuthology.orchestra.run.smithi094.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:28:33.169 INFO:teuthology.orchestra.run.smithi094.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:28:33.169 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:28:33.169 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.169 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.169 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.170 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.170 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:33.170 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.170 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.170 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.170 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:33.171 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:33.171 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:33.171 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:33.171 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:33.171 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.172 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.172 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:33.172 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.172 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.172 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.172 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:33.172 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:33.172 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.173 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.173 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:28:33.173 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:28:33.173 INFO:teuthology.orchestra.run.smithi094.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:28:33.173 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:28:33.173 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:28:33.174 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:28:33.174 INFO:teuthology.orchestra.run.smithi094.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:28:33.174 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:28:33.174 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:28:33.174 INFO:teuthology.orchestra.run.smithi094.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:28:33.175 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:28:33.175 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:28:33.175 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.175 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.175 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.175 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:28:33.176 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:28:33.176 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:28:33.176 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:28:33.176 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:28:33.176 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.177 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.177 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:28:33.177 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.177 INFO:teuthology.orchestra.run.smithi094.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:28:33.177 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:28:33.177 INFO:teuthology.orchestra.run.smithi094.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:28:33.177 INFO:teuthology.orchestra.run.smithi094.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:28:33.177 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:28:33.178 INFO:teuthology.orchestra.run.smithi094.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:28:33.178 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:28:33.178 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:28:33.178 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:28:33.178 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:28:33.178 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:28:33.178 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:28:33.178 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:28:33.179 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:28:33.179 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:28:33.179 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:28:33.179 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:28:33.179 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:28:33.179 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:28:33.179 INFO:teuthology.orchestra.run.smithi094.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:28:33.180 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:28:33.180 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:28:33.180 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:28:33.180 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.180 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.180 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.180 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:28:33.181 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:28:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.183 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.184 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:28:33.184 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:28:33.184 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:28:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:28:33.186 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:28:33.187 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-21T08:28:33.536 DEBUG:teuthology.parallel:result is None 2024-04-21T08:28:33.536 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:28:33.680 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:28:33.702 INFO:teuthology.orchestra.run.smithi094.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:28:33.703 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:28:33.703 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:28:33.705 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:28:33.853 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:28:33.879 INFO:teuthology.orchestra.run.smithi111.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:28:33.879 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:28:33.879 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:28:33.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:28:34.101 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:28:34.127 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:28:34.127 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:28:34.127 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:28:34.129 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-21T08:28:34.129 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:28:34.129 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:28:34.155 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:28:34.155 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:28:34.186 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:28:34.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:28:34.216 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-21T08:28:34.216 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:28:34.216 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:28:34.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:28:34.304 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:28:34.304 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:28:34.332 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:28:34.399 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:28:34.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:28:34.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:28:34.492 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-21T08:28:34.493 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:28:34.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:28:34.516 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:28:34.582 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:28:34.582 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:28:34.606 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:28:34.671 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:28:34.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:28:34.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:28:34.765 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-21T08:28:34.765 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:28:34.765 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:28:34.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:28:34.853 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:28:34.853 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:28:34.884 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:28:34.948 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:28:34.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:28:34.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:28:35.041 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-21T08:28:35.148 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-21T08:28:35.148 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:28:35.148 INFO:tasks.cephadm:Cluster fsid is 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:28:35.148 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-21T08:28:35.149 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.94', 'mon.b': '172.21.15.111', 'mon.c': '172.21.15.173'} 2024-04-21T08:28:35.149 INFO:tasks.cephadm:First mon is mon.a on smithi094 2024-04-21T08:28:35.149 INFO:tasks.cephadm:First mgr is x 2024-04-21T08:28:35.149 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-21T08:28:35.149 DEBUG:teuthology.orchestra.run.smithi094:> sudo hostname $(hostname -s) 2024-04-21T08:28:35.173 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2024-04-21T08:28:35.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-04-21T08:28:35.226 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-21T08:28:35.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:28:35.446 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78740/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3154-ga630765c', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3154.ga630765c'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-19 11:13:53.331234', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-21T08:28:35.587 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 a630765c1f93c6e61d9b40c4b76437611d5ef29e from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:28:35.589 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm 2024-04-21T08:28:35.589 DEBUG:teuthology.orchestra.run.smithi094:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:28:36.194 INFO:teuthology.orchestra.run.smithi094.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:28 /home/ubuntu/cephtest/cephadm 2024-04-21T08:28:36.194 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:28:36.638 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:28 /home/ubuntu/cephtest/cephadm 2024-04-21T08:28:36.639 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:28:37.257 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:28 /home/ubuntu/cephtest/cephadm 2024-04-21T08:28:37.258 DEBUG:teuthology.orchestra.run.smithi094:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:28:37.276 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:28:37.296 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:28:37.331 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e on all hosts... 2024-04-21T08:28:37.332 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:28:37.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:28:37.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:28:38.845 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:28:38.898 INFO:teuthology.orchestra.run.smithi094.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:28:38.959 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:29:06.790 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-21T08:29:06.790 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:29:06.791 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:29:06.791 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-04-21T08:29:06.791 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:29:06.791 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-04-21T08:29:06.791 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-21T08:29:07.681 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T08:29:07.681 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:29:07.681 INFO:teuthology.orchestra.run.smithi111.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:29:07.681 INFO:teuthology.orchestra.run.smithi111.stdout: "repo_digests": [ 2024-04-21T08:29:07.681 INFO:teuthology.orchestra.run.smithi111.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:29:07.681 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-21T08:29:07.681 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T08:29:14.637 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:29:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:29:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:29:14.637 INFO:teuthology.orchestra.run.smithi094.stdout: "repo_digests": [ 2024-04-21T08:29:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:29:14.638 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:29:14.638 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:29:14.661 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph 2024-04-21T08:29:14.710 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2024-04-21T08:29:14.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-04-21T08:29:14.774 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /etc/ceph 2024-04-21T08:29:14.798 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2024-04-21T08:29:14.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-04-21T08:29:14.854 INFO:tasks.cephadm:Writing seed config... 2024-04-21T08:29:14.855 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-21T08:29:14.855 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-21T08:29:14.855 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-21T08:29:14.855 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-21T08:29:14.855 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-21T08:29:14.855 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-21T08:29:14.855 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-21T08:29:14.856 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-21T08:29:14.857 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-21T08:29:14.858 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-21T08:29:14.859 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:29:14.859 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-21T08:29:14.874 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2518ed94-ffb9-11ee-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-21T08:29:14.874 DEBUG:teuthology.orchestra.run.smithi094:mon.a> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.a.service 2024-04-21T08:29:14.916 DEBUG:teuthology.orchestra.run.smithi094:mgr.x> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.x.service 2024-04-21T08:29:14.960 INFO:tasks.cephadm:Bootstrapping... 2024-04-21T08:29:14.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e -v bootstrap --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.94 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:29:15.147 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:29:15.148 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e', '-v', 'bootstrap', '--fsid', '2518ed94-ffb9-11ee-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.94', '--skip-admin-label'] 2024-04-21T08:29:15.148 INFO:teuthology.orchestra.run.smithi094.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-21T08:29:15.148 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying podman|docker is present... 2024-04-21T08:29:15.178 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T08:29:15.178 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying lvm2 is present... 2024-04-21T08:29:15.178 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying time synchronization is in place... 2024-04-21T08:29:15.185 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T08:29:15.186 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T08:29:15.191 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T08:29:15.191 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-21T08:29:15.198 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout enabled 2024-04-21T08:29:15.206 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout active 2024-04-21T08:29:15.206 INFO:teuthology.orchestra.run.smithi094.stdout:Unit chronyd.service is enabled and running 2024-04-21T08:29:15.206 INFO:teuthology.orchestra.run.smithi094.stdout:Repeating the final host check... 2024-04-21T08:29:15.228 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T08:29:15.228 INFO:teuthology.orchestra.run.smithi094.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-21T08:29:15.228 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl is present 2024-04-21T08:29:15.228 INFO:teuthology.orchestra.run.smithi094.stdout:lvcreate is present 2024-04-21T08:29:15.234 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T08:29:15.234 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T08:29:15.240 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T08:29:15.240 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-21T08:29:15.246 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout enabled 2024-04-21T08:29:15.252 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout active 2024-04-21T08:29:15.252 INFO:teuthology.orchestra.run.smithi094.stdout:Unit chronyd.service is enabled and running 2024-04-21T08:29:15.252 INFO:teuthology.orchestra.run.smithi094.stdout:Host looks OK 2024-04-21T08:29:15.252 INFO:teuthology.orchestra.run.smithi094.stdout:Cluster fsid: 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:29:15.252 INFO:teuthology.orchestra.run.smithi094.stdout:Acquiring lock 140417418083488 on /run/cephadm/2518ed94-ffb9-11ee-bc93-c7b262605968.lock 2024-04-21T08:29:15.252 INFO:teuthology.orchestra.run.smithi094.stdout:Lock 140417418083488 acquired on /run/cephadm/2518ed94-ffb9-11ee-bc93-c7b262605968.lock 2024-04-21T08:29:15.253 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying IP 172.21.15.94 port 3300 ... 2024-04-21T08:29:15.253 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying IP 172.21.15.94 port 6789 ... 2024-04-21T08:29:15.253 INFO:teuthology.orchestra.run.smithi094.stdout:Base mon IP(s) is [172.21.15.94:3300, 172.21.15.94:6789], mon addrv is [v2:172.21.15.94:3300,v1:172.21.15.94:6789] 2024-04-21T08:29:15.256 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.94 metric 100 2024-04-21T08:29:15.257 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.94 metric 100 2024-04-21T08:29:15.259 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-21T08:29:15.259 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-21T08:29:15.259 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-21T08:29:15.261 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-21T08:29:15.261 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-21T08:29:15.261 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T08:29:15.262 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-21T08:29:15.262 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout inet6 fe80::cabc:3417:31e5:23d5/64 scope link noprefixroute 2024-04-21T08:29:15.262 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T08:29:15.262 INFO:teuthology.orchestra.run.smithi094.stdout:Mon IP `172.21.15.94` is in CIDR network `172.21.0.0/20` 2024-04-21T08:29:15.262 INFO:teuthology.orchestra.run.smithi094.stdout:Mon IP `172.21.15.94` is in CIDR network `172.21.0.0/20` 2024-04-21T08:29:15.262 INFO:teuthology.orchestra.run.smithi094.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-21T08:29:15.263 INFO:teuthology.orchestra.run.smithi094.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-21T08:29:15.263 INFO:teuthology.orchestra.run.smithi094.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:29:15.707 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stdout b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 2024-04-21T08:29:15.707 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:29:15.707 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Getting image source signatures 2024-04-21T08:29:15.707 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Copying blob sha256:7eab8a2da10d06ac88aea1fd8998a2559b4de9d07c585e3b1e6ee6c4d35fc119 2024-04-21T08:29:15.707 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-21T08:29:15.707 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Copying config sha256:b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 2024-04-21T08:29:15.707 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-21T08:29:16.628 INFO:teuthology.orchestra.run.smithi094.stdout:ceph: stdout ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev) 2024-04-21T08:29:16.628 INFO:teuthology.orchestra.run.smithi094.stdout:Ceph version: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev) 2024-04-21T08:29:16.628 INFO:teuthology.orchestra.run.smithi094.stdout:Extracting ceph user uid/gid from container image... 2024-04-21T08:29:17.434 INFO:teuthology.orchestra.run.smithi094.stdout:stat: stdout 167 167 2024-04-21T08:29:17.434 INFO:teuthology.orchestra.run.smithi094.stdout:Creating initial keys... 2024-04-21T08:29:18.297 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph-authtool: stdout AQBdziRmsffZLRAAiiyhcc6tETQaVBpS8x3tTg== 2024-04-21T08:29:19.296 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph-authtool: stdout AQBeziRmXczJIRAAhFP1ObSLGmR02Q9732bBcQ== 2024-04-21T08:29:20.961 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph-authtool: stdout AQBfziRmR7KtIhAA1hfR/FhUlsXysX0PQht2pg== 2024-04-21T08:29:20.962 INFO:teuthology.orchestra.run.smithi094.stdout:Creating initial monmap... 2024-04-21T08:29:30.879 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T08:29:30.880 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-21T08:29:30.880 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:29:30.880 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T08:29:30.880 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool for a [v2:172.21.15.94:3300,v1:172.21.15.94:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T08:29:30.880 INFO:teuthology.orchestra.run.smithi094.stdout:setting min_mon_release = quincy 2024-04-21T08:29:30.880 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: set fsid to 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:29:30.880 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T08:29:30.880 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:29:30.880 INFO:teuthology.orchestra.run.smithi094.stdout:Creating mon... 2024-04-21T08:29:33.634 INFO:teuthology.orchestra.run.smithi094.stdout:create mon.a on 2024-04-21T08:29:33.847 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-21T08:29:33.987 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-21T08:29:34.171 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2518ed94-ffb9-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-2518ed94-ffb9-11ee-bc93-c7b262605968.target. 2024-04-21T08:29:34.171 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-2518ed94-ffb9-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-2518ed94-ffb9-11ee-bc93-c7b262605968.target. 2024-04-21T08:29:34.432 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.a 2024-04-21T08:29:34.432 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.a.service: Unit ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.a.service not loaded. 2024-04-21T08:29:34.592 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2518ed94-ffb9-11ee-bc93-c7b262605968.target.wants/ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@.service. 2024-04-21T08:29:35.133 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:29:35.134 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-04-21T08:29:35.140 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:29:35.140 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-21T08:29:35.140 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-04-21T08:29:35.141 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T08:29:35.141 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mon to start... 2024-04-21T08:29:35.141 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mon... 2024-04-21T08:29:36.178 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout cluster: 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout id: 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout services: 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.582947s) 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout data: 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout pgs: 2024-04-21T08:29:36.179 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.595+0000 7fbe83281700 1 Processor -- start 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.595+0000 7fbe83281700 1 -- start start 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.595+0000 7fbe83281700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c10b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.595+0000 7fbe83281700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7c07be60 con 0x7fbe7c108fe0 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.595+0000 7fbe8101d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c10b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.595+0000 7fbe8101d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c10b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55196/0 (socket says 172.21.15.94:55196) 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.595+0000 7fbe8101d700 1 -- 172.21.15.94:0/419738790 learned_addr learned my addr 172.21.15.94:0/419738790 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.596+0000 7fbe8101d700 1 -- 172.21.15.94:0/419738790 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7c10b910 con 0x7fbe7c108fe0 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.596+0000 7fbe8101d700 1 --2- 172.21.15.94:0/419738790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c10b3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbe780098a0 tx=0x7fbe78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd85ad1eb03db78e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.596+0000 7fbe73fff700 1 -- 172.21.15.94:0/419738790 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe78004160 con 0x7fbe7c108fe0 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.596+0000 7fbe73fff700 1 -- 172.21.15.94:0/419738790 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe78004300 con 0x7fbe7c108fe0 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.596+0000 7fbe73fff700 1 -- 172.21.15.94:0/419738790 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe780036a0 con 0x7fbe7c108fe0 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.597+0000 7fbe83281700 1 -- 172.21.15.94:0/419738790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 msgr2=0x7fbe7c10b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:36.180 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.597+0000 7fbe83281700 1 --2- 172.21.15.94:0/419738790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c10b3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbe780098a0 tx=0x7fbe78009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.597+0000 7fbe83281700 1 -- 172.21.15.94:0/419738790 shutdown_connections 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.597+0000 7fbe83281700 1 --2- 172.21.15.94:0/419738790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c10b3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.597+0000 7fbe83281700 1 -- 172.21.15.94:0/419738790 >> 172.21.15.94:0/419738790 conn(0x7fbe7c102a70 msgr2=0x7fbe7c104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.597+0000 7fbe83281700 1 -- 172.21.15.94:0/419738790 shutdown_connections 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.597+0000 7fbe83281700 1 -- 172.21.15.94:0/419738790 wait complete. 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.597+0000 7fbe83281700 1 Processor -- start 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.597+0000 7fbe83281700 1 -- start start 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe83281700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c19beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe83281700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7c10bd80 con 0x7fbe7c108fe0 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe8101d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c19beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe8101d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c19beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55200/0 (socket says 172.21.15.94:55200) 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe8101d700 1 -- 172.21.15.94:0/2409630113 learned_addr learned my addr 172.21.15.94:0/2409630113 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe8101d700 1 -- 172.21.15.94:0/2409630113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7c19c3f0 con 0x7fbe7c108fe0 2024-04-21T08:29:36.181 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe8101d700 1 --2- 172.21.15.94:0/2409630113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c19beb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbe78009fb0 tx=0x7fbe78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe727fc700 1 -- 172.21.15.94:0/2409630113 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe78003de0 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe727fc700 1 -- 172.21.15.94:0/2409630113 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe78011630 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe83281700 1 -- 172.21.15.94:0/2409630113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe7c19c680 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe727fc700 1 -- 172.21.15.94:0/2409630113 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe78011930 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe83281700 1 -- 172.21.15.94:0/2409630113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7c19f370 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.598+0000 7fbe727fc700 1 -- 172.21.15.94:0/2409630113 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbe78011ad0 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.599+0000 7fbe727fc700 1 -- 172.21.15.94:0/2409630113 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbe7801b450 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.599+0000 7fbe83281700 1 -- 172.21.15.94:0/2409630113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe7c0640a0 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.602+0000 7fbe727fc700 1 -- 172.21.15.94:0/2409630113 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fbe78004260 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.672+0000 7fbe83281700 1 -- 172.21.15.94:0/2409630113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fbe7c068b90 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.672+0000 7fbe727fc700 1 -- 172.21.15.94:0/2409630113 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fbe78025480 con 0x7fbe7c108fe0 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.673+0000 7fbe83281700 1 -- 172.21.15.94:0/2409630113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 msgr2=0x7fbe7c19beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:36.182 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.673+0000 7fbe83281700 1 --2- 172.21.15.94:0/2409630113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c19beb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbe78009fb0 tx=0x7fbe78004750 comp rx=0 tx=0).stop 2024-04-21T08:29:36.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.673+0000 7fbe83281700 1 -- 172.21.15.94:0/2409630113 shutdown_connections 2024-04-21T08:29:36.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.673+0000 7fbe83281700 1 --2- 172.21.15.94:0/2409630113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7c108fe0 0x7fbe7c19beb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:36.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.673+0000 7fbe83281700 1 -- 172.21.15.94:0/2409630113 >> 172.21.15.94:0/2409630113 conn(0x7fbe7c102a70 msgr2=0x7fbe7c104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:36.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.673+0000 7fbe83281700 1 -- 172.21.15.94:0/2409630113 shutdown_connections 2024-04-21T08:29:36.183 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:35.673+0000 7fbe83281700 1 -- 172.21.15.94:0/2409630113 wait complete. 2024-04-21T08:29:36.183 INFO:teuthology.orchestra.run.smithi094.stdout:mon is available 2024-04-21T08:29:36.183 INFO:teuthology.orchestra.run.smithi094.stdout:Assimilating anything we can from ceph.conf... 2024-04-21T08:29:37.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:37 smithi094 ceph-mon[27611]: from='client.? 172.21.15.94:0/1336136689' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout fsid = 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.94:3300,v1:172.21.15.94:6789] 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:37.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.627+0000 7fd34299d700 1 Processor -- start 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.627+0000 7fd34299d700 1 -- start start 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.628+0000 7fd34299d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c07c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.628+0000 7fd34299d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c07c870 con 0x7fd33c07de30 2024-04-21T08:29:37.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.628+0000 7fd34199b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c07c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.628+0000 7fd34199b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c07c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55212/0 (socket says 172.21.15.94:55212) 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.628+0000 7fd34199b700 1 -- 172.21.15.94:0/3687709554 learned_addr learned my addr 172.21.15.94:0/3687709554 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.628+0000 7fd34199b700 1 -- 172.21.15.94:0/3687709554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33c07c9f0 con 0x7fd33c07de30 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.628+0000 7fd34199b700 1 --2- 172.21.15.94:0/3687709554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c07c330 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0098a0 tx=0x7fd32c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262d9ff0e208f77a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.628+0000 7fd340999700 1 -- 172.21.15.94:0/3687709554 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd32c004160 con 0x7fd33c07de30 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.628+0000 7fd340999700 1 -- 172.21.15.94:0/3687709554 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd32c004300 con 0x7fd33c07de30 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.628+0000 7fd340999700 1 -- 172.21.15.94:0/3687709554 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd32c0036a0 con 0x7fd33c07de30 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.629+0000 7fd34299d700 1 -- 172.21.15.94:0/3687709554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 msgr2=0x7fd33c07c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.629+0000 7fd34299d700 1 --2- 172.21.15.94:0/3687709554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c07c330 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0098a0 tx=0x7fd32c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.629+0000 7fd34299d700 1 -- 172.21.15.94:0/3687709554 shutdown_connections 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.629+0000 7fd34299d700 1 --2- 172.21.15.94:0/3687709554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c07c330 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.629+0000 7fd34299d700 1 -- 172.21.15.94:0/3687709554 >> 172.21.15.94:0/3687709554 conn(0x7fd33c103940 msgr2=0x7fd33c105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:37.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.629+0000 7fd34299d700 1 -- 172.21.15.94:0/3687709554 shutdown_connections 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.629+0000 7fd34299d700 1 -- 172.21.15.94:0/3687709554 wait complete. 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.629+0000 7fd34299d700 1 Processor -- start 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd34299d700 1 -- start start 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd34299d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c19ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd34299d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c07cf20 con 0x7fd33c07de30 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd34199b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c19ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd34199b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c19ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55222/0 (socket says 172.21.15.94:55222) 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd34199b700 1 -- 172.21.15.94:0/1336136689 learned_addr learned my addr 172.21.15.94:0/1336136689 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd34199b700 1 -- 172.21.15.94:0/1336136689 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33c19e410 con 0x7fd33c07de30 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd34199b700 1 --2- 172.21.15.94:0/1336136689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c19ded0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd32c000c00 tx=0x7fd32c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd332ffd700 1 -- 172.21.15.94:0/1336136689 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd32c003db0 con 0x7fd33c07de30 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd332ffd700 1 -- 172.21.15.94:0/1336136689 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd32c011630 con 0x7fd33c07de30 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd332ffd700 1 -- 172.21.15.94:0/1336136689 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd32c011950 con 0x7fd33c07de30 2024-04-21T08:29:37.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd34299d700 1 -- 172.21.15.94:0/1336136689 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd33c19e6a0 con 0x7fd33c07de30 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.630+0000 7fd34299d700 1 -- 172.21.15.94:0/1336136689 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd33c1a1390 con 0x7fd33c07de30 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.631+0000 7fd332ffd700 1 -- 172.21.15.94:0/1336136689 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd32c011af0 con 0x7fd33c07de30 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.631+0000 7fd332ffd700 1 -- 172.21.15.94:0/1336136689 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd32c01c450 con 0x7fd33c07de30 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.631+0000 7fd34299d700 1 -- 172.21.15.94:0/1336136689 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd33c069b90 con 0x7fd33c07de30 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.634+0000 7fd332ffd700 1 -- 172.21.15.94:0/1336136689 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd32c004260 con 0x7fd33c07de30 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.702+0000 7fd34299d700 1 -- 172.21.15.94:0/1336136689 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fd33c1a26e0 con 0x7fd33c07de30 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.705+0000 7fd332ffd700 1 -- 172.21.15.94:0/1336136689 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7fd32c01b450 con 0x7fd33c07de30 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.705+0000 7fd332ffd700 1 -- 172.21.15.94:0/1336136689 <== mon.0 v2:172.21.15.94:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd32c01b630 con 0x7fd33c07de30 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.705+0000 7fd34299d700 1 -- 172.21.15.94:0/1336136689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 msgr2=0x7fd33c19ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.705+0000 7fd34299d700 1 --2- 172.21.15.94:0/1336136689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c19ded0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd32c000c00 tx=0x7fd32c004750 comp rx=0 tx=0).stop 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.706+0000 7fd34299d700 1 -- 172.21.15.94:0/1336136689 shutdown_connections 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.706+0000 7fd34299d700 1 --2- 172.21.15.94:0/1336136689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c07de30 0x7fd33c19ded0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.706+0000 7fd34299d700 1 -- 172.21.15.94:0/1336136689 >> 172.21.15.94:0/1336136689 conn(0x7fd33c103940 msgr2=0x7fd33c105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:37.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.706+0000 7fd34299d700 1 -- 172.21.15.94:0/1336136689 shutdown_connections 2024-04-21T08:29:37.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:36.706+0000 7fd34299d700 1 -- 172.21.15.94:0/1336136689 wait complete. 2024-04-21T08:29:37.325 INFO:teuthology.orchestra.run.smithi094.stdout:Generating new minimal ceph.conf... 2024-04-21T08:29:38.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:38 smithi094 ceph-mon[27611]: from='client.? 172.21.15.94:0/2979921209' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:29:40.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.726+0000 7f27bf93b700 1 Processor -- start 2024-04-21T08:29:40.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.727+0000 7f27bf93b700 1 -- start start 2024-04-21T08:29:40.045 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.727+0000 7f27bf93b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b810b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.727+0000 7f27bf93b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f27b807be80 con 0x7f27b8108fc0 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.727+0000 7f27bd6d7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b810b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.727+0000 7f27bd6d7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b810b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55234/0 (socket says 172.21.15.94:55234) 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.727+0000 7f27bd6d7700 1 -- 172.21.15.94:0/3874683719 learned_addr learned my addr 172.21.15.94:0/3874683719 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.727+0000 7f27bd6d7700 1 -- 172.21.15.94:0/3874683719 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b810b8f0 con 0x7f27b8108fc0 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.727+0000 7f27bd6d7700 1 --2- 172.21.15.94:0/3874683719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b810b3b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f27b40098a0 tx=0x7f27b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cbb14c629b6046a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.728+0000 7f27affff700 1 -- 172.21.15.94:0/3874683719 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27b4004160 con 0x7f27b8108fc0 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.728+0000 7f27affff700 1 -- 172.21.15.94:0/3874683719 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f27b4004300 con 0x7f27b8108fc0 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.728+0000 7f27affff700 1 -- 172.21.15.94:0/3874683719 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27b4011af0 con 0x7f27b8108fc0 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.728+0000 7f27bf93b700 1 -- 172.21.15.94:0/3874683719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 msgr2=0x7f27b810b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.728+0000 7f27bf93b700 1 --2- 172.21.15.94:0/3874683719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b810b3b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f27b40098a0 tx=0x7f27b4009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:40.046 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.728+0000 7f27bf93b700 1 -- 172.21.15.94:0/3874683719 shutdown_connections 2024-04-21T08:29:40.047 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.728+0000 7f27bf93b700 1 --2- 172.21.15.94:0/3874683719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b810b3b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:40.047 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.728+0000 7f27bf93b700 1 -- 172.21.15.94:0/3874683719 >> 172.21.15.94:0/3874683719 conn(0x7f27b8102a70 msgr2=0x7f27b8104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:40.047 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.728+0000 7f27bf93b700 1 -- 172.21.15.94:0/3874683719 shutdown_connections 2024-04-21T08:29:40.047 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.728+0000 7f27bf93b700 1 -- 172.21.15.94:0/3874683719 wait complete. 2024-04-21T08:29:40.047 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.729+0000 7f27bf93b700 1 Processor -- start 2024-04-21T08:29:40.047 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.729+0000 7f27bf93b700 1 -- start start 2024-04-21T08:29:40.047 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.729+0000 7f27bf93b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b8108840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:40.047 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.729+0000 7f27bf93b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f27b810c8b0 con 0x7f27b8108fc0 2024-04-21T08:29:40.047 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.729+0000 7f27bd6d7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b8108840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:40.047 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.729+0000 7f27bd6d7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b8108840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55250/0 (socket says 172.21.15.94:55250) 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.729+0000 7f27bd6d7700 1 -- 172.21.15.94:0/2979921209 learned_addr learned my addr 172.21.15.94:0/2979921209 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.729+0000 7f27bd6d7700 1 -- 172.21.15.94:0/2979921209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b8108d80 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.729+0000 7f27bd6d7700 1 --2- 172.21.15.94:0/2979921209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b8108840 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f27b4011ac0 tx=0x7f27b4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.729+0000 7f27ae7fc700 1 -- 172.21.15.94:0/2979921209 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27b4003b20 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.730+0000 7f27ae7fc700 1 -- 172.21.15.94:0/2979921209 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f27b401a8c0 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.730+0000 7f27bf93b700 1 -- 172.21.15.94:0/2979921209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b8106f70 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.730+0000 7f27ae7fc700 1 -- 172.21.15.94:0/2979921209 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27b40248d0 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.730+0000 7f27bf93b700 1 -- 172.21.15.94:0/2979921209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b81074b0 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.730+0000 7f27ae7fc700 1 -- 172.21.15.94:0/2979921209 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f27b4022020 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.730+0000 7f27ae7fc700 1 -- 172.21.15.94:0/2979921209 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f27b40193c0 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.730+0000 7f27bf93b700 1 -- 172.21.15.94:0/2979921209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27b80640a0 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.733+0000 7f27ae7fc700 1 -- 172.21.15.94:0/2979921209 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f27b402ddf0 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.799+0000 7f27bf93b700 1 -- 172.21.15.94:0/2979921209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f27b8195de0 con 0x7f27b8108fc0 2024-04-21T08:29:40.048 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.799+0000 7f27ae7fc700 1 -- 172.21.15.94:0/2979921209 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f27b401aa70 con 0x7f27b8108fc0 2024-04-21T08:29:40.049 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.800+0000 7f27bf93b700 1 -- 172.21.15.94:0/2979921209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 msgr2=0x7f27b8108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:40.049 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.800+0000 7f27bf93b700 1 --2- 172.21.15.94:0/2979921209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b8108840 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f27b4011ac0 tx=0x7f27b4011ba0 comp rx=0 tx=0).stop 2024-04-21T08:29:40.049 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.800+0000 7f27bf93b700 1 -- 172.21.15.94:0/2979921209 shutdown_connections 2024-04-21T08:29:40.049 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.800+0000 7f27bf93b700 1 --2- 172.21.15.94:0/2979921209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b8108fc0 0x7f27b8108840 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:40.049 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.800+0000 7f27bf93b700 1 -- 172.21.15.94:0/2979921209 >> 172.21.15.94:0/2979921209 conn(0x7f27b8102a70 msgr2=0x7f27b81036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:40.049 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.800+0000 7f27bf93b700 1 -- 172.21.15.94:0/2979921209 shutdown_connections 2024-04-21T08:29:40.049 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:37.800+0000 7f27bf93b700 1 -- 172.21.15.94:0/2979921209 wait complete. 2024-04-21T08:29:40.049 INFO:teuthology.orchestra.run.smithi094.stdout:Restarting the monitor... 2024-04-21T08:29:40.324 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:40 smithi094 systemd[1]: Stopping Ceph mon.a for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:29:40.674 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:40 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a[27607]: 2024-04-21T08:29:40.324+0000 7f34aac34700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:29:40.674 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:40 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a[27607]: 2024-04-21T08:29:40.324+0000 7f34aac34700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-21T08:29:40.674 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:40 smithi094 podman[27857]: 2024-04-21 08:29:40.406978148 +0000 UTC m=+0.192862220 container died 65e7b1a0aa87cd8e39d45b83e8e6263b40d056dcee94c33be92c7ef9f391aefc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T08:29:41.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:40 smithi094 podman[27857]: 2024-04-21 08:29:40.774885949 +0000 UTC m=+0.560770008 container cleanup 65e7b1a0aa87cd8e39d45b83e8e6263b40d056dcee94c33be92c7ef9f391aefc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:29:41.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:40 smithi094 bash[27857]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a 2024-04-21T08:29:41.347 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 podman[27871]: 2024-04-21 08:29:41.017574797 +0000 UTC m=+0.610359488 container remove 65e7b1a0aa87cd8e39d45b83e8e6263b40d056dcee94c33be92c7ef9f391aefc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:29:41.347 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 systemd[1]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-21T08:29:41.347 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 systemd[1]: Stopped Ceph mon.a for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:29:41.347 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 systemd[1]: Starting Ceph mon.a for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:29:41.599 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 podman[27969]: 2024-04-21 08:29:41.251504638 +0000 UTC m=+0.017753722 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:29:41.600 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 podman[27969]: 2024-04-21 08:29:41.351916003 +0000 UTC m=+0.118165111 container create 6dd38797f9a6c816fc29822477c71bc45161baa2b0b7f439ff07d335e6b57fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:29:41.600 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 podman[27969]: 2024-04-21 08:29:41.571005859 +0000 UTC m=+0.337254924 container init 6dd38797f9a6c816fc29822477c71bc45161baa2b0b7f439ff07d335e6b57fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:29:41.600 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 podman[27969]: 2024-04-21 08:29:41.573984975 +0000 UTC m=+0.340234044 container start 6dd38797f9a6c816fc29822477c71bc45161baa2b0b7f439ff07d335e6b57fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:29:41.600 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T08:29:41.600 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev), process ceph-mon, pid 2 2024-04-21T08:29:41.600 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: pidfile_write: ignore empty --pid-file 2024-04-21T08:29:41.600 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: load: jerasure load: lrc 2024-04-21T08:29:41.632 INFO:teuthology.orchestra.run.smithi094.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-21T08:29:41.868 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: RocksDB version: 7.9.2 2024-04-21T08:29:41.868 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Git sha 0 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Compile date 2024-04-19 14:19:22 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: DB SUMMARY 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: DB Session ID: IVAWSVZ1H0ROFUYTIZ0Q 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: CURRENT file: CURRENT 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105682 ; 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.error_if_exists: 0 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.create_if_missing: 0 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.paranoid_checks: 1 2024-04-21T08:29:41.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T08:29:41.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T08:29:41.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T08:29:41.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.env: 0x5596857d7cc0 2024-04-21T08:29:41.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T08:29:41.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.info_log: 0x559687a41be0 2024-04-21T08:29:41.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T08:29:41.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.statistics: (nil) 2024-04-21T08:29:41.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.use_fsync: 0 2024-04-21T08:29:41.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_log_file_size: 0 2024-04-21T08:29:41.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.allow_fallocate: 1 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.use_direct_reads: 0 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.db_log_dir: 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.wal_dir: 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T08:29:41.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T08:29:41.873 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T08:29:41.873 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.write_buffer_manager: 0x5596879d50e0 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.unordered_write: 0 2024-04-21T08:29:41.874 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.row_cache: None 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.wal_filter: None 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.two_write_queues: 0 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.wal_compression: 0 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.atomic_flush: 0 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T08:29:41.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.log_readahead_size: 0 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_background_jobs: 2 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_background_compactions: -1 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_subcompactions: 1 2024-04-21T08:29:41.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_open_files: -1 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_background_flushes: -1 2024-04-21T08:29:41.877 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Compression algorithms supported: 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: kZSTD supported: 0 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: kXpressCompression supported: 0 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: kZlibCompression supported: 1 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: kSnappyCompression supported: 1 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: kLZ4Compression supported: 1 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: kBZip2Compression supported: 0 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-21T08:29:41.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.merge_operator: 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_filter: None 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559687a41dc0) 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout: cache_index_and_filter_blocks: 1 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout: pin_top_level_index_and_filter: 1 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout: index_type: 0 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout: data_block_index_type: 0 2024-04-21T08:29:41.879 INFO:journalctl@ceph.mon.a.smithi094.stdout: index_shortening: 1 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: checksum: 4 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: no_block_cache: 0 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache: 0x559687a071f0 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache_name: BinnedLRUCache 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache_options: 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: capacity : 536870912 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: num_shard_bits : 4 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: strict_capacity_limit : 0 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: high_pri_pool_ratio: 0.000 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache_compressed: (nil) 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: persistent_cache: (nil) 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_size: 4096 2024-04-21T08:29:41.880 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_size_deviation: 10 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_restart_interval: 16 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: index_block_restart_interval: 1 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: metadata_block_size: 4096 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: partition_filters: 0 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: use_delta_encoding: 1 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: filter_policy: bloomfilter 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: whole_key_filtering: 1 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: verify_compression: 0 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: read_amp_bytes_per_bit: 0 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: format_version: 5 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: enable_index_compression: 1 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_align: 0 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: max_auto_readahead_size: 262144 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: prepopulate_block_cache: 0 2024-04-21T08:29:41.881 INFO:journalctl@ceph.mon.a.smithi094.stdout: initial_auto_readahead_size: 8192 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compression: NoCompression 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.num_levels: 7 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T08:29:41.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T08:29:41.883 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:29:41.883 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T08:29:41.883 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T08:29:41.883 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:29:41.883 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:29:41.883 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T08:29:41.883 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T08:29:41.883 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T08:29:41.883 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T08:29:41.883 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:29:41.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:29:41.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T08:29:41.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T08:29:41.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:29:41.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T08:29:41.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T08:29:41.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T08:29:41.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T08:29:41.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T08:29:41.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T08:29:41.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T08:29:41.886 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T08:29:41.887 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.inplace_update_support: 0 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.bloom_locality: 0 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.max_successive_merges: 0 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.ttl: 2592000 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T08:29:41.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.enable_blob_files: false 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.min_blob_size: 0 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ab54bb33-7487-4010-bae0-191b92a7e594 2024-04-21T08:29:41.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688181603413, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688181604521, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100789, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98677, "index_size": 304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14302, "raw_average_key_size": 49, "raw_value_size": 90895, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713688181, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ab54bb33-7487-4010-bae0-191b92a7e594", "db_session_id": "IVAWSVZ1H0ROFUYTIZ0Q", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688181604609, "job": 1, "event": "recovery_finished"} 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559687b1c000 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: DB pointer 0x559687b06000 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** DB Stats ** 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:29:41.890 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** Compaction Stats [default] ** 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: L0 2/0 100.29 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: Sum 2/0 100.29 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** Compaction Stats [default] ** 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T08:29:41.891 INFO:journalctl@ceph.mon.a.smithi094.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative compaction: 0.00 GB write, 30.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval compaction: 0.00 GB write, 30.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: Block cache BinnedLRUCache@0x559687a071f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-21T08:29:41.892 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: starting mon.a rank 0 at public addrs [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] at bind addrs [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: mon.a@-1(???) e1 preinit fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: mon.a@-1(???).mds e1 new map 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: mon.a@-1(???).mds e1 print_map 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout: e1 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout: legacy client fscid: -1 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout: No filesystems configured 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:29:41.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 bash[27969]: 6dd38797f9a6c816fc29822477c71bc45161baa2b0b7f439ff07d335e6b57fd6 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 systemd[1]: Started Ceph mon.a for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: monmap epoch 1 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: last_changed 2024-04-21T08:29:23.995760+0000 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: created 2024-04-21T08:29:23.995760+0000 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: min_mon_release 19 (squid) 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: election_strategy: 1 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: fsmap 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: osdmap e1: 0 total, 0 up, 0 in 2024-04-21T08:29:41.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:41 smithi094 ceph-mon[27988]: mgrmap e1: no daemons active 2024-04-21T08:29:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.083+0000 7efdd7b76700 1 Processor -- start 2024-04-21T08:29:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.083+0000 7efdd7b76700 1 -- start start 2024-04-21T08:29:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.083+0000 7efdd7b76700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd007c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.083+0000 7efdd7b76700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efdd007c8e0 con 0x7efdd007de60 2024-04-21T08:29:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.084+0000 7efdd5912700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd007c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.084+0000 7efdd5912700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd007c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55258/0 (socket says 172.21.15.94:55258) 2024-04-21T08:29:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.084+0000 7efdd5912700 1 -- 172.21.15.94:0/1996845724 learned_addr learned my addr 172.21.15.94:0/1996845724 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.084+0000 7efdd5912700 1 -- 172.21.15.94:0/1996845724 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd007ca60 con 0x7efdd007de60 2024-04-21T08:29:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.084+0000 7efdd5912700 1 --2- 172.21.15.94:0/1996845724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd007c310 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0098a0 tx=0x7efdcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f214d80ea84b6a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.084+0000 7efdd4910700 1 -- 172.21.15.94:0/1996845724 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdcc004160 con 0x7efdd007de60 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.084+0000 7efdd4910700 1 -- 172.21.15.94:0/1996845724 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efdcc004300 con 0x7efdd007de60 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.084+0000 7efdd4910700 1 -- 172.21.15.94:0/1996845724 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdcc011af0 con 0x7efdd007de60 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.085+0000 7efdd7b76700 1 -- 172.21.15.94:0/1996845724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 msgr2=0x7efdd007c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.085+0000 7efdd7b76700 1 --2- 172.21.15.94:0/1996845724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd007c310 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0098a0 tx=0x7efdcc009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.085+0000 7efdd7b76700 1 -- 172.21.15.94:0/1996845724 shutdown_connections 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.085+0000 7efdd7b76700 1 --2- 172.21.15.94:0/1996845724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd007c310 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.085+0000 7efdd7b76700 1 -- 172.21.15.94:0/1996845724 >> 172.21.15.94:0/1996845724 conn(0x7efdd0103b70 msgr2=0x7efdd0105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.085+0000 7efdd7b76700 1 -- 172.21.15.94:0/1996845724 shutdown_connections 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.085+0000 7efdd7b76700 1 -- 172.21.15.94:0/1996845724 wait complete. 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdd7b76700 1 Processor -- start 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdd7b76700 1 -- start start 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdd7b76700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:42.688 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdd7b76700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efdd010ea60 con 0x7efdd007de60 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdd5912700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdd5912700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd019e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55262/0 (socket says 172.21.15.94:55262) 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdd5912700 1 -- 172.21.15.94:0/2840507903 learned_addr learned my addr 172.21.15.94:0/2840507903 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdd5912700 1 -- 172.21.15.94:0/2840507903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd019ee50 con 0x7efdd007de60 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdd5912700 1 --2- 172.21.15.94:0/2840507903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd019e910 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efdcc011ac0 tx=0x7efdcc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdc6ffd700 1 -- 172.21.15.94:0/2840507903 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdcc003b20 con 0x7efdd007de60 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdc6ffd700 1 -- 172.21.15.94:0/2840507903 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efdcc01a8c0 con 0x7efdd007de60 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.086+0000 7efdc6ffd700 1 -- 172.21.15.94:0/2840507903 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdcc0248d0 con 0x7efdd007de60 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.087+0000 7efdd7b76700 1 -- 172.21.15.94:0/2840507903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd019f0e0 con 0x7efdd007de60 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.087+0000 7efdd7b76700 1 -- 172.21.15.94:0/2840507903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd01a1dd0 con 0x7efdd007de60 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.087+0000 7efdc6ffd700 1 -- 172.21.15.94:0/2840507903 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efdcc022020 con 0x7efdd007de60 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.087+0000 7efdc6ffd700 1 -- 172.21.15.94:0/2840507903 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efdcc0193c0 con 0x7efdd007de60 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.087+0000 7efdd7b76700 1 -- 172.21.15.94:0/2840507903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd00640a0 con 0x7efdd007de60 2024-04-21T08:29:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.090+0000 7efdc6ffd700 1 -- 172.21.15.94:0/2840507903 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efdcc042e50 con 0x7efdd007de60 2024-04-21T08:29:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.157+0000 7efdd7b76700 1 -- 172.21.15.94:0/2840507903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7efdd002df80 con 0x7efdd007de60 2024-04-21T08:29:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.159+0000 7efdc6ffd700 1 -- 172.21.15.94:0/2840507903 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7efdcc031020 con 0x7efdd007de60 2024-04-21T08:29:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.160+0000 7efdd7b76700 1 -- 172.21.15.94:0/2840507903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 msgr2=0x7efdd019e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.160+0000 7efdd7b76700 1 --2- 172.21.15.94:0/2840507903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd019e910 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efdcc011ac0 tx=0x7efdcc011ba0 comp rx=0 tx=0).stop 2024-04-21T08:29:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.160+0000 7efdd7b76700 1 -- 172.21.15.94:0/2840507903 shutdown_connections 2024-04-21T08:29:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.160+0000 7efdd7b76700 1 --2- 172.21.15.94:0/2840507903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdd007de60 0x7efdd019e910 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.160+0000 7efdd7b76700 1 -- 172.21.15.94:0/2840507903 >> 172.21.15.94:0/2840507903 conn(0x7efdd0103b70 msgr2=0x7efdd0104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.160+0000 7efdd7b76700 1 -- 172.21.15.94:0/2840507903 shutdown_connections 2024-04-21T08:29:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:42.160+0000 7efdd7b76700 1 -- 172.21.15.94:0/2840507903 wait complete. 2024-04-21T08:29:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-21T08:29:42.709 INFO:teuthology.orchestra.run.smithi094.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:29:42.710 INFO:teuthology.orchestra.run.smithi094.stdout:Creating mgr... 2024-04-21T08:29:42.710 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-21T08:29:42.710 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-21T08:29:43.186 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.x 2024-04-21T08:29:43.187 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.x.service: Unit ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-21T08:29:43.333 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2518ed94-ffb9-11ee-bc93-c7b262605968.target.wants/ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@.service. 2024-04-21T08:29:43.352 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:43 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2840507903' entity='client.admin' 2024-04-21T08:29:43.886 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:29:43.886 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-04-21T08:29:43.893 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:29:43.893 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-21T08:29:43.893 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-04-21T08:29:43.893 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T08:29:43.901 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:29:43.901 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-04-21T08:29:43.908 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:29:43.908 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-21T08:29:43.909 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-04-21T08:29:43.909 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-21T08:29:43.909 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr to start... 2024-04-21T08:29:43.909 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr... 2024-04-21T08:29:44.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:43 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:43.945+0000 7f6ee83e8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:29:44.291 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:44 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:44.130+0000 7f6ee83e8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:29:44.291 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:44 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:44.236+0000 7f6ee83e8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:29:44.563 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:44 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3268938305' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:29:44.888 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:44 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:44.563+0000 7f6ee83e8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:29:44.888 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:44 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:44.650+0000 7f6ee83e8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:45.162 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:29:45.163 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:29:45.163 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:29:45.163 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:29:45.163 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:29:45.163 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:29:45.163 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:45.163 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:29:45.170 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:29:45.170 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:29:45.171 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:29:35.092204+0000", 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:29:45.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.355+0000 7f85ddecc700 1 Processor -- start 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.355+0000 7f85ddecc700 1 -- start start 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.355+0000 7f85ddecc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d8106bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.355+0000 7f85ddecc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f85d81071a0 con 0x7f85d81067b0 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.355+0000 7f85d77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d8106bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.355+0000 7f85d77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d8106bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45856/0 (socket says 172.21.15.94:45856) 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.355+0000 7f85d77fe700 1 -- 172.21.15.94:0/2641232309 learned_addr learned my addr 172.21.15.94:0/2641232309 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.356+0000 7f85d77fe700 1 -- 172.21.15.94:0/2641232309 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d8107320 con 0x7f85d81067b0 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.356+0000 7f85d77fe700 1 --2- 172.21.15.94:0/2641232309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d8106bd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0098a0 tx=0x7f85cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f673ba5879aed server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.356+0000 7f85d67fc700 1 -- 172.21.15.94:0/2641232309 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85cc004160 con 0x7f85d81067b0 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.356+0000 7f85d67fc700 1 -- 172.21.15.94:0/2641232309 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f85cc004300 con 0x7f85d81067b0 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.356+0000 7f85d67fc700 1 -- 172.21.15.94:0/2641232309 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85cc011af0 con 0x7f85d81067b0 2024-04-21T08:29:45.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.357+0000 7f85ddecc700 1 -- 172.21.15.94:0/2641232309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 msgr2=0x7f85d8106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.357+0000 7f85ddecc700 1 --2- 172.21.15.94:0/2641232309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d8106bd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0098a0 tx=0x7f85cc009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.357+0000 7f85ddecc700 1 -- 172.21.15.94:0/2641232309 shutdown_connections 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.357+0000 7f85ddecc700 1 --2- 172.21.15.94:0/2641232309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d8106bd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.357+0000 7f85ddecc700 1 -- 172.21.15.94:0/2641232309 >> 172.21.15.94:0/2641232309 conn(0x7f85d81021b0 msgr2=0x7f85d81045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.357+0000 7f85ddecc700 1 -- 172.21.15.94:0/2641232309 shutdown_connections 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.357+0000 7f85ddecc700 1 -- 172.21.15.94:0/2641232309 wait complete. 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.357+0000 7f85ddecc700 1 Processor -- start 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85ddecc700 1 -- start start 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85ddecc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d81090b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85ddecc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f85d810a610 con 0x7f85d81067b0 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85d77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d81090b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85d77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d81090b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45858/0 (socket says 172.21.15.94:45858) 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85d77fe700 1 -- 172.21.15.94:0/3268938305 learned_addr learned my addr 172.21.15.94:0/3268938305 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:45.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85d77fe700 1 -- 172.21.15.94:0/3268938305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d81095f0 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85d77fe700 1 --2- 172.21.15.94:0/3268938305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d81090b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f85cc003f40 tx=0x7f85cc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85d4ff9700 1 -- 172.21.15.94:0/3268938305 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85cc0038b0 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85ddecc700 1 -- 172.21.15.94:0/3268938305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85d81077e0 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85d4ff9700 1 -- 172.21.15.94:0/3268938305 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f85cc003a50 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.358+0000 7f85d4ff9700 1 -- 172.21.15.94:0/3268938305 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85cc01a5c0 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.359+0000 7f85ddecc700 1 -- 172.21.15.94:0/3268938305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85d8107d20 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.359+0000 7f85d4ff9700 1 -- 172.21.15.94:0/3268938305 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f85cc01ac50 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.359+0000 7f85d4ff9700 1 -- 172.21.15.94:0/3268938305 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f85cc019700 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.359+0000 7f85ddecc700 1 -- 172.21.15.94:0/3268938305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85d810ca60 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.362+0000 7f85d4ff9700 1 -- 172.21.15.94:0/3268938305 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f85cc032080 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.433+0000 7f85ddecc700 1 -- 172.21.15.94:0/3268938305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f85d8195db0 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.433+0000 7f85d4ff9700 1 -- 172.21.15.94:0/3268938305 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f85cc0199a0 con 0x7f85d81067b0 2024-04-21T08:29:45.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.434+0000 7f85ddecc700 1 -- 172.21.15.94:0/3268938305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 msgr2=0x7f85d81090b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:45.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.434+0000 7f85ddecc700 1 --2- 172.21.15.94:0/3268938305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d81090b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f85cc003f40 tx=0x7f85cc011890 comp rx=0 tx=0).stop 2024-04-21T08:29:45.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.434+0000 7f85ddecc700 1 -- 172.21.15.94:0/3268938305 shutdown_connections 2024-04-21T08:29:45.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.434+0000 7f85ddecc700 1 --2- 172.21.15.94:0/3268938305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85d81067b0 0x7f85d81090b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:45.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.434+0000 7f85ddecc700 1 -- 172.21.15.94:0/3268938305 >> 172.21.15.94:0/3268938305 conn(0x7f85d81021b0 msgr2=0x7f85d8102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:45.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.434+0000 7f85ddecc700 1 -- 172.21.15.94:0/3268938305 shutdown_connections 2024-04-21T08:29:45.176 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:44.434+0000 7f85ddecc700 1 -- 172.21.15.94:0/3268938305 wait complete. 2024-04-21T08:29:45.176 INFO:teuthology.orchestra.run.smithi094.stdout:mgr not available, waiting (1/15)... 2024-04-21T08:29:46.162 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:45 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:45.831+0000 7f6ee83e8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:29:46.162 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:45 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:45.997+0000 7f6ee83e8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:29:46.162 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:46 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:46.162+0000 7f6ee83e8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:29:46.733 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:46 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:46.445+0000 7f6ee83e8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:29:46.733 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:46 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:46.535+0000 7f6ee83e8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:29:46.733 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:46 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:46.733+0000 7f6ee83e8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:29:47.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:46 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:46.922+0000 7f6ee83e8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:29:47.607 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:47 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:47.531+0000 7f6ee83e8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:29:47.970 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:47 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3861476912' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:29:48.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:47 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:47.970+0000 7f6ee83e8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:29:48.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:48 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:48.066+0000 7f6ee83e8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:29:48.500 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:48.500 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:29:48.500 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-21T08:29:48.501 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:48.502 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:48.503 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:29:35.092204+0000", 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:29:48.504 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.614+0000 7f8da05cc700 1 Processor -- start 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.614+0000 7f8da05cc700 1 -- start start 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.614+0000 7f8da05cc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d981071e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.614+0000 7f8da05cc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9807be40 con 0x7f8d98106dc0 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.615+0000 7f8d9e368700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d981071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.615+0000 7f8d9e368700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d981071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45870/0 (socket says 172.21.15.94:45870) 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.615+0000 7f8d9e368700 1 -- 172.21.15.94:0/3662483324 learned_addr learned my addr 172.21.15.94:0/3662483324 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.615+0000 7f8d9e368700 1 -- 172.21.15.94:0/3662483324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d98107720 con 0x7f8d98106dc0 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.615+0000 7f8d9e368700 1 --2- 172.21.15.94:0/3662483324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d981071e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8d940098a0 tx=0x7f8d94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbbfc875e91be77 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:48.505 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.615+0000 7f8d9d366700 1 -- 172.21.15.94:0/3662483324 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d94004160 con 0x7f8d98106dc0 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.615+0000 7f8d9d366700 1 -- 172.21.15.94:0/3662483324 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8d94004300 con 0x7f8d98106dc0 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.615+0000 7f8d9d366700 1 -- 172.21.15.94:0/3662483324 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d94011af0 con 0x7f8d98106dc0 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.616+0000 7f8da05cc700 1 -- 172.21.15.94:0/3662483324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 msgr2=0x7f8d981071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.616+0000 7f8da05cc700 1 --2- 172.21.15.94:0/3662483324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d981071e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8d940098a0 tx=0x7f8d94009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.616+0000 7f8da05cc700 1 -- 172.21.15.94:0/3662483324 shutdown_connections 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.616+0000 7f8da05cc700 1 --2- 172.21.15.94:0/3662483324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d981071e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.616+0000 7f8da05cc700 1 -- 172.21.15.94:0/3662483324 >> 172.21.15.94:0/3662483324 conn(0x7f8d981029c0 msgr2=0x7f8d98104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.616+0000 7f8da05cc700 1 -- 172.21.15.94:0/3662483324 shutdown_connections 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.616+0000 7f8da05cc700 1 -- 172.21.15.94:0/3662483324 wait complete. 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.617+0000 7f8da05cc700 1 Processor -- start 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.617+0000 7f8da05cc700 1 -- start start 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.617+0000 7f8da05cc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d9819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:48.506 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.617+0000 7f8da05cc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8d98108750 con 0x7f8d98106dc0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.617+0000 7f8d9e368700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d9819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.617+0000 7f8d9e368700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d9819e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45872/0 (socket says 172.21.15.94:45872) 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.617+0000 7f8d9e368700 1 -- 172.21.15.94:0/3861476912 learned_addr learned my addr 172.21.15.94:0/3861476912 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.617+0000 7f8d9e368700 1 -- 172.21.15.94:0/3861476912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9819ee70 con 0x7f8d98106dc0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.617+0000 7f8d9e368700 1 --2- 172.21.15.94:0/3861476912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d9819e930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8d94003f40 tx=0x7f8d94011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.617+0000 7f8d8f7fe700 1 -- 172.21.15.94:0/3861476912 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d940038b0 con 0x7f8d98106dc0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.618+0000 7f8d8f7fe700 1 -- 172.21.15.94:0/3861476912 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8d94003a50 con 0x7f8d98106dc0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.618+0000 7f8d8f7fe700 1 -- 172.21.15.94:0/3861476912 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d9401a5c0 con 0x7f8d98106dc0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.618+0000 7f8da05cc700 1 -- 172.21.15.94:0/3861476912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9819f100 con 0x7f8d98106dc0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.618+0000 7f8da05cc700 1 -- 172.21.15.94:0/3861476912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d981a1df0 con 0x7f8d98106dc0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.618+0000 7f8d8f7fe700 1 -- 172.21.15.94:0/3861476912 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d9401ac50 con 0x7f8d98106dc0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.618+0000 7f8d8f7fe700 1 -- 172.21.15.94:0/3861476912 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8d94019a40 con 0x7f8d98106dc0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.619+0000 7f8da05cc700 1 -- 172.21.15.94:0/3861476912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d980640a0 con 0x7f8d98106dc0 2024-04-21T08:29:48.507 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.621+0000 7f8d8f7fe700 1 -- 172.21.15.94:0/3861476912 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8d94019ce0 con 0x7f8d98106dc0 2024-04-21T08:29:48.508 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.692+0000 7f8da05cc700 1 -- 172.21.15.94:0/3861476912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8d9802df80 con 0x7f8d98106dc0 2024-04-21T08:29:48.508 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.692+0000 7f8d8f7fe700 1 -- 172.21.15.94:0/3861476912 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8d94024430 con 0x7f8d98106dc0 2024-04-21T08:29:48.508 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.693+0000 7f8da05cc700 1 -- 172.21.15.94:0/3861476912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 msgr2=0x7f8d9819e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:48.508 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.693+0000 7f8da05cc700 1 --2- 172.21.15.94:0/3861476912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d9819e930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8d94003f40 tx=0x7f8d94011890 comp rx=0 tx=0).stop 2024-04-21T08:29:48.508 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.693+0000 7f8da05cc700 1 -- 172.21.15.94:0/3861476912 shutdown_connections 2024-04-21T08:29:48.508 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.693+0000 7f8da05cc700 1 --2- 172.21.15.94:0/3861476912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d98106dc0 0x7f8d9819e930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:48.508 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.693+0000 7f8da05cc700 1 -- 172.21.15.94:0/3861476912 >> 172.21.15.94:0/3861476912 conn(0x7f8d981029c0 msgr2=0x7f8d981952d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:48.508 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.693+0000 7f8da05cc700 1 -- 172.21.15.94:0/3861476912 shutdown_connections 2024-04-21T08:29:48.508 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:47.693+0000 7f8da05cc700 1 -- 172.21.15.94:0/3861476912 wait complete. 2024-04-21T08:29:48.508 INFO:teuthology.orchestra.run.smithi094.stdout:mgr not available, waiting (2/15)... 2024-04-21T08:29:48.760 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:48 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:48.272+0000 7f6ee83e8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:29:49.129 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:48 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:48.797+0000 7f6ee83e8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:29:49.129 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:48 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:48.888+0000 7f6ee83e8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:29:49.404 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:49 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:49.130+0000 7f6ee83e8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:29:49.404 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:49 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:49.269+0000 7f6ee83e8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:29:49.404 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:49 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:49.368+0000 7f6ee83e8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:29:49.760 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:49 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:49.489+0000 7f6ee83e8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:29:50.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:49 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:49.799+0000 7f6ee83e8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:29:50.620 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:50.407+0000 7f6ee83e8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:29:50.945 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:50.627+0000 7f6ee83e8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:29:50.945 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: Activating manager daemon x 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: mgrmap e2: x(active, starting, since 0.0030993s) 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: from='mgr.14100 172.21.15.94:0/922475940' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: from='mgr.14100 172.21.15.94:0/922475940' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: from='mgr.14100 172.21.15.94:0/922475940' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: from='mgr.14100 172.21.15.94:0/922475940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: from='mgr.14100 172.21.15.94:0/922475940' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: Manager daemon x is now available 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: from='mgr.14100 172.21.15.94:0/922475940' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: from='mgr.14100 172.21.15.94:0/922475940' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: from='mgr.14100 172.21.15.94:0/922475940' entity='mgr.x' 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: from='mgr.14100 172.21.15.94:0/922475940' entity='mgr.x' 2024-04-21T08:29:50.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:50 smithi094 ceph-mon[27988]: from='mgr.14100 172.21.15.94:0/922475940' entity='mgr.x' 2024-04-21T08:29:51.631 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:29:51.632 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:29:51.633 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:29:51.634 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:29:51.635 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:29:35.092204+0000", 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.945+0000 7f56b9c7c700 1 Processor -- start 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.945+0000 7f56b9c7c700 1 -- start start 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.945+0000 7f56b9c7c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b4108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.945+0000 7f56b9c7c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f56b4108f00 con 0x7f56b4108510 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.945+0000 7f56b37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b4108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.945+0000 7f56b37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b4108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45944/0 (socket says 172.21.15.94:45944) 2024-04-21T08:29:51.636 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.945+0000 7f56b37fe700 1 -- 172.21.15.94:0/4007407984 learned_addr learned my addr 172.21.15.94:0/4007407984 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.945+0000 7f56b37fe700 1 -- 172.21.15.94:0/4007407984 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b41096c0 con 0x7f56b4108510 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b37fe700 1 --2- 172.21.15.94:0/4007407984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b4108930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f56a40098a0 tx=0x7f56a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58106656591d396f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b27fc700 1 -- 172.21.15.94:0/4007407984 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56a4004160 con 0x7f56b4108510 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b27fc700 1 -- 172.21.15.94:0/4007407984 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f56a4004300 con 0x7f56b4108510 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b27fc700 1 -- 172.21.15.94:0/4007407984 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56a4011af0 con 0x7f56b4108510 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b9c7c700 1 -- 172.21.15.94:0/4007407984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 msgr2=0x7f56b4108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b9c7c700 1 --2- 172.21.15.94:0/4007407984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b4108930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f56a40098a0 tx=0x7f56a4009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b9c7c700 1 -- 172.21.15.94:0/4007407984 shutdown_connections 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b9c7c700 1 --2- 172.21.15.94:0/4007407984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b4108930 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b9c7c700 1 -- 172.21.15.94:0/4007407984 >> 172.21.15.94:0/4007407984 conn(0x7f56b4103b70 msgr2=0x7f56b4105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b9c7c700 1 -- 172.21.15.94:0/4007407984 shutdown_connections 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.946+0000 7f56b9c7c700 1 -- 172.21.15.94:0/4007407984 wait complete. 2024-04-21T08:29:51.637 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.947+0000 7f56b9c7c700 1 Processor -- start 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.947+0000 7f56b9c7c700 1 -- start start 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.947+0000 7f56b9c7c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.947+0000 7f56b9c7c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f56b410ca70 con 0x7f56b4108510 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.947+0000 7f56b37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.947+0000 7f56b37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b419c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45954/0 (socket says 172.21.15.94:45954) 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.947+0000 7f56b37fe700 1 -- 172.21.15.94:0/3924705486 learned_addr learned my addr 172.21.15.94:0/3924705486 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.948+0000 7f56b37fe700 1 -- 172.21.15.94:0/3924705486 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b419ccc0 con 0x7f56b4108510 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.948+0000 7f56b37fe700 1 --2- 172.21.15.94:0/3924705486 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b419c780 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f56a4003f40 tx=0x7f56a4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.948+0000 7f56b0ff9700 1 -- 172.21.15.94:0/3924705486 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56a40038b0 con 0x7f56b4108510 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.948+0000 7f56b0ff9700 1 -- 172.21.15.94:0/3924705486 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f56a4003a50 con 0x7f56b4108510 2024-04-21T08:29:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.948+0000 7f56b0ff9700 1 -- 172.21.15.94:0/3924705486 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56a401a5c0 con 0x7f56b4108510 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.948+0000 7f56b9c7c700 1 -- 172.21.15.94:0/3924705486 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b419cf50 con 0x7f56b4108510 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.948+0000 7f56b9c7c700 1 -- 172.21.15.94:0/3924705486 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b419fbf0 con 0x7f56b4108510 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.949+0000 7f56b0ff9700 1 -- 172.21.15.94:0/3924705486 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 2) ==== 46314+0+0 (secure 0 0 0) 0x7f56a401a870 con 0x7f56b4108510 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.949+0000 7f56b0ff9700 1 -- 172.21.15.94:0/3924705486 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f56a404d6c0 con 0x7f56b4108510 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.949+0000 7f56b9c7c700 1 -- 172.21.15.94:0/3924705486 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b40640a0 con 0x7f56b4108510 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:50.951+0000 7f56b0ff9700 1 -- 172.21.15.94:0/3924705486 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f56a4019c70 con 0x7f56b4108510 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:51.023+0000 7f56b9c7c700 1 -- 172.21.15.94:0/3924705486 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f56b402df80 con 0x7f56b4108510 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:51.023+0000 7f56b0ff9700 1 -- 172.21.15.94:0/3924705486 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f56a402cbc0 con 0x7f56b4108510 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:51.024+0000 7f56b9c7c700 1 -- 172.21.15.94:0/3924705486 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 msgr2=0x7f56b419c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:51.024+0000 7f56b9c7c700 1 --2- 172.21.15.94:0/3924705486 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b419c780 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f56a4003f40 tx=0x7f56a4011890 comp rx=0 tx=0).stop 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:51.024+0000 7f56b9c7c700 1 -- 172.21.15.94:0/3924705486 shutdown_connections 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:51.024+0000 7f56b9c7c700 1 --2- 172.21.15.94:0/3924705486 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56b4108510 0x7f56b419c780 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:51.024+0000 7f56b9c7c700 1 -- 172.21.15.94:0/3924705486 >> 172.21.15.94:0/3924705486 conn(0x7f56b4103b70 msgr2=0x7f56b41048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:51.024+0000 7f56b9c7c700 1 -- 172.21.15.94:0/3924705486 shutdown_connections 2024-04-21T08:29:51.639 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:51.024+0000 7f56b9c7c700 1 -- 172.21.15.94:0/3924705486 wait complete. 2024-04-21T08:29:51.640 INFO:teuthology.orchestra.run.smithi094.stdout:mgr not available, waiting (3/15)... 2024-04-21T08:29:52.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:51 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3924705486' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:29:52.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:51 smithi094 ceph-mon[27988]: mgrmap e3: x(active, since 1.00678s) 2024-04-21T08:29:53.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:53 smithi094 ceph-mon[27988]: mgrmap e4: x(active, since 2s) 2024-04-21T08:29:54.889 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:54.889 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:29:54.889 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:29:54.889 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:29:54.889 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:29:54.889 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:29:54.889 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:29:54.889 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:29:54.890 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:54.891 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:29:54.891 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:29:54.891 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:29:54.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:29:54.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:29:35.092204+0000", 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.065+0000 7fdf8a8e2700 1 Processor -- start 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.065+0000 7fdf8a8e2700 1 -- start start 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.066+0000 7fdf8a8e2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf8410b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.066+0000 7fdf8a8e2700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8407be60 con 0x7fdf84108fe0 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.066+0000 7fdf83fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf8410b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:54.902 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.066+0000 7fdf83fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf8410b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60492/0 (socket says 172.21.15.94:60492) 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.066+0000 7fdf83fff700 1 -- 172.21.15.94:0/657623499 learned_addr learned my addr 172.21.15.94:0/657623499 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.066+0000 7fdf83fff700 1 -- 172.21.15.94:0/657623499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8410b910 con 0x7fdf84108fe0 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.066+0000 7fdf83fff700 1 --2- 172.21.15.94:0/657623499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf8410b3d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdf780098a0 tx=0x7fdf78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49cc8a637ff23d8f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.067+0000 7fdf82ffd700 1 -- 172.21.15.94:0/657623499 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf78004160 con 0x7fdf84108fe0 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.067+0000 7fdf82ffd700 1 -- 172.21.15.94:0/657623499 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdf78004300 con 0x7fdf84108fe0 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.067+0000 7fdf82ffd700 1 -- 172.21.15.94:0/657623499 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf78011af0 con 0x7fdf84108fe0 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.067+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/657623499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 msgr2=0x7fdf8410b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.067+0000 7fdf8a8e2700 1 --2- 172.21.15.94:0/657623499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf8410b3d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdf780098a0 tx=0x7fdf78009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.067+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/657623499 shutdown_connections 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.067+0000 7fdf8a8e2700 1 --2- 172.21.15.94:0/657623499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf8410b3d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.067+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/657623499 >> 172.21.15.94:0/657623499 conn(0x7fdf84102a70 msgr2=0x7fdf84104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.067+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/657623499 shutdown_connections 2024-04-21T08:29:54.903 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.067+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/657623499 wait complete. 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.068+0000 7fdf8a8e2700 1 Processor -- start 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.068+0000 7fdf8a8e2700 1 -- start start 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.068+0000 7fdf8a8e2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf84108860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.068+0000 7fdf8a8e2700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8410c880 con 0x7fdf84108fe0 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.068+0000 7fdf83fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf84108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.068+0000 7fdf83fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf84108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60504/0 (socket says 172.21.15.94:60504) 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.068+0000 7fdf83fff700 1 -- 172.21.15.94:0/716635637 learned_addr learned my addr 172.21.15.94:0/716635637 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.069+0000 7fdf83fff700 1 -- 172.21.15.94:0/716635637 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf84108da0 con 0x7fdf84108fe0 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.069+0000 7fdf83fff700 1 --2- 172.21.15.94:0/716635637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf84108860 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdf78003f40 tx=0x7fdf78011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.069+0000 7fdf817fa700 1 -- 172.21.15.94:0/716635637 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf780038b0 con 0x7fdf84108fe0 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.069+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/716635637 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf84106f90 con 0x7fdf84108fe0 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.069+0000 7fdf817fa700 1 -- 172.21.15.94:0/716635637 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdf78003a50 con 0x7fdf84108fe0 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.069+0000 7fdf817fa700 1 -- 172.21.15.94:0/716635637 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf7801a5c0 con 0x7fdf84108fe0 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.069+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/716635637 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf841074d0 con 0x7fdf84108fe0 2024-04-21T08:29:54.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.070+0000 7fdf817fa700 1 -- 172.21.15.94:0/716635637 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fdf7801a8c0 con 0x7fdf84108fe0 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.070+0000 7fdf817fa700 1 --2- 172.21.15.94:0/716635637 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fdf6c039510 0x7fdf6c03b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.070+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/716635637 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf840640a0 con 0x7fdf84108fe0 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.070+0000 7fdf817fa700 1 -- 172.21.15.94:0/716635637 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdf7804dd00 con 0x7fdf84108fe0 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.070+0000 7fdf837fe700 1 --2- 172.21.15.94:0/716635637 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fdf6c039510 0x7fdf6c03b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.070+0000 7fdf837fe700 1 --2- 172.21.15.94:0/716635637 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fdf6c039510 0x7fdf6c03b9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf74006fd0 tx=0x7fdf74006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.075+0000 7fdf817fa700 1 -- 172.21.15.94:0/716635637 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf78019c70 con 0x7fdf84108fe0 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.348+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/716635637 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fdf84195ee0 con 0x7fdf84108fe0 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.349+0000 7fdf817fa700 1 -- 172.21.15.94:0/716635637 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fdf780248c0 con 0x7fdf84108fe0 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.352+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/716635637 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fdf6c039510 msgr2=0x7fdf6c03b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.352+0000 7fdf8a8e2700 1 --2- 172.21.15.94:0/716635637 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fdf6c039510 0x7fdf6c03b9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf74006fd0 tx=0x7fdf74006e40 comp rx=0 tx=0).stop 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.352+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/716635637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 msgr2=0x7fdf84108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.352+0000 7fdf8a8e2700 1 --2- 172.21.15.94:0/716635637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf84108860 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdf78003f40 tx=0x7fdf78011890 comp rx=0 tx=0).stop 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.352+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/716635637 shutdown_connections 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.352+0000 7fdf8a8e2700 1 --2- 172.21.15.94:0/716635637 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fdf6c039510 0x7fdf6c03b9d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:54.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.352+0000 7fdf8a8e2700 1 --2- 172.21.15.94:0/716635637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf84108fe0 0x7fdf84108860 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:54.906 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.352+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/716635637 >> 172.21.15.94:0/716635637 conn(0x7fdf84102a70 msgr2=0x7fdf841036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:54.906 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.352+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/716635637 shutdown_connections 2024-04-21T08:29:54.906 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:54.352+0000 7fdf8a8e2700 1 -- 172.21.15.94:0/716635637 wait complete. 2024-04-21T08:29:54.906 INFO:teuthology.orchestra.run.smithi094.stdout:mgr is available 2024-04-21T08:29:55.001 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:54 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/716635637' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:29:55.982 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:55 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3200504277' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T08:29:55.991 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:55.991 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T08:29:55.991 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout fsid = 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:29:55.991 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-21T08:29:55.991 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T08:29:55.991 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T08:29:55.991 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T08:29:55.991 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T08:29:55.991 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:55.992 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T08:29:55.992 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T08:29:55.992 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-21T08:29:55.992 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T08:29:55.992 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T08:29:55.992 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T08:29:55.992 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T08:29:55.992 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.302+0000 7fc44881d700 1 Processor -- start 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.303+0000 7fc44881d700 1 -- start start 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.303+0000 7fc44881d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc444108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.303+0000 7fc44881d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc444108f00 con 0x7fc444108510 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.303+0000 7fc44259c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc444108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.303+0000 7fc44259c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc444108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60512/0 (socket says 172.21.15.94:60512) 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.303+0000 7fc44259c700 1 -- 172.21.15.94:0/3062060647 learned_addr learned my addr 172.21.15.94:0/3062060647 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.303+0000 7fc44259c700 1 -- 172.21.15.94:0/3062060647 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4441096c0 con 0x7fc444108510 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44259c700 1 --2- 172.21.15.94:0/3062060647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc444108930 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc4340098a0 tx=0x7fc434009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1cd951478ba0d88 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44159a700 1 -- 172.21.15.94:0/3062060647 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc434004160 con 0x7fc444108510 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44159a700 1 -- 172.21.15.94:0/3062060647 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc434004300 con 0x7fc444108510 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44159a700 1 -- 172.21.15.94:0/3062060647 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc434011af0 con 0x7fc444108510 2024-04-21T08:29:55.993 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44881d700 1 -- 172.21.15.94:0/3062060647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 msgr2=0x7fc444108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44881d700 1 --2- 172.21.15.94:0/3062060647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc444108930 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc4340098a0 tx=0x7fc434009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44881d700 1 -- 172.21.15.94:0/3062060647 shutdown_connections 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44881d700 1 --2- 172.21.15.94:0/3062060647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc444108930 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44881d700 1 -- 172.21.15.94:0/3062060647 >> 172.21.15.94:0/3062060647 conn(0x7fc444103b70 msgr2=0x7fc444105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44881d700 1 -- 172.21.15.94:0/3062060647 shutdown_connections 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.304+0000 7fc44881d700 1 -- 172.21.15.94:0/3062060647 wait complete. 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.305+0000 7fc44881d700 1 Processor -- start 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.305+0000 7fc44881d700 1 -- start start 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.305+0000 7fc44881d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc44419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.305+0000 7fc44881d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc44410ca70 con 0x7fc444108510 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.305+0000 7fc44259c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc44419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.305+0000 7fc44259c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc44419c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60522/0 (socket says 172.21.15.94:60522) 2024-04-21T08:29:55.994 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.305+0000 7fc44259c700 1 -- 172.21.15.94:0/3200504277 learned_addr learned my addr 172.21.15.94:0/3200504277 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.306+0000 7fc44259c700 1 -- 172.21.15.94:0/3200504277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44419ccc0 con 0x7fc444108510 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.306+0000 7fc44259c700 1 --2- 172.21.15.94:0/3200504277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc44419c780 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc434003f40 tx=0x7fc434011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.306+0000 7fc4337fe700 1 -- 172.21.15.94:0/3200504277 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4340038b0 con 0x7fc444108510 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.306+0000 7fc4337fe700 1 -- 172.21.15.94:0/3200504277 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc434003a50 con 0x7fc444108510 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.306+0000 7fc44881d700 1 -- 172.21.15.94:0/3200504277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc44419cf50 con 0x7fc444108510 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.306+0000 7fc4337fe700 1 -- 172.21.15.94:0/3200504277 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc43401a5c0 con 0x7fc444108510 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.306+0000 7fc44881d700 1 -- 172.21.15.94:0/3200504277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc44419fbf0 con 0x7fc444108510 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.307+0000 7fc4337fe700 1 -- 172.21.15.94:0/3200504277 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fc43401a870 con 0x7fc444108510 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.307+0000 7fc4337fe700 1 --2- 172.21.15.94:0/3200504277 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc42c039560 0x7fc42c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.307+0000 7fc4337fe700 1 -- 172.21.15.94:0/3200504277 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc43404e5a0 con 0x7fc444108510 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.307+0000 7fc441d9b700 1 --2- 172.21.15.94:0/3200504277 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc42c039560 0x7fc42c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.307+0000 7fc44881d700 1 -- 172.21.15.94:0/3200504277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4440640a0 con 0x7fc444108510 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.307+0000 7fc441d9b700 1 --2- 172.21.15.94:0/3200504277 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc42c039560 0x7fc42c03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc438006fd0 tx=0x7fc438006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.312+0000 7fc4337fe700 1 -- 172.21.15.94:0/3200504277 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc434019c70 con 0x7fc444108510 2024-04-21T08:29:55.995 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.509+0000 7fc44881d700 1 -- 172.21.15.94:0/3200504277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fc444195390 con 0x7fc444108510 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.510+0000 7fc4337fe700 1 -- 172.21.15.94:0/3200504277 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7fc43404e360 con 0x7fc444108510 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.512+0000 7fc44881d700 1 -- 172.21.15.94:0/3200504277 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc42c039560 msgr2=0x7fc42c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.512+0000 7fc44881d700 1 --2- 172.21.15.94:0/3200504277 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc42c039560 0x7fc42c03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc438006fd0 tx=0x7fc438006e40 comp rx=0 tx=0).stop 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.512+0000 7fc44881d700 1 -- 172.21.15.94:0/3200504277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 msgr2=0x7fc44419c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.512+0000 7fc44881d700 1 --2- 172.21.15.94:0/3200504277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc44419c780 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc434003f40 tx=0x7fc434011890 comp rx=0 tx=0).stop 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.512+0000 7fc44881d700 1 -- 172.21.15.94:0/3200504277 shutdown_connections 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.512+0000 7fc44881d700 1 --2- 172.21.15.94:0/3200504277 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc42c039560 0x7fc42c03ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.512+0000 7fc44881d700 1 --2- 172.21.15.94:0/3200504277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc444108510 0x7fc44419c780 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.512+0000 7fc44881d700 1 -- 172.21.15.94:0/3200504277 >> 172.21.15.94:0/3200504277 conn(0x7fc444103b70 msgr2=0x7fc4441048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.512+0000 7fc44881d700 1 -- 172.21.15.94:0/3200504277 shutdown_connections 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:55.512+0000 7fc44881d700 1 -- 172.21.15.94:0/3200504277 wait complete. 2024-04-21T08:29:55.996 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling cephadm module... 2024-04-21T08:29:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:56 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/470200999' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-21T08:29:56.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: ignoring --setuser ceph since I am not root 2024-04-21T08:29:56.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: ignoring --setgroup ceph since I am not root 2024-04-21T08:29:56.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:56.853+0000 7fd4882e0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:29:57.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:57.039+0000 7fd4882e0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:29:57.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:57.142+0000 7fd4882e0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:29:57.305 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.413+0000 7ff174ff3700 1 Processor -- start 2024-04-21T08:29:57.305 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.413+0000 7ff174ff3700 1 -- start start 2024-04-21T08:29:57.305 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.413+0000 7ff174ff3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff170108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.413+0000 7ff174ff3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff170108f00 con 0x7ff170108510 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.413+0000 7ff16e59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff170108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.413+0000 7ff16e59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff170108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60536/0 (socket says 172.21.15.94:60536) 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.413+0000 7ff16e59c700 1 -- 172.21.15.94:0/1692125098 learned_addr learned my addr 172.21.15.94:0/1692125098 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.414+0000 7ff16e59c700 1 -- 172.21.15.94:0/1692125098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1701096c0 con 0x7ff170108510 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.414+0000 7ff16e59c700 1 --2- 172.21.15.94:0/1692125098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff170108930 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff1600098a0 tx=0x7ff160009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28791dbfad78881 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.414+0000 7ff16d59a700 1 -- 172.21.15.94:0/1692125098 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff160004160 con 0x7ff170108510 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.414+0000 7ff16d59a700 1 -- 172.21.15.94:0/1692125098 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff160004300 con 0x7ff170108510 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.414+0000 7ff16d59a700 1 -- 172.21.15.94:0/1692125098 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff160011af0 con 0x7ff170108510 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.414+0000 7ff174ff3700 1 -- 172.21.15.94:0/1692125098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 msgr2=0x7ff170108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.414+0000 7ff174ff3700 1 --2- 172.21.15.94:0/1692125098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff170108930 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff1600098a0 tx=0x7ff160009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:57.306 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.415+0000 7ff174ff3700 1 -- 172.21.15.94:0/1692125098 shutdown_connections 2024-04-21T08:29:57.307 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.415+0000 7ff174ff3700 1 --2- 172.21.15.94:0/1692125098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff170108930 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:57.307 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.415+0000 7ff174ff3700 1 -- 172.21.15.94:0/1692125098 >> 172.21.15.94:0/1692125098 conn(0x7ff170103b70 msgr2=0x7ff170105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.415+0000 7ff174ff3700 1 -- 172.21.15.94:0/1692125098 shutdown_connections 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.415+0000 7ff174ff3700 1 -- 172.21.15.94:0/1692125098 wait complete. 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.415+0000 7ff174ff3700 1 Processor -- start 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.415+0000 7ff174ff3700 1 -- start start 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.415+0000 7ff174ff3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff17019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff174ff3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff17010ca70 con 0x7ff170108510 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff16e59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff17019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff16e59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff17019c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60546/0 (socket says 172.21.15.94:60546) 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff16e59c700 1 -- 172.21.15.94:0/470200999 learned_addr learned my addr 172.21.15.94:0/470200999 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff16e59c700 1 -- 172.21.15.94:0/470200999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17019cca0 con 0x7ff170108510 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff16e59c700 1 --2- 172.21.15.94:0/470200999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff17019c760 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff160003f40 tx=0x7ff160011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff15f7fe700 1 -- 172.21.15.94:0/470200999 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1600038b0 con 0x7ff170108510 2024-04-21T08:29:57.308 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff15f7fe700 1 -- 172.21.15.94:0/470200999 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff160003a50 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff174ff3700 1 -- 172.21.15.94:0/470200999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff17019cf30 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff15f7fe700 1 -- 172.21.15.94:0/470200999 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff16001a5c0 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.416+0000 7ff174ff3700 1 -- 172.21.15.94:0/470200999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff17019fbd0 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.417+0000 7ff15f7fe700 1 -- 172.21.15.94:0/470200999 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7ff16001a870 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.417+0000 7ff174ff3700 1 -- 172.21.15.94:0/470200999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1700640a0 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.417+0000 7ff15f7fe700 1 --2- 172.21.15.94:0/470200999 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7ff158039560 0x7ff15803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.417+0000 7ff15f7fe700 1 -- 172.21.15.94:0/470200999 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff16004da50 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.417+0000 7ff16dd9b700 1 --2- 172.21.15.94:0/470200999 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7ff158039560 0x7ff15803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.418+0000 7ff16dd9b700 1 --2- 172.21.15.94:0/470200999 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7ff158039560 0x7ff15803ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff164006fd0 tx=0x7ff164006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.422+0000 7ff15f7fe700 1 -- 172.21.15.94:0/470200999 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff160019c70 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.676+0000 7ff174ff3700 1 -- 172.21.15.94:0/470200999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7ff17002df80 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.687+0000 7ff15f7fe700 1 -- 172.21.15.94:0/470200999 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7ff16004d470 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.689+0000 7ff15f7fe700 1 -- 172.21.15.94:0/470200999 <== mon.0 v2:172.21.15.94:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7ff16002d910 con 0x7ff170108510 2024-04-21T08:29:57.309 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.690+0000 7ff174ff3700 1 -- 172.21.15.94:0/470200999 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7ff158039560 msgr2=0x7ff15803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:57.310 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.690+0000 7ff174ff3700 1 --2- 172.21.15.94:0/470200999 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7ff158039560 0x7ff15803ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff164006fd0 tx=0x7ff164006e40 comp rx=0 tx=0).stop 2024-04-21T08:29:57.310 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.690+0000 7ff174ff3700 1 -- 172.21.15.94:0/470200999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 msgr2=0x7ff17019c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:57.310 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.690+0000 7ff174ff3700 1 --2- 172.21.15.94:0/470200999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff17019c760 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff160003f40 tx=0x7ff160011890 comp rx=0 tx=0).stop 2024-04-21T08:29:57.310 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.690+0000 7ff174ff3700 1 -- 172.21.15.94:0/470200999 shutdown_connections 2024-04-21T08:29:57.310 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.690+0000 7ff174ff3700 1 --2- 172.21.15.94:0/470200999 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7ff158039560 0x7ff15803ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:57.310 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.690+0000 7ff174ff3700 1 --2- 172.21.15.94:0/470200999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff170108510 0x7ff17019c760 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:57.310 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.690+0000 7ff174ff3700 1 -- 172.21.15.94:0/470200999 >> 172.21.15.94:0/470200999 conn(0x7ff170103b70 msgr2=0x7ff170104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:57.310 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.690+0000 7ff174ff3700 1 -- 172.21.15.94:0/470200999 shutdown_connections 2024-04-21T08:29:57.310 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:56.690+0000 7ff174ff3700 1 -- 172.21.15.94:0/470200999 wait complete. 2024-04-21T08:29:57.548 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:57.458+0000 7fd4882e0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:29:57.548 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:57.544+0000 7fd4882e0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:29:57.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:57 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/470200999' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-21T08:29:57.957 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:57 smithi094 ceph-mon[27988]: mgrmap e5: x(active, since 6s) 2024-04-21T08:29:58.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:29:58.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-21T08:29:58.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:29:58.620 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.752+0000 7fc435b88700 1 Processor -- start 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.752+0000 7fc435b88700 1 -- start start 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.752+0000 7fc435b88700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc430108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.752+0000 7fc435b88700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc430108f00 con 0x7fc430108510 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.753+0000 7fc42f7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc430108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.753+0000 7fc42f7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc430108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60578/0 (socket says 172.21.15.94:60578) 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.753+0000 7fc42f7fe700 1 -- 172.21.15.94:0/2051084880 learned_addr learned my addr 172.21.15.94:0/2051084880 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.753+0000 7fc42f7fe700 1 -- 172.21.15.94:0/2051084880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4301096c0 con 0x7fc430108510 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.753+0000 7fc42f7fe700 1 --2- 172.21.15.94:0/2051084880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc430108930 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc4240098a0 tx=0x7fc424009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970e92081ef2b4cd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.753+0000 7fc42e7fc700 1 -- 172.21.15.94:0/2051084880 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc424004160 con 0x7fc430108510 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.753+0000 7fc42e7fc700 1 -- 172.21.15.94:0/2051084880 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc424004300 con 0x7fc430108510 2024-04-21T08:29:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.753+0000 7fc42e7fc700 1 -- 172.21.15.94:0/2051084880 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc424011af0 con 0x7fc430108510 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.754+0000 7fc435b88700 1 -- 172.21.15.94:0/2051084880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 msgr2=0x7fc430108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.754+0000 7fc435b88700 1 --2- 172.21.15.94:0/2051084880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc430108930 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc4240098a0 tx=0x7fc424009bb0 comp rx=0 tx=0).stop 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.754+0000 7fc435b88700 1 -- 172.21.15.94:0/2051084880 shutdown_connections 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.754+0000 7fc435b88700 1 --2- 172.21.15.94:0/2051084880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc430108930 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.754+0000 7fc435b88700 1 -- 172.21.15.94:0/2051084880 >> 172.21.15.94:0/2051084880 conn(0x7fc430103b70 msgr2=0x7fc430105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.754+0000 7fc435b88700 1 -- 172.21.15.94:0/2051084880 shutdown_connections 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.754+0000 7fc435b88700 1 -- 172.21.15.94:0/2051084880 wait complete. 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.755+0000 7fc435b88700 1 Processor -- start 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.755+0000 7fc435b88700 1 -- start start 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.755+0000 7fc435b88700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc43019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.755+0000 7fc435b88700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc43010ca70 con 0x7fc430108510 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.755+0000 7fc42f7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc43019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.755+0000 7fc42f7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc43019c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60580/0 (socket says 172.21.15.94:60580) 2024-04-21T08:29:58.622 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.755+0000 7fc42f7fe700 1 -- 172.21.15.94:0/2197491152 learned_addr learned my addr 172.21.15.94:0/2197491152 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.756+0000 7fc42f7fe700 1 -- 172.21.15.94:0/2197491152 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43019ccc0 con 0x7fc430108510 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.756+0000 7fc42f7fe700 1 --2- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc43019c780 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc424000c00 tx=0x7fc424011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.756+0000 7fc42cff9700 1 -- 172.21.15.94:0/2197491152 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc424003b20 con 0x7fc430108510 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.756+0000 7fc42cff9700 1 -- 172.21.15.94:0/2197491152 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc42401a8c0 con 0x7fc430108510 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.756+0000 7fc435b88700 1 -- 172.21.15.94:0/2197491152 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc43019cf50 con 0x7fc430108510 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.756+0000 7fc42cff9700 1 -- 172.21.15.94:0/2197491152 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc424024940 con 0x7fc430108510 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.756+0000 7fc435b88700 1 -- 172.21.15.94:0/2197491152 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43019fbf0 con 0x7fc430108510 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.757+0000 7fc42cff9700 1 -- 172.21.15.94:0/2197491152 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fc424022020 con 0x7fc430108510 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.757+0000 7fc435b88700 1 -- 172.21.15.94:0/2197491152 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4300640a0 con 0x7fc430108510 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.757+0000 7fc42cff9700 1 --2- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc4180395b0 0x7fc41803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.757+0000 7fc42cff9700 1 -- 172.21.15.94:0/2197491152 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc42404d4a0 con 0x7fc430108510 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.757+0000 7fc42effd700 1 -- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc4180395b0 msgr2=0x7fc41803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/2629131251 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.757+0000 7fc42effd700 1 --2- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc4180395b0 0x7fc41803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:29:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.762+0000 7fc42cff9700 1 -- 172.21.15.94:0/2197491152 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc42404d200 con 0x7fc430108510 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.958+0000 7fc42effd700 1 -- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc4180395b0 msgr2=0x7fc41803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/2629131251 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:57.958+0000 7fc42effd700 1 --2- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc4180395b0 0x7fc41803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.059+0000 7fc435b88700 1 -- 172.21.15.94:0/2197491152 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc43002df80 con 0x7fc430108510 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.059+0000 7fc42cff9700 1 -- 172.21.15.94:0/2197491152 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fc424021d80 con 0x7fc430108510 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.061+0000 7fc435b88700 1 -- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc4180395b0 msgr2=0x7fc41803ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.061+0000 7fc435b88700 1 --2- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc4180395b0 0x7fc41803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.061+0000 7fc435b88700 1 -- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 msgr2=0x7fc43019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.061+0000 7fc435b88700 1 --2- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc43019c780 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc424000c00 tx=0x7fc424011b70 comp rx=0 tx=0).stop 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.061+0000 7fc435b88700 1 -- 172.21.15.94:0/2197491152 shutdown_connections 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.061+0000 7fc435b88700 1 --2- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fc4180395b0 0x7fc41803ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.061+0000 7fc435b88700 1 --2- 172.21.15.94:0/2197491152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc430108510 0x7fc43019c780 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.061+0000 7fc435b88700 1 -- 172.21.15.94:0/2197491152 >> 172.21.15.94:0/2197491152 conn(0x7fc430103b70 msgr2=0x7fc4301048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.062+0000 7fc435b88700 1 -- 172.21.15.94:0/2197491152 shutdown_connections 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:58.062+0000 7fc435b88700 1 -- 172.21.15.94:0/2197491152 wait complete. 2024-04-21T08:29:58.624 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for the mgr to restart... 2024-04-21T08:29:58.625 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr epoch 5... 2024-04-21T08:29:58.865 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:29:58 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2197491152' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T08:29:58.866 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:58.756+0000 7fd4882e0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:29:59.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:58.934+0000 7fd4882e0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:29:59.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:59.110+0000 7fd4882e0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:29:59.665 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:59.398+0000 7fd4882e0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:29:59.666 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:59.491+0000 7fd4882e0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:30:00.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:59.692+0000 7fd4882e0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:30:00.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:29:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:29:59.889+0000 7fd4882e0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:30:00.760 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:00.488+0000 7fd4882e0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:30:01.198 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:00.906+0000 7fd4882e0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:30:01.199 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:01 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:01.000+0000 7fd4882e0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:30:01.199 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:01 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:01.200+0000 7fd4882e0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:30:02.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:01 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:01.716+0000 7fd4882e0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:30:02.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:01 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:01.809+0000 7fd4882e0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:30:02.389 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:02.050+0000 7fd4882e0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:30:02.389 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:02.182+0000 7fd4882e0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:30:02.389 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:02.277+0000 7fd4882e0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:30:02.389 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:02.390+0000 7fd4882e0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:30:03.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:02.697+0000 7fd4882e0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:30:03.560 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:03.295+0000 7fd4882e0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:30:03.561 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:03.509+0000 7fd4882e0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:30:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: Active manager daemon x restarted 2024-04-21T08:30:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: Activating manager daemon x 2024-04-21T08:30:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: osdmap e2: 0 total, 0 up, 0 in 2024-04-21T08:30:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: mgrmap e6: x(active, starting, since 0.00338173s) 2024-04-21T08:30:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:30:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:30:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:30:04.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:30:04.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:30:04.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: Manager daemon x is now available 2024-04-21T08:30:04.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:04.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:04.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:30:04.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:30:04.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:03 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:30:04.776 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:04 smithi094 ceph-mon[27988]: Found migration_current of "None". Setting to last migration. 2024-04-21T08:30:04.776 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:04 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:30:04.777 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:04 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:04.777 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:04 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:04.777 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:04 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:30:04.777 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:04 smithi094 ceph-mon[27988]: mgrmap e7: x(active, since 1.00612s) 2024-04-21T08:30:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:30:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-21T08:30:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T08:30:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:30:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.060+0000 7fd9adf1b700 1 Processor -- start 2024-04-21T08:30:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.060+0000 7fd9adf1b700 1 -- start start 2024-04-21T08:30:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.061+0000 7fd9adf1b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a810b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.061+0000 7fd9adf1b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a807be80 con 0x7fd9a8108f40 2024-04-21T08:30:05.071 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.061+0000 7fd9a77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a810b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.061+0000 7fd9a77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a810b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60586/0 (socket says 172.21.15.94:60586) 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.061+0000 7fd9a77fe700 1 -- 172.21.15.94:0/1681452816 learned_addr learned my addr 172.21.15.94:0/1681452816 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.061+0000 7fd9a77fe700 1 -- 172.21.15.94:0/1681452816 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a810b870 con 0x7fd9a8108f40 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.061+0000 7fd9a77fe700 1 --2- 172.21.15.94:0/1681452816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a810b330 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd9980098a0 tx=0x7fd998009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0ed5dc93bafbda server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.062+0000 7fd9a67fc700 1 -- 172.21.15.94:0/1681452816 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd998004160 con 0x7fd9a8108f40 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.062+0000 7fd9a67fc700 1 -- 172.21.15.94:0/1681452816 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd998004300 con 0x7fd9a8108f40 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.062+0000 7fd9a67fc700 1 -- 172.21.15.94:0/1681452816 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd998011af0 con 0x7fd9a8108f40 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.062+0000 7fd9adf1b700 1 -- 172.21.15.94:0/1681452816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 msgr2=0x7fd9a810b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.062+0000 7fd9adf1b700 1 --2- 172.21.15.94:0/1681452816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a810b330 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd9980098a0 tx=0x7fd998009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.062+0000 7fd9adf1b700 1 -- 172.21.15.94:0/1681452816 shutdown_connections 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.062+0000 7fd9adf1b700 1 --2- 172.21.15.94:0/1681452816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a810b330 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.062+0000 7fd9adf1b700 1 -- 172.21.15.94:0/1681452816 >> 172.21.15.94:0/1681452816 conn(0x7fd9a81029b0 msgr2=0x7fd9a8104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.062+0000 7fd9adf1b700 1 -- 172.21.15.94:0/1681452816 shutdown_connections 2024-04-21T08:30:05.072 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.062+0000 7fd9adf1b700 1 -- 172.21.15.94:0/1681452816 wait complete. 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.063+0000 7fd9adf1b700 1 Processor -- start 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.063+0000 7fd9adf1b700 1 -- start start 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.063+0000 7fd9adf1b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.063+0000 7fd9adf1b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a810c830 con 0x7fd9a8108f40 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.063+0000 7fd9a77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.063+0000 7fd9a77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a819c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60588/0 (socket says 172.21.15.94:60588) 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.063+0000 7fd9a77fe700 1 -- 172.21.15.94:0/597462770 learned_addr learned my addr 172.21.15.94:0/597462770 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.064+0000 7fd9a77fe700 1 -- 172.21.15.94:0/597462770 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a819c990 con 0x7fd9a8108f40 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.064+0000 7fd9a77fe700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a819c450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd998009fb0 tx=0x7fd998011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.064+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9980036a0 con 0x7fd9a8108f40 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.064+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd99801a430 con 0x7fd9a8108f40 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.064+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd998019550 con 0x7fd9a8108f40 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.064+0000 7fd9adf1b700 1 -- 172.21.15.94:0/597462770 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a819cc20 con 0x7fd9a8108f40 2024-04-21T08:30:05.073 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.064+0000 7fd9adf1b700 1 -- 172.21.15.94:0/597462770 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a819f910 con 0x7fd9a8108f40 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.065+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fd998019850 con 0x7fd9a8108f40 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.065+0000 7fd9a4ff9700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 0x7fd99003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.065+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 --> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd99003c130 con 0x7fd990039560 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.065+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd99804da90 con 0x7fd9a8108f40 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.065+0000 7fd9a6ffd700 1 -- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 msgr2=0x7fd99003ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/2629131251 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.065+0000 7fd9a6ffd700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 0x7fd99003ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.266+0000 7fd9a6ffd700 1 -- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 msgr2=0x7fd99003ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/2629131251 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.266+0000 7fd9a6ffd700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 0x7fd99003ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.667+0000 7fd9a6ffd700 1 -- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 msgr2=0x7fd99003ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/2629131251 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:29:59.667+0000 7fd9a6ffd700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 0x7fd99003ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:00.468+0000 7fd9a6ffd700 1 -- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 msgr2=0x7fd99003ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/2629131251 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:00.468+0000 7fd9a6ffd700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 0x7fd99003ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:02.069+0000 7fd9a6ffd700 1 -- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 msgr2=0x7fd99003ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/2629131251 2024-04-21T08:30:05.074 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:02.069+0000 7fd9a6ffd700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 0x7fd99003ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:03.513+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7fd998019b90 con 0x7fd9a8108f40 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:03.513+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 msgr2=0x7fd99003ba20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:03.513+0000 7fd9a4ff9700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 0x7fd99003ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.514+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fd99804f5b0 con 0x7fd9a8108f40 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.514+0000 7fd9a4ff9700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd99003cf70 0x7fd99003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.514+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 --> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd998016e70 con 0x7fd99003cf70 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.515+0000 7fd9a6ffd700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd99003cf70 0x7fd99003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.515+0000 7fd9a6ffd700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd99003cf70 0x7fd99003f360 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd99c003a10 tx=0x7fd99c009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.516+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 <== mgr.14120 v2:172.21.15.94:6800/3877440809 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd998016e70 con 0x7fd99003cf70 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9adf1b700 1 -- 172.21.15.94:0/597462770 --> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd9a8069b90 con 0x7fd99003cf70 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9a4ff9700 1 -- 172.21.15.94:0/597462770 <== mgr.14120 v2:172.21.15.94:6800/3877440809 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fd9a8069b90 con 0x7fd99003cf70 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9adf1b700 1 -- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd99003cf70 msgr2=0x7fd99003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9adf1b700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd99003cf70 0x7fd99003f360 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd99c003a10 tx=0x7fd99c009290 comp rx=0 tx=0).stop 2024-04-21T08:30:05.075 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9adf1b700 1 -- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 msgr2=0x7fd9a819c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:05.076 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9adf1b700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a819c450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd998009fb0 tx=0x7fd998011940 comp rx=0 tx=0).stop 2024-04-21T08:30:05.076 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9adf1b700 1 -- 172.21.15.94:0/597462770 shutdown_connections 2024-04-21T08:30:05.076 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9adf1b700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd99003cf70 0x7fd99003f360 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:05.076 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9adf1b700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a8108f40 0x7fd9a819c450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:05.076 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9adf1b700 1 --2- 172.21.15.94:0/597462770 >> [v2:172.21.15.94:6800/2629131251,v1:172.21.15.94:6801/2629131251] conn(0x7fd990039560 0x7fd99003ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:05.076 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.523+0000 7fd9adf1b700 1 -- 172.21.15.94:0/597462770 >> 172.21.15.94:0/597462770 conn(0x7fd9a81029b0 msgr2=0x7fd9a8103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:05.076 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.524+0000 7fd9adf1b700 1 -- 172.21.15.94:0/597462770 shutdown_connections 2024-04-21T08:30:05.076 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:04.524+0000 7fd9adf1b700 1 -- 172.21.15.94:0/597462770 wait complete. 2024-04-21T08:30:05.076 INFO:teuthology.orchestra.run.smithi094.stdout:mgr epoch 5 is available 2024-04-21T08:30:05.076 INFO:teuthology.orchestra.run.smithi094.stdout:Setting orchestrator backend to cephadm... 2024-04-21T08:30:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:05 smithi094 ceph-mon[27988]: [21/Apr/2024:08:30:03] ENGINE Bus STARTING 2024-04-21T08:30:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:05 smithi094 ceph-mon[27988]: [21/Apr/2024:08:30:04] ENGINE Serving on https://172.21.15.94:7150 2024-04-21T08:30:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:05 smithi094 ceph-mon[27988]: [21/Apr/2024:08:30:04] ENGINE Serving on http://172.21.15.94:8765 2024-04-21T08:30:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:05 smithi094 ceph-mon[27988]: [21/Apr/2024:08:30:04] ENGINE Bus STARTED 2024-04-21T08:30:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:05 smithi094 ceph-mon[27988]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T08:30:05.711 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:05 smithi094 ceph-mon[27988]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.483+0000 7f01938bb700 1 Processor -- start 2024-04-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.483+0000 7f01938bb700 1 -- start start 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.483+0000 7f01938bb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c107230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.483+0000 7f01938bb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f018c07be20 con 0x7f018c106e10 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.483+0000 7f0191657700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c107230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.483+0000 7f0191657700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c107230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51836/0 (socket says 172.21.15.94:51836) 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.483+0000 7f0191657700 1 -- 172.21.15.94:0/1957572123 learned_addr learned my addr 172.21.15.94:0/1957572123 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.484+0000 7f0191657700 1 -- 172.21.15.94:0/1957572123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c107770 con 0x7f018c106e10 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.484+0000 7f0191657700 1 --2- 172.21.15.94:0/1957572123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c107230 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f01880098a0 tx=0x7f0188009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c298a4085c12da68 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.484+0000 7f0183fff700 1 -- 172.21.15.94:0/1957572123 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0188004160 con 0x7f018c106e10 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.484+0000 7f0183fff700 1 -- 172.21.15.94:0/1957572123 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0188004300 con 0x7f018c106e10 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.484+0000 7f0183fff700 1 -- 172.21.15.94:0/1957572123 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0188011af0 con 0x7f018c106e10 2024-04-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.484+0000 7f01938bb700 1 -- 172.21.15.94:0/1957572123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 msgr2=0x7f018c107230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.484+0000 7f01938bb700 1 --2- 172.21.15.94:0/1957572123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c107230 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f01880098a0 tx=0x7f0188009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.484+0000 7f01938bb700 1 -- 172.21.15.94:0/1957572123 shutdown_connections 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.485+0000 7f01938bb700 1 --2- 172.21.15.94:0/1957572123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c107230 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.485+0000 7f01938bb700 1 -- 172.21.15.94:0/1957572123 >> 172.21.15.94:0/1957572123 conn(0x7f018c102a70 msgr2=0x7f018c104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.485+0000 7f01938bb700 1 -- 172.21.15.94:0/1957572123 shutdown_connections 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.485+0000 7f01938bb700 1 -- 172.21.15.94:0/1957572123 wait complete. 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.485+0000 7f01938bb700 1 Processor -- start 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.485+0000 7f01938bb700 1 -- start start 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.485+0000 7f01938bb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.485+0000 7f01938bb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f018c1087a0 con 0x7f018c106e10 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.486+0000 7f0191657700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.486+0000 7f0191657700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c19e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51846/0 (socket says 172.21.15.94:51846) 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.486+0000 7f0191657700 1 -- 172.21.15.94:0/4120983155 learned_addr learned my addr 172.21.15.94:0/4120983155 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:06.285 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.486+0000 7f0191657700 1 -- 172.21.15.94:0/4120983155 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c19ee70 con 0x7f018c106e10 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.486+0000 7f0191657700 1 --2- 172.21.15.94:0/4120983155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c19e930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0188011ac0 tx=0x7f0188011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.486+0000 7f01827fc700 1 -- 172.21.15.94:0/4120983155 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0188003b20 con 0x7f018c106e10 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.486+0000 7f01827fc700 1 -- 172.21.15.94:0/4120983155 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f018801a8c0 con 0x7f018c106e10 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.486+0000 7f01938bb700 1 -- 172.21.15.94:0/4120983155 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018c19f100 con 0x7f018c106e10 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.486+0000 7f01827fc700 1 -- 172.21.15.94:0/4120983155 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01880248d0 con 0x7f018c106e10 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.486+0000 7f01938bb700 1 -- 172.21.15.94:0/4120983155 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018c1a1df0 con 0x7f018c106e10 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.487+0000 7f01827fc700 1 -- 172.21.15.94:0/4120983155 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f0188022020 con 0x7f018c106e10 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.487+0000 7f01938bb700 1 -- 172.21.15.94:0/4120983155 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f018c0640a0 con 0x7f018c106e10 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.487+0000 7f01827fc700 1 --2- 172.21.15.94:0/4120983155 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f0178039490 0x7f017803b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.487+0000 7f01827fc700 1 -- 172.21.15.94:0/4120983155 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f018804d860 con 0x7f018c106e10 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.487+0000 7f0190e56700 1 --2- 172.21.15.94:0/4120983155 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f0178039490 0x7f017803b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.488+0000 7f0190e56700 1 --2- 172.21.15.94:0/4120983155 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f0178039490 0x7f017803b950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f017c006fd0 tx=0x7f017c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:06.286 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.492+0000 7f01827fc700 1 -- 172.21.15.94:0/4120983155 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f018802cdf0 con 0x7f018c106e10 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.567+0000 7f01827fc700 1 -- 172.21.15.94:0/4120983155 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f018802dc90 con 0x7f018c106e10 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.712+0000 7f01938bb700 1 -- 172.21.15.94:0/4120983155 --> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f018c02dff0 con 0x7f0178039490 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.717+0000 7f01827fc700 1 -- 172.21.15.94:0/4120983155 <== mgr.14120 v2:172.21.15.94:6800/3877440809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f018c02dff0 con 0x7f0178039490 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.719+0000 7f01938bb700 1 -- 172.21.15.94:0/4120983155 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f0178039490 msgr2=0x7f017803b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.719+0000 7f01938bb700 1 --2- 172.21.15.94:0/4120983155 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f0178039490 0x7f017803b950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f017c006fd0 tx=0x7f017c006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.719+0000 7f01938bb700 1 -- 172.21.15.94:0/4120983155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 msgr2=0x7f018c19e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.719+0000 7f01938bb700 1 --2- 172.21.15.94:0/4120983155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c19e930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0188011ac0 tx=0x7f0188011ba0 comp rx=0 tx=0).stop 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.719+0000 7f01938bb700 1 -- 172.21.15.94:0/4120983155 shutdown_connections 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.719+0000 7f01938bb700 1 --2- 172.21.15.94:0/4120983155 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f0178039490 0x7f017803b950 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.719+0000 7f01938bb700 1 --2- 172.21.15.94:0/4120983155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f018c106e10 0x7f018c19e930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.719+0000 7f01938bb700 1 -- 172.21.15.94:0/4120983155 >> 172.21.15.94:0/4120983155 conn(0x7f018c102a70 msgr2=0x7f018c195280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.719+0000 7f01938bb700 1 -- 172.21.15.94:0/4120983155 shutdown_connections 2024-04-21T08:30:06.287 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:05.719+0000 7f01938bb700 1 -- 172.21.15.94:0/4120983155 wait complete. 2024-04-21T08:30:06.903 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:06 smithi094 ceph-mon[27988]: mgrmap e8: x(active, since 2s) 2024-04-21T08:30:06.903 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:06 smithi094 ceph-mon[27988]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:06.903 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:06 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:06.904 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:06 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:30:07.515 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-21T08:30:07.515 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.695+0000 7f01ed131700 1 Processor -- start 2024-04-21T08:30:07.515 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.695+0000 7f01ed131700 1 -- start start 2024-04-21T08:30:07.515 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.695+0000 7f01ed131700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8107d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:07.515 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.695+0000 7f01ed131700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f01e8108320 con 0x7f01e8107930 2024-04-21T08:30:07.515 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.695+0000 7f01e6d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8107d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:07.515 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.695+0000 7f01e6d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8107d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51848/0 (socket says 172.21.15.94:51848) 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.695+0000 7f01e6d9d700 1 -- 172.21.15.94:0/1514676860 learned_addr learned my addr 172.21.15.94:0/1514676860 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.696+0000 7f01e6d9d700 1 -- 172.21.15.94:0/1514676860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e8108650 con 0x7f01e8107930 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.696+0000 7f01e6d9d700 1 --2- 172.21.15.94:0/1514676860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8107d50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f01d80098a0 tx=0x7f01d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c62960ec74a831c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.696+0000 7f01e659c700 1 -- 172.21.15.94:0/1514676860 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01d8004160 con 0x7f01e8107930 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.696+0000 7f01e659c700 1 -- 172.21.15.94:0/1514676860 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f01d8004300 con 0x7f01e8107930 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.696+0000 7f01e659c700 1 -- 172.21.15.94:0/1514676860 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01d8011af0 con 0x7f01e8107930 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.697+0000 7f01ed131700 1 -- 172.21.15.94:0/1514676860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 msgr2=0x7f01e8107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.697+0000 7f01ed131700 1 --2- 172.21.15.94:0/1514676860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8107d50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f01d80098a0 tx=0x7f01d8009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.697+0000 7f01ed131700 1 -- 172.21.15.94:0/1514676860 shutdown_connections 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.697+0000 7f01ed131700 1 --2- 172.21.15.94:0/1514676860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8107d50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.697+0000 7f01ed131700 1 -- 172.21.15.94:0/1514676860 >> 172.21.15.94:0/1514676860 conn(0x7f01e81021b0 msgr2=0x7f01e81045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.697+0000 7f01ed131700 1 -- 172.21.15.94:0/1514676860 shutdown_connections 2024-04-21T08:30:07.516 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.697+0000 7f01ed131700 1 -- 172.21.15.94:0/1514676860 wait complete. 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.697+0000 7f01ed131700 1 Processor -- start 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01ed131700 1 -- start start 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01ed131700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01ed131700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f01e810ba00 con 0x7f01e8107930 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01e6d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01e6d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51852/0 (socket says 172.21.15.94:51852) 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01e6d9d700 1 -- 172.21.15.94:0/3194556744 learned_addr learned my addr 172.21.15.94:0/3194556744 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01e6d9d700 1 -- 172.21.15.94:0/3194556744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e819f4b0 con 0x7f01e8107930 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01e6d9d700 1 --2- 172.21.15.94:0/3194556744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8106ee0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f01d8011ac0 tx=0x7f01d8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01e4d99700 1 -- 172.21.15.94:0/3194556744 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01d8003b20 con 0x7f01e8107930 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01e4d99700 1 -- 172.21.15.94:0/3194556744 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f01d801a8c0 con 0x7f01e8107930 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01ed131700 1 -- 172.21.15.94:0/3194556744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e819f680 con 0x7f01e8107930 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.698+0000 7f01e4d99700 1 -- 172.21.15.94:0/3194556744 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01d80248d0 con 0x7f01e8107930 2024-04-21T08:30:07.517 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.699+0000 7f01ed131700 1 -- 172.21.15.94:0/3194556744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e819fa20 con 0x7f01e8107930 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.700+0000 7f01e4d99700 1 -- 172.21.15.94:0/3194556744 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f01d8022020 con 0x7f01e8107930 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.700+0000 7f01e4d99700 1 --2- 172.21.15.94:0/3194556744 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f01d40395b0 0x7f01d403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.700+0000 7f01ed131700 1 -- 172.21.15.94:0/3194556744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e8069b90 con 0x7f01e8107930 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.700+0000 7f01e4d99700 1 -- 172.21.15.94:0/3194556744 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f01d804d8c0 con 0x7f01e8107930 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.700+0000 7f01de59c700 1 --2- 172.21.15.94:0/3194556744 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f01d40395b0 0x7f01d403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.700+0000 7f01de59c700 1 --2- 172.21.15.94:0/3194556744 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f01d40395b0 0x7f01d403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01d0006fd0 tx=0x7f01d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.705+0000 7f01e4d99700 1 -- 172.21.15.94:0/3194556744 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f01d800b5a0 con 0x7f01e8107930 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.904+0000 7f01ed131700 1 -- 172.21.15.94:0/3194556744 --> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f01e802e100 con 0x7f01d40395b0 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.905+0000 7f01e4d99700 1 -- 172.21.15.94:0/3194556744 <== mgr.14120 v2:172.21.15.94:6800/3877440809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f01e802e100 con 0x7f01d40395b0 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.908+0000 7f01ed131700 1 -- 172.21.15.94:0/3194556744 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f01d40395b0 msgr2=0x7f01d403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.908+0000 7f01ed131700 1 --2- 172.21.15.94:0/3194556744 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f01d40395b0 0x7f01d403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01d0006fd0 tx=0x7f01d0006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.908+0000 7f01ed131700 1 -- 172.21.15.94:0/3194556744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 msgr2=0x7f01e8106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:07.518 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.908+0000 7f01ed131700 1 --2- 172.21.15.94:0/3194556744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8106ee0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f01d8011ac0 tx=0x7f01d8011ba0 comp rx=0 tx=0).stop 2024-04-21T08:30:07.519 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.908+0000 7f01ed131700 1 -- 172.21.15.94:0/3194556744 shutdown_connections 2024-04-21T08:30:07.519 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.908+0000 7f01ed131700 1 --2- 172.21.15.94:0/3194556744 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f01d40395b0 0x7f01d403ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:07.519 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.908+0000 7f01ed131700 1 --2- 172.21.15.94:0/3194556744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e8107930 0x7f01e8106ee0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:07.519 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.908+0000 7f01ed131700 1 -- 172.21.15.94:0/3194556744 >> 172.21.15.94:0/3194556744 conn(0x7f01e81021b0 msgr2=0x7f01e8102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:07.519 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.908+0000 7f01ed131700 1 -- 172.21.15.94:0/3194556744 shutdown_connections 2024-04-21T08:30:07.519 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:06.908+0000 7f01ed131700 1 -- 172.21.15.94:0/3194556744 wait complete. 2024-04-21T08:30:07.519 INFO:teuthology.orchestra.run.smithi094.stdout:Generating ssh key... 2024-04-21T08:30:08.109 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:07 smithi094 ceph-mon[27988]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:08.510 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: Generating public/private rsa key pair. 2024-04-21T08:30:08.510 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: Your identification has been saved in /tmp/tmpjonect_y/key. 2024-04-21T08:30:08.510 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: Your public key has been saved in /tmp/tmpjonect_y/key.pub. 2024-04-21T08:30:08.510 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: The key fingerprint is: 2024-04-21T08:30:08.510 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: SHA256:HxR8VNVvdA5lLcCw6vEPip8LYUqmw7MByVG+f0dXpKg ceph-2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:30:08.510 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: The key's randomart image is: 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: +---[RSA 3072]----+ 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: | . .o+o+.o*| 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: | o oo= o.=| 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: | . . .oo . =o| 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: |... . .o . +| 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: |o. .o oES... . | 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: | o +.o o.+.. | 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: | * ......+ | 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: | = .o.o o | 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: | . ..=. . | 2024-04-21T08:30:08.511 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: +----[SHA256]-----+ 2024-04-21T08:30:08.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.903+0000 7f60fe036700 1 Processor -- start 2024-04-21T08:30:08.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.903+0000 7f60fe036700 1 -- start start 2024-04-21T08:30:08.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.903+0000 7f60fe036700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f8108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.903+0000 7f60fe036700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f60f8108f00 con 0x7f60f8108510 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.903+0000 7f60f77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f8108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.903+0000 7f60f77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f8108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51866/0 (socket says 172.21.15.94:51866) 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.903+0000 7f60f77fe700 1 -- 172.21.15.94:0/1896024144 learned_addr learned my addr 172.21.15.94:0/1896024144 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.903+0000 7f60f77fe700 1 -- 172.21.15.94:0/1896024144 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f81096c0 con 0x7f60f8108510 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.903+0000 7f60f77fe700 1 --2- 172.21.15.94:0/1896024144 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f8108930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f60e80098a0 tx=0x7f60e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0d6fee9a2d7fd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.904+0000 7f60f67fc700 1 -- 172.21.15.94:0/1896024144 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60e8004160 con 0x7f60f8108510 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.904+0000 7f60f67fc700 1 -- 172.21.15.94:0/1896024144 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f60e8004300 con 0x7f60f8108510 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.904+0000 7f60f67fc700 1 -- 172.21.15.94:0/1896024144 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60e8011af0 con 0x7f60f8108510 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.904+0000 7f60fe036700 1 -- 172.21.15.94:0/1896024144 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 msgr2=0x7f60f8108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.904+0000 7f60fe036700 1 --2- 172.21.15.94:0/1896024144 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f8108930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f60e80098a0 tx=0x7f60e8009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:08.847 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.904+0000 7f60fe036700 1 -- 172.21.15.94:0/1896024144 shutdown_connections 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.904+0000 7f60fe036700 1 --2- 172.21.15.94:0/1896024144 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f8108930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.904+0000 7f60fe036700 1 -- 172.21.15.94:0/1896024144 >> 172.21.15.94:0/1896024144 conn(0x7f60f8103b70 msgr2=0x7f60f8105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.904+0000 7f60fe036700 1 -- 172.21.15.94:0/1896024144 shutdown_connections 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.904+0000 7f60fe036700 1 -- 172.21.15.94:0/1896024144 wait complete. 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.905+0000 7f60fe036700 1 Processor -- start 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.905+0000 7f60fe036700 1 -- start start 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.905+0000 7f60fe036700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.905+0000 7f60fe036700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f60f810ca70 con 0x7f60f8108510 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.905+0000 7f60f77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.905+0000 7f60f77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f819c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51874/0 (socket says 172.21.15.94:51874) 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.905+0000 7f60f77fe700 1 -- 172.21.15.94:0/429751093 learned_addr learned my addr 172.21.15.94:0/429751093 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.906+0000 7f60f77fe700 1 -- 172.21.15.94:0/429751093 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f819ccc0 con 0x7f60f8108510 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.906+0000 7f60f77fe700 1 --2- 172.21.15.94:0/429751093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f819c780 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f60e8000c00 tx=0x7f60e8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:08.848 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.906+0000 7f60f4ff9700 1 -- 172.21.15.94:0/429751093 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60e8003b20 con 0x7f60f8108510 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.906+0000 7f60f4ff9700 1 -- 172.21.15.94:0/429751093 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f60e801a8c0 con 0x7f60f8108510 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.906+0000 7f60fe036700 1 -- 172.21.15.94:0/429751093 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f819cf50 con 0x7f60f8108510 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.906+0000 7f60f4ff9700 1 -- 172.21.15.94:0/429751093 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60e8024940 con 0x7f60f8108510 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.906+0000 7f60fe036700 1 -- 172.21.15.94:0/429751093 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f819fbf0 con 0x7f60f8108510 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.907+0000 7f60f4ff9700 1 -- 172.21.15.94:0/429751093 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f60e8022020 con 0x7f60f8108510 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.907+0000 7f60f4ff9700 1 --2- 172.21.15.94:0/429751093 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f60e0039600 0x7f60e003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.907+0000 7f60f4ff9700 1 -- 172.21.15.94:0/429751093 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f60e804dac0 con 0x7f60f8108510 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.907+0000 7f60f6ffd700 1 --2- 172.21.15.94:0/429751093 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f60e0039600 0x7f60e003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.907+0000 7f60fe036700 1 -- 172.21.15.94:0/429751093 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60f80640a0 con 0x7f60f8108510 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.907+0000 7f60f6ffd700 1 --2- 172.21.15.94:0/429751093 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f60e0039600 0x7f60e003bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60ec006fd0 tx=0x7f60ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:07.912+0000 7f60f4ff9700 1 -- 172.21.15.94:0/429751093 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60e801aa70 con 0x7f60f8108510 2024-04-21T08:30:08.849 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.111+0000 7f60fe036700 1 -- 172.21.15.94:0/429751093 --> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f60f8062e80 con 0x7f60e0039600 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.297+0000 7f60f4ff9700 1 -- 172.21.15.94:0/429751093 <== mgr.14120 v2:172.21.15.94:6800/3877440809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f60f8062e80 con 0x7f60e0039600 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.299+0000 7f60fe036700 1 -- 172.21.15.94:0/429751093 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f60e0039600 msgr2=0x7f60e003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.299+0000 7f60fe036700 1 --2- 172.21.15.94:0/429751093 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f60e0039600 0x7f60e003bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60ec006fd0 tx=0x7f60ec006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.299+0000 7f60fe036700 1 -- 172.21.15.94:0/429751093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 msgr2=0x7f60f819c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.299+0000 7f60fe036700 1 --2- 172.21.15.94:0/429751093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f819c780 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f60e8000c00 tx=0x7f60e8011b70 comp rx=0 tx=0).stop 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.300+0000 7f60fe036700 1 -- 172.21.15.94:0/429751093 shutdown_connections 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.300+0000 7f60fe036700 1 --2- 172.21.15.94:0/429751093 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f60e0039600 0x7f60e003bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.300+0000 7f60fe036700 1 --2- 172.21.15.94:0/429751093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60f8108510 0x7f60f819c780 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.300+0000 7f60fe036700 1 -- 172.21.15.94:0/429751093 >> 172.21.15.94:0/429751093 conn(0x7f60f8103b70 msgr2=0x7f60f81048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.300+0000 7f60fe036700 1 -- 172.21.15.94:0/429751093 shutdown_connections 2024-04-21T08:30:08.850 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:08.300+0000 7f60fe036700 1 -- 172.21.15.94:0/429751093 wait complete. 2024-04-21T08:30:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:09 smithi094 ceph-mon[27988]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:09 smithi094 ceph-mon[27988]: Generating ssh key... 2024-04-21T08:30:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:09 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:09 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:10.011 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd5BGgFqwdS0Pe1MHfgnTwKJImIzYikZc7luxu4v6nVx0oRIdb95xeNm9oL4Me5L5VJXTfWm0mUr30zIOS3CR3au6Dwl452WJ2orRxd4lWe3Me4TlQDPw0T5qMFXCCr6NZpgbs1hASg04f+ti1QJLnlpzeSU97YEFKQqucDmf8AlcsdCVCqSCACrbJGA/AYiK3Rf8oKojSnLWd8oL45kk2sWDsyLc0tSLdGn33niSGccwQ8bvzAbVCoRUVktMz8LMOusQNiaBJ8RSAteRoxCpCO3og0uXLPVnoguaY6ZHlZl2To/KaUFKKKhDP/+241D0nFza1Ats29w5mA0y56mWvdT0uKyKYa1Y5R3OgxD4HrNC6L5sq8APd8Oqne/QAVE/TI/74djWOJ31TiUWeyRIPzrEtHs5kPk4SSObucpP5JVA9f7bPh027HwzIsqeXqg77sMuqQkmXopEbZuqD5uWjDY2QlvOf9toNGakGFgc5wSYlBJ1Wrus3hov3uwHkMOE= ceph-2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:30:10.011 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.240+0000 7fe454f49700 1 Processor -- start 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.240+0000 7fe454f49700 1 -- start start 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.240+0000 7fe454f49700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe450106c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.240+0000 7fe454f49700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe450107210 con 0x7fe450106820 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.240+0000 7fe44e59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe450106c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.240+0000 7fe44e59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe450106c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51884/0 (socket says 172.21.15.94:51884) 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.240+0000 7fe44e59c700 1 -- 172.21.15.94:0/2586344854 learned_addr learned my addr 172.21.15.94:0/2586344854 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.240+0000 7fe44e59c700 1 -- 172.21.15.94:0/2586344854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4501097b0 con 0x7fe450106820 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.241+0000 7fe44e59c700 1 --2- 172.21.15.94:0/2586344854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe450106c40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe4440098a0 tx=0x7fe444009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58bda7feb3cb6e56 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.241+0000 7fe44d59a700 1 -- 172.21.15.94:0/2586344854 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe444004160 con 0x7fe450106820 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.241+0000 7fe44d59a700 1 -- 172.21.15.94:0/2586344854 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe444004300 con 0x7fe450106820 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.241+0000 7fe44d59a700 1 -- 172.21.15.94:0/2586344854 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe444011af0 con 0x7fe450106820 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.241+0000 7fe454f49700 1 -- 172.21.15.94:0/2586344854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 msgr2=0x7fe450106c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.241+0000 7fe454f49700 1 --2- 172.21.15.94:0/2586344854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe450106c40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe4440098a0 tx=0x7fe444009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:10.012 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.242+0000 7fe454f49700 1 -- 172.21.15.94:0/2586344854 shutdown_connections 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.242+0000 7fe454f49700 1 --2- 172.21.15.94:0/2586344854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe450106c40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.242+0000 7fe454f49700 1 -- 172.21.15.94:0/2586344854 >> 172.21.15.94:0/2586344854 conn(0x7fe4501021b0 msgr2=0x7fe4501045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.242+0000 7fe454f49700 1 -- 172.21.15.94:0/2586344854 shutdown_connections 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.242+0000 7fe454f49700 1 -- 172.21.15.94:0/2586344854 wait complete. 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.242+0000 7fe454f49700 1 Processor -- start 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.242+0000 7fe454f49700 1 -- start start 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.242+0000 7fe454f49700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe45019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.242+0000 7fe454f49700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe45010cb60 con 0x7fe450106820 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.243+0000 7fe44e59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe45019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.243+0000 7fe44e59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe45019c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51900/0 (socket says 172.21.15.94:51900) 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.243+0000 7fe44e59c700 1 -- 172.21.15.94:0/387629902 learned_addr learned my addr 172.21.15.94:0/387629902 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.243+0000 7fe44e59c700 1 -- 172.21.15.94:0/387629902 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45019ccc0 con 0x7fe450106820 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.243+0000 7fe44e59c700 1 --2- 172.21.15.94:0/387629902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe45019c780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe444003f40 tx=0x7fe444011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:10.013 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.243+0000 7fe43f7fe700 1 -- 172.21.15.94:0/387629902 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4440038b0 con 0x7fe450106820 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.243+0000 7fe454f49700 1 -- 172.21.15.94:0/387629902 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe45019cf50 con 0x7fe450106820 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.243+0000 7fe43f7fe700 1 -- 172.21.15.94:0/387629902 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe444003a50 con 0x7fe450106820 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.243+0000 7fe43f7fe700 1 -- 172.21.15.94:0/387629902 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe44401a5c0 con 0x7fe450106820 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.243+0000 7fe454f49700 1 -- 172.21.15.94:0/387629902 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe45019fc40 con 0x7fe450106820 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.244+0000 7fe43f7fe700 1 -- 172.21.15.94:0/387629902 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe44401a8c0 con 0x7fe450106820 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.244+0000 7fe43f7fe700 1 --2- 172.21.15.94:0/387629902 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fe438039560 0x7fe43803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.244+0000 7fe454f49700 1 -- 172.21.15.94:0/387629902 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4500640a0 con 0x7fe450106820 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.244+0000 7fe43f7fe700 1 -- 172.21.15.94:0/387629902 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe44404db40 con 0x7fe450106820 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.244+0000 7fe44dd9b700 1 --2- 172.21.15.94:0/387629902 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fe438039560 0x7fe43803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.245+0000 7fe44dd9b700 1 --2- 172.21.15.94:0/387629902 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fe438039560 0x7fe43803ba20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe440006fd0 tx=0x7fe440006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.249+0000 7fe43f7fe700 1 -- 172.21.15.94:0/387629902 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe444019c70 con 0x7fe450106820 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.448+0000 7fe454f49700 1 -- 172.21.15.94:0/387629902 --> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fe450062e80 con 0x7fe438039560 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.449+0000 7fe43f7fe700 1 -- 172.21.15.94:0/387629902 <== mgr.14120 v2:172.21.15.94:6800/3877440809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fe450062e80 con 0x7fe438039560 2024-04-21T08:30:10.014 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.451+0000 7fe454f49700 1 -- 172.21.15.94:0/387629902 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fe438039560 msgr2=0x7fe43803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.451+0000 7fe454f49700 1 --2- 172.21.15.94:0/387629902 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fe438039560 0x7fe43803ba20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe440006fd0 tx=0x7fe440006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.451+0000 7fe454f49700 1 -- 172.21.15.94:0/387629902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 msgr2=0x7fe45019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.451+0000 7fe454f49700 1 --2- 172.21.15.94:0/387629902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe45019c780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe444003f40 tx=0x7fe444011890 comp rx=0 tx=0).stop 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.451+0000 7fe454f49700 1 -- 172.21.15.94:0/387629902 shutdown_connections 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.451+0000 7fe454f49700 1 --2- 172.21.15.94:0/387629902 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fe438039560 0x7fe43803ba20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.451+0000 7fe454f49700 1 --2- 172.21.15.94:0/387629902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450106820 0x7fe45019c780 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.451+0000 7fe454f49700 1 -- 172.21.15.94:0/387629902 >> 172.21.15.94:0/387629902 conn(0x7fe4501021b0 msgr2=0x7fe450102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.451+0000 7fe454f49700 1 -- 172.21.15.94:0/387629902 shutdown_connections 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:09.451+0000 7fe454f49700 1 -- 172.21.15.94:0/387629902 wait complete. 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:Adding key to root@localhost authorized_keys... 2024-04-21T08:30:10.015 INFO:teuthology.orchestra.run.smithi094.stdout:Adding host smithi094... 2024-04-21T08:30:10.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:10 smithi094 ceph-mon[27988]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:11.503 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:11 smithi094 ceph-mon[27988]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi094", "addr": "172.21.15.94", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:12.759 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:12 smithi094 ceph-mon[27988]: Deploying cephadm binary to smithi094 2024-04-21T08:30:15.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Added host 'smithi094' with addr '172.21.15.94' 2024-04-21T08:30:15.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.441+0000 7fd71eea8700 1 Processor -- start 2024-04-21T08:30:15.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd71eea8700 1 -- start start 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd71eea8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71807c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd71eea8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd71807c920 con 0x7fd71807dea0 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd71cc44700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71807c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd71cc44700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71807c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51906/0 (socket says 172.21.15.94:51906) 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd71cc44700 1 -- 172.21.15.94:0/110220535 learned_addr learned my addr 172.21.15.94:0/110220535 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd71cc44700 1 -- 172.21.15.94:0/110220535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71807caa0 con 0x7fd71807dea0 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd71cc44700 1 --2- 172.21.15.94:0/110220535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71807c350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0098a0 tx=0x7fd70c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b7ed8f39bcdd07 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd7177fe700 1 -- 172.21.15.94:0/110220535 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd70c004160 con 0x7fd71807dea0 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd7177fe700 1 -- 172.21.15.94:0/110220535 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd70c004300 con 0x7fd71807dea0 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.442+0000 7fd7177fe700 1 -- 172.21.15.94:0/110220535 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd70c011af0 con 0x7fd71807dea0 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.443+0000 7fd71eea8700 1 -- 172.21.15.94:0/110220535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 msgr2=0x7fd71807c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.443+0000 7fd71eea8700 1 --2- 172.21.15.94:0/110220535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71807c350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0098a0 tx=0x7fd70c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.443+0000 7fd71eea8700 1 -- 172.21.15.94:0/110220535 shutdown_connections 2024-04-21T08:30:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.443+0000 7fd71eea8700 1 --2- 172.21.15.94:0/110220535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71807c350 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.443+0000 7fd71eea8700 1 -- 172.21.15.94:0/110220535 >> 172.21.15.94:0/110220535 conn(0x7fd718103b70 msgr2=0x7fd718105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.443+0000 7fd71eea8700 1 -- 172.21.15.94:0/110220535 shutdown_connections 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.443+0000 7fd71eea8700 1 -- 172.21.15.94:0/110220535 wait complete. 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd71eea8700 1 Processor -- start 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd71eea8700 1 -- start start 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd71eea8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71819e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd71eea8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd71810ea80 con 0x7fd71807dea0 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd71cc44700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71819e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd71cc44700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71819e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51918/0 (socket says 172.21.15.94:51918) 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd71cc44700 1 -- 172.21.15.94:0/2277913308 learned_addr learned my addr 172.21.15.94:0/2277913308 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd71cc44700 1 -- 172.21.15.94:0/2277913308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71819ee50 con 0x7fd71807dea0 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd71cc44700 1 --2- 172.21.15.94:0/2277913308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71819e910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd70c003f40 tx=0x7fd70c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:15.265 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd715ffb700 1 -- 172.21.15.94:0/2277913308 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd70c0038b0 con 0x7fd71807dea0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd715ffb700 1 -- 172.21.15.94:0/2277913308 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd70c003a50 con 0x7fd71807dea0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd715ffb700 1 -- 172.21.15.94:0/2277913308 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd70c01a5c0 con 0x7fd71807dea0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.444+0000 7fd71eea8700 1 -- 172.21.15.94:0/2277913308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71819f0e0 con 0x7fd71807dea0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.445+0000 7fd71eea8700 1 -- 172.21.15.94:0/2277913308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7181a1dd0 con 0x7fd71807dea0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.445+0000 7fd715ffb700 1 -- 172.21.15.94:0/2277913308 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd70c024430 con 0x7fd71807dea0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.445+0000 7fd71eea8700 1 -- 172.21.15.94:0/2277913308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7180640a0 con 0x7fd71807dea0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.446+0000 7fd715ffb700 1 --2- 172.21.15.94:0/2277913308 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd708039560 0x7fd70803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.446+0000 7fd715ffb700 1 -- 172.21.15.94:0/2277913308 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd70c04dae0 con 0x7fd71807dea0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.446+0000 7fd717fff700 1 --2- 172.21.15.94:0/2277913308 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd708039560 0x7fd70803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.446+0000 7fd717fff700 1 --2- 172.21.15.94:0/2277913308 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd708039560 0x7fd70803ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd704006fd0 tx=0x7fd704006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.451+0000 7fd715ffb700 1 -- 172.21.15.94:0/2277913308 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd70c02d910 con 0x7fd71807dea0 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:10.660+0000 7fd71eea8700 1 -- 172.21.15.94:0/2277913308 --> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi094", "addr": "172.21.15.94", "target": ["mon-mgr", ""]}) -- 0x7fd71802dff0 con 0x7fd708039560 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.684+0000 7fd715ffb700 1 -- 172.21.15.94:0/2277913308 <== mgr.14120 v2:172.21.15.94:6800/3877440809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fd71802dff0 con 0x7fd708039560 2024-04-21T08:30:15.266 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.686+0000 7fd71eea8700 1 -- 172.21.15.94:0/2277913308 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd708039560 msgr2=0x7fd70803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:15.267 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.686+0000 7fd71eea8700 1 --2- 172.21.15.94:0/2277913308 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd708039560 0x7fd70803ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd704006fd0 tx=0x7fd704006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:15.267 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.686+0000 7fd71eea8700 1 -- 172.21.15.94:0/2277913308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 msgr2=0x7fd71819e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:15.267 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.686+0000 7fd71eea8700 1 --2- 172.21.15.94:0/2277913308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71819e910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd70c003f40 tx=0x7fd70c011890 comp rx=0 tx=0).stop 2024-04-21T08:30:15.267 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.686+0000 7fd71eea8700 1 -- 172.21.15.94:0/2277913308 shutdown_connections 2024-04-21T08:30:15.267 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.686+0000 7fd71eea8700 1 --2- 172.21.15.94:0/2277913308 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fd708039560 0x7fd70803ba20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:15.267 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.686+0000 7fd71eea8700 1 --2- 172.21.15.94:0/2277913308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd71807dea0 0x7fd71819e910 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:15.267 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.686+0000 7fd71eea8700 1 -- 172.21.15.94:0/2277913308 >> 172.21.15.94:0/2277913308 conn(0x7fd718103b70 msgr2=0x7fd718104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:15.267 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.686+0000 7fd71eea8700 1 -- 172.21.15.94:0/2277913308 shutdown_connections 2024-04-21T08:30:15.267 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:14.687+0000 7fd71eea8700 1 -- 172.21.15.94:0/2277913308 wait complete. 2024-04-21T08:30:15.267 INFO:teuthology.orchestra.run.smithi094.stdout:Deploying unmanaged mon service... 2024-04-21T08:30:15.920 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:15 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:15.920 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:15 smithi094 ceph-mon[27988]: Added host smithi094 2024-04-21T08:30:15.920 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:15 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:30:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-21T08:30:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.694+0000 7fc304fca700 1 Processor -- start 2024-04-21T08:30:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.694+0000 7fc304fca700 1 -- start start 2024-04-21T08:30:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.694+0000 7fc304fca700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc300108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.694+0000 7fc304fca700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc300108f00 con 0x7fc300108510 2024-04-21T08:30:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.694+0000 7fc2fe59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc300108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.694+0000 7fc2fe59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc300108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55256/0 (socket says 172.21.15.94:55256) 2024-04-21T08:30:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.694+0000 7fc2fe59c700 1 -- 172.21.15.94:0/1687868929 learned_addr learned my addr 172.21.15.94:0/1687868929 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.695+0000 7fc2fe59c700 1 -- 172.21.15.94:0/1687868929 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3001096c0 con 0x7fc300108510 2024-04-21T08:30:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.695+0000 7fc2fe59c700 1 --2- 172.21.15.94:0/1687868929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc300108930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00098a0 tx=0x7fc2f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd589212ae5d13f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.695+0000 7fc2fd59a700 1 -- 172.21.15.94:0/1687868929 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2f0004160 con 0x7fc300108510 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.695+0000 7fc2fd59a700 1 -- 172.21.15.94:0/1687868929 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc2f0004300 con 0x7fc300108510 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.695+0000 7fc2fd59a700 1 -- 172.21.15.94:0/1687868929 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2f0011af0 con 0x7fc300108510 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.695+0000 7fc304fca700 1 -- 172.21.15.94:0/1687868929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 msgr2=0x7fc300108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.695+0000 7fc304fca700 1 --2- 172.21.15.94:0/1687868929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc300108930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00098a0 tx=0x7fc2f0009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.696+0000 7fc304fca700 1 -- 172.21.15.94:0/1687868929 shutdown_connections 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.696+0000 7fc304fca700 1 --2- 172.21.15.94:0/1687868929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc300108930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.696+0000 7fc304fca700 1 -- 172.21.15.94:0/1687868929 >> 172.21.15.94:0/1687868929 conn(0x7fc300103b70 msgr2=0x7fc300105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.696+0000 7fc304fca700 1 -- 172.21.15.94:0/1687868929 shutdown_connections 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.696+0000 7fc304fca700 1 -- 172.21.15.94:0/1687868929 wait complete. 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.696+0000 7fc304fca700 1 Processor -- start 2024-04-21T08:30:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.696+0000 7fc304fca700 1 -- start start 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.696+0000 7fc304fca700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc30007db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc304fca700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc30010ca70 con 0x7fc300108510 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc2fe59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc30007db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc2fe59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc30007db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55260/0 (socket says 172.21.15.94:55260) 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc2fe59c700 1 -- 172.21.15.94:0/781635611 learned_addr learned my addr 172.21.15.94:0/781635611 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc2fe59c700 1 -- 172.21.15.94:0/781635611 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30007e070 con 0x7fc300108510 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc2fe59c700 1 --2- 172.21.15.94:0/781635611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc30007db30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0000c00 tx=0x7fc2f0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc2ef7fe700 1 -- 172.21.15.94:0/781635611 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2f0003b20 con 0x7fc300108510 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc2ef7fe700 1 -- 172.21.15.94:0/781635611 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc2f001a8c0 con 0x7fc300108510 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc304fca700 1 -- 172.21.15.94:0/781635611 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30007c2b0 con 0x7fc300108510 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc2ef7fe700 1 -- 172.21.15.94:0/781635611 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2f0024940 con 0x7fc300108510 2024-04-21T08:30:16.533 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.697+0000 7fc304fca700 1 -- 172.21.15.94:0/781635611 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30007c790 con 0x7fc300108510 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.698+0000 7fc2ef7fe700 1 -- 172.21.15.94:0/781635611 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc2f0022020 con 0x7fc300108510 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.698+0000 7fc304fca700 1 -- 172.21.15.94:0/781635611 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3000640a0 con 0x7fc300108510 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.698+0000 7fc2ef7fe700 1 --2- 172.21.15.94:0/781635611 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fc2e80395b0 0x7fc2e803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.698+0000 7fc2ef7fe700 1 -- 172.21.15.94:0/781635611 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc2f004d810 con 0x7fc300108510 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.698+0000 7fc2fdd9b700 1 --2- 172.21.15.94:0/781635611 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fc2e80395b0 0x7fc2e803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.699+0000 7fc2fdd9b700 1 --2- 172.21.15.94:0/781635611 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fc2e80395b0 0x7fc2e803ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4006fd0 tx=0x7fc2f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.703+0000 7fc2ef7fe700 1 -- 172.21.15.94:0/781635611 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2f0021d80 con 0x7fc300108510 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.920+0000 7fc304fca700 1 -- 172.21.15.94:0/781635611 --> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc300195ef0 con 0x7fc2e80395b0 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.925+0000 7fc2ef7fe700 1 -- 172.21.15.94:0/781635611 <== mgr.14120 v2:172.21.15.94:6800/3877440809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc300195ef0 con 0x7fc2e80395b0 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.927+0000 7fc304fca700 1 -- 172.21.15.94:0/781635611 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fc2e80395b0 msgr2=0x7fc2e803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:16.534 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.927+0000 7fc304fca700 1 --2- 172.21.15.94:0/781635611 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fc2e80395b0 0x7fc2e803ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4006fd0 tx=0x7fc2f4006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:16.535 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.927+0000 7fc304fca700 1 -- 172.21.15.94:0/781635611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 msgr2=0x7fc30007db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:16.535 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.927+0000 7fc304fca700 1 --2- 172.21.15.94:0/781635611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc30007db30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0000c00 tx=0x7fc2f0011b70 comp rx=0 tx=0).stop 2024-04-21T08:30:16.535 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.927+0000 7fc304fca700 1 -- 172.21.15.94:0/781635611 shutdown_connections 2024-04-21T08:30:16.535 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.927+0000 7fc304fca700 1 --2- 172.21.15.94:0/781635611 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fc2e80395b0 0x7fc2e803ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:16.535 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.927+0000 7fc304fca700 1 --2- 172.21.15.94:0/781635611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc300108510 0x7fc30007db30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:16.535 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.927+0000 7fc304fca700 1 -- 172.21.15.94:0/781635611 >> 172.21.15.94:0/781635611 conn(0x7fc300103b70 msgr2=0x7fc300104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:16.535 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.927+0000 7fc304fca700 1 -- 172.21.15.94:0/781635611 shutdown_connections 2024-04-21T08:30:16.535 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:15.927+0000 7fc304fca700 1 -- 172.21.15.94:0/781635611 wait complete. 2024-04-21T08:30:16.535 INFO:teuthology.orchestra.run.smithi094.stdout:Deploying unmanaged mgr service... 2024-04-21T08:30:17.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:16 smithi094 ceph-mon[27988]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:17.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:16 smithi094 ceph-mon[27988]: Saving service mon spec with placement count:5 2024-04-21T08:30:17.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:16 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:18.751 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-21T08:30:18.751 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.003+0000 7fad608e8700 1 Processor -- start 2024-04-21T08:30:18.751 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.003+0000 7fad608e8700 1 -- start start 2024-04-21T08:30:18.751 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.004+0000 7fad608e8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c1071b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:18.751 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.004+0000 7fad608e8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c07be40 con 0x7fad5c106d90 2024-04-21T08:30:18.751 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.004+0000 7fad5a59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c1071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.004+0000 7fad5a59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c1071b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55264/0 (socket says 172.21.15.94:55264) 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.004+0000 7fad5a59c700 1 -- 172.21.15.94:0/2988720252 learned_addr learned my addr 172.21.15.94:0/2988720252 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.004+0000 7fad5a59c700 1 -- 172.21.15.94:0/2988720252 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5c1076f0 con 0x7fad5c106d90 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.004+0000 7fad5a59c700 1 --2- 172.21.15.94:0/2988720252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c1071b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0098a0 tx=0x7fad4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=103281c607022780 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.004+0000 7fad5959a700 1 -- 172.21.15.94:0/2988720252 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad4c004160 con 0x7fad5c106d90 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.004+0000 7fad5959a700 1 -- 172.21.15.94:0/2988720252 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad4c004300 con 0x7fad5c106d90 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.004+0000 7fad5959a700 1 -- 172.21.15.94:0/2988720252 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad4c011a20 con 0x7fad5c106d90 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.005+0000 7fad608e8700 1 -- 172.21.15.94:0/2988720252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 msgr2=0x7fad5c1071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.005+0000 7fad608e8700 1 --2- 172.21.15.94:0/2988720252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c1071b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0098a0 tx=0x7fad4c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.005+0000 7fad608e8700 1 -- 172.21.15.94:0/2988720252 shutdown_connections 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.005+0000 7fad608e8700 1 --2- 172.21.15.94:0/2988720252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c1071b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.005+0000 7fad608e8700 1 -- 172.21.15.94:0/2988720252 >> 172.21.15.94:0/2988720252 conn(0x7fad5c102970 msgr2=0x7fad5c104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.005+0000 7fad608e8700 1 -- 172.21.15.94:0/2988720252 shutdown_connections 2024-04-21T08:30:18.752 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.005+0000 7fad608e8700 1 -- 172.21.15.94:0/2988720252 wait complete. 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.006+0000 7fad608e8700 1 Processor -- start 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.006+0000 7fad608e8700 1 -- start start 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.006+0000 7fad608e8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.006+0000 7fad608e8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c108760 con 0x7fad5c106d90 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.006+0000 7fad5a59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.006+0000 7fad5a59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c19e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55268/0 (socket says 172.21.15.94:55268) 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.006+0000 7fad5a59c700 1 -- 172.21.15.94:0/1876032583 learned_addr learned my addr 172.21.15.94:0/1876032583 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.007+0000 7fad5a59c700 1 -- 172.21.15.94:0/1876032583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5c19eb60 con 0x7fad5c106d90 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.007+0000 7fad5a59c700 1 --2- 172.21.15.94:0/1876032583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c19e620 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0116e0 tx=0x7fad4c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.007+0000 7fad4b7fe700 1 -- 172.21.15.94:0/1876032583 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad4c003b20 con 0x7fad5c106d90 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.007+0000 7fad4b7fe700 1 -- 172.21.15.94:0/1876032583 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad4c01b8f0 con 0x7fad5c106d90 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.007+0000 7fad608e8700 1 -- 172.21.15.94:0/1876032583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5c19edf0 con 0x7fad5c106d90 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.007+0000 7fad4b7fe700 1 -- 172.21.15.94:0/1876032583 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad4c0238c0 con 0x7fad5c106d90 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.007+0000 7fad608e8700 1 -- 172.21.15.94:0/1876032583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad5c1a1ae0 con 0x7fad5c106d90 2024-04-21T08:30:18.753 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.008+0000 7fad608e8700 1 -- 172.21.15.94:0/1876032583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad5c0640a0 con 0x7fad5c106d90 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.008+0000 7fad4b7fe700 1 -- 172.21.15.94:0/1876032583 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fad4c023b70 con 0x7fad5c106d90 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.008+0000 7fad4b7fe700 1 --2- 172.21.15.94:0/1876032583 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fad44039560 0x7fad4403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.008+0000 7fad4b7fe700 1 -- 172.21.15.94:0/1876032583 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fad4c04e020 con 0x7fad5c106d90 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.009+0000 7fad59d9b700 1 --2- 172.21.15.94:0/1876032583 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fad44039560 0x7fad4403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.009+0000 7fad59d9b700 1 --2- 172.21.15.94:0/1876032583 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fad44039560 0x7fad4403ba20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fad50006fd0 tx=0x7fad50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.013+0000 7fad4b7fe700 1 -- 172.21.15.94:0/1876032583 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad4c01f030 con 0x7fad5c106d90 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.235+0000 7fad608e8700 1 -- 172.21.15.94:0/1876032583 --> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fad5c02e240 con 0x7fad44039560 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.240+0000 7fad4b7fe700 1 -- 172.21.15.94:0/1876032583 <== mgr.14120 v2:172.21.15.94:6800/3877440809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fad5c02e240 con 0x7fad44039560 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.242+0000 7fad608e8700 1 -- 172.21.15.94:0/1876032583 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fad44039560 msgr2=0x7fad4403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.242+0000 7fad608e8700 1 --2- 172.21.15.94:0/1876032583 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fad44039560 0x7fad4403ba20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fad50006fd0 tx=0x7fad50006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.242+0000 7fad608e8700 1 -- 172.21.15.94:0/1876032583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 msgr2=0x7fad5c19e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.242+0000 7fad608e8700 1 --2- 172.21.15.94:0/1876032583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c19e620 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0116e0 tx=0x7fad4c011aa0 comp rx=0 tx=0).stop 2024-04-21T08:30:18.754 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.242+0000 7fad608e8700 1 -- 172.21.15.94:0/1876032583 shutdown_connections 2024-04-21T08:30:18.755 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.242+0000 7fad608e8700 1 --2- 172.21.15.94:0/1876032583 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fad44039560 0x7fad4403ba20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:18.755 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.242+0000 7fad608e8700 1 --2- 172.21.15.94:0/1876032583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad5c106d90 0x7fad5c19e620 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:18.755 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.242+0000 7fad608e8700 1 -- 172.21.15.94:0/1876032583 >> 172.21.15.94:0/1876032583 conn(0x7fad5c102970 msgr2=0x7fad5c195200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:18.755 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.242+0000 7fad608e8700 1 -- 172.21.15.94:0/1876032583 shutdown_connections 2024-04-21T08:30:18.755 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:18.242+0000 7fad608e8700 1 -- 172.21.15.94:0/1876032583 wait complete. 2024-04-21T08:30:18.844 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:18 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:18.844 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:18 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:18.844 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:18 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:19.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:19 smithi094 ceph-mon[27988]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:19.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:19 smithi094 ceph-mon[27988]: Saving service mgr spec with placement count:2 2024-04-21T08:30:19.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:19 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2701201940' entity='client.admin' 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.123+0000 7fdf9e680700 1 Processor -- start 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.123+0000 7fdf9e680700 1 -- start start 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.123+0000 7fdf9e680700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf98108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.123+0000 7fdf9e680700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdf98108f00 con 0x7fdf98108510 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.123+0000 7fdf97fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf98108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.123+0000 7fdf97fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf98108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55282/0 (socket says 172.21.15.94:55282) 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.123+0000 7fdf97fff700 1 -- 172.21.15.94:0/657769330 learned_addr learned my addr 172.21.15.94:0/657769330 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.124+0000 7fdf97fff700 1 -- 172.21.15.94:0/657769330 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf981096c0 con 0x7fdf98108510 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.124+0000 7fdf97fff700 1 --2- 172.21.15.94:0/657769330 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf98108930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdf880098a0 tx=0x7fdf88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c8196a58db4192 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.124+0000 7fdf96ffd700 1 -- 172.21.15.94:0/657769330 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf88004160 con 0x7fdf98108510 2024-04-21T08:30:19.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.124+0000 7fdf96ffd700 1 -- 172.21.15.94:0/657769330 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf88004300 con 0x7fdf98108510 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.124+0000 7fdf96ffd700 1 -- 172.21.15.94:0/657769330 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf88011a20 con 0x7fdf98108510 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.125+0000 7fdf9e680700 1 -- 172.21.15.94:0/657769330 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 msgr2=0x7fdf98108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.125+0000 7fdf9e680700 1 --2- 172.21.15.94:0/657769330 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf98108930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdf880098a0 tx=0x7fdf88009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.125+0000 7fdf9e680700 1 -- 172.21.15.94:0/657769330 shutdown_connections 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.125+0000 7fdf9e680700 1 --2- 172.21.15.94:0/657769330 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf98108930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.125+0000 7fdf9e680700 1 -- 172.21.15.94:0/657769330 >> 172.21.15.94:0/657769330 conn(0x7fdf98103b70 msgr2=0x7fdf98105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.125+0000 7fdf9e680700 1 -- 172.21.15.94:0/657769330 shutdown_connections 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.125+0000 7fdf9e680700 1 -- 172.21.15.94:0/657769330 wait complete. 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.125+0000 7fdf9e680700 1 Processor -- start 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.125+0000 7fdf9e680700 1 -- start start 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.125+0000 7fdf9e680700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf9819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf9e680700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9810ca70 con 0x7fdf98108510 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf97fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf9819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:19.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf97fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf9819c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55298/0 (socket says 172.21.15.94:55298) 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf97fff700 1 -- 172.21.15.94:0/2701201940 learned_addr learned my addr 172.21.15.94:0/2701201940 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf97fff700 1 -- 172.21.15.94:0/2701201940 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9819cc40 con 0x7fdf98108510 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf97fff700 1 --2- 172.21.15.94:0/2701201940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf9819c700 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009520 tx=0x7fdf88011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf957fa700 1 -- 172.21.15.94:0/2701201940 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf880039e0 con 0x7fdf98108510 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf9e680700 1 -- 172.21.15.94:0/2701201940 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9819ced0 con 0x7fdf98108510 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf957fa700 1 -- 172.21.15.94:0/2701201940 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf8801b8f0 con 0x7fdf98108510 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf957fa700 1 -- 172.21.15.94:0/2701201940 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf88019770 con 0x7fdf98108510 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.126+0000 7fdf9e680700 1 -- 172.21.15.94:0/2701201940 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9819fb70 con 0x7fdf98108510 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.127+0000 7fdf957fa700 1 -- 172.21.15.94:0/2701201940 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fdf88019a70 con 0x7fdf98108510 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.127+0000 7fdf9e680700 1 -- 172.21.15.94:0/2701201940 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf980640a0 con 0x7fdf98108510 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.127+0000 7fdf957fa700 1 --2- 172.21.15.94:0/2701201940 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fdf800395b0 0x7fdf8003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.127+0000 7fdf957fa700 1 -- 172.21.15.94:0/2701201940 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdf8804ded0 con 0x7fdf98108510 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.127+0000 7fdf977fe700 1 --2- 172.21.15.94:0/2701201940 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fdf800395b0 0x7fdf8003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:19.953 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.128+0000 7fdf977fe700 1 --2- 172.21.15.94:0/2701201940 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fdf800395b0 0x7fdf8003ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c006fd0 tx=0x7fdf8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.132+0000 7fdf957fa700 1 -- 172.21.15.94:0/2701201940 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf8801f030 con 0x7fdf98108510 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.364+0000 7fdf9e680700 1 -- 172.21.15.94:0/2701201940 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fdf9802df80 con 0x7fdf98108510 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.367+0000 7fdf957fa700 1 -- 172.21.15.94:0/2701201940 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fdf880172b0 con 0x7fdf98108510 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.369+0000 7fdf9e680700 1 -- 172.21.15.94:0/2701201940 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fdf800395b0 msgr2=0x7fdf8003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.369+0000 7fdf9e680700 1 --2- 172.21.15.94:0/2701201940 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fdf800395b0 0x7fdf8003ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c006fd0 tx=0x7fdf8c006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.369+0000 7fdf9e680700 1 -- 172.21.15.94:0/2701201940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 msgr2=0x7fdf9819c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.369+0000 7fdf9e680700 1 --2- 172.21.15.94:0/2701201940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf9819c700 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009520 tx=0x7fdf88011aa0 comp rx=0 tx=0).stop 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.369+0000 7fdf9e680700 1 -- 172.21.15.94:0/2701201940 shutdown_connections 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.369+0000 7fdf9e680700 1 --2- 172.21.15.94:0/2701201940 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7fdf800395b0 0x7fdf8003ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.369+0000 7fdf9e680700 1 --2- 172.21.15.94:0/2701201940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf98108510 0x7fdf9819c700 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:19.954 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.369+0000 7fdf9e680700 1 -- 172.21.15.94:0/2701201940 >> 172.21.15.94:0/2701201940 conn(0x7fdf98103b70 msgr2=0x7fdf98104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:19.955 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.369+0000 7fdf9e680700 1 -- 172.21.15.94:0/2701201940 shutdown_connections 2024-04-21T08:30:19.955 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:19.369+0000 7fdf9e680700 1 -- 172.21.15.94:0/2701201940 wait complete. 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.349+0000 7f4c82657700 1 Processor -- start 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.349+0000 7f4c82657700 1 -- start start 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.349+0000 7f4c82657700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c108dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.349+0000 7f4c82657700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c1093a0 con 0x7f4c7c1089b0 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.349+0000 7f4c7bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c108dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.349+0000 7f4c7bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c108dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55302/0 (socket says 172.21.15.94:55302) 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.349+0000 7f4c7bfff700 1 -- 172.21.15.94:0/2290315226 learned_addr learned my addr 172.21.15.94:0/2290315226 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.349+0000 7f4c7bfff700 1 -- 172.21.15.94:0/2290315226 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c107660 con 0x7f4c7c1089b0 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.349+0000 7f4c7bfff700 1 --2- 172.21.15.94:0/2290315226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c108dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4c700098a0 tx=0x7f4c70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6da2f1b41301d53 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.350+0000 7f4c7affd700 1 -- 172.21.15.94:0/2290315226 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c70004160 con 0x7f4c7c1089b0 2024-04-21T08:30:21.436 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.350+0000 7f4c7affd700 1 -- 172.21.15.94:0/2290315226 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c70004300 con 0x7f4c7c1089b0 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.350+0000 7f4c7affd700 1 -- 172.21.15.94:0/2290315226 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c70011a20 con 0x7f4c7c1089b0 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.350+0000 7f4c82657700 1 -- 172.21.15.94:0/2290315226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 msgr2=0x7f4c7c108dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.350+0000 7f4c82657700 1 --2- 172.21.15.94:0/2290315226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c108dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4c700098a0 tx=0x7f4c70009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.350+0000 7f4c82657700 1 -- 172.21.15.94:0/2290315226 shutdown_connections 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.350+0000 7f4c82657700 1 --2- 172.21.15.94:0/2290315226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c108dd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.350+0000 7f4c82657700 1 -- 172.21.15.94:0/2290315226 >> 172.21.15.94:0/2290315226 conn(0x7f4c7c102110 msgr2=0x7f4c7c104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.350+0000 7f4c82657700 1 -- 172.21.15.94:0/2290315226 shutdown_connections 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.350+0000 7f4c82657700 1 -- 172.21.15.94:0/2290315226 wait complete. 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.351+0000 7f4c82657700 1 Processor -- start 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.351+0000 7f4c82657700 1 -- start start 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.351+0000 7f4c82657700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.351+0000 7f4c82657700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c10ba20 con 0x7f4c7c1089b0 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.351+0000 7f4c7bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:21.437 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.351+0000 7f4c7bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55312/0 (socket says 172.21.15.94:55312) 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.351+0000 7f4c7bfff700 1 -- 172.21.15.94:0/3728650972 learned_addr learned my addr 172.21.15.94:0/3728650972 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.352+0000 7f4c7bfff700 1 -- 172.21.15.94:0/3728650972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c19ca80 con 0x7f4c7c1089b0 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.352+0000 7f4c7bfff700 1 --2- 172.21.15.94:0/3728650972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c19c540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4c700119f0 tx=0x7f4c70011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.352+0000 7f4c797fa700 1 -- 172.21.15.94:0/3728650972 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c700039e0 con 0x7f4c7c1089b0 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.352+0000 7f4c797fa700 1 -- 172.21.15.94:0/3728650972 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c7001b8f0 con 0x7f4c7c1089b0 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.352+0000 7f4c82657700 1 -- 172.21.15.94:0/3728650972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7c19cd10 con 0x7f4c7c1089b0 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.352+0000 7f4c82657700 1 -- 172.21.15.94:0/3728650972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7c19d170 con 0x7f4c7c1089b0 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.352+0000 7f4c797fa700 1 -- 172.21.15.94:0/3728650972 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c700237a0 con 0x7f4c7c1089b0 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.353+0000 7f4c797fa700 1 -- 172.21.15.94:0/3728650972 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4c7002c3f0 con 0x7f4c7c1089b0 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.353+0000 7f4c82657700 1 -- 172.21.15.94:0/3728650972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c7c0640a0 con 0x7f4c7c1089b0 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.353+0000 7f4c797fa700 1 --2- 172.21.15.94:0/3728650972 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4c640395b0 0x7f4c6403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.353+0000 7f4c797fa700 1 -- 172.21.15.94:0/3728650972 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4c7004de50 con 0x7f4c7c1089b0 2024-04-21T08:30:21.438 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.353+0000 7f4c7b7fe700 1 --2- 172.21.15.94:0/3728650972 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4c640395b0 0x7f4c6403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.354+0000 7f4c7b7fe700 1 --2- 172.21.15.94:0/3728650972 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4c640395b0 0x7f4c6403ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c006fd0 tx=0x7f4c6c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.358+0000 7f4c797fa700 1 -- 172.21.15.94:0/3728650972 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c7001f030 con 0x7f4c7c1089b0 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.564+0000 7f4c82657700 1 -- 172.21.15.94:0/3728650972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f4c7c063650 con 0x7f4c7c1089b0 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.567+0000 7f4c797fa700 1 -- 172.21.15.94:0/3728650972 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f4c7004f070 con 0x7f4c7c1089b0 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.569+0000 7f4c82657700 1 -- 172.21.15.94:0/3728650972 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4c640395b0 msgr2=0x7f4c6403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.569+0000 7f4c82657700 1 --2- 172.21.15.94:0/3728650972 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4c640395b0 0x7f4c6403ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c006fd0 tx=0x7f4c6c006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.569+0000 7f4c82657700 1 -- 172.21.15.94:0/3728650972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 msgr2=0x7f4c7c19c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.569+0000 7f4c82657700 1 --2- 172.21.15.94:0/3728650972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c19c540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4c700119f0 tx=0x7f4c70011ad0 comp rx=0 tx=0).stop 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.569+0000 7f4c82657700 1 -- 172.21.15.94:0/3728650972 shutdown_connections 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.569+0000 7f4c82657700 1 --2- 172.21.15.94:0/3728650972 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4c640395b0 0x7f4c6403ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.569+0000 7f4c82657700 1 --2- 172.21.15.94:0/3728650972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c7c1089b0 0x7f4c7c19c540 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.569+0000 7f4c82657700 1 -- 172.21.15.94:0/3728650972 >> 172.21.15.94:0/3728650972 conn(0x7f4c7c102110 msgr2=0x7f4c7c102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.570+0000 7f4c82657700 1 -- 172.21.15.94:0/3728650972 shutdown_connections 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:20.570+0000 7f4c82657700 1 -- 172.21.15.94:0/3728650972 wait complete. 2024-04-21T08:30:21.439 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling the dashboard module... 2024-04-21T08:30:21.669 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:21 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3728650972' entity='client.admin' 2024-04-21T08:30:21.670 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:21 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:22.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:22 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/600779153' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-21T08:30:22.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:22 smithi094 ceph-mon[27988]: from='mgr.14120 172.21.15.94:0/1179268932' entity='mgr.x' 2024-04-21T08:30:22.894 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: ignoring --setuser ceph since I am not root 2024-04-21T08:30:22.894 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: ignoring --setgroup ceph since I am not root 2024-04-21T08:30:22.894 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:22.752+0000 7fbd8b237200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:30:23.187 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:22.944+0000 7fbd8b237200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:30:23.187 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:23 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:23.046+0000 7fbd8b237200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:30:23.195 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.919+0000 7f8e990e4700 1 Processor -- start 2024-04-21T08:30:23.195 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.919+0000 7f8e990e4700 1 -- start start 2024-04-21T08:30:23.195 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.919+0000 7f8e990e4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e94106f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:23.195 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.919+0000 7f8e990e4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8e94107520 con 0x7f8e94106b30 2024-04-21T08:30:23.195 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.919+0000 7f8e92d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e94106f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:23.195 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.919+0000 7f8e92d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e94106f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55328/0 (socket says 172.21.15.94:55328) 2024-04-21T08:30:23.195 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.919+0000 7f8e92d9d700 1 -- 172.21.15.94:0/1446720870 learned_addr learned my addr 172.21.15.94:0/1446720870 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:23.195 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.920+0000 7f8e92d9d700 1 -- 172.21.15.94:0/1446720870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e94107d30 con 0x7f8e94106b30 2024-04-21T08:30:23.195 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.920+0000 7f8e92d9d700 1 --2- 172.21.15.94:0/1446720870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e94106f50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8e840098a0 tx=0x7f8e84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42b482a3a7361c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.920+0000 7f8e91d9b700 1 -- 172.21.15.94:0/1446720870 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e84004160 con 0x7f8e94106b30 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.920+0000 7f8e91d9b700 1 -- 172.21.15.94:0/1446720870 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e84004300 con 0x7f8e94106b30 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.920+0000 7f8e91d9b700 1 -- 172.21.15.94:0/1446720870 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e84011a20 con 0x7f8e94106b30 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.921+0000 7f8e990e4700 1 -- 172.21.15.94:0/1446720870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 msgr2=0x7f8e94106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.921+0000 7f8e990e4700 1 --2- 172.21.15.94:0/1446720870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e94106f50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8e840098a0 tx=0x7f8e84009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.921+0000 7f8e990e4700 1 -- 172.21.15.94:0/1446720870 shutdown_connections 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.921+0000 7f8e990e4700 1 --2- 172.21.15.94:0/1446720870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e94106f50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.921+0000 7f8e990e4700 1 -- 172.21.15.94:0/1446720870 >> 172.21.15.94:0/1446720870 conn(0x7f8e94102110 msgr2=0x7f8e94104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.921+0000 7f8e990e4700 1 -- 172.21.15.94:0/1446720870 shutdown_connections 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.921+0000 7f8e990e4700 1 -- 172.21.15.94:0/1446720870 wait complete. 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.921+0000 7f8e990e4700 1 Processor -- start 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.922+0000 7f8e990e4700 1 -- start start 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.922+0000 7f8e990e4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e9419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:23.196 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.922+0000 7f8e990e4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8e941084a0 con 0x7f8e94106b30 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.922+0000 7f8e92d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e9419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.922+0000 7f8e92d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e9419c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55334/0 (socket says 172.21.15.94:55334) 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.922+0000 7f8e92d9d700 1 -- 172.21.15.94:0/600779153 learned_addr learned my addr 172.21.15.94:0/600779153 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.922+0000 7f8e92d9d700 1 -- 172.21.15.94:0/600779153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9419c980 con 0x7f8e94106b30 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.922+0000 7f8e92d9d700 1 --2- 172.21.15.94:0/600779153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e9419c440 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8e840119f0 tx=0x7f8e84011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.923+0000 7f8e7bfff700 1 -- 172.21.15.94:0/600779153 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e840039e0 con 0x7f8e94106b30 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.923+0000 7f8e990e4700 1 -- 172.21.15.94:0/600779153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e9419cc10 con 0x7f8e94106b30 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.923+0000 7f8e7bfff700 1 -- 172.21.15.94:0/600779153 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e8401b8f0 con 0x7f8e94106b30 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.923+0000 7f8e7bfff700 1 -- 172.21.15.94:0/600779153 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e840237a0 con 0x7f8e94106b30 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.923+0000 7f8e990e4700 1 -- 172.21.15.94:0/600779153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e9419f900 con 0x7f8e94106b30 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.924+0000 7f8e7bfff700 1 -- 172.21.15.94:0/600779153 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f8e84023aa0 con 0x7f8e94106b30 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.924+0000 7f8e990e4700 1 -- 172.21.15.94:0/600779153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e940640a0 con 0x7f8e94106b30 2024-04-21T08:30:23.197 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.924+0000 7f8e7bfff700 1 --2- 172.21.15.94:0/600779153 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f8e7c039560 0x7f8e7c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.924+0000 7f8e7bfff700 1 -- 172.21.15.94:0/600779153 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8e8404de70 con 0x7f8e94106b30 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.924+0000 7f8e9259c700 1 --2- 172.21.15.94:0/600779153 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f8e7c039560 0x7f8e7c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.925+0000 7f8e9259c700 1 --2- 172.21.15.94:0/600779153 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f8e7c039560 0x7f8e7c03ba20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8e88006fd0 tx=0x7f8e88006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:21.929+0000 7f8e7bfff700 1 -- 172.21.15.94:0/600779153 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e8401f030 con 0x7f8e94106b30 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.193+0000 7f8e990e4700 1 -- 172.21.15.94:0/600779153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f8e9402e210 con 0x7f8e94106b30 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.569+0000 7f8e7bfff700 1 -- 172.21.15.94:0/600779153 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f8e84018e30 con 0x7f8e94106b30 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.571+0000 7f8e7bfff700 1 -- 172.21.15.94:0/600779153 <== mon.0 v2:172.21.15.94:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f8e8402c910 con 0x7f8e94106b30 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.572+0000 7f8e990e4700 1 -- 172.21.15.94:0/600779153 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f8e7c039560 msgr2=0x7f8e7c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.572+0000 7f8e990e4700 1 --2- 172.21.15.94:0/600779153 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f8e7c039560 0x7f8e7c03ba20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8e88006fd0 tx=0x7f8e88006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.572+0000 7f8e990e4700 1 -- 172.21.15.94:0/600779153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 msgr2=0x7f8e9419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.572+0000 7f8e990e4700 1 --2- 172.21.15.94:0/600779153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e9419c440 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8e840119f0 tx=0x7f8e84011ad0 comp rx=0 tx=0).stop 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.572+0000 7f8e990e4700 1 -- 172.21.15.94:0/600779153 shutdown_connections 2024-04-21T08:30:23.198 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.572+0000 7f8e990e4700 1 --2- 172.21.15.94:0/600779153 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f8e7c039560 0x7f8e7c03ba20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:23.199 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.572+0000 7f8e990e4700 1 --2- 172.21.15.94:0/600779153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e94106b30 0x7f8e9419c440 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:23.199 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.572+0000 7f8e990e4700 1 -- 172.21.15.94:0/600779153 >> 172.21.15.94:0/600779153 conn(0x7f8e94102110 msgr2=0x7f8e941930a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:23.199 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.573+0000 7f8e990e4700 1 -- 172.21.15.94:0/600779153 shutdown_connections 2024-04-21T08:30:23.199 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:22.573+0000 7f8e990e4700 1 -- 172.21.15.94:0/600779153 wait complete. 2024-04-21T08:30:23.450 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:23 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:23.360+0000 7fbd8b237200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:30:23.450 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:23 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:23.450+0000 7fbd8b237200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:30:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:23 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/600779153' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-21T08:30:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:23 smithi094 ceph-mon[27988]: mgrmap e9: x(active, since 19s) 2024-04-21T08:30:24.849 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:24 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/607978941' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T08:30:24.849 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:24 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:24.677+0000 7fbd8b237200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:30:24.849 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:24 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:24.849+0000 7fbd8b237200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:30:24.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:30:24.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-21T08:30:24.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:30:24.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T08:30:24.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T08:30:24.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:30:24.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.601+0000 7f89d9f1d700 1 Processor -- start 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.601+0000 7f89d9f1d700 1 -- start start 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.601+0000 7f89d9f1d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d41077b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.601+0000 7f89d9f1d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4107d80 con 0x7f89d4107390 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.602+0000 7f89d37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d41077b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.602+0000 7f89d37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d41077b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34926/0 (socket says 172.21.15.94:34926) 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.602+0000 7f89d37fe700 1 -- 172.21.15.94:0/1822505020 learned_addr learned my addr 172.21.15.94:0/1822505020 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.602+0000 7f89d37fe700 1 -- 172.21.15.94:0/1822505020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d4108590 con 0x7f89d4107390 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.602+0000 7f89d37fe700 1 --2- 172.21.15.94:0/1822505020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d41077b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f89c800bb40 tx=0x7f89c800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee6a85c36d6d13f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.602+0000 7f89d27fc700 1 -- 172.21.15.94:0/1822505020 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c800dbe0 con 0x7f89d4107390 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.602+0000 7f89d27fc700 1 -- 172.21.15.94:0/1822505020 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89c80044e0 con 0x7f89d4107390 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.602+0000 7f89d27fc700 1 -- 172.21.15.94:0/1822505020 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c8010470 con 0x7f89d4107390 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.603+0000 7f89d9f1d700 1 -- 172.21.15.94:0/1822505020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 msgr2=0x7f89d41077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:24.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.603+0000 7f89d9f1d700 1 --2- 172.21.15.94:0/1822505020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d41077b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f89c800bb40 tx=0x7f89c800be50 comp rx=0 tx=0).stop 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.603+0000 7f89d9f1d700 1 -- 172.21.15.94:0/1822505020 shutdown_connections 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.603+0000 7f89d9f1d700 1 --2- 172.21.15.94:0/1822505020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d41077b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.603+0000 7f89d9f1d700 1 -- 172.21.15.94:0/1822505020 >> 172.21.15.94:0/1822505020 conn(0x7f89d41029b0 msgr2=0x7f89d4104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.603+0000 7f89d9f1d700 1 -- 172.21.15.94:0/1822505020 shutdown_connections 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.603+0000 7f89d9f1d700 1 -- 172.21.15.94:0/1822505020 wait complete. 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.604+0000 7f89d9f1d700 1 Processor -- start 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.604+0000 7f89d9f1d700 1 -- start start 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.604+0000 7f89d9f1d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.604+0000 7f89d9f1d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f89d407afe0 con 0x7f89d4107390 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.604+0000 7f89d37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.604+0000 7f89d37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34934/0 (socket says 172.21.15.94:34934) 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.604+0000 7f89d37fe700 1 -- 172.21.15.94:0/607978941 learned_addr learned my addr 172.21.15.94:0/607978941 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.604+0000 7f89d37fe700 1 -- 172.21.15.94:0/607978941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d419a790 con 0x7f89d4107390 2024-04-21T08:30:24.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.604+0000 7f89d37fe700 1 --2- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d419a250 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f89c800b340 tx=0x7f89c8003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.605+0000 7f89d0ff9700 1 -- 172.21.15.94:0/607978941 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c8027070 con 0x7f89d4107390 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.605+0000 7f89d9f1d700 1 -- 172.21.15.94:0/607978941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d419aa20 con 0x7f89d4107390 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.605+0000 7f89d0ff9700 1 -- 172.21.15.94:0/607978941 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89c8010c20 con 0x7f89d4107390 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.605+0000 7f89d0ff9700 1 -- 172.21.15.94:0/607978941 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c8022b70 con 0x7f89d4107390 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.605+0000 7f89d9f1d700 1 -- 172.21.15.94:0/607978941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d419d710 con 0x7f89d4107390 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.605+0000 7f89d0ff9700 1 -- 172.21.15.94:0/607978941 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f89c802a070 con 0x7f89d4107390 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.606+0000 7f89d0ff9700 1 --2- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f89bc0395b0 0x7f89bc03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.606+0000 7f89d0ff9700 1 -- 172.21.15.94:0/607978941 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f89c801f070 con 0x7f89d4107390 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.606+0000 7f89d2ffd700 1 -- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f89bc0395b0 msgr2=0x7f89bc03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3877440809 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.606+0000 7f89d2ffd700 1 --2- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f89bc0395b0 0x7f89bc03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.606+0000 7f89d9f1d700 1 -- 172.21.15.94:0/607978941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d40640a0 con 0x7f89d4107390 2024-04-21T08:30:24.926 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.611+0000 7f89d0ff9700 1 -- 172.21.15.94:0/607978941 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f89c8015070 con 0x7f89d4107390 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.807+0000 7f89d2ffd700 1 -- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f89bc0395b0 msgr2=0x7f89bc03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3877440809 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.807+0000 7f89d2ffd700 1 --2- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f89bc0395b0 0x7f89bc03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.926+0000 7f89d9f1d700 1 -- 172.21.15.94:0/607978941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f89d419d8e0 con 0x7f89d4107390 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.926+0000 7f89d0ff9700 1 -- 172.21.15.94:0/607978941 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f89c8010dd0 con 0x7f89d4107390 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.928+0000 7f89d9f1d700 1 -- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f89bc0395b0 msgr2=0x7f89bc03ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.928+0000 7f89d9f1d700 1 --2- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f89bc0395b0 0x7f89bc03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.928+0000 7f89d9f1d700 1 -- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 msgr2=0x7f89d419a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.928+0000 7f89d9f1d700 1 --2- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d419a250 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f89c800b340 tx=0x7f89c8003dc0 comp rx=0 tx=0).stop 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.928+0000 7f89d9f1d700 1 -- 172.21.15.94:0/607978941 shutdown_connections 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.928+0000 7f89d9f1d700 1 --2- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f89bc0395b0 0x7f89bc03ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.928+0000 7f89d9f1d700 1 --2- 172.21.15.94:0/607978941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4107390 0x7f89d419a250 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.928+0000 7f89d9f1d700 1 -- 172.21.15.94:0/607978941 >> 172.21.15.94:0/607978941 conn(0x7f89d41029b0 msgr2=0x7f89d4190870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.929+0000 7f89d9f1d700 1 -- 172.21.15.94:0/607978941 shutdown_connections 2024-04-21T08:30:24.928 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:23.929+0000 7f89d9f1d700 1 -- 172.21.15.94:0/607978941 wait complete. 2024-04-21T08:30:24.928 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for the mgr to restart... 2024-04-21T08:30:24.928 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr epoch 9... 2024-04-21T08:30:25.136 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:25.028+0000 7fbd8b237200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:30:25.418 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:25.325+0000 7fbd8b237200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:30:25.418 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:25.418+0000 7fbd8b237200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:30:25.727 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:25.623+0000 7fbd8b237200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:30:26.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:25.826+0000 7fbd8b237200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:30:26.760 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:26 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:26.425+0000 7fbd8b237200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:30:27.151 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:26 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:26.852+0000 7fbd8b237200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:30:27.152 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:26 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:26.950+0000 7fbd8b237200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:30:27.510 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:27 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:27.153+0000 7fbd8b237200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:30:28.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:27 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:27.675+0000 7fbd8b237200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:30:28.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:27 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:27.769+0000 7fbd8b237200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:30:28.358 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:28 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:28.014+0000 7fbd8b237200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:30:28.358 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:28 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:28.147+0000 7fbd8b237200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:30:28.358 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:28 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:28.243+0000 7fbd8b237200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:30:28.359 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:28 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:28.358+0000 7fbd8b237200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:30:29.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:28 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:28.657+0000 7fbd8b237200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:30:29.534 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:29.262+0000 7fbd8b237200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:30:29.534 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:30:29.481+0000 7fbd8b237200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:30:30.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: Active manager daemon x restarted 2024-04-21T08:30:30.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: Activating manager daemon x 2024-04-21T08:30:30.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: osdmap e3: 0 total, 0 up, 0 in 2024-04-21T08:30:30.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: mgrmap e10: x(active, starting, since 0.00351959s) 2024-04-21T08:30:30.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:30:30.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:30:30.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:30:30.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:30:30.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:30:30.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: Manager daemon x is now available 2024-04-21T08:30:30.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:30:30.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:30:30.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:30:30.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:30 smithi094 ceph-mon[27988]: mgrmap e11: x(active, since 1.00598s) 2024-04-21T08:30:31.420 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:30:31.420 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-21T08:30:31.420 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T08:30:31.420 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:30:31.420 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.744+0000 7f424582f700 1 Processor -- start 2024-04-21T08:30:31.421 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.744+0000 7f424582f700 1 -- start start 2024-04-21T08:30:31.421 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.744+0000 7f424582f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424010b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:31.421 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.744+0000 7f424582f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f424007be60 con 0x7f4240108fe0 2024-04-21T08:30:31.421 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.745+0000 7f423effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424010b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:31.421 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.745+0000 7f423effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424010b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34940/0 (socket says 172.21.15.94:34940) 2024-04-21T08:30:31.422 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.745+0000 7f423effd700 1 -- 172.21.15.94:0/2716134406 learned_addr learned my addr 172.21.15.94:0/2716134406 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:31.422 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.745+0000 7f423effd700 1 -- 172.21.15.94:0/2716134406 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424010b910 con 0x7f4240108fe0 2024-04-21T08:30:31.422 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.745+0000 7f423effd700 1 --2- 172.21.15.94:0/2716134406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424010b3d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f42340098a0 tx=0x7f4234009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf9641a74534d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:31.424 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.745+0000 7f423dffb700 1 -- 172.21.15.94:0/2716134406 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4234004160 con 0x7f4240108fe0 2024-04-21T08:30:31.424 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.745+0000 7f423dffb700 1 -- 172.21.15.94:0/2716134406 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4234004300 con 0x7f4240108fe0 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.745+0000 7f423dffb700 1 -- 172.21.15.94:0/2716134406 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4234011a20 con 0x7f4240108fe0 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.746+0000 7f424582f700 1 -- 172.21.15.94:0/2716134406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 msgr2=0x7f424010b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.746+0000 7f424582f700 1 --2- 172.21.15.94:0/2716134406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424010b3d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f42340098a0 tx=0x7f4234009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.746+0000 7f424582f700 1 -- 172.21.15.94:0/2716134406 shutdown_connections 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.746+0000 7f424582f700 1 --2- 172.21.15.94:0/2716134406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424010b3d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.746+0000 7f424582f700 1 -- 172.21.15.94:0/2716134406 >> 172.21.15.94:0/2716134406 conn(0x7f4240102a70 msgr2=0x7f4240104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.746+0000 7f424582f700 1 -- 172.21.15.94:0/2716134406 shutdown_connections 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.746+0000 7f424582f700 1 -- 172.21.15.94:0/2716134406 wait complete. 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.747+0000 7f424582f700 1 Processor -- start 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.747+0000 7f424582f700 1 -- start start 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.747+0000 7f424582f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.747+0000 7f424582f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f424010c8c0 con 0x7f4240108fe0 2024-04-21T08:30:31.425 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.747+0000 7f423effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:31.426 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.747+0000 7f423effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424019c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34944/0 (socket says 172.21.15.94:34944) 2024-04-21T08:30:31.426 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.747+0000 7f423effd700 1 -- 172.21.15.94:0/3037813127 learned_addr learned my addr 172.21.15.94:0/3037813127 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:31.426 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.748+0000 7f423effd700 1 -- 172.21.15.94:0/3037813127 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424019cc00 con 0x7f4240108fe0 2024-04-21T08:30:31.426 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.748+0000 7f423effd700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424019c6c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4234000c00 tx=0x7f4234011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:31.426 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.748+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4234003aa0 con 0x7f4240108fe0 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.748+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f423401b8f0 con 0x7f4240108fe0 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.748+0000 7f424582f700 1 -- 172.21.15.94:0/3037813127 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424019ce90 con 0x7f4240108fe0 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.748+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4234023870 con 0x7f4240108fe0 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.748+0000 7f424582f700 1 -- 172.21.15.94:0/3037813127 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f424019fb30 con 0x7f4240108fe0 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.749+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f4234023b70 con 0x7f4240108fe0 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.749+0000 7f424482d700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 0x7f422803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.749+0000 7f424582f700 1 -- 172.21.15.94:0/3037813127 --> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4240069b90 con 0x7f4228039600 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.749+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f423404f200 con 0x7f4240108fe0 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.749+0000 7f423e7fc700 1 -- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 msgr2=0x7f422803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3877440809 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.749+0000 7f423e7fc700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 0x7f422803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:30:31.427 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.950+0000 7f423e7fc700 1 -- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 msgr2=0x7f422803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3877440809 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:25.950+0000 7f423e7fc700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 0x7f422803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:26.351+0000 7f423e7fc700 1 -- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 msgr2=0x7f422803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3877440809 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:26.351+0000 7f423e7fc700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 0x7f422803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:27.152+0000 7f423e7fc700 1 -- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 msgr2=0x7f422803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3877440809 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:27.152+0000 7f423e7fc700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 0x7f422803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:28.753+0000 7f423e7fc700 1 -- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 msgr2=0x7f422803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3877440809 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:28.753+0000 7f423e7fc700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 0x7f422803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:29.484+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f4234016bd0 con 0x7f4240108fe0 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:29.484+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 msgr2=0x7f422803bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:29.484+0000 7f424482d700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 0x7f422803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.486+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f4234016260 con 0x7f4240108fe0 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.487+0000 7f424482d700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f422803ce90 0x7f422803f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:31.428 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.487+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4234004300 con 0x7f422803ce90 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.487+0000 7f423e7fc700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f422803ce90 0x7f422803f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.488+0000 7f423e7fc700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f422803ce90 0x7f422803f280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4230003a10 tx=0x7f42300092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.488+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f4234004300 con 0x7f422803ce90 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.494+0000 7f424582f700 1 -- 172.21.15.94:0/3037813127 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f4240068a40 con 0x7f422803ce90 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424482d700 1 -- 172.21.15.94:0/3037813127 <== mgr.14152 v2:172.21.15.94:6800/3394521972 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f4240068a40 con 0x7f422803ce90 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 -- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f422803ce90 msgr2=0x7f422803f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f422803ce90 0x7f422803f280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4230003a10 tx=0x7f42300092b0 comp rx=0 tx=0).stop 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 -- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 msgr2=0x7f424019c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424019c6c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4234000c00 tx=0x7f4234011aa0 comp rx=0 tx=0).stop 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 -- 172.21.15.94:0/3037813127 shutdown_connections 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f422803ce90 0x7f422803f280 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4240108fe0 0x7f424019c6c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:31.429 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 --2- 172.21.15.94:0/3037813127 >> [v2:172.21.15.94:6800/3877440809,v1:172.21.15.94:6801/3877440809] conn(0x7f4228039600 0x7f422803bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:31.430 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 -- 172.21.15.94:0/3037813127 >> 172.21.15.94:0/3037813127 conn(0x7f4240102a70 msgr2=0x7f4240103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:31.430 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 -- 172.21.15.94:0/3037813127 shutdown_connections 2024-04-21T08:30:31.430 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:30.495+0000 7f424582f700 1 -- 172.21.15.94:0/3037813127 wait complete. 2024-04-21T08:30:31.430 INFO:teuthology.orchestra.run.smithi094.stdout:mgr epoch 9 is available 2024-04-21T08:30:31.430 INFO:teuthology.orchestra.run.smithi094.stdout:Generating a dashboard self-signed certificate... 2024-04-21T08:30:31.660 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:31 smithi094 ceph-mon[27988]: [21/Apr/2024:08:30:30] ENGINE Bus STARTING 2024-04-21T08:30:31.660 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:31 smithi094 ceph-mon[27988]: [21/Apr/2024:08:30:30] ENGINE Serving on https://172.21.15.94:7150 2024-04-21T08:30:31.660 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:31 smithi094 ceph-mon[27988]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T08:30:31.660 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:31 smithi094 ceph-mon[27988]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T08:30:31.661 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:31 smithi094 ceph-mon[27988]: [21/Apr/2024:08:30:30] ENGINE Serving on http://172.21.15.94:8765 2024-04-21T08:30:31.661 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:31 smithi094 ceph-mon[27988]: [21/Apr/2024:08:30:30] ENGINE Bus STARTED 2024-04-21T08:30:32.739 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-21T08:30:32.739 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.879+0000 7f3db3b6f700 1 Processor -- start 2024-04-21T08:30:32.740 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.880+0000 7f3db3b6f700 1 -- start start 2024-04-21T08:30:32.740 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.880+0000 7f3db3b6f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac10b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:32.740 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.880+0000 7f3db3b6f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac07be60 con 0x7f3dac108f60 2024-04-21T08:30:32.740 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.880+0000 7f3db190b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac10b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:32.740 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.880+0000 7f3db190b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac10b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35002/0 (socket says 172.21.15.94:35002) 2024-04-21T08:30:32.741 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.880+0000 7f3db190b700 1 -- 172.21.15.94:0/4033789278 learned_addr learned my addr 172.21.15.94:0/4033789278 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:32.741 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.880+0000 7f3db190b700 1 -- 172.21.15.94:0/4033789278 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac10b890 con 0x7f3dac108f60 2024-04-21T08:30:32.741 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.880+0000 7f3db190b700 1 --2- 172.21.15.94:0/4033789278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac10b350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3da80098a0 tx=0x7f3da8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5b591e4f02c022 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:32.741 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.881+0000 7f3db0909700 1 -- 172.21.15.94:0/4033789278 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3da8004160 con 0x7f3dac108f60 2024-04-21T08:30:32.741 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.881+0000 7f3db0909700 1 -- 172.21.15.94:0/4033789278 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3da8004300 con 0x7f3dac108f60 2024-04-21T08:30:32.741 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.881+0000 7f3db0909700 1 -- 172.21.15.94:0/4033789278 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3da8011a20 con 0x7f3dac108f60 2024-04-21T08:30:32.741 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.881+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4033789278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 msgr2=0x7f3dac10b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:32.741 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.881+0000 7f3db3b6f700 1 --2- 172.21.15.94:0/4033789278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac10b350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3da80098a0 tx=0x7f3da8009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:32.742 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.881+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4033789278 shutdown_connections 2024-04-21T08:30:32.742 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.881+0000 7f3db3b6f700 1 --2- 172.21.15.94:0/4033789278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac10b350 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:32.742 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.881+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4033789278 >> 172.21.15.94:0/4033789278 conn(0x7f3dac1029b0 msgr2=0x7f3dac104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:32.742 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.882+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4033789278 shutdown_connections 2024-04-21T08:30:32.742 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.882+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4033789278 wait complete. 2024-04-21T08:30:32.742 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.882+0000 7f3db3b6f700 1 Processor -- start 2024-04-21T08:30:32.742 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.882+0000 7f3db3b6f700 1 -- start start 2024-04-21T08:30:32.742 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.882+0000 7f3db3b6f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac19c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:32.743 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.882+0000 7f3db3b6f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac10c840 con 0x7f3dac108f60 2024-04-21T08:30:32.743 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.882+0000 7f3db190b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac19c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:32.743 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.882+0000 7f3db190b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac19c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35014/0 (socket says 172.21.15.94:35014) 2024-04-21T08:30:32.743 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.882+0000 7f3db190b700 1 -- 172.21.15.94:0/4252823591 learned_addr learned my addr 172.21.15.94:0/4252823591 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:32.743 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.883+0000 7f3db190b700 1 -- 172.21.15.94:0/4252823591 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac19c920 con 0x7f3dac108f60 2024-04-21T08:30:32.743 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.883+0000 7f3db190b700 1 --2- 172.21.15.94:0/4252823591 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac19c3e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3da8004660 tx=0x7f3da8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:32.743 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.883+0000 7f3da2ffd700 1 -- 172.21.15.94:0/4252823591 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3da8003aa0 con 0x7f3dac108f60 2024-04-21T08:30:32.743 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.883+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4252823591 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dac19cbb0 con 0x7f3dac108f60 2024-04-21T08:30:32.743 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.883+0000 7f3da2ffd700 1 -- 172.21.15.94:0/4252823591 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3da801b8f0 con 0x7f3dac108f60 2024-04-21T08:30:32.743 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.884+0000 7f3da2ffd700 1 -- 172.21.15.94:0/4252823591 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3da8023900 con 0x7f3dac108f60 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.884+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4252823591 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dac19f8a0 con 0x7f3dac108f60 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.884+0000 7f3da2ffd700 1 -- 172.21.15.94:0/4252823591 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f3da8023b70 con 0x7f3dac108f60 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.884+0000 7f3da2ffd700 1 --2- 172.21.15.94:0/4252823591 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3d980396b0 0x7f3d9803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.884+0000 7f3da2ffd700 1 -- 172.21.15.94:0/4252823591 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3da801baa0 con 0x7f3dac108f60 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.884+0000 7f3db110a700 1 --2- 172.21.15.94:0/4252823591 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3d980396b0 0x7f3d9803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.885+0000 7f3db110a700 1 --2- 172.21.15.94:0/4252823591 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3d980396b0 0x7f3d9803bb70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c006fd0 tx=0x7f3d9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.885+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4252823591 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dac0640a0 con 0x7f3dac108f60 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:31.890+0000 7f3da2ffd700 1 -- 172.21.15.94:0/4252823591 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3da801f030 con 0x7f3dac108f60 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.099+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4252823591 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f3dac103c00 con 0x7f3d980396b0 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.183+0000 7f3da2ffd700 1 -- 172.21.15.94:0/4252823591 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f3dac103c00 con 0x7f3d980396b0 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.185+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4252823591 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3d980396b0 msgr2=0x7f3d9803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.185+0000 7f3db3b6f700 1 --2- 172.21.15.94:0/4252823591 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3d980396b0 0x7f3d9803bb70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c006fd0 tx=0x7f3d9c006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:32.744 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.185+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4252823591 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 msgr2=0x7f3dac19c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:32.745 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.185+0000 7f3db3b6f700 1 --2- 172.21.15.94:0/4252823591 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac19c3e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3da8004660 tx=0x7f3da8011aa0 comp rx=0 tx=0).stop 2024-04-21T08:30:32.745 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.185+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4252823591 shutdown_connections 2024-04-21T08:30:32.745 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.185+0000 7f3db3b6f700 1 --2- 172.21.15.94:0/4252823591 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3d980396b0 0x7f3d9803bb70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:32.745 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.185+0000 7f3db3b6f700 1 --2- 172.21.15.94:0/4252823591 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3dac108f60 0x7f3dac19c3e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:32.745 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.185+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4252823591 >> 172.21.15.94:0/4252823591 conn(0x7f3dac1029b0 msgr2=0x7f3dac1034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:32.745 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.186+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4252823591 shutdown_connections 2024-04-21T08:30:32.745 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:32.186+0000 7f3db3b6f700 1 -- 172.21.15.94:0/4252823591 wait complete. 2024-04-21T08:30:32.745 INFO:teuthology.orchestra.run.smithi094.stdout:Creating initial admin user... 2024-04-21T08:30:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:32 smithi094 ceph-mon[27988]: mgrmap e12: x(active, since 2s) 2024-04-21T08:30:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:33.719 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:33 smithi094 ceph-mon[27988]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:33.721 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:33 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:34.808 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:34 smithi094 ceph-mon[27988]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:34.808 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:34.816 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$r0f4Tto8a5r.c/j9/igXIuoW408tVXwLYTlChZPsqH5sZ3GTAUHga", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713688233, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-21T08:30:34.816 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.178+0000 7fc1ca0ed700 1 Processor -- start 2024-04-21T08:30:34.816 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.178+0000 7fc1ca0ed700 1 -- start start 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.178+0000 7fc1ca0ed700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c4107830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.178+0000 7fc1ca0ed700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c4107e00 con 0x7fc1c4107410 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.179+0000 7fc1c37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c4107830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.179+0000 7fc1c37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c4107830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35030/0 (socket says 172.21.15.94:35030) 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.179+0000 7fc1c37fe700 1 -- 172.21.15.94:0/530350516 learned_addr learned my addr 172.21.15.94:0/530350516 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.179+0000 7fc1c37fe700 1 -- 172.21.15.94:0/530350516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c4108610 con 0x7fc1c4107410 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.179+0000 7fc1c37fe700 1 --2- 172.21.15.94:0/530350516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c4107830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800bb40 tx=0x7fc1b800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16785a7195129202 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.179+0000 7fc1c27fc700 1 -- 172.21.15.94:0/530350516 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1b800dbe0 con 0x7fc1c4107410 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.179+0000 7fc1c27fc700 1 -- 172.21.15.94:0/530350516 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1b80044e0 con 0x7fc1c4107410 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.179+0000 7fc1c27fc700 1 -- 172.21.15.94:0/530350516 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1b8010470 con 0x7fc1c4107410 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.180+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/530350516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 msgr2=0x7fc1c4107830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.180+0000 7fc1ca0ed700 1 --2- 172.21.15.94:0/530350516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c4107830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800bb40 tx=0x7fc1b800be50 comp rx=0 tx=0).stop 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.180+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/530350516 shutdown_connections 2024-04-21T08:30:34.817 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.180+0000 7fc1ca0ed700 1 --2- 172.21.15.94:0/530350516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c4107830 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.180+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/530350516 >> 172.21.15.94:0/530350516 conn(0x7fc1c4102a70 msgr2=0x7fc1c4104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.180+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/530350516 shutdown_connections 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.180+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/530350516 wait complete. 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.181+0000 7fc1ca0ed700 1 Processor -- start 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.181+0000 7fc1ca0ed700 1 -- start start 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.181+0000 7fc1ca0ed700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.181+0000 7fc1ca0ed700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c407afe0 con 0x7fc1c4107410 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.181+0000 7fc1c37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.181+0000 7fc1c37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c419e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35046/0 (socket says 172.21.15.94:35046) 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.181+0000 7fc1c37fe700 1 -- 172.21.15.94:0/2066800345 learned_addr learned my addr 172.21.15.94:0/2066800345 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.181+0000 7fc1c37fe700 1 -- 172.21.15.94:0/2066800345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c419ee50 con 0x7fc1c4107410 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.182+0000 7fc1c37fe700 1 --2- 172.21.15.94:0/2066800345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c419e910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800d650 tx=0x7fc1b8003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.182+0000 7fc1c0ff9700 1 -- 172.21.15.94:0/2066800345 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1b8010770 con 0x7fc1c4107410 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.182+0000 7fc1c0ff9700 1 -- 172.21.15.94:0/2066800345 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1b801d070 con 0x7fc1c4107410 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.182+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/2066800345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c419f0e0 con 0x7fc1c4107410 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.182+0000 7fc1c0ff9700 1 -- 172.21.15.94:0/2066800345 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1b8021bf0 con 0x7fc1c4107410 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.182+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/2066800345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c41a1dd0 con 0x7fc1c4107410 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.183+0000 7fc1c0ff9700 1 -- 172.21.15.94:0/2066800345 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fc1b801f030 con 0x7fc1c4107410 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.183+0000 7fc1c0ff9700 1 --2- 172.21.15.94:0/2066800345 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc1ac0395b0 0x7fc1ac03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.183+0000 7fc1c0ff9700 1 -- 172.21.15.94:0/2066800345 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc1b804db30 con 0x7fc1c4107410 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.183+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/2066800345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c40640a0 con 0x7fc1c4107410 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.183+0000 7fc1c2ffd700 1 --2- 172.21.15.94:0/2066800345 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc1ac0395b0 0x7fc1ac03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.184+0000 7fc1c2ffd700 1 --2- 172.21.15.94:0/2066800345 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc1ac0395b0 0x7fc1ac03ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4006fd0 tx=0x7fc1b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.188+0000 7fc1c0ff9700 1 -- 172.21.15.94:0/2066800345 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc1b8015070 con 0x7fc1c4107410 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.416+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/2066800345 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fc1c419f5c0 con 0x7fc1ac0395b0 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.640+0000 7fc1c0ff9700 1 -- 172.21.15.94:0/2066800345 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fc1c419f5c0 con 0x7fc1ac0395b0 2024-04-21T08:30:34.819 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.642+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/2066800345 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc1ac0395b0 msgr2=0x7fc1ac03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:34.820 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.642+0000 7fc1ca0ed700 1 --2- 172.21.15.94:0/2066800345 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc1ac0395b0 0x7fc1ac03ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4006fd0 tx=0x7fc1b4006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:34.820 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.642+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/2066800345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 msgr2=0x7fc1c419e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:34.820 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.642+0000 7fc1ca0ed700 1 --2- 172.21.15.94:0/2066800345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c419e910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800d650 tx=0x7fc1b8003ff0 comp rx=0 tx=0).stop 2024-04-21T08:30:34.820 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.642+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/2066800345 shutdown_connections 2024-04-21T08:30:34.820 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.642+0000 7fc1ca0ed700 1 --2- 172.21.15.94:0/2066800345 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc1ac0395b0 0x7fc1ac03ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:34.820 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.642+0000 7fc1ca0ed700 1 --2- 172.21.15.94:0/2066800345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1c4107410 0x7fc1c419e910 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:34.820 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.642+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/2066800345 >> 172.21.15.94:0/2066800345 conn(0x7fc1c4102a70 msgr2=0x7fc1c4190b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:34.820 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.642+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/2066800345 shutdown_connections 2024-04-21T08:30:34.820 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:33.642+0000 7fc1ca0ed700 1 -- 172.21.15.94:0/2066800345 wait complete. 2024-04-21T08:30:34.820 INFO:teuthology.orchestra.run.smithi094.stdout:Fetching dashboard port number... 2024-04-21T08:30:35.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:35 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2771388047' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-21T08:30:36.155 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 8443 2024-04-21T08:30:36.155 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.278+0000 7f024ceaf700 1 Processor -- start 2024-04-21T08:30:36.155 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.278+0000 7f024ceaf700 1 -- start start 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.278+0000 7f024ceaf700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f0248108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.278+0000 7f024ceaf700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0248108f00 con 0x7f0248108510 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.278+0000 7f024659c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f0248108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.278+0000 7f024659c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f0248108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35004/0 (socket says 172.21.15.94:35004) 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.278+0000 7f024659c700 1 -- 172.21.15.94:0/4195198894 learned_addr learned my addr 172.21.15.94:0/4195198894 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.278+0000 7f024659c700 1 -- 172.21.15.94:0/4195198894 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02481096c0 con 0x7f0248108510 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024659c700 1 --2- 172.21.15.94:0/4195198894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f0248108930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f023c0098a0 tx=0x7f023c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b494693ba4c2a41e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024559a700 1 -- 172.21.15.94:0/4195198894 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f023c004160 con 0x7f0248108510 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024559a700 1 -- 172.21.15.94:0/4195198894 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f023c004300 con 0x7f0248108510 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024559a700 1 -- 172.21.15.94:0/4195198894 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f023c011a20 con 0x7f0248108510 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024ceaf700 1 -- 172.21.15.94:0/4195198894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 msgr2=0x7f0248108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024ceaf700 1 --2- 172.21.15.94:0/4195198894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f0248108930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f023c0098a0 tx=0x7f023c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024ceaf700 1 -- 172.21.15.94:0/4195198894 shutdown_connections 2024-04-21T08:30:36.156 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024ceaf700 1 --2- 172.21.15.94:0/4195198894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f0248108930 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024ceaf700 1 -- 172.21.15.94:0/4195198894 >> 172.21.15.94:0/4195198894 conn(0x7f0248103b70 msgr2=0x7f0248105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024ceaf700 1 -- 172.21.15.94:0/4195198894 shutdown_connections 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.279+0000 7f024ceaf700 1 -- 172.21.15.94:0/4195198894 wait complete. 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.280+0000 7f024ceaf700 1 Processor -- start 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.280+0000 7f024ceaf700 1 -- start start 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.280+0000 7f024ceaf700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f024819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.280+0000 7f024ceaf700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f024810ca70 con 0x7f0248108510 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.280+0000 7f024659c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f024819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.280+0000 7f024659c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f024819c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35012/0 (socket says 172.21.15.94:35012) 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.280+0000 7f024659c700 1 -- 172.21.15.94:0/2771388047 learned_addr learned my addr 172.21.15.94:0/2771388047 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.281+0000 7f024659c700 1 -- 172.21.15.94:0/2771388047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024819cbe0 con 0x7f0248108510 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.281+0000 7f024659c700 1 --2- 172.21.15.94:0/2771388047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f024819c6a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f023c009520 tx=0x7f023c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.281+0000 7f02377fe700 1 -- 172.21.15.94:0/2771388047 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f023c0039e0 con 0x7f0248108510 2024-04-21T08:30:36.157 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.281+0000 7f024ceaf700 1 -- 172.21.15.94:0/2771388047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f024819ce70 con 0x7f0248108510 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.281+0000 7f02377fe700 1 -- 172.21.15.94:0/2771388047 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f023c01b8f0 con 0x7f0248108510 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.281+0000 7f02377fe700 1 -- 172.21.15.94:0/2771388047 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f023c019770 con 0x7f0248108510 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.281+0000 7f024ceaf700 1 -- 172.21.15.94:0/2771388047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f024819fb60 con 0x7f0248108510 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.282+0000 7f02377fe700 1 -- 172.21.15.94:0/2771388047 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f023c019a20 con 0x7f0248108510 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.282+0000 7f02377fe700 1 --2- 172.21.15.94:0/2771388047 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0230039650 0x7f023003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.282+0000 7f024ceaf700 1 -- 172.21.15.94:0/2771388047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02480640a0 con 0x7f0248108510 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.282+0000 7f02377fe700 1 -- 172.21.15.94:0/2771388047 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f023c04dff0 con 0x7f0248108510 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.282+0000 7f0245d9b700 1 --2- 172.21.15.94:0/2771388047 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0230039650 0x7f023003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.282+0000 7f0245d9b700 1 --2- 172.21.15.94:0/2771388047 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0230039650 0x7f023003bb10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0238006fd0 tx=0x7f0238006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.287+0000 7f02377fe700 1 -- 172.21.15.94:0/2771388047 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f023c01f030 con 0x7f0248108510 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.483+0000 7f024ceaf700 1 -- 172.21.15.94:0/2771388047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f024802df80 con 0x7f0248108510 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.484+0000 7f02377fe700 1 -- 172.21.15.94:0/2771388047 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f023c0172b0 con 0x7f0248108510 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.486+0000 7f024ceaf700 1 -- 172.21.15.94:0/2771388047 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0230039650 msgr2=0x7f023003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:36.158 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.486+0000 7f024ceaf700 1 --2- 172.21.15.94:0/2771388047 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0230039650 0x7f023003bb10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0238006fd0 tx=0x7f0238006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:36.159 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.486+0000 7f024ceaf700 1 -- 172.21.15.94:0/2771388047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 msgr2=0x7f024819c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:36.159 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.486+0000 7f024ceaf700 1 --2- 172.21.15.94:0/2771388047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f024819c6a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f023c009520 tx=0x7f023c011aa0 comp rx=0 tx=0).stop 2024-04-21T08:30:36.159 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.486+0000 7f024ceaf700 1 -- 172.21.15.94:0/2771388047 shutdown_connections 2024-04-21T08:30:36.159 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.486+0000 7f024ceaf700 1 --2- 172.21.15.94:0/2771388047 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0230039650 0x7f023003bb10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:36.159 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.486+0000 7f024ceaf700 1 --2- 172.21.15.94:0/2771388047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0248108510 0x7f024819c6a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:36.159 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.486+0000 7f024ceaf700 1 -- 172.21.15.94:0/2771388047 >> 172.21.15.94:0/2771388047 conn(0x7f0248103b70 msgr2=0x7f0248104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:36.159 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.486+0000 7f024ceaf700 1 -- 172.21.15.94:0/2771388047 shutdown_connections 2024-04-21T08:30:36.159 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:35.486+0000 7f024ceaf700 1 -- 172.21.15.94:0/2771388047 wait complete. 2024-04-21T08:30:36.162 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:30:36.162 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-04-21T08:30:36.169 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:30:36.169 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-21T08:30:36.169 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-04-21T08:30:36.169 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-21T08:30:36.171 INFO:teuthology.orchestra.run.smithi094.stdout:Ceph Dashboard is now available at: 2024-04-21T08:30:36.171 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:36.171 INFO:teuthology.orchestra.run.smithi094.stdout: URL: https://smithi094:8443/ 2024-04-21T08:30:36.171 INFO:teuthology.orchestra.run.smithi094.stdout: User: admin 2024-04-21T08:30:36.171 INFO:teuthology.orchestra.run.smithi094.stdout: Password: wuc6clkx3g 2024-04-21T08:30:36.171 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:36.171 INFO:teuthology.orchestra.run.smithi094.stdout:Saving cluster configuration to /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config directory 2024-04-21T08:30:36.172 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling autotune for osd_memory_target 2024-04-21T08:30:36.569 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:36 smithi094 ceph-mon[27988]: mgrmap e13: x(active, since 6s) 2024-04-21T08:30:37.404 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.569+0000 7fcca745e700 1 Processor -- start 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.569+0000 7fcca745e700 1 -- start start 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.570+0000 7fcca745e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca01077b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.570+0000 7fcca745e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcca0107d80 con 0x7fcca0107390 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.570+0000 7fcca51fa700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca01077b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.570+0000 7fcca51fa700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca01077b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35020/0 (socket says 172.21.15.94:35020) 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.570+0000 7fcca51fa700 1 -- 172.21.15.94:0/1362046209 learned_addr learned my addr 172.21.15.94:0/1362046209 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.570+0000 7fcca51fa700 1 -- 172.21.15.94:0/1362046209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca0108590 con 0x7fcca0107390 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.570+0000 7fcca51fa700 1 --2- 172.21.15.94:0/1362046209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca01077b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009e80 tx=0x7fcc9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11c12dc8fc6ca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.570+0000 7fcc97fff700 1 -- 172.21.15.94:0/1362046209 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc9c017070 con 0x7fcca0107390 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.570+0000 7fcc97fff700 1 -- 172.21.15.94:0/1362046209 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc9c00b8c0 con 0x7fcca0107390 2024-04-21T08:30:37.405 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.570+0000 7fcc97fff700 1 -- 172.21.15.94:0/1362046209 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc9c004490 con 0x7fcca0107390 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.571+0000 7fcca745e700 1 -- 172.21.15.94:0/1362046209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 msgr2=0x7fcca01077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.571+0000 7fcca745e700 1 --2- 172.21.15.94:0/1362046209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca01077b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009e80 tx=0x7fcc9c00b330 comp rx=0 tx=0).stop 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.571+0000 7fcca745e700 1 -- 172.21.15.94:0/1362046209 shutdown_connections 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.571+0000 7fcca745e700 1 --2- 172.21.15.94:0/1362046209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca01077b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.571+0000 7fcca745e700 1 -- 172.21.15.94:0/1362046209 >> 172.21.15.94:0/1362046209 conn(0x7fcca01029b0 msgr2=0x7fcca0104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.571+0000 7fcca745e700 1 -- 172.21.15.94:0/1362046209 shutdown_connections 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.571+0000 7fcca745e700 1 -- 172.21.15.94:0/1362046209 wait complete. 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.572+0000 7fcca745e700 1 Processor -- start 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.572+0000 7fcca745e700 1 -- start start 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.572+0000 7fcca745e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca0192500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.572+0000 7fcca745e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcca010b940 con 0x7fcca0107390 2024-04-21T08:30:37.406 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.572+0000 7fcca51fa700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca0192500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.572+0000 7fcca51fa700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca0192500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35036/0 (socket says 172.21.15.94:35036) 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.572+0000 7fcca51fa700 1 -- 172.21.15.94:0/85717558 learned_addr learned my addr 172.21.15.94:0/85717558 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.572+0000 7fcca51fa700 1 -- 172.21.15.94:0/85717558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca0192a40 con 0x7fcca0107390 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.572+0000 7fcca51fa700 1 --2- 172.21.15.94:0/85717558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca0192500 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009e50 tx=0x7fcc9c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.573+0000 7fcc967fc700 1 -- 172.21.15.94:0/85717558 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc9c020020 con 0x7fcca0107390 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.573+0000 7fcc967fc700 1 -- 172.21.15.94:0/85717558 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc9c003b50 con 0x7fcca0107390 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.573+0000 7fcca745e700 1 -- 172.21.15.94:0/85717558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca0190c30 con 0x7fcca0107390 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.573+0000 7fcc967fc700 1 -- 172.21.15.94:0/85717558 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc9c017040 con 0x7fcca0107390 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.573+0000 7fcca745e700 1 -- 172.21.15.94:0/85717558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca0191170 con 0x7fcca0107390 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.574+0000 7fcca745e700 1 -- 172.21.15.94:0/85717558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca00640a0 con 0x7fcca0107390 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.574+0000 7fcc967fc700 1 -- 172.21.15.94:0/85717558 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fcc9c023020 con 0x7fcca0107390 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.574+0000 7fcc967fc700 1 --2- 172.21.15.94:0/85717558 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcc8c0396b0 0x7fcc8c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.574+0000 7fcc967fc700 1 -- 172.21.15.94:0/85717558 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcc9c00f050 con 0x7fcca0107390 2024-04-21T08:30:37.407 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.574+0000 7fcca49f9700 1 --2- 172.21.15.94:0/85717558 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcc8c0396b0 0x7fcc8c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:37.408 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.574+0000 7fcca49f9700 1 --2- 172.21.15.94:0/85717558 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcc8c0396b0 0x7fcc8c03bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcc90006fd0 tx=0x7fcc90006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:37.408 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.579+0000 7fcc967fc700 1 -- 172.21.15.94:0/85717558 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc9c02a020 con 0x7fcca0107390 2024-04-21T08:30:37.408 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.774+0000 7fcca745e700 1 -- 172.21.15.94:0/85717558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fcca01962e0 con 0x7fcca0107390 2024-04-21T08:30:37.408 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.774+0000 7fcc967fc700 1 -- 172.21.15.94:0/85717558 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fcc9c02a200 con 0x7fcca0107390 2024-04-21T08:30:37.408 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.776+0000 7fcca745e700 1 -- 172.21.15.94:0/85717558 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcc8c0396b0 msgr2=0x7fcc8c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:37.408 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.776+0000 7fcca745e700 1 --2- 172.21.15.94:0/85717558 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcc8c0396b0 0x7fcc8c03bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcc90006fd0 tx=0x7fcc90006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:37.408 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.776+0000 7fcca745e700 1 -- 172.21.15.94:0/85717558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 msgr2=0x7fcca0192500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:37.408 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.776+0000 7fcca745e700 1 --2- 172.21.15.94:0/85717558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca0192500 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009e50 tx=0x7fcc9c00bd50 comp rx=0 tx=0).stop 2024-04-21T08:30:37.409 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.776+0000 7fcca745e700 1 -- 172.21.15.94:0/85717558 shutdown_connections 2024-04-21T08:30:37.409 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.776+0000 7fcca745e700 1 --2- 172.21.15.94:0/85717558 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcc8c0396b0 0x7fcc8c03bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:37.409 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.776+0000 7fcca745e700 1 --2- 172.21.15.94:0/85717558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca0107390 0x7fcca0192500 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:37.409 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.776+0000 7fcca745e700 1 -- 172.21.15.94:0/85717558 >> 172.21.15.94:0/85717558 conn(0x7fcca01029b0 msgr2=0x7fcca01038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:37.409 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.776+0000 7fcca745e700 1 -- 172.21.15.94:0/85717558 shutdown_connections 2024-04-21T08:30:37.409 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:36.776+0000 7fcca745e700 1 -- 172.21.15.94:0/85717558 wait complete. 2024-04-21T08:30:38.734 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.835+0000 7fe88eccd700 1 Processor -- start 2024-04-21T08:30:38.734 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.835+0000 7fe88eccd700 1 -- start start 2024-04-21T08:30:38.734 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.835+0000 7fe88eccd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe888108e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:38.734 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.836+0000 7fe88eccd700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe888109400 con 0x7fe888108a10 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.836+0000 7fe88ca69700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe888108e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.836+0000 7fe88ca69700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe888108e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35040/0 (socket says 172.21.15.94:35040) 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.836+0000 7fe88ca69700 1 -- 172.21.15.94:0/1950473970 learned_addr learned my addr 172.21.15.94:0/1950473970 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.836+0000 7fe88ca69700 1 -- 172.21.15.94:0/1950473970 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8881076c0 con 0x7fe888108a10 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.836+0000 7fe88ca69700 1 --2- 172.21.15.94:0/1950473970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe888108e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00bb40 tx=0x7fe87c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a64a30d4087778a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.836+0000 7fe8877fe700 1 -- 172.21.15.94:0/1950473970 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe87c00dbe0 con 0x7fe888108a10 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.836+0000 7fe8877fe700 1 -- 172.21.15.94:0/1950473970 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe87c0044e0 con 0x7fe888108a10 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.836+0000 7fe8877fe700 1 -- 172.21.15.94:0/1950473970 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe87c010470 con 0x7fe888108a10 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.837+0000 7fe88eccd700 1 -- 172.21.15.94:0/1950473970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 msgr2=0x7fe888108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.837+0000 7fe88eccd700 1 --2- 172.21.15.94:0/1950473970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe888108e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00bb40 tx=0x7fe87c00be50 comp rx=0 tx=0).stop 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.837+0000 7fe88eccd700 1 -- 172.21.15.94:0/1950473970 shutdown_connections 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.837+0000 7fe88eccd700 1 --2- 172.21.15.94:0/1950473970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe888108e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.837+0000 7fe88eccd700 1 -- 172.21.15.94:0/1950473970 >> 172.21.15.94:0/1950473970 conn(0x7fe8881021b0 msgr2=0x7fe8881045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.837+0000 7fe88eccd700 1 -- 172.21.15.94:0/1950473970 shutdown_connections 2024-04-21T08:30:38.735 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.837+0000 7fe88eccd700 1 -- 172.21.15.94:0/1950473970 wait complete. 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.837+0000 7fe88eccd700 1 Processor -- start 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.837+0000 7fe88eccd700 1 -- start start 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe88eccd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe88819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe88eccd700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881065b0 con 0x7fe888108a10 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe88ca69700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe88819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe88ca69700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe88819a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35048/0 (socket says 172.21.15.94:35048) 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe88ca69700 1 -- 172.21.15.94:0/2102005007 learned_addr learned my addr 172.21.15.94:0/2102005007 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe88ca69700 1 -- 172.21.15.94:0/2102005007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88819aa20 con 0x7fe888108a10 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe88ca69700 1 --2- 172.21.15.94:0/2102005007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe88819a4e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00d650 tx=0x7fe87c003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe885ffb700 1 -- 172.21.15.94:0/2102005007 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe87c027070 con 0x7fe888108a10 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe885ffb700 1 -- 172.21.15.94:0/2102005007 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe87c010c20 con 0x7fe888108a10 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe885ffb700 1 -- 172.21.15.94:0/2102005007 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe87c022b10 con 0x7fe888108a10 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe88eccd700 1 -- 172.21.15.94:0/2102005007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88819acb0 con 0x7fe888108a10 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.838+0000 7fe88eccd700 1 -- 172.21.15.94:0/2102005007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88819d9a0 con 0x7fe888108a10 2024-04-21T08:30:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.839+0000 7fe885ffb700 1 -- 172.21.15.94:0/2102005007 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe87c02a070 con 0x7fe888108a10 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.839+0000 7fe88eccd700 1 -- 172.21.15.94:0/2102005007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8880640a0 con 0x7fe888108a10 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.839+0000 7fe885ffb700 1 --2- 172.21.15.94:0/2102005007 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe878039650 0x7fe87803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.839+0000 7fe885ffb700 1 -- 172.21.15.94:0/2102005007 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe87c01f070 con 0x7fe888108a10 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.840+0000 7fe887fff700 1 --2- 172.21.15.94:0/2102005007 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe878039650 0x7fe87803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.840+0000 7fe887fff700 1 --2- 172.21.15.94:0/2102005007 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe878039650 0x7fe87803bb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe874006fd0 tx=0x7fe874006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:37.845+0000 7fe885ffb700 1 -- 172.21.15.94:0/2102005007 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe87c015070 con 0x7fe888108a10 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.156+0000 7fe88eccd700 1 -- 172.21.15.94:0/2102005007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fe888193ae0 con 0x7fe888108a10 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.159+0000 7fe885ffb700 1 -- 172.21.15.94:0/2102005007 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fe888193ae0 con 0x7fe888108a10 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.161+0000 7fe88eccd700 1 -- 172.21.15.94:0/2102005007 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe878039650 msgr2=0x7fe87803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.161+0000 7fe88eccd700 1 --2- 172.21.15.94:0/2102005007 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe878039650 0x7fe87803bb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe874006fd0 tx=0x7fe874006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.161+0000 7fe88eccd700 1 -- 172.21.15.94:0/2102005007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 msgr2=0x7fe88819a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.161+0000 7fe88eccd700 1 --2- 172.21.15.94:0/2102005007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe88819a4e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00d650 tx=0x7fe87c003ff0 comp rx=0 tx=0).stop 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.161+0000 7fe88eccd700 1 -- 172.21.15.94:0/2102005007 shutdown_connections 2024-04-21T08:30:38.737 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.161+0000 7fe88eccd700 1 --2- 172.21.15.94:0/2102005007 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe878039650 0x7fe87803bb10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.161+0000 7fe88eccd700 1 --2- 172.21.15.94:0/2102005007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe888108a10 0x7fe88819a4e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.161+0000 7fe88eccd700 1 -- 172.21.15.94:0/2102005007 >> 172.21.15.94:0/2102005007 conn(0x7fe8881021b0 msgr2=0x7fe888190ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.161+0000 7fe88eccd700 1 -- 172.21.15.94:0/2102005007 shutdown_connections 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-21T08:30:38.161+0000 7fe88eccd700 1 -- 172.21.15.94:0/2102005007 wait complete. 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout:Or, if you are only running a single cluster on this host: 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-21T08:30:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:38.739 INFO:teuthology.orchestra.run.smithi094.stdout: ceph telemetry on 2024-04-21T08:30:38.739 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:38.739 INFO:teuthology.orchestra.run.smithi094.stdout:For more information see: 2024-04-21T08:30:38.739 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:38.739 INFO:teuthology.orchestra.run.smithi094.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-21T08:30:38.739 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:38.739 INFO:teuthology.orchestra.run.smithi094.stdout:Bootstrap complete. 2024-04-21T08:30:38.771 INFO:tasks.cephadm:Fetching config... 2024-04-21T08:30:38.772 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:30:38.772 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-21T08:30:38.787 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-21T08:30:38.787 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:30:38.787 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-21T08:30:38.843 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-21T08:30:38.843 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:30:38.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-21T08:30:38.907 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-21T08:30:38.908 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:30:38.908 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-21T08:30:38.964 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-21T08:30:38.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd5BGgFqwdS0Pe1MHfgnTwKJImIzYikZc7luxu4v6nVx0oRIdb95xeNm9oL4Me5L5VJXTfWm0mUr30zIOS3CR3au6Dwl452WJ2orRxd4lWe3Me4TlQDPw0T5qMFXCCr6NZpgbs1hASg04f+ti1QJLnlpzeSU97YEFKQqucDmf8AlcsdCVCqSCACrbJGA/AYiK3Rf8oKojSnLWd8oL45kk2sWDsyLc0tSLdGn33niSGccwQ8bvzAbVCoRUVktMz8LMOusQNiaBJ8RSAteRoxCpCO3og0uXLPVnoguaY6ZHlZl2To/KaUFKKKhDP/+241D0nFza1Ats29w5mA0y56mWvdT0uKyKYa1Y5R3OgxD4HrNC6L5sq8APd8Oqne/QAVE/TI/74djWOJ31TiUWeyRIPzrEtHs5kPk4SSObucpP5JVA9f7bPh027HwzIsqeXqg77sMuqQkmXopEbZuqD5uWjDY2QlvOf9toNGakGFgc5wSYlBJ1Wrus3hov3uwHkMOE= ceph-2518ed94-ffb9-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:30:39.035 INFO:teuthology.orchestra.run.smithi094.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd5BGgFqwdS0Pe1MHfgnTwKJImIzYikZc7luxu4v6nVx0oRIdb95xeNm9oL4Me5L5VJXTfWm0mUr30zIOS3CR3au6Dwl452WJ2orRxd4lWe3Me4TlQDPw0T5qMFXCCr6NZpgbs1hASg04f+ti1QJLnlpzeSU97YEFKQqucDmf8AlcsdCVCqSCACrbJGA/AYiK3Rf8oKojSnLWd8oL45kk2sWDsyLc0tSLdGn33niSGccwQ8bvzAbVCoRUVktMz8LMOusQNiaBJ8RSAteRoxCpCO3og0uXLPVnoguaY6ZHlZl2To/KaUFKKKhDP/+241D0nFza1Ats29w5mA0y56mWvdT0uKyKYa1Y5R3OgxD4HrNC6L5sq8APd8Oqne/QAVE/TI/74djWOJ31TiUWeyRIPzrEtHs5kPk4SSObucpP5JVA9f7bPh027HwzIsqeXqg77sMuqQkmXopEbZuqD5uWjDY2QlvOf9toNGakGFgc5wSYlBJ1Wrus3hov3uwHkMOE= ceph-2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:30:39.046 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd5BGgFqwdS0Pe1MHfgnTwKJImIzYikZc7luxu4v6nVx0oRIdb95xeNm9oL4Me5L5VJXTfWm0mUr30zIOS3CR3au6Dwl452WJ2orRxd4lWe3Me4TlQDPw0T5qMFXCCr6NZpgbs1hASg04f+ti1QJLnlpzeSU97YEFKQqucDmf8AlcsdCVCqSCACrbJGA/AYiK3Rf8oKojSnLWd8oL45kk2sWDsyLc0tSLdGn33niSGccwQ8bvzAbVCoRUVktMz8LMOusQNiaBJ8RSAteRoxCpCO3og0uXLPVnoguaY6ZHlZl2To/KaUFKKKhDP/+241D0nFza1Ats29w5mA0y56mWvdT0uKyKYa1Y5R3OgxD4HrNC6L5sq8APd8Oqne/QAVE/TI/74djWOJ31TiUWeyRIPzrEtHs5kPk4SSObucpP5JVA9f7bPh027HwzIsqeXqg77sMuqQkmXopEbZuqD5uWjDY2QlvOf9toNGakGFgc5wSYlBJ1Wrus3hov3uwHkMOE= ceph-2518ed94-ffb9-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:30:39.087 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd5BGgFqwdS0Pe1MHfgnTwKJImIzYikZc7luxu4v6nVx0oRIdb95xeNm9oL4Me5L5VJXTfWm0mUr30zIOS3CR3au6Dwl452WJ2orRxd4lWe3Me4TlQDPw0T5qMFXCCr6NZpgbs1hASg04f+ti1QJLnlpzeSU97YEFKQqucDmf8AlcsdCVCqSCACrbJGA/AYiK3Rf8oKojSnLWd8oL45kk2sWDsyLc0tSLdGn33niSGccwQ8bvzAbVCoRUVktMz8LMOusQNiaBJ8RSAteRoxCpCO3og0uXLPVnoguaY6ZHlZl2To/KaUFKKKhDP/+241D0nFza1Ats29w5mA0y56mWvdT0uKyKYa1Y5R3OgxD4HrNC6L5sq8APd8Oqne/QAVE/TI/74djWOJ31TiUWeyRIPzrEtHs5kPk4SSObucpP5JVA9f7bPh027HwzIsqeXqg77sMuqQkmXopEbZuqD5uWjDY2QlvOf9toNGakGFgc5wSYlBJ1Wrus3hov3uwHkMOE= ceph-2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:30:39.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd5BGgFqwdS0Pe1MHfgnTwKJImIzYikZc7luxu4v6nVx0oRIdb95xeNm9oL4Me5L5VJXTfWm0mUr30zIOS3CR3au6Dwl452WJ2orRxd4lWe3Me4TlQDPw0T5qMFXCCr6NZpgbs1hASg04f+ti1QJLnlpzeSU97YEFKQqucDmf8AlcsdCVCqSCACrbJGA/AYiK3Rf8oKojSnLWd8oL45kk2sWDsyLc0tSLdGn33niSGccwQ8bvzAbVCoRUVktMz8LMOusQNiaBJ8RSAteRoxCpCO3og0uXLPVnoguaY6ZHlZl2To/KaUFKKKhDP/+241D0nFza1Ats29w5mA0y56mWvdT0uKyKYa1Y5R3OgxD4HrNC6L5sq8APd8Oqne/QAVE/TI/74djWOJ31TiUWeyRIPzrEtHs5kPk4SSObucpP5JVA9f7bPh027HwzIsqeXqg77sMuqQkmXopEbZuqD5uWjDY2QlvOf9toNGakGFgc5wSYlBJ1Wrus3hov3uwHkMOE= ceph-2518ed94-ffb9-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:30:39.140 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd5BGgFqwdS0Pe1MHfgnTwKJImIzYikZc7luxu4v6nVx0oRIdb95xeNm9oL4Me5L5VJXTfWm0mUr30zIOS3CR3au6Dwl452WJ2orRxd4lWe3Me4TlQDPw0T5qMFXCCr6NZpgbs1hASg04f+ti1QJLnlpzeSU97YEFKQqucDmf8AlcsdCVCqSCACrbJGA/AYiK3Rf8oKojSnLWd8oL45kk2sWDsyLc0tSLdGn33niSGccwQ8bvzAbVCoRUVktMz8LMOusQNiaBJ8RSAteRoxCpCO3og0uXLPVnoguaY6ZHlZl2To/KaUFKKKhDP/+241D0nFza1Ats29w5mA0y56mWvdT0uKyKYa1Y5R3OgxD4HrNC6L5sq8APd8Oqne/QAVE/TI/74djWOJ31TiUWeyRIPzrEtHs5kPk4SSObucpP5JVA9f7bPh027HwzIsqeXqg77sMuqQkmXopEbZuqD5uWjDY2QlvOf9toNGakGFgc5wSYlBJ1Wrus3hov3uwHkMOE= ceph-2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:30:39.152 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-21T08:30:39.388 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:30:39.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:39 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2102005007' entity='client.admin' 2024-04-21T08:30:41.908 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:41.908 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:41.909 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi094", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:30:41.909 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:41.909 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:30:41.909 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.017+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2625172405 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5cc100790 msgr2=0x7fe5cc100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.017+0000 7fe5d0b46700 1 --2- 172.21.15.94:0/2625172405 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5cc100790 0x7fe5cc100bb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009910 tx=0x7fe5c0009c20 comp rx=0 tx=0).stop 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.017+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2625172405 shutdown_connections 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.017+0000 7fe5d0b46700 1 --2- 172.21.15.94:0/2625172405 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5cc100790 0x7fe5cc100bb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.017+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2625172405 >> 172.21.15.94:0/2625172405 conn(0x7fe5cc0fc0f0 msgr2=0x7fe5cc0fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.018+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2625172405 shutdown_connections 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.018+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2625172405 wait complete. 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.018+0000 7fe5d0b46700 1 Processor -- start 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.018+0000 7fe5d0b46700 1 -- start start 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.018+0000 7fe5d0b46700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5cc100790 0x7fe5cc197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5d0b46700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc108410 con 0x7fe5cc100790 2024-04-21T08:30:42.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5ca59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5cc100790 0x7fe5cc197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5ca59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5cc100790 0x7fe5cc197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35076/0 (socket says 172.21.15.94:35076) 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5ca59c700 1 -- 172.21.15.94:0/2832661384 learned_addr learned my addr 172.21.15.94:0/2832661384 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5ca59c700 1 -- 172.21.15.94:0/2832661384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc198000 con 0x7fe5cc100790 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5ca59c700 1 --2- 172.21.15.94:0/2832661384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5cc100790 0x7fe5cc197ac0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0005800 tx=0x7fe5c0009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5bb7fe700 1 -- 172.21.15.94:0/2832661384 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5c0027070 con 0x7fe5cc100790 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5bb7fe700 1 -- 172.21.15.94:0/2832661384 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5c000be10 con 0x7fe5cc100790 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2832661384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5cc198290 con 0x7fe5cc100790 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5bb7fe700 1 -- 172.21.15.94:0/2832661384 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5c000f460 con 0x7fe5cc100790 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.019+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2832661384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5cc19af80 con 0x7fe5cc100790 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.020+0000 7fe5bb7fe700 1 -- 172.21.15.94:0/2832661384 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe5c002a070 con 0x7fe5cc100790 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.020+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2832661384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5cc108b90 con 0x7fe5cc100790 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.020+0000 7fe5bb7fe700 1 --2- 172.21.15.94:0/2832661384 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5b4039650 0x7fe5b403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.020+0000 7fe5bb7fe700 1 -- 172.21.15.94:0/2832661384 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe5c001e030 con 0x7fe5cc100790 2024-04-21T08:30:42.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.020+0000 7fe5c9d9b700 1 --2- 172.21.15.94:0/2832661384 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5b4039650 0x7fe5b403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:42.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.021+0000 7fe5c9d9b700 1 --2- 172.21.15.94:0/2832661384 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5b4039650 0x7fe5b403bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc006fd0 tx=0x7fe5bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:42.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.025+0000 7fe5bb7fe700 1 -- 172.21.15.94:0/2832661384 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe5c0013070 con 0x7fe5cc100790 2024-04-21T08:30:42.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.232+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2832661384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fe5cc02e090 con 0x7fe5cc100790 2024-04-21T08:30:42.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.235+0000 7fe5bb7fe700 1 -- 172.21.15.94:0/2832661384 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fe5c0031070 con 0x7fe5cc100790 2024-04-21T08:30:42.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.237+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2832661384 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5b4039650 msgr2=0x7fe5b403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:42.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.237+0000 7fe5d0b46700 1 --2- 172.21.15.94:0/2832661384 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5b4039650 0x7fe5b403bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc006fd0 tx=0x7fe5bc006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:42.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.237+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2832661384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5cc100790 msgr2=0x7fe5cc197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:42.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.237+0000 7fe5d0b46700 1 --2- 172.21.15.94:0/2832661384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5cc100790 0x7fe5cc197ac0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0005800 tx=0x7fe5c0009f60 comp rx=0 tx=0).stop 2024-04-21T08:30:42.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.237+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2832661384 shutdown_connections 2024-04-21T08:30:42.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.237+0000 7fe5d0b46700 1 --2- 172.21.15.94:0/2832661384 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5b4039650 0x7fe5b403bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:42.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.237+0000 7fe5d0b46700 1 --2- 172.21.15.94:0/2832661384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5cc100790 0x7fe5cc197ac0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:42.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.237+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2832661384 >> 172.21.15.94:0/2832661384 conn(0x7fe5cc0fc0f0 msgr2=0x7fe5cc0fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:42.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.237+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2832661384 shutdown_connections 2024-04-21T08:30:42.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:42.237+0000 7fe5d0b46700 1 -- 172.21.15.94:0/2832661384 wait complete. 2024-04-21T08:30:43.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:43 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2832661384' entity='client.admin' 2024-04-21T08:30:43.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:43 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:30:43.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:43 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:43.382 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-21T08:30:43.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-21T08:30:43.617 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:30:45.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.007+0000 7fa0c1cad700 1 -- 172.21.15.94:0/2230261038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0bc102f50 msgr2=0x7fa0bc105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:45.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.007+0000 7fa0c1cad700 1 --2- 172.21.15.94:0/2230261038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0bc102f50 0x7fa0bc105340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009910 tx=0x7fa0ac009c20 comp rx=0 tx=0).stop 2024-04-21T08:30:45.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.007+0000 7fa0c1cad700 1 -- 172.21.15.94:0/2230261038 shutdown_connections 2024-04-21T08:30:45.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.007+0000 7fa0c1cad700 1 --2- 172.21.15.94:0/2230261038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0bc102f50 0x7fa0bc105340 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:45.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.007+0000 7fa0c1cad700 1 -- 172.21.15.94:0/2230261038 >> 172.21.15.94:0/2230261038 conn(0x7fa0bc0fc980 msgr2=0x7fa0bc0feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:45.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.008+0000 7fa0c1cad700 1 -- 172.21.15.94:0/2230261038 shutdown_connections 2024-04-21T08:30:45.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.008+0000 7fa0c1cad700 1 -- 172.21.15.94:0/2230261038 wait complete. 2024-04-21T08:30:45.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.008+0000 7fa0c1cad700 1 Processor -- start 2024-04-21T08:30:45.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.008+0000 7fa0c1cad700 1 -- start start 2024-04-21T08:30:45.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0c1cad700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0bc102f50 0x7fa0bc197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:45.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0c1cad700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc1083a0 con 0x7fa0bc102f50 2024-04-21T08:30:45.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0bb7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0bc102f50 0x7fa0bc197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:45.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0bb7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0bc102f50 0x7fa0bc197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48836/0 (socket says 172.21.15.94:48836) 2024-04-21T08:30:45.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0bb7fe700 1 -- 172.21.15.94:0/1853148280 learned_addr learned my addr 172.21.15.94:0/1853148280 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:45.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0bb7fe700 1 -- 172.21.15.94:0/1853148280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0bc198040 con 0x7fa0bc102f50 2024-04-21T08:30:45.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0bb7fe700 1 --2- 172.21.15.94:0/1853148280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0bc102f50 0x7fa0bc197b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac005b40 tx=0x7fa0ac004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:45.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0b8ff9700 1 -- 172.21.15.94:0/1853148280 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0ac027070 con 0x7fa0bc102f50 2024-04-21T08:30:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0b8ff9700 1 -- 172.21.15.94:0/1853148280 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0ac00be10 con 0x7fa0bc102f50 2024-04-21T08:30:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0b8ff9700 1 -- 172.21.15.94:0/1853148280 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0ac022d80 con 0x7fa0bc102f50 2024-04-21T08:30:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0c1cad700 1 -- 172.21.15.94:0/1853148280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0bc198330 con 0x7fa0bc102f50 2024-04-21T08:30:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.009+0000 7fa0c1cad700 1 -- 172.21.15.94:0/1853148280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0bc19afc0 con 0x7fa0bc102f50 2024-04-21T08:30:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.010+0000 7fa0b8ff9700 1 -- 172.21.15.94:0/1853148280 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa0ac02a070 con 0x7fa0bc102f50 2024-04-21T08:30:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.010+0000 7fa0b8ff9700 1 --2- 172.21.15.94:0/1853148280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0a4039650 0x7fa0a403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.010+0000 7fa0c1cad700 1 -- 172.21.15.94:0/1853148280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0bc108b20 con 0x7fa0bc102f50 2024-04-21T08:30:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.010+0000 7fa0b8ff9700 1 -- 172.21.15.94:0/1853148280 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa0ac01e030 con 0x7fa0bc102f50 2024-04-21T08:30:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.010+0000 7fa0baffd700 1 --2- 172.21.15.94:0/1853148280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0a4039650 0x7fa0a403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:45.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.011+0000 7fa0baffd700 1 --2- 172.21.15.94:0/1853148280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0a4039650 0x7fa0a403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0006fd0 tx=0x7fa0b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:45.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.015+0000 7fa0b8ff9700 1 -- 172.21.15.94:0/1853148280 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0ac013070 con 0x7fa0bc102f50 2024-04-21T08:30:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.216+0000 7fa0c1cad700 1 -- 172.21.15.94:0/1853148280 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fa0bc02e120 con 0x7fa0a4039650 2024-04-21T08:30:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.219+0000 7fa0b8ff9700 1 -- 172.21.15.94:0/1853148280 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa0bc02e120 con 0x7fa0a4039650 2024-04-21T08:30:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.222+0000 7fa0c1cad700 1 -- 172.21.15.94:0/1853148280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0a4039650 msgr2=0x7fa0a403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.222+0000 7fa0c1cad700 1 --2- 172.21.15.94:0/1853148280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0a4039650 0x7fa0a403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0006fd0 tx=0x7fa0b0006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.222+0000 7fa0c1cad700 1 -- 172.21.15.94:0/1853148280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0bc102f50 msgr2=0x7fa0bc197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.222+0000 7fa0c1cad700 1 --2- 172.21.15.94:0/1853148280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0bc102f50 0x7fa0bc197b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac005b40 tx=0x7fa0ac004f80 comp rx=0 tx=0).stop 2024-04-21T08:30:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.222+0000 7fa0c1cad700 1 -- 172.21.15.94:0/1853148280 shutdown_connections 2024-04-21T08:30:45.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.222+0000 7fa0c1cad700 1 --2- 172.21.15.94:0/1853148280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0a4039650 0x7fa0a403bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:45.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.222+0000 7fa0c1cad700 1 --2- 172.21.15.94:0/1853148280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0bc102f50 0x7fa0bc197b00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:45.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.222+0000 7fa0c1cad700 1 -- 172.21.15.94:0/1853148280 >> 172.21.15.94:0/1853148280 conn(0x7fa0bc0fc980 msgr2=0x7fa0bc0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:45.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.222+0000 7fa0c1cad700 1 -- 172.21.15.94:0/1853148280 shutdown_connections 2024-04-21T08:30:45.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:45.222+0000 7fa0c1cad700 1 -- 172.21.15.94:0/1853148280 wait complete. 2024-04-21T08:30:45.997 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi111 2024-04-21T08:30:45.997 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:30:45.997 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.conf 2024-04-21T08:30:46.016 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:30:46.016 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:30:46.074 INFO:tasks.cephadm:Adding host smithi111 to orchestrator... 2024-04-21T08:30:46.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch host add smithi111 2024-04-21T08:30:46.309 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:30:46.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:46 smithi094 ceph-mon[27988]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:46.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:46.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:30:46.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:30:46.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:30:46.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:46 smithi094 ceph-mon[27988]: Updating smithi094:/etc/ceph/ceph.conf 2024-04-21T08:30:46.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:46 smithi094 ceph-mon[27988]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:30:46.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:46.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:46.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:47.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:47 smithi094 ceph-mon[27988]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:30:47.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:47 smithi094 ceph-mon[27988]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:30:47.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.729+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3002335090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a9c074de0 msgr2=0x7f4a9c0751c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:47.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.729+0000 7f4aa3e33700 1 --2- 172.21.15.94:0/3002335090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a9c074de0 0x7f4a9c0751c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400b0a0 tx=0x7f4a9400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:30:47.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3002335090 shutdown_connections 2024-04-21T08:30:47.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa3e33700 1 --2- 172.21.15.94:0/3002335090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a9c074de0 0x7f4a9c0751c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:47.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3002335090 >> 172.21.15.94:0/3002335090 conn(0x7f4a9c0fdf70 msgr2=0x7f4a9c100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:47.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3002335090 shutdown_connections 2024-04-21T08:30:47.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3002335090 wait complete. 2024-04-21T08:30:47.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa3e33700 1 Processor -- start 2024-04-21T08:30:47.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa3e33700 1 -- start start 2024-04-21T08:30:47.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa3e33700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a9c074de0 0x7f4a9c19e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa3e33700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c112fd0 con 0x7f4a9c074de0 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa1bcf700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a9c074de0 0x7f4a9c19e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.730+0000 7f4aa1bcf700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a9c074de0 0x7f4a9c19e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48850/0 (socket says 172.21.15.94:48850) 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.731+0000 7f4aa1bcf700 1 -- 172.21.15.94:0/3089003984 learned_addr learned my addr 172.21.15.94:0/3089003984 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.731+0000 7f4aa1bcf700 1 -- 172.21.15.94:0/3089003984 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c19ea00 con 0x7f4a9c074de0 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.731+0000 7f4aa1bcf700 1 --2- 172.21.15.94:0/3089003984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a9c074de0 0x7f4a9c19e4c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4a94015040 tx=0x7f4a94009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.731+0000 7f4a92ffd700 1 -- 172.21.15.94:0/3089003984 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a94021020 con 0x7f4a9c074de0 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.731+0000 7f4a92ffd700 1 -- 172.21.15.94:0/3089003984 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a94004000 con 0x7f4a9c074de0 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.731+0000 7f4a92ffd700 1 -- 172.21.15.94:0/3089003984 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a94017040 con 0x7f4a9c074de0 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.731+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3089003984 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9c19ec90 con 0x7f4a9c074de0 2024-04-21T08:30:47.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.731+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3089003984 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a9c1a2d90 con 0x7f4a9c074de0 2024-04-21T08:30:47.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.732+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3089003984 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a9c076520 con 0x7f4a9c074de0 2024-04-21T08:30:47.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.732+0000 7f4a92ffd700 1 -- 172.21.15.94:0/3089003984 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4a94024020 con 0x7f4a9c074de0 2024-04-21T08:30:47.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.732+0000 7f4a92ffd700 1 --2- 172.21.15.94:0/3089003984 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a880396b0 0x7f4a8803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:47.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.732+0000 7f4a92ffd700 1 -- 172.21.15.94:0/3089003984 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4a9400f050 con 0x7f4a9c074de0 2024-04-21T08:30:47.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.732+0000 7f4aa13ce700 1 --2- 172.21.15.94:0/3089003984 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a880396b0 0x7f4a8803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:47.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.733+0000 7f4aa13ce700 1 --2- 172.21.15.94:0/3089003984 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a880396b0 0x7f4a8803bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006fd0 tx=0x7f4a8c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:47.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.737+0000 7f4a92ffd700 1 -- 172.21.15.94:0/3089003984 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a9402b020 con 0x7f4a9c074de0 2024-04-21T08:30:47.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:47.937+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3089003984 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}) -- 0x7f4a9c19f820 con 0x7f4a880396b0 2024-04-21T08:30:49.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:48 smithi094 ceph-mon[27988]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:49.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:48 smithi094 ceph-mon[27988]: Deploying cephadm binary to smithi111 2024-04-21T08:30:50.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:50 smithi094 ceph-mon[27988]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:30:52.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.039+0000 7f4a92ffd700 1 -- 172.21.15.94:0/3089003984 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f4a9c19f820 con 0x7f4a880396b0 2024-04-21T08:30:52.039 INFO:teuthology.orchestra.run.smithi094.stdout:Added host 'smithi111' with addr '172.21.15.111' 2024-04-21T08:30:52.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.041+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3089003984 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a880396b0 msgr2=0x7f4a8803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:52.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.041+0000 7f4aa3e33700 1 --2- 172.21.15.94:0/3089003984 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a880396b0 0x7f4a8803bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006fd0 tx=0x7f4a8c006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:52.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.041+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3089003984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a9c074de0 msgr2=0x7f4a9c19e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:52.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.041+0000 7f4aa3e33700 1 --2- 172.21.15.94:0/3089003984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a9c074de0 0x7f4a9c19e4c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4a94015040 tx=0x7f4a94009800 comp rx=0 tx=0).stop 2024-04-21T08:30:52.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.041+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3089003984 shutdown_connections 2024-04-21T08:30:52.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.041+0000 7f4aa3e33700 1 --2- 172.21.15.94:0/3089003984 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a880396b0 0x7f4a8803bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:52.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.041+0000 7f4aa3e33700 1 --2- 172.21.15.94:0/3089003984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a9c074de0 0x7f4a9c19e4c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:52.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.041+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3089003984 >> 172.21.15.94:0/3089003984 conn(0x7f4a9c0fdf70 msgr2=0x7f4a9c0ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:52.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.041+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3089003984 shutdown_connections 2024-04-21T08:30:52.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:52.041+0000 7f4aa3e33700 1 -- 172.21.15.94:0/3089003984 wait complete. 2024-04-21T08:30:52.679 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:52 smithi094 ceph-mon[27988]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:30:52.679 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:52 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:52.680 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:52 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:30:52.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T08:30:52.945 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:30:53.751 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:53 smithi094 ceph-mon[27988]: Added host smithi111 2024-04-21T08:30:53.751 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:53.751 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:54.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.490+0000 7f8d28b02700 1 -- 172.21.15.94:0/805478292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d24106330 msgr2=0x7f8d24106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:54.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.490+0000 7f8d28b02700 1 --2- 172.21.15.94:0/805478292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d24106330 0x7f8d24106710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009910 tx=0x7f8d0c009c20 comp rx=0 tx=0).stop 2024-04-21T08:30:54.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.490+0000 7f8d28b02700 1 -- 172.21.15.94:0/805478292 shutdown_connections 2024-04-21T08:30:54.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.490+0000 7f8d28b02700 1 --2- 172.21.15.94:0/805478292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d24106330 0x7f8d24106710 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:54.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.490+0000 7f8d28b02700 1 -- 172.21.15.94:0/805478292 >> 172.21.15.94:0/805478292 conn(0x7f8d24077ac0 msgr2=0x7f8d24077ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:54.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.490+0000 7f8d28b02700 1 -- 172.21.15.94:0/805478292 shutdown_connections 2024-04-21T08:30:54.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.490+0000 7f8d28b02700 1 -- 172.21.15.94:0/805478292 wait complete. 2024-04-21T08:30:54.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.491+0000 7f8d28b02700 1 Processor -- start 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.491+0000 7f8d28b02700 1 -- start start 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.491+0000 7f8d28b02700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d24106330 0x7f8d24197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.491+0000 7f8d28b02700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2410b960 con 0x7f8d24106330 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.491+0000 7f8d2259c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d24106330 0x7f8d24197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.491+0000 7f8d2259c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d24106330 0x7f8d24197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50800/0 (socket says 172.21.15.94:50800) 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.491+0000 7f8d2259c700 1 -- 172.21.15.94:0/1796486930 learned_addr learned my addr 172.21.15.94:0/1796486930 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.491+0000 7f8d2259c700 1 -- 172.21.15.94:0/1796486930 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d24198410 con 0x7f8d24106330 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.491+0000 7f8d2259c700 1 --2- 172.21.15.94:0/1796486930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d24106330 0x7f8d24197ed0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0056d0 tx=0x7f8d0c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.492+0000 7f8d1b7fe700 1 -- 172.21.15.94:0/1796486930 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d0c026070 con 0x7f8d24106330 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.492+0000 7f8d1b7fe700 1 -- 172.21.15.94:0/1796486930 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d0c00bc00 con 0x7f8d24106330 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.492+0000 7f8d28b02700 1 -- 172.21.15.94:0/1796486930 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d241986a0 con 0x7f8d24106330 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.492+0000 7f8d1b7fe700 1 -- 172.21.15.94:0/1796486930 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d0c003920 con 0x7f8d24106330 2024-04-21T08:30:54.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.492+0000 7f8d28b02700 1 -- 172.21.15.94:0/1796486930 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d24198b00 con 0x7f8d24106330 2024-04-21T08:30:54.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.493+0000 7f8d1b7fe700 1 -- 172.21.15.94:0/1796486930 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8d0c029070 con 0x7f8d24106330 2024-04-21T08:30:54.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.493+0000 7f8d28b02700 1 -- 172.21.15.94:0/1796486930 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d24102af0 con 0x7f8d24106330 2024-04-21T08:30:54.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.493+0000 7f8d1b7fe700 1 --2- 172.21.15.94:0/1796486930 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8d10039650 0x7f8d1003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:54.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.493+0000 7f8d1b7fe700 1 -- 172.21.15.94:0/1796486930 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8d0c01e070 con 0x7f8d24106330 2024-04-21T08:30:54.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.493+0000 7f8d21d9b700 1 --2- 172.21.15.94:0/1796486930 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8d10039650 0x7f8d1003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:54.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.493+0000 7f8d21d9b700 1 --2- 172.21.15.94:0/1796486930 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8d10039650 0x7f8d1003bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d14006fd0 tx=0x7f8d14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:54.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.499+0000 7f8d1b7fe700 1 -- 172.21.15.94:0/1796486930 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d0c013070 con 0x7f8d24106330 2024-04-21T08:30:54.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.699+0000 7f8d28b02700 1 -- 172.21.15.94:0/1796486930 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8d241992a0 con 0x7f8d10039650 2024-04-21T08:30:54.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.700+0000 7f8d1b7fe700 1 -- 172.21.15.94:0/1796486930 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f8d241992a0 con 0x7f8d10039650 2024-04-21T08:30:54.700 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:30:54.700 INFO:teuthology.orchestra.run.smithi094.stdout:[{"addr": "172.21.15.94", "hostname": "smithi094", "labels": [], "status": ""}, {"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}] 2024-04-21T08:30:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.702+0000 7f8d28b02700 1 -- 172.21.15.94:0/1796486930 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8d10039650 msgr2=0x7f8d1003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.702+0000 7f8d28b02700 1 --2- 172.21.15.94:0/1796486930 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8d10039650 0x7f8d1003bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d14006fd0 tx=0x7f8d14006e40 comp rx=0 tx=0).stop 2024-04-21T08:30:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.702+0000 7f8d28b02700 1 -- 172.21.15.94:0/1796486930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d24106330 msgr2=0x7f8d24197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.702+0000 7f8d28b02700 1 --2- 172.21.15.94:0/1796486930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d24106330 0x7f8d24197ed0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0056d0 tx=0x7f8d0c0057b0 comp rx=0 tx=0).stop 2024-04-21T08:30:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.703+0000 7f8d28b02700 1 -- 172.21.15.94:0/1796486930 shutdown_connections 2024-04-21T08:30:54.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.703+0000 7f8d28b02700 1 --2- 172.21.15.94:0/1796486930 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8d10039650 0x7f8d1003bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:54.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.703+0000 7f8d28b02700 1 --2- 172.21.15.94:0/1796486930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d24106330 0x7f8d24197ed0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:54.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.703+0000 7f8d28b02700 1 -- 172.21.15.94:0/1796486930 >> 172.21.15.94:0/1796486930 conn(0x7f8d24077ac0 msgr2=0x7f8d240fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:54.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.703+0000 7f8d28b02700 1 -- 172.21.15.94:0/1796486930 shutdown_connections 2024-04-21T08:30:54.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:54.703+0000 7f8d28b02700 1 -- 172.21.15.94:0/1796486930 wait complete. 2024-04-21T08:30:54.713 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:54 smithi094 ceph-mon[27988]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:30:55.293 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-04-21T08:30:55.293 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:30:55.293 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-04-21T08:30:55.312 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:30:55.312 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:30:55.370 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-04-21T08:30:55.370 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch host add smithi173 2024-04-21T08:30:55.602 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:30:56.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:55 smithi094 ceph-mon[27988]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:30:56.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:55 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:56.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:55 smithi094 ceph-mon[27988]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:30:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.096+0000 7f1905c85700 1 -- 172.21.15.94:0/3027061050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19001084e0 msgr2=0x7f19001088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:30:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.096+0000 7f1905c85700 1 --2- 172.21.15.94:0/3027061050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19001084e0 0x7f19001088c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18f4009910 tx=0x7f18f4009c20 comp rx=0 tx=0).stop 2024-04-21T08:30:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.096+0000 7f1905c85700 1 -- 172.21.15.94:0/3027061050 shutdown_connections 2024-04-21T08:30:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.096+0000 7f1905c85700 1 --2- 172.21.15.94:0/3027061050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19001084e0 0x7f19001088c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:30:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.096+0000 7f1905c85700 1 -- 172.21.15.94:0/3027061050 >> 172.21.15.94:0/3027061050 conn(0x7f19000fdf80 msgr2=0x7f19001003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:30:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.096+0000 7f1905c85700 1 -- 172.21.15.94:0/3027061050 shutdown_connections 2024-04-21T08:30:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.096+0000 7f1905c85700 1 -- 172.21.15.94:0/3027061050 wait complete. 2024-04-21T08:30:57.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.096+0000 7f1905c85700 1 Processor -- start 2024-04-21T08:30:57.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f1905c85700 1 -- start start 2024-04-21T08:30:57.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f1905c85700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19001084e0 0x7f1900197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f1905c85700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f190010c940 con 0x7f19001084e0 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f18ff7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19001084e0 0x7f1900197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f18ff7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19001084e0 0x7f1900197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50830/0 (socket says 172.21.15.94:50830) 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f18ff7fe700 1 -- 172.21.15.94:0/1048355415 learned_addr learned my addr 172.21.15.94:0/1048355415 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f18ff7fe700 1 -- 172.21.15.94:0/1048355415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19001983f0 con 0x7f19001084e0 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f18ff7fe700 1 --2- 172.21.15.94:0/1048355415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19001084e0 0x7f1900197eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f18f40056d0 tx=0x7f18f40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f18fcff9700 1 -- 172.21.15.94:0/1048355415 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18f4026070 con 0x7f19001084e0 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f1905c85700 1 -- 172.21.15.94:0/1048355415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19001986e0 con 0x7f19001084e0 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f18fcff9700 1 -- 172.21.15.94:0/1048355415 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18f400bba0 con 0x7f19001084e0 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.097+0000 7f18fcff9700 1 -- 172.21.15.94:0/1048355415 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18f4003920 con 0x7f19001084e0 2024-04-21T08:30:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.098+0000 7f1905c85700 1 -- 172.21.15.94:0/1048355415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f190019c780 con 0x7f19001084e0 2024-04-21T08:30:57.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.099+0000 7f18fcff9700 1 -- 172.21.15.94:0/1048355415 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f18f4029070 con 0x7f19001084e0 2024-04-21T08:30:57.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.099+0000 7f1905c85700 1 -- 172.21.15.94:0/1048355415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1900103b90 con 0x7f19001084e0 2024-04-21T08:30:57.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.099+0000 7f18fcff9700 1 --2- 172.21.15.94:0/1048355415 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f18e80396b0 0x7f18e803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:30:57.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.099+0000 7f18fcff9700 1 -- 172.21.15.94:0/1048355415 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f18f401e070 con 0x7f19001084e0 2024-04-21T08:30:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.099+0000 7f18feffd700 1 --2- 172.21.15.94:0/1048355415 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f18e80396b0 0x7f18e803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:30:57.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.099+0000 7f18feffd700 1 --2- 172.21.15.94:0/1048355415 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f18e80396b0 0x7f18e803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f18f0006fd0 tx=0x7f18f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:30:57.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.104+0000 7f18fcff9700 1 -- 172.21.15.94:0/1048355415 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18f4013070 con 0x7f19001084e0 2024-04-21T08:30:57.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:30:57.306+0000 7f1905c85700 1 -- 172.21.15.94:0/1048355415 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) -- 0x7f1900199210 con 0x7f18e80396b0 2024-04-21T08:30:59.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:58 smithi094 ceph-mon[27988]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:30:59.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:58 smithi094 ceph-mon[27988]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:30:59.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:59.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:59.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:59.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:30:59.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:30:59.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:30:59.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:59 smithi094 ceph-mon[27988]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T08:31:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:59 smithi094 ceph-mon[27988]: Deploying cephadm binary to smithi173 2024-04-21T08:31:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:59 smithi094 ceph-mon[27988]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:59 smithi094 ceph-mon[27988]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:31:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:30:59 smithi094 ceph-mon[27988]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:31:01.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:00 smithi094 ceph-mon[27988]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:01.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.501+0000 7f18fcff9700 1 -- 172.21.15.94:0/1048355415 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f1900199210 con 0x7f18e80396b0 2024-04-21T08:31:01.499 INFO:teuthology.orchestra.run.smithi094.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-04-21T08:31:01.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.503+0000 7f1905c85700 1 -- 172.21.15.94:0/1048355415 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f18e80396b0 msgr2=0x7f18e803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:01.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.503+0000 7f1905c85700 1 --2- 172.21.15.94:0/1048355415 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f18e80396b0 0x7f18e803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f18f0006fd0 tx=0x7f18f0006e40 comp rx=0 tx=0).stop 2024-04-21T08:31:01.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.503+0000 7f1905c85700 1 -- 172.21.15.94:0/1048355415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19001084e0 msgr2=0x7f1900197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:01.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.503+0000 7f1905c85700 1 --2- 172.21.15.94:0/1048355415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19001084e0 0x7f1900197eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f18f40056d0 tx=0x7f18f40057b0 comp rx=0 tx=0).stop 2024-04-21T08:31:01.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.503+0000 7f1905c85700 1 -- 172.21.15.94:0/1048355415 shutdown_connections 2024-04-21T08:31:01.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.503+0000 7f1905c85700 1 --2- 172.21.15.94:0/1048355415 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f18e80396b0 0x7f18e803bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:01.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.503+0000 7f1905c85700 1 --2- 172.21.15.94:0/1048355415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19001084e0 0x7f1900197eb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:01.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.503+0000 7f1905c85700 1 -- 172.21.15.94:0/1048355415 >> 172.21.15.94:0/1048355415 conn(0x7f19000fdf80 msgr2=0x7f190010a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:01.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.503+0000 7f1905c85700 1 -- 172.21.15.94:0/1048355415 shutdown_connections 2024-04-21T08:31:01.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:01.503+0000 7f1905c85700 1 -- 172.21.15.94:0/1048355415 wait complete. 2024-04-21T08:31:02.071 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:02.071 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:02.071 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:02.071 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:02 smithi094 ceph-mon[27988]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:02.071 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:02.071 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:02 smithi094 ceph-mon[27988]: Added host smithi173 2024-04-21T08:31:02.071 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:02.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T08:31:02.304 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:31:03.442 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:03 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:03.442 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:03 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:03.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.757+0000 7fd6afaf7700 1 -- 172.21.15.94:0/1857789570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6a8074d60 msgr2=0x7fd6a8075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:03.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.757+0000 7fd6afaf7700 1 --2- 172.21.15.94:0/1857789570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6a8074d60 0x7fd6a8075140 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd69800b0a0 tx=0x7fd69800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.757+0000 7fd6afaf7700 1 -- 172.21.15.94:0/1857789570 shutdown_connections 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.757+0000 7fd6afaf7700 1 --2- 172.21.15.94:0/1857789570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6a8074d60 0x7fd6a8075140 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.757+0000 7fd6afaf7700 1 -- 172.21.15.94:0/1857789570 >> 172.21.15.94:0/1857789570 conn(0x7fd6a80fdf60 msgr2=0x7fd6a8100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.757+0000 7fd6afaf7700 1 -- 172.21.15.94:0/1857789570 shutdown_connections 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.757+0000 7fd6afaf7700 1 -- 172.21.15.94:0/1857789570 wait complete. 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.758+0000 7fd6afaf7700 1 Processor -- start 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.758+0000 7fd6afaf7700 1 -- start start 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.758+0000 7fd6afaf7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6a8074d60 0x7fd6a819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.758+0000 7fd6afaf7700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a810eb90 con 0x7fd6a8074d60 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.758+0000 7fd6ad893700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6a8074d60 0x7fd6a819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.758+0000 7fd6ad893700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6a8074d60 0x7fd6a819a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45498/0 (socket says 172.21.15.94:45498) 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.758+0000 7fd6ad893700 1 -- 172.21.15.94:0/641590146 learned_addr learned my addr 172.21.15.94:0/641590146 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:31:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.758+0000 7fd6ad893700 1 -- 172.21.15.94:0/641590146 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a819a5a0 con 0x7fd6a8074d60 2024-04-21T08:31:03.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.759+0000 7fd6ad893700 1 --2- 172.21.15.94:0/641590146 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6a8074d60 0x7fd6a819a060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd698015040 tx=0x7fd698009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:03.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.759+0000 7fd6a6ffd700 1 -- 172.21.15.94:0/641590146 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd698021020 con 0x7fd6a8074d60 2024-04-21T08:31:03.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.759+0000 7fd6a6ffd700 1 -- 172.21.15.94:0/641590146 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd698004000 con 0x7fd6a8074d60 2024-04-21T08:31:03.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.759+0000 7fd6a6ffd700 1 -- 172.21.15.94:0/641590146 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd698017040 con 0x7fd6a8074d60 2024-04-21T08:31:03.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.759+0000 7fd6afaf7700 1 -- 172.21.15.94:0/641590146 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a819a830 con 0x7fd6a8074d60 2024-04-21T08:31:03.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.759+0000 7fd6afaf7700 1 -- 172.21.15.94:0/641590146 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a819e930 con 0x7fd6a8074d60 2024-04-21T08:31:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.760+0000 7fd6a6ffd700 1 -- 172.21.15.94:0/641590146 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd698024020 con 0x7fd6a8074d60 2024-04-21T08:31:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.760+0000 7fd6afaf7700 1 -- 172.21.15.94:0/641590146 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a80764a0 con 0x7fd6a8074d60 2024-04-21T08:31:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.760+0000 7fd6a6ffd700 1 --2- 172.21.15.94:0/641590146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6900396b0 0x7fd69003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.760+0000 7fd6a6ffd700 1 -- 172.21.15.94:0/641590146 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd69800f050 con 0x7fd6a8074d60 2024-04-21T08:31:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.760+0000 7fd6ad092700 1 --2- 172.21.15.94:0/641590146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6900396b0 0x7fd69003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.760+0000 7fd6ad092700 1 --2- 172.21.15.94:0/641590146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6900396b0 0x7fd69003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd69c006fd0 tx=0x7fd69c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.765+0000 7fd6a6ffd700 1 -- 172.21.15.94:0/641590146 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd69802b020 con 0x7fd6a8074d60 2024-04-21T08:31:03.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.967+0000 7fd6afaf7700 1 -- 172.21.15.94:0/641590146 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd6a80640a0 con 0x7fd6900396b0 2024-04-21T08:31:03.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.968+0000 7fd6a6ffd700 1 -- 172.21.15.94:0/641590146 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fd6a80640a0 con 0x7fd6900396b0 2024-04-21T08:31:03.967 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:31:03.967 INFO:teuthology.orchestra.run.smithi094.stdout:[{"addr": "172.21.15.94", "hostname": "smithi094", "labels": [], "status": ""}, {"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-04-21T08:31:03.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.970+0000 7fd6afaf7700 1 -- 172.21.15.94:0/641590146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6900396b0 msgr2=0x7fd69003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:03.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.970+0000 7fd6afaf7700 1 --2- 172.21.15.94:0/641590146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6900396b0 0x7fd69003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd69c006fd0 tx=0x7fd69c006e40 comp rx=0 tx=0).stop 2024-04-21T08:31:03.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.970+0000 7fd6afaf7700 1 -- 172.21.15.94:0/641590146 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6a8074d60 msgr2=0x7fd6a819a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:03.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.970+0000 7fd6afaf7700 1 --2- 172.21.15.94:0/641590146 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6a8074d60 0x7fd6a819a060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd698015040 tx=0x7fd698009800 comp rx=0 tx=0).stop 2024-04-21T08:31:03.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.970+0000 7fd6afaf7700 1 -- 172.21.15.94:0/641590146 shutdown_connections 2024-04-21T08:31:03.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.970+0000 7fd6afaf7700 1 --2- 172.21.15.94:0/641590146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6900396b0 0x7fd69003bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:03.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.970+0000 7fd6afaf7700 1 --2- 172.21.15.94:0/641590146 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6a8074d60 0x7fd6a819a060 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:03.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.970+0000 7fd6afaf7700 1 -- 172.21.15.94:0/641590146 >> 172.21.15.94:0/641590146 conn(0x7fd6a80fdf60 msgr2=0x7fd6a8106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:03.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.970+0000 7fd6afaf7700 1 -- 172.21.15.94:0/641590146 shutdown_connections 2024-04-21T08:31:03.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:03.970+0000 7fd6afaf7700 1 -- 172.21.15.94:0/641590146 wait complete. 2024-04-21T08:31:04.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:04 smithi094 ceph-mon[27988]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:04.592 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-21T08:31:04.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-21T08:31:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:31:05.259 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:05 smithi094 ceph-mon[27988]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:31:05.259 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:05 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:06.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.317+0000 7f672fce2700 1 -- 172.21.15.94:0/3561902457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281035f0 msgr2=0x7f67281039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:06.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.317+0000 7f672fce2700 1 --2- 172.21.15.94:0/3561902457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281035f0 0x7f67281039d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f672400b0a0 tx=0x7f672400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:31:06.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.317+0000 7f672fce2700 1 -- 172.21.15.94:0/3561902457 shutdown_connections 2024-04-21T08:31:06.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.317+0000 7f672fce2700 1 --2- 172.21.15.94:0/3561902457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281035f0 0x7f67281039d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:06.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.317+0000 7f672fce2700 1 -- 172.21.15.94:0/3561902457 >> 172.21.15.94:0/3561902457 conn(0x7f67280fce40 msgr2=0x7f67280ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:06.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.317+0000 7f672fce2700 1 -- 172.21.15.94:0/3561902457 shutdown_connections 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.317+0000 7f672fce2700 1 -- 172.21.15.94:0/3561902457 wait complete. 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.317+0000 7f672fce2700 1 Processor -- start 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f672fce2700 1 -- start start 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f672fce2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281035f0 0x7f672819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f672fce2700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f672810c990 con 0x7f67281035f0 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f672da7e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281035f0 0x7f672819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f672da7e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281035f0 0x7f672819a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45522/0 (socket says 172.21.15.94:45522) 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f672da7e700 1 -- 172.21.15.94:0/3348099528 learned_addr learned my addr 172.21.15.94:0/3348099528 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f672da7e700 1 -- 172.21.15.94:0/3348099528 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672819a650 con 0x7f67281035f0 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f672da7e700 1 --2- 172.21.15.94:0/3348099528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281035f0 0x7f672819a110 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6724015040 tx=0x7f6724009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f671effd700 1 -- 172.21.15.94:0/3348099528 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6724021020 con 0x7f67281035f0 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f671effd700 1 -- 172.21.15.94:0/3348099528 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6724004000 con 0x7f67281035f0 2024-04-21T08:31:06.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f672fce2700 1 -- 172.21.15.94:0/3348099528 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672819a8e0 con 0x7f67281035f0 2024-04-21T08:31:06.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f671effd700 1 -- 172.21.15.94:0/3348099528 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6724017040 con 0x7f67281035f0 2024-04-21T08:31:06.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.318+0000 7f672fce2700 1 -- 172.21.15.94:0/3348099528 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f672819e990 con 0x7f67281035f0 2024-04-21T08:31:06.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.319+0000 7f671effd700 1 -- 172.21.15.94:0/3348099528 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6724024020 con 0x7f67281035f0 2024-04-21T08:31:06.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.319+0000 7f671effd700 1 --2- 172.21.15.94:0/3348099528 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6714039650 0x7f671403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:06.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.319+0000 7f671effd700 1 -- 172.21.15.94:0/3348099528 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f672400f050 con 0x7f67281035f0 2024-04-21T08:31:06.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.319+0000 7f672fce2700 1 -- 172.21.15.94:0/3348099528 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f672806afd0 con 0x7f67281035f0 2024-04-21T08:31:06.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.319+0000 7f672d27d700 1 --2- 172.21.15.94:0/3348099528 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6714039650 0x7f671403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:06.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.320+0000 7f672d27d700 1 --2- 172.21.15.94:0/3348099528 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6714039650 0x7f671403bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6718006fd0 tx=0x7f6718006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:06.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.324+0000 7f671effd700 1 -- 172.21.15.94:0/3348099528 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f672402b020 con 0x7f67281035f0 2024-04-21T08:31:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:06 smithi094 ceph-mon[27988]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:06.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:06.527+0000 7f672fce2700 1 -- 172.21.15.94:0/3348099528 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f67280640a0 con 0x7f67281035f0 2024-04-21T08:31:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.122+0000 7f671effd700 1 -- 172.21.15.94:0/3348099528 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f672402e350 con 0x7f67281035f0 2024-04-21T08:31:07.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.125+0000 7f672fce2700 1 -- 172.21.15.94:0/3348099528 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6714039650 msgr2=0x7f671403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:07.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.125+0000 7f672fce2700 1 --2- 172.21.15.94:0/3348099528 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6714039650 0x7f671403bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6718006fd0 tx=0x7f6718006e40 comp rx=0 tx=0).stop 2024-04-21T08:31:07.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.125+0000 7f672fce2700 1 -- 172.21.15.94:0/3348099528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281035f0 msgr2=0x7f672819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:07.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.125+0000 7f672fce2700 1 --2- 172.21.15.94:0/3348099528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281035f0 0x7f672819a110 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6724015040 tx=0x7f6724009800 comp rx=0 tx=0).stop 2024-04-21T08:31:07.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.125+0000 7f672fce2700 1 -- 172.21.15.94:0/3348099528 shutdown_connections 2024-04-21T08:31:07.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.125+0000 7f672fce2700 1 --2- 172.21.15.94:0/3348099528 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6714039650 0x7f671403bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:07.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.125+0000 7f672fce2700 1 --2- 172.21.15.94:0/3348099528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281035f0 0x7f672819a110 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:07.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.125+0000 7f672fce2700 1 -- 172.21.15.94:0/3348099528 >> 172.21.15.94:0/3348099528 conn(0x7f67280fce40 msgr2=0x7f67281087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:07.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.125+0000 7f672fce2700 1 -- 172.21.15.94:0/3348099528 shutdown_connections 2024-04-21T08:31:07.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:07.125+0000 7f672fce2700 1 -- 172.21.15.94:0/3348099528 wait complete. 2024-04-21T08:31:07.125 INFO:teuthology.orchestra.run.smithi094.stderr:adjusted tunables profile to default 2024-04-21T08:31:07.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:07 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3348099528' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T08:31:07.795 INFO:tasks.cephadm:Adding mon.a on smithi094 2024-04-21T08:31:07.795 INFO:tasks.cephadm:Adding mon.b on smithi111 2024-04-21T08:31:07.795 INFO:tasks.cephadm:Adding mon.c on smithi173 2024-04-21T08:31:07.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch apply mon '3;smithi094:172.21.15.94=a;smithi111:172.21.15.111=b;smithi173:172.21.15.173=c' 2024-04-21T08:31:08.029 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:08.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3348099528' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T08:31:08.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:31:08.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:08.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:08.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:09.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.471+0000 7f903293d700 1 -- 172.21.15.173:0/3334494969 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f902c104310 msgr2=0x7f902c1046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:09.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.471+0000 7f903293d700 1 --2- 172.21.15.173:0/3334494969 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f902c104310 0x7f902c1046f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f901c009910 tx=0x7f901c009c20 comp rx=0 tx=0).stop 2024-04-21T08:31:09.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.471+0000 7f903293d700 1 -- 172.21.15.173:0/3334494969 shutdown_connections 2024-04-21T08:31:09.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.471+0000 7f903293d700 1 --2- 172.21.15.173:0/3334494969 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f902c104310 0x7f902c1046f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:09.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.471+0000 7f903293d700 1 -- 172.21.15.173:0/3334494969 >> 172.21.15.173:0/3334494969 conn(0x7f902c0fc5b0 msgr2=0x7f902c0fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:09.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.471+0000 7f903293d700 1 -- 172.21.15.173:0/3334494969 shutdown_connections 2024-04-21T08:31:09.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.471+0000 7f903293d700 1 -- 172.21.15.173:0/3334494969 wait complete. 2024-04-21T08:31:09.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.472+0000 7f903293d700 1 Processor -- start 2024-04-21T08:31:09.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.472+0000 7f903293d700 1 -- start start 2024-04-21T08:31:09.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.472+0000 7f903293d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f902c104310 0x7f902c101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:09.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.472+0000 7f903293d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f902c10c9c0 con 0x7f902c104310 2024-04-21T08:31:09.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.473+0000 7f902bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f902c104310 0x7f902c101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:09.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.473+0000 7f902bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f902c104310 0x7f902c101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.173:41676/0 (socket says 172.21.15.173:41676) 2024-04-21T08:31:09.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.473+0000 7f902bfff700 1 -- 172.21.15.173:0/3159451978 learned_addr learned my addr 172.21.15.173:0/3159451978 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-21T08:31:09.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.473+0000 7f902bfff700 1 -- 172.21.15.173:0/3159451978 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902c103a60 con 0x7f902c104310 2024-04-21T08:31:09.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.473+0000 7f902bfff700 1 --2- 172.21.15.173:0/3159451978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f902c104310 0x7f902c101b70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f901c0056d0 tx=0x7f901c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:09.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.473+0000 7f90297fa700 1 -- 172.21.15.173:0/3159451978 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c026070 con 0x7f902c104310 2024-04-21T08:31:09.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.473+0000 7f903293d700 1 -- 172.21.15.173:0/3159451978 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902c102110 con 0x7f902c104310 2024-04-21T08:31:09.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.473+0000 7f90297fa700 1 -- 172.21.15.173:0/3159451978 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f901c00bc00 con 0x7f902c104310 2024-04-21T08:31:09.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.473+0000 7f90297fa700 1 -- 172.21.15.173:0/3159451978 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c003920 con 0x7f902c104310 2024-04-21T08:31:09.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.474+0000 7f903293d700 1 -- 172.21.15.173:0/3159451978 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f902c1025d0 con 0x7f902c104310 2024-04-21T08:31:09.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.474+0000 7f903293d700 1 -- 172.21.15.173:0/3159451978 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f902c0640a0 con 0x7f902c104310 2024-04-21T08:31:09.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.474+0000 7f90297fa700 1 -- 172.21.15.173:0/3159451978 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f901c029070 con 0x7f902c104310 2024-04-21T08:31:09.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.475+0000 7f90297fa700 1 --2- 172.21.15.173:0/3159451978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9014039650 0x7f901403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:09.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.475+0000 7f90297fa700 1 -- 172.21.15.173:0/3159451978 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f901c01e070 con 0x7f902c104310 2024-04-21T08:31:09.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.475+0000 7f902b7fe700 1 --2- 172.21.15.173:0/3159451978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9014039650 0x7f901403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:09.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.475+0000 7f902b7fe700 1 --2- 172.21.15.173:0/3159451978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9014039650 0x7f901403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9020006fd0 tx=0x7f9020006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:09.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.481+0000 7f90297fa700 1 -- 172.21.15.173:0/3159451978 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f901c013070 con 0x7f902c104310 2024-04-21T08:31:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:09 smithi094 ceph-mon[27988]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:31:09.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.687+0000 7f903293d700 1 -- 172.21.15.173:0/3159451978 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi094:172.21.15.94=a;smithi111:172.21.15.111=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}) -- 0x7f902c103c90 con 0x7f9014039650 2024-04-21T08:31:09.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.694+0000 7f90297fa700 1 -- 172.21.15.173:0/3159451978 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f902c103c90 con 0x7f9014039650 2024-04-21T08:31:09.694 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-04-21T08:31:09.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.697+0000 7f903293d700 1 -- 172.21.15.173:0/3159451978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9014039650 msgr2=0x7f901403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:09.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.697+0000 7f903293d700 1 --2- 172.21.15.173:0/3159451978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9014039650 0x7f901403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9020006fd0 tx=0x7f9020006e40 comp rx=0 tx=0).stop 2024-04-21T08:31:09.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.697+0000 7f903293d700 1 -- 172.21.15.173:0/3159451978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f902c104310 msgr2=0x7f902c101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:09.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.697+0000 7f903293d700 1 --2- 172.21.15.173:0/3159451978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f902c104310 0x7f902c101b70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f901c0056d0 tx=0x7f901c0057b0 comp rx=0 tx=0).stop 2024-04-21T08:31:09.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.697+0000 7f903293d700 1 -- 172.21.15.173:0/3159451978 shutdown_connections 2024-04-21T08:31:09.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.697+0000 7f903293d700 1 --2- 172.21.15.173:0/3159451978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9014039650 0x7f901403bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:09.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.697+0000 7f903293d700 1 --2- 172.21.15.173:0/3159451978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f902c104310 0x7f902c101b70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:09.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.697+0000 7f903293d700 1 -- 172.21.15.173:0/3159451978 >> 172.21.15.173:0/3159451978 conn(0x7f902c0fc5b0 msgr2=0x7f902c0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:09.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.697+0000 7f903293d700 1 -- 172.21.15.173:0/3159451978 shutdown_connections 2024-04-21T08:31:09.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:09.697+0000 7f903293d700 1 -- 172.21.15.173:0/3159451978 wait complete. 2024-04-21T08:31:10.211 DEBUG:teuthology.orchestra.run.smithi111:mon.b> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.b.service 2024-04-21T08:31:10.214 DEBUG:teuthology.orchestra.run.smithi173:mon.c> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.c.service 2024-04-21T08:31:10.217 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:31:10.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:31:10.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:10 smithi094 ceph-mon[27988]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:10.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:10.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:10.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:10.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:10.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:10.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:10.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:11.005 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:11.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:11 smithi094 ceph-mon[27988]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi094:172.21.15.94=a;smithi111:172.21.15.111=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:31:11.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:11 smithi094 ceph-mon[27988]: Saving service mon spec with placement smithi094:172.21.15.94=a;smithi111:172.21.15.111=b;smithi173:172.21.15.173=c;count:3 2024-04-21T08:31:11.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:11 smithi094 ceph-mon[27988]: Deploying daemon mon.c on smithi173 2024-04-21T08:31:12.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:12 smithi094 ceph-mon[27988]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.234+0000 7f61cf22d700 1 -- 172.21.15.173:0/3497188001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61c81015b0 msgr2=0x7f61c8101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.234+0000 7f61cf22d700 1 --2- 172.21.15.173:0/3497188001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61c81015b0 0x7f61c8101990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f61bc009910 tx=0x7f61bc009c20 comp rx=0 tx=0).stop 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.234+0000 7f61cf22d700 1 -- 172.21.15.173:0/3497188001 shutdown_connections 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.234+0000 7f61cf22d700 1 --2- 172.21.15.173:0/3497188001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61c81015b0 0x7f61c8101990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.234+0000 7f61cf22d700 1 -- 172.21.15.173:0/3497188001 >> 172.21.15.173:0/3497188001 conn(0x7f61c80fce60 msgr2=0x7f61c80ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.234+0000 7f61cf22d700 1 -- 172.21.15.173:0/3497188001 shutdown_connections 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.234+0000 7f61cf22d700 1 -- 172.21.15.173:0/3497188001 wait complete. 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.235+0000 7f61cf22d700 1 Processor -- start 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.235+0000 7f61cf22d700 1 -- start start 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.235+0000 7f61cf22d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61c81015b0 0x7f61c8198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.235+0000 7f61cf22d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f61c810b830 con 0x7f61c81015b0 2024-04-21T08:31:14.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.235+0000 7f61ccfc9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61c81015b0 0x7f61c8198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:14.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.236+0000 7f61ccfc9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61c81015b0 0x7f61c8198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.173:41704/0 (socket says 172.21.15.173:41704) 2024-04-21T08:31:14.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.236+0000 7f61ccfc9700 1 -- 172.21.15.173:0/2587600367 learned_addr learned my addr 172.21.15.173:0/2587600367 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-21T08:31:14.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.236+0000 7f61ccfc9700 1 -- 172.21.15.173:0/2587600367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c8199230 con 0x7f61c81015b0 2024-04-21T08:31:14.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.236+0000 7f61ccfc9700 1 --2- 172.21.15.173:0/2587600367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61c81015b0 0x7f61c8198cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0056d0 tx=0x7f61bc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:14.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.236+0000 7f61c5ffb700 1 -- 172.21.15.173:0/2587600367 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61bc026070 con 0x7f61c81015b0 2024-04-21T08:31:14.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.236+0000 7f61cf22d700 1 -- 172.21.15.173:0/2587600367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c8199520 con 0x7f61c81015b0 2024-04-21T08:31:14.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.236+0000 7f61c5ffb700 1 -- 172.21.15.173:0/2587600367 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61bc00bba0 con 0x7f61c81015b0 2024-04-21T08:31:14.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.236+0000 7f61c5ffb700 1 -- 172.21.15.173:0/2587600367 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61bc003920 con 0x7f61c81015b0 2024-04-21T08:31:14.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.236+0000 7f61cf22d700 1 -- 172.21.15.173:0/2587600367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c81999e0 con 0x7f61c81015b0 2024-04-21T08:31:14.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.237+0000 7f61cf22d700 1 -- 172.21.15.173:0/2587600367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c8102a90 con 0x7f61c81015b0 2024-04-21T08:31:14.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.237+0000 7f61c5ffb700 1 -- 172.21.15.173:0/2587600367 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f61bc029070 con 0x7f61c81015b0 2024-04-21T08:31:14.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.238+0000 7f61c5ffb700 1 --2- 172.21.15.173:0/2587600367 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61b8039650 0x7f61b803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:14.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.238+0000 7f61c5ffb700 1 -- 172.21.15.173:0/2587600367 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f61bc01e070 con 0x7f61c81015b0 2024-04-21T08:31:14.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.238+0000 7f61c7fff700 1 --2- 172.21.15.173:0/2587600367 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61b8039650 0x7f61b803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:14.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.238+0000 7f61c7fff700 1 --2- 172.21.15.173:0/2587600367 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61b8039650 0x7f61b803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61b4006fd0 tx=0x7f61b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:14.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.243+0000 7f61c5ffb700 1 -- 172.21.15.173:0/2587600367 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61bc013070 con 0x7f61c81015b0 2024-04-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.537+0000 7f61cf22d700 1 -- 172.21.15.173:0/2587600367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f61c80640a0 con 0x7f61c81015b0 2024-04-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.538+0000 7f61c5ffb700 1 -- 172.21.15.173:0/2587600367 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f61bc003bb0 con 0x7f61c81015b0 2024-04-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","modified":"2024-04-21T08:29:23.995760Z","created":"2024-04-21T08:29:23.995760Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:3300","nonce":0},{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.540+0000 7f61cf22d700 1 -- 172.21.15.173:0/2587600367 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61b8039650 msgr2=0x7f61b803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.540+0000 7f61cf22d700 1 --2- 172.21.15.173:0/2587600367 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61b8039650 0x7f61b803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61b4006fd0 tx=0x7f61b4006e40 comp rx=0 tx=0).stop 2024-04-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.540+0000 7f61cf22d700 1 -- 172.21.15.173:0/2587600367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61c81015b0 msgr2=0x7f61c8198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.540+0000 7f61cf22d700 1 --2- 172.21.15.173:0/2587600367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61c81015b0 0x7f61c8198cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0056d0 tx=0x7f61bc0057b0 comp rx=0 tx=0).stop 2024-04-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.540+0000 7f61cf22d700 1 -- 172.21.15.173:0/2587600367 shutdown_connections 2024-04-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.540+0000 7f61cf22d700 1 --2- 172.21.15.173:0/2587600367 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61b8039650 0x7f61b803bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.540+0000 7f61cf22d700 1 --2- 172.21.15.173:0/2587600367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61c81015b0 0x7f61c8198cf0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.540+0000 7f61cf22d700 1 -- 172.21.15.173:0/2587600367 >> 172.21.15.173:0/2587600367 conn(0x7f61c80fce60 msgr2=0x7f61c8102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.540+0000 7f61cf22d700 1 -- 172.21.15.173:0/2587600367 shutdown_connections 2024-04-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:14.540+0000 7f61cf22d700 1 -- 172.21.15.173:0/2587600367 wait complete. 2024-04-21T08:31:14.541 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-04-21T08:31:15.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:14 smithi094 ceph-mon[27988]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:15.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:14 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/2587600367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:31:15.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:15 smithi173 ceph-mon[31428]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-21T08:31:16.141 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:31:16.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:31:16.403 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:31:20.686 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:31:20.686 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: mon.a calling monitor election 2024-04-21T08:31:20.686 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.686 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:20.686 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.686 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: mon.c calling monitor election 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: monmap epoch 2 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: last_changed 2024-04-21T08:31:15.286533+0000 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: created 2024-04-21T08:29:23.995760+0000 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: min_mon_release 19 (squid) 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: election_strategy: 1 2024-04-21T08:31:20.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-21T08:31:20.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-21T08:31:20.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: fsmap 2024-04-21T08:31:20.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:31:20.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: mgrmap e13: x(active, since 50s) 2024-04-21T08:31:20.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: overall HEALTH_OK 2024-04-21T08:31:20.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:20.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:20.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:20.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:20.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:20.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: mon.a calling monitor election 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: mon.c calling monitor election 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:20.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: monmap epoch 2 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: last_changed 2024-04-21T08:31:15.286533+0000 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: created 2024-04-21T08:29:23.995760+0000 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: min_mon_release 19 (squid) 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: election_strategy: 1 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: fsmap 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: mgrmap e13: x(active, since 50s) 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: overall HEALTH_OK 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:20.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:20.763 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:20.763 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:20.763 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:20.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.920+0000 7f8f320e4700 1 -- 172.21.15.173:0/2874895909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f140035c0 msgr2=0x7f8f14005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:20.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.920+0000 7f8f320e4700 1 --2- 172.21.15.173:0/2874895909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f140035c0 0x7f8f14005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c102490 tx=0x7f8f1c00feb0 comp rx=0 tx=0).stop 2024-04-21T08:31:20.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.920+0000 7f8f320e4700 1 -- 172.21.15.173:0/2874895909 shutdown_connections 2024-04-21T08:31:20.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.920+0000 7f8f320e4700 1 --2- 172.21.15.173:0/2874895909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f140035c0 0x7f8f14005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:20.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.920+0000 7f8f320e4700 1 -- 172.21.15.173:0/2874895909 >> 172.21.15.173:0/2874895909 conn(0x7f8f2c0fe050 msgr2=0x7f8f2c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:20.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.920+0000 7f8f320e4700 1 -- 172.21.15.173:0/2874895909 shutdown_connections 2024-04-21T08:31:20.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.920+0000 7f8f320e4700 1 -- 172.21.15.173:0/2874895909 wait complete. 2024-04-21T08:31:20.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.921+0000 7f8f320e4700 1 Processor -- start 2024-04-21T08:31:20.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.921+0000 7f8f320e4700 1 -- start start 2024-04-21T08:31:20.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.921+0000 7f8f320e4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f140035c0 0x7f8f2c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:20.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.921+0000 7f8f320e4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f2c1984d0 0x7f8f2c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:20.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.921+0000 7f8f320e4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c10a4b0 con 0x7f8f2c1984d0 2024-04-21T08:31:20.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.921+0000 7f8f320e4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2c10a330 con 0x7f8f140035c0 2024-04-21T08:31:20.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.921+0000 7f8f2b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f140035c0 0x7f8f2c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:20.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f140035c0 0x7f8f2c197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51338/0 (socket says 172.21.15.173:51338) 2024-04-21T08:31:20.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2b7fe700 1 -- 172.21.15.173:0/709339531 learned_addr learned my addr 172.21.15.173:0/709339531 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2affd700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f2c1984d0 0x7f8f2c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2b7fe700 1 -- 172.21.15.173:0/709339531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f140035c0 msgr2=0x7f8f2c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2b7fe700 1 -- 172.21.15.173:0/709339531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f140035c0 msgr2=0x7f8f2c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2b7fe700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f140035c0 0x7f8f2c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2b7fe700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f140035c0 0x7f8f2c197f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2affd700 1 -- 172.21.15.173:0/709339531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f140035c0 msgr2=0x7f8f2c197f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2affd700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f140035c0 0x7f8f2c197f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2affd700 1 -- 172.21.15.173:0/709339531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2c198b10 con 0x7f8f2c1984d0 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f2affd700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f2c1984d0 0x7f8f2c19c860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8f2000b4d0 tx=0x7f8f2000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f28ff9700 1 -- 172.21.15.173:0/709339531 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8f20011630 con 0x7f8f2c1984d0 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f28ff9700 1 -- 172.21.15.173:0/709339531 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f20011c80 con 0x7f8f2c1984d0 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f320e4700 1 -- 172.21.15.173:0/709339531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2c19cda0 con 0x7f8f2c1984d0 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f28ff9700 1 -- 172.21.15.173:0/709339531 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8f20010da0 con 0x7f8f2c1984d0 2024-04-21T08:31:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.922+0000 7f8f320e4700 1 -- 172.21.15.173:0/709339531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f2c19d380 con 0x7f8f2c1984d0 2024-04-21T08:31:20.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.923+0000 7f8f28ff9700 1 -- 172.21.15.173:0/709339531 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8f20018020 con 0x7f8f2c1984d0 2024-04-21T08:31:20.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.924+0000 7f8f28ff9700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8f18039440 0x7f8f1803b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:20.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.924+0000 7f8f320e4700 1 -- 172.21.15.173:0/709339531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f2c19d690 con 0x7f8f2c1984d0 2024-04-21T08:31:20.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.924+0000 7f8f28ff9700 1 -- 172.21.15.173:0/709339531 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8f2004c1f0 con 0x7f8f2c1984d0 2024-04-21T08:31:20.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.924+0000 7f8f2b7fe700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8f18039440 0x7f8f1803b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:20.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.924+0000 7f8f2b7fe700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8f18039440 0x7f8f1803b900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00fa60 tx=0x7f8f1c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:20.928 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:20.929+0000 7f8f28ff9700 1 -- 172.21.15.173:0/709339531 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8f2c19d690 con 0x7f8f2c1984d0 2024-04-21T08:31:21.213 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.214+0000 7f8f320e4700 1 -- 172.21.15.173:0/709339531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f8f2c1992f0 con 0x7f8f2c1984d0 2024-04-21T08:31:21.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.214+0000 7f8f28ff9700 1 -- 172.21.15.173:0/709339531 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f8f2001f020 con 0x7f8f2c1984d0 2024-04-21T08:31:21.214 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:31:21.214 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","modified":"2024-04-21T08:31:15.286533Z","created":"2024-04-21T08:29:23.995760Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:3300","nonce":0},{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T08:31:21.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.216+0000 7f8f320e4700 1 -- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8f18039440 msgr2=0x7f8f1803b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:21.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.216+0000 7f8f320e4700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8f18039440 0x7f8f1803b900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00fa60 tx=0x7f8f1c005960 comp rx=0 tx=0).stop 2024-04-21T08:31:21.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.216+0000 7f8f320e4700 1 -- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f2c1984d0 msgr2=0x7f8f2c19c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:21.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.216+0000 7f8f320e4700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f2c1984d0 0x7f8f2c19c860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8f2000b4d0 tx=0x7f8f2000b890 comp rx=0 tx=0).stop 2024-04-21T08:31:21.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.216+0000 7f8f320e4700 1 -- 172.21.15.173:0/709339531 shutdown_connections 2024-04-21T08:31:21.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.217+0000 7f8f320e4700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8f18039440 0x7f8f1803b900 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:21.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.217+0000 7f8f320e4700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f140035c0 0x7f8f2c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:21.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.217+0000 7f8f320e4700 1 --2- 172.21.15.173:0/709339531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f2c1984d0 0x7f8f2c19c860 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:21.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.217+0000 7f8f320e4700 1 -- 172.21.15.173:0/709339531 >> 172.21.15.173:0/709339531 conn(0x7f8f2c0fe050 msgr2=0x7f8f2c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:21.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.217+0000 7f8f320e4700 1 -- 172.21.15.173:0/709339531 shutdown_connections 2024-04-21T08:31:21.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:21.217+0000 7f8f320e4700 1 -- 172.21.15.173:0/709339531 wait complete. 2024-04-21T08:31:21.217 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-04-21T08:31:21.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:21 smithi173 ceph-mon[31428]: Deploying daemon mon.b on smithi111 2024-04-21T08:31:21.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:21 smithi173 ceph-mon[31428]: from='client.? 172.21.15.173:0/709339531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:31:21.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:21.617 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:21 smithi094 ceph-mon[27988]: Deploying daemon mon.b on smithi111 2024-04-21T08:31:21.617 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:21 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/709339531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:31:21.617 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:22.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:22 smithi094 ceph-mon[27988]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:22.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:22 smithi173 ceph-mon[31428]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:22.874 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:31:22.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:31:23.116 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:31:24.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:24 smithi111 systemd[1]: Started Ceph mon.b for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:31:29.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:31:29.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: mon.a calling monitor election 2024-04-21T08:31:29.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: mon.c calling monitor election 2024-04-21T08:31:29.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: mon.b calling monitor election 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: monmap epoch 3 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: last_changed 2024-04-21T08:31:24.188426+0000 2024-04-21T08:31:29.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: created 2024-04-21T08:29:23.995760+0000 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: min_mon_release 19 (squid) 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: election_strategy: 1 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: fsmap 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: mgrmap e13: x(active, since 59s) 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: overall HEALTH_OK 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:29.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: mon.a calling monitor election 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: mon.c calling monitor election 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: mon.b calling monitor election 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: monmap epoch 3 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: last_changed 2024-04-21T08:31:24.188426+0000 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: created 2024-04-21T08:29:23.995760+0000 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: min_mon_release 19 (squid) 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: election_strategy: 1 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: fsmap 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:31:29.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: mgrmap e13: x(active, since 59s) 2024-04-21T08:31:29.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: overall HEALTH_OK 2024-04-21T08:31:29.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:29.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:29.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:31:29.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: mon.a calling monitor election 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: mon.c calling monitor election 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: mon.b calling monitor election 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: monmap epoch 3 2024-04-21T08:31:29.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: last_changed 2024-04-21T08:31:24.188426+0000 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: created 2024-04-21T08:29:23.995760+0000 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: min_mon_release 19 (squid) 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: election_strategy: 1 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: fsmap 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: mgrmap e13: x(active, since 59s) 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: overall HEALTH_OK 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:29.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:30 smithi094 ceph-mon[27988]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:30 smithi173 ceph-mon[31428]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:30.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.610+0000 7f6123499700 1 -- 172.21.15.173:0/381133692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6108006230 msgr2=0x7f61080086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:30.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.610+0000 7f6123499700 1 --2- 172.21.15.173:0/381133692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6108006230 0x7f61080086a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f611c06aeb0 tx=0x7f611800fe90 comp rx=0 tx=0).stop 2024-04-21T08:31:30.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.610+0000 7f6123499700 1 -- 172.21.15.173:0/381133692 shutdown_connections 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.610+0000 7f6123499700 1 --2- 172.21.15.173:0/381133692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6108006230 0x7f61080086a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.610+0000 7f6123499700 1 -- 172.21.15.173:0/381133692 >> 172.21.15.173:0/381133692 conn(0x7f611c0fcf40 msgr2=0x7f611c0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.610+0000 7f6123499700 1 -- 172.21.15.173:0/381133692 shutdown_connections 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.610+0000 7f6123499700 1 -- 172.21.15.173:0/381133692 wait complete. 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.610+0000 7f6123499700 1 Processor -- start 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.610+0000 7f6123499700 1 -- start start 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6123499700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6108006230 0x7f611c1a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6123499700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f611c1a4980 0x7f611c19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6123499700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f611c19eb40 0x7f611c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6123499700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f611c110a40 con 0x7f6108006230 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6123499700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f611c1108c0 con 0x7f611c1a4980 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6123499700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f611c110bc0 con 0x7f611c19eb40 2024-04-21T08:31:30.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121a36700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f611c19eb40 0x7f611c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121a36700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f611c19eb40 0x7f611c19eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:50140/0 (socket says 172.21.15.173:50140) 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121a36700 1 -- 172.21.15.173:0/1644692971 learned_addr learned my addr 172.21.15.173:0/1644692971 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121235700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6108006230 0x7f611c1a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6120a34700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f611c1a4980 0x7f611c19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121235700 1 -- 172.21.15.173:0/1644692971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f611c19eb40 msgr2=0x7f611c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121235700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f611c19eb40 0x7f611c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121235700 1 -- 172.21.15.173:0/1644692971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f611c1a4980 msgr2=0x7f611c19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121235700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f611c1a4980 0x7f611c19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121235700 1 -- 172.21.15.173:0/1644692971 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f611c1ab310 con 0x7f6108006230 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121a36700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f611c19eb40 0x7f611c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:31:30.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.611+0000 7f6121235700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6108006230 0x7f611c1a4440 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6118010010 tx=0x7f6118012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.612+0000 7f61127fc700 1 -- 172.21.15.173:0/1644692971 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6118011d20 con 0x7f6108006230 2024-04-21T08:31:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.612+0000 7f61127fc700 1 -- 172.21.15.173:0/1644692971 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f611802a070 con 0x7f6108006230 2024-04-21T08:31:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.612+0000 7f6123499700 1 -- 172.21.15.173:0/1644692971 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f611c1ab4e0 con 0x7f6108006230 2024-04-21T08:31:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.612+0000 7f61127fc700 1 -- 172.21.15.173:0/1644692971 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611801d9a0 con 0x7f6108006230 2024-04-21T08:31:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.612+0000 7f6123499700 1 -- 172.21.15.173:0/1644692971 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f611c1ab9c0 con 0x7f6108006230 2024-04-21T08:31:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.612+0000 7f61127fc700 1 -- 172.21.15.173:0/1644692971 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f611802c020 con 0x7f6108006230 2024-04-21T08:31:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.612+0000 7f6123499700 1 -- 172.21.15.173:0/1644692971 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f611c0640a0 con 0x7f6108006230 2024-04-21T08:31:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.613+0000 7f61127fc700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6100039ae0 0x7f610003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.613+0000 7f61127fc700 1 -- 172.21.15.173:0/1644692971 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f611804ef90 con 0x7f6108006230 2024-04-21T08:31:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.613+0000 7f6120a34700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6100039ae0 0x7f610003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:30.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.613+0000 7f6120a34700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6100039ae0 0x7f610003bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f610c009850 tx=0x7f610c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:30.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.618+0000 7f61127fc700 1 -- 172.21.15.173:0/1644692971 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6118024d90 con 0x7f6108006230 2024-04-21T08:31:30.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:30.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:30.690 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:30.690 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:30 smithi111 ceph-mon[31493]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:30.690 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:30.690 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:31:30.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.914+0000 7f6123499700 1 -- 172.21.15.173:0/1644692971 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f611c1abc80 con 0x7f6108006230 2024-04-21T08:31:30.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.915+0000 7f61127fc700 1 -- 172.21.15.173:0/1644692971 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7f6118037770 con 0x7f6108006230 2024-04-21T08:31:30.915 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-21T08:31:30.915 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","modified":"2024-04-21T08:31:24.188426Z","created":"2024-04-21T08:29:23.995760Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:3300","nonce":0},{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-21T08:31:30.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.917+0000 7f6123499700 1 -- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6100039ae0 msgr2=0x7f610003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:30.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.917+0000 7f6123499700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6100039ae0 0x7f610003bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f610c009850 tx=0x7f610c008040 comp rx=0 tx=0).stop 2024-04-21T08:31:30.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.917+0000 7f6123499700 1 -- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6108006230 msgr2=0x7f611c1a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:30.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.917+0000 7f6123499700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6108006230 0x7f611c1a4440 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6118010010 tx=0x7f6118012c00 comp rx=0 tx=0).stop 2024-04-21T08:31:30.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.917+0000 7f6123499700 1 -- 172.21.15.173:0/1644692971 shutdown_connections 2024-04-21T08:31:30.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.917+0000 7f6123499700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6100039ae0 0x7f610003bfa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:30.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.917+0000 7f6123499700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f611c19eb40 0x7f611c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:30.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.917+0000 7f6123499700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6108006230 0x7f611c1a4440 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:30.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.917+0000 7f6123499700 1 --2- 172.21.15.173:0/1644692971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f611c1a4980 0x7f611c19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:30.918 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.917+0000 7f6123499700 1 -- 172.21.15.173:0/1644692971 >> 172.21.15.173:0/1644692971 conn(0x7f611c0fcf40 msgr2=0x7f611c1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:30.918 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.918+0000 7f6123499700 1 -- 172.21.15.173:0/1644692971 shutdown_connections 2024-04-21T08:31:30.918 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:30.918+0000 7f6123499700 1 -- 172.21.15.173:0/1644692971 wait complete. 2024-04-21T08:31:30.919 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-04-21T08:31:31.565 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-21T08:31:31.565 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-21T08:31:31.815 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:31:31.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='client.? 172.21.15.173:0/1644692971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:31.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:31:31.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:31.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:31.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:31.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/1644692971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:31:31.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:31:31.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:32.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='client.? 172.21.15.173:0/1644692971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:32.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:32.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:31:32.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:32 smithi111 ceph-mon[31493]: Updating smithi094:/etc/ceph/ceph.conf 2024-04-21T08:31:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:32 smithi111 ceph-mon[31493]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T08:31:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:32 smithi111 ceph-mon[31493]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-21T08:31:32.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:32 smithi111 ceph-mon[31493]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:32.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:32 smithi111 ceph-mon[31493]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:32.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:32 smithi111 ceph-mon[31493]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:32.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:32 smithi111 ceph-mon[31493]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:31:32.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:32 smithi111 ceph-mon[31493]: Reconfiguring daemon mon.a on smithi094 2024-04-21T08:31:32.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:32 smithi111 ceph-mon[31493]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:32.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:32 smithi094 ceph-mon[27988]: Updating smithi094:/etc/ceph/ceph.conf 2024-04-21T08:31:32.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:32 smithi094 ceph-mon[27988]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T08:31:32.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:32 smithi094 ceph-mon[27988]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-21T08:31:32.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:32 smithi094 ceph-mon[27988]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:32.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:32 smithi094 ceph-mon[27988]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:32.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:32 smithi094 ceph-mon[27988]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:32.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:32 smithi094 ceph-mon[27988]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:31:32.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:32 smithi094 ceph-mon[27988]: Reconfiguring daemon mon.a on smithi094 2024-04-21T08:31:32.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:32 smithi094 ceph-mon[27988]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:33.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:32 smithi173 ceph-mon[31428]: Updating smithi094:/etc/ceph/ceph.conf 2024-04-21T08:31:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:32 smithi173 ceph-mon[31428]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T08:31:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:32 smithi173 ceph-mon[31428]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-21T08:31:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:32 smithi173 ceph-mon[31428]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:32 smithi173 ceph-mon[31428]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:32 smithi173 ceph-mon[31428]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:31:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:32 smithi173 ceph-mon[31428]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:31:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:32 smithi173 ceph-mon[31428]: Reconfiguring daemon mon.a on smithi094 2024-04-21T08:31:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:32 smithi173 ceph-mon[31428]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:34.201 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:34.202 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:34.202 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:34 smithi094 ceph-mon[27988]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:31:34.202 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:34.202 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:31:34.202 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:34.202 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:34 smithi094 ceph-mon[27988]: Reconfiguring daemon mon.b on smithi111 2024-04-21T08:31:34.202 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:34 smithi094 ceph-mon[27988]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.235+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/1418221885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a44102300 msgr2=0x7f5a4410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.235+0000 7f5a4bb6f700 1 --2- 172.21.15.94:0/1418221885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a44102300 0x7f5a4410e7f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00b0a0 tx=0x7f5a3c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:31:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.235+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/1418221885 shutdown_connections 2024-04-21T08:31:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.235+0000 7f5a4bb6f700 1 --2- 172.21.15.94:0/1418221885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a44107990 0x7f5a44107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.235+0000 7f5a4bb6f700 1 --2- 172.21.15.94:0/1418221885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a44102300 0x7f5a4410e7f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.235+0000 7f5a4bb6f700 1 --2- 172.21.15.94:0/1418221885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a44101940 0x7f5a44101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.235+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/1418221885 >> 172.21.15.94:0/1418221885 conn(0x7f5a440788f0 msgr2=0x7f5a44078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.235+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/1418221885 shutdown_connections 2024-04-21T08:31:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.235+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/1418221885 wait complete. 2024-04-21T08:31:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4bb6f700 1 Processor -- start 2024-04-21T08:31:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4bb6f700 1 -- start start 2024-04-21T08:31:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4bb6f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a44101940 0x7f5a44198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4bb6f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a44102300 0x7f5a441985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4bb6f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a44107990 0x7f5a4419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4bb6f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5a44111b50 con 0x7f5a44101940 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4bb6f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5a441119d0 con 0x7f5a44107990 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4bb6f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5a44111cd0 con 0x7f5a44102300 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4990b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a44101940 0x7f5a44198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4990b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a44101940 0x7f5a44198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46134/0 (socket says 172.21.15.94:46134) 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.236+0000 7f5a4990b700 1 -- 172.21.15.94:0/2298312395 learned_addr learned my addr 172.21.15.94:0/2298312395 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4a10c700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a44107990 0x7f5a4419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4910a700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a44102300 0x7f5a441985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4990b700 1 -- 172.21.15.94:0/2298312395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a44102300 msgr2=0x7f5a441985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4990b700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a44102300 0x7f5a441985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4990b700 1 -- 172.21.15.94:0/2298312395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a44107990 msgr2=0x7f5a4419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4990b700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a44107990 0x7f5a4419c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4990b700 1 -- 172.21.15.94:0/2298312395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a4419d070 con 0x7f5a44101940 2024-04-21T08:31:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4910a700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a44102300 0x7f5a441985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:31:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4990b700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a44101940 0x7f5a44198060 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c920 tx=0x7f5a3400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a3affd700 1 -- 172.21.15.94:0/2298312395 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34007b00 con 0x7f5a44101940 2024-04-21T08:31:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a3affd700 1 -- 172.21.15.94:0/2298312395 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a34004ce0 con 0x7f5a44101940 2024-04-21T08:31:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/2298312395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a4419d360 con 0x7f5a44101940 2024-04-21T08:31:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a3affd700 1 -- 172.21.15.94:0/2298312395 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34005550 con 0x7f5a44101940 2024-04-21T08:31:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.237+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/2298312395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a4406a720 con 0x7f5a44101940 2024-04-21T08:31:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.238+0000 7f5a3affd700 1 -- 172.21.15.94:0/2298312395 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5a34004830 con 0x7f5a44101940 2024-04-21T08:31:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.238+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/2298312395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a44069730 con 0x7f5a44101940 2024-04-21T08:31:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.238+0000 7f5a3affd700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5a30039890 0x7f5a3003bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.238+0000 7f5a3affd700 1 -- 172.21.15.94:0/2298312395 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5a3404c850 con 0x7f5a44101940 2024-04-21T08:31:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.239+0000 7f5a4910a700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5a30039890 0x7f5a3003bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.239+0000 7f5a4910a700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5a30039890 0x7f5a3003bd50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a44078d10 tx=0x7f5a40008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.244+0000 7f5a3affd700 1 -- 172.21.15.94:0/2298312395 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a34014030 con 0x7f5a44101940 2024-04-21T08:31:34.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:34 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:34.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:34 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:34.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:34 smithi111 ceph-mon[31493]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:31:34.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:34 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:34.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:34 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:31:34.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:34 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:34.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:34 smithi111 ceph-mon[31493]: Reconfiguring daemon mon.b on smithi111 2024-04-21T08:31:34.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:34 smithi111 ceph-mon[31493]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:34.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.443+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/2298312395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f5a4406b210 con 0x7f5a44101940 2024-04-21T08:31:34.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.443+0000 7f5a3affd700 1 -- 172.21.15.94:0/2298312395 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f5a340131b0 con 0x7f5a44101940 2024-04-21T08:31:34.443 INFO:teuthology.orchestra.run.smithi094.stdout:# minimal ceph.conf for 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:31:34.443 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2024-04-21T08:31:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:31:34.443 INFO:teuthology.orchestra.run.smithi094.stdout: mon_host = [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-04-21T08:31:34.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/2298312395 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5a30039890 msgr2=0x7f5a3003bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:34.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5a30039890 0x7f5a3003bd50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a44078d10 tx=0x7f5a40008040 comp rx=0 tx=0).stop 2024-04-21T08:31:34.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/2298312395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a44101940 msgr2=0x7f5a44198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:34.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a44101940 0x7f5a44198060 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c920 tx=0x7f5a3400cc30 comp rx=0 tx=0).stop 2024-04-21T08:31:34.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/2298312395 shutdown_connections 2024-04-21T08:31:34.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5a30039890 0x7f5a3003bd50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:34.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a44107990 0x7f5a4419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:34.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a44101940 0x7f5a44198060 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:34.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 --2- 172.21.15.94:0/2298312395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a44102300 0x7f5a441985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:34.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/2298312395 >> 172.21.15.94:0/2298312395 conn(0x7f5a440788f0 msgr2=0x7f5a440ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:34.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/2298312395 shutdown_connections 2024-04-21T08:31:34.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:34.445+0000 7f5a4bb6f700 1 -- 172.21.15.94:0/2298312395 wait complete. 2024-04-21T08:31:34.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:34 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:34.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:34 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:34.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:34 smithi173 ceph-mon[31428]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:31:34.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:34 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:34.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:34 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:31:34.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:34 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:34.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:34 smithi173 ceph-mon[31428]: Reconfiguring daemon mon.b on smithi111 2024-04-21T08:31:34.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:34 smithi173 ceph-mon[31428]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:35.046 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-21T08:31:35.047 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:31:35.047 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:31:35.074 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:31:35.074 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:31:35.138 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:31:35.138 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:31:35.169 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:31:35.169 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:31:35.237 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:31:35.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:31:35.264 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:31:35.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:31:35.327 INFO:tasks.cephadm:Adding mgr.x on smithi094 2024-04-21T08:31:35.327 INFO:tasks.cephadm:Adding mgr.y on smithi111 2024-04-21T08:31:35.327 INFO:tasks.cephadm:Adding mgr.z on smithi173 2024-04-21T08:31:35.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi094=x;smithi111=y;smithi173=z' 2024-04-21T08:31:35.370 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:35 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2298312395' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:35.370 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:35 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:35.370 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:35 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:35.370 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:35 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:35.370 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:35 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:31:35.370 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:35 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:35.419 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:35 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2298312395' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:35.419 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:35 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:35.419 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:35 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:35.419 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:35 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:35.419 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:35 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:31:35.419 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:35 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:35.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:35 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2298312395' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:35.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:35 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:35.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:35 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:35.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:35 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:31:35.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:35 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:31:35.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:35 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:36.211 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:36 smithi173 ceph-mon[31428]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:31:36.211 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:36 smithi173 ceph-mon[31428]: Reconfiguring daemon mon.c on smithi173 2024-04-21T08:31:36.211 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:36 smithi173 ceph-mon[31428]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:36.328 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:31:36.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:36 smithi111 ceph-mon[31493]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:31:36.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:36 smithi111 ceph-mon[31493]: Reconfiguring daemon mon.c on smithi173 2024-04-21T08:31:36.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:36 smithi111 ceph-mon[31493]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:36.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:36 smithi094 ceph-mon[27988]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:31:36.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:36 smithi094 ceph-mon[27988]: Reconfiguring daemon mon.c on smithi173 2024-04-21T08:31:36.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:36 smithi094 ceph-mon[27988]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:37.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:37 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:37.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:37 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:37.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:37 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:37.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:37 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:37.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:37 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:37.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:37 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:37 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:37 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:37 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:37 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:37 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:37 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:37.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.682+0000 7f14d4d7c700 1 -- 172.21.15.173:0/1346216016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14d0105ed0 msgr2=0x7f14d01082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:37.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.682+0000 7f14d4d7c700 1 --2- 172.21.15.173:0/1346216016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14d0105ed0 0x7f14d01082c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009990 tx=0x7f14c4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.682+0000 7f14d4d7c700 1 -- 172.21.15.173:0/1346216016 shutdown_connections 2024-04-21T08:31:37.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.682+0000 7f14d4d7c700 1 --2- 172.21.15.173:0/1346216016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14d0108800 0x7f14d010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.682+0000 7f14d4d7c700 1 --2- 172.21.15.173:0/1346216016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d01035a0 0x7f14d0105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.682+0000 7f14d4d7c700 1 --2- 172.21.15.173:0/1346216016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14d0105ed0 0x7f14d01082c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.682+0000 7f14d4d7c700 1 -- 172.21.15.173:0/1346216016 >> 172.21.15.173:0/1346216016 conn(0x7f14d00fcf70 msgr2=0x7f14d00ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:37.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.682+0000 7f14d4d7c700 1 -- 172.21.15.173:0/1346216016 shutdown_connections 2024-04-21T08:31:37.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.682+0000 7f14d4d7c700 1 -- 172.21.15.173:0/1346216016 wait complete. 2024-04-21T08:31:37.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14d4d7c700 1 Processor -- start 2024-04-21T08:31:37.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14d4d7c700 1 -- start start 2024-04-21T08:31:37.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14d4d7c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14d01035a0 0x7f14d0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:37.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14d4d7c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14d0105ed0 0x7f14d01991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:37.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14d4d7c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d0108800 0x7f14d01a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:37.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14d4d7c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f14d010e540 con 0x7f14d0108800 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14d4d7c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f14d010e3c0 con 0x7f14d01035a0 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14d4d7c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f14d010e6c0 con 0x7f14d0105ed0 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14cdd9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14d0105ed0 0x7f14d01991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14cdd9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14d0105ed0 0x7f14d01991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:50156/0 (socket says 172.21.15.173:50156) 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14cdd9b700 1 -- 172.21.15.173:0/5547728 learned_addr learned my addr 172.21.15.173:0/5547728 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14ce59c700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14d01035a0 0x7f14d0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14ced9d700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d0108800 0x7f14d01a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14cdd9b700 1 -- 172.21.15.173:0/5547728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14d01035a0 msgr2=0x7f14d0198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14cdd9b700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14d01035a0 0x7f14d0198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14cdd9b700 1 -- 172.21.15.173:0/5547728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d0108800 msgr2=0x7f14d01a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14cdd9b700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d0108800 0x7f14d01a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14cdd9b700 1 -- 172.21.15.173:0/5547728 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d01a08b0 con 0x7f14d0105ed0 2024-04-21T08:31:37.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14cdd9b700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14d0105ed0 0x7f14d01991a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f14c4006010 tx=0x7f14c40047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.683+0000 7f14bf7fe700 1 -- 172.21.15.173:0/5547728 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c4027070 con 0x7f14d0105ed0 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.684+0000 7f14bf7fe700 1 -- 172.21.15.173:0/5547728 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14c4004080 con 0x7f14d0105ed0 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.684+0000 7f14bf7fe700 1 -- 172.21.15.173:0/5547728 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c4022d70 con 0x7f14d0105ed0 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.684+0000 7f14d4d7c700 1 -- 172.21.15.173:0/5547728 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d01a0b40 con 0x7f14d0105ed0 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.684+0000 7f14d4d7c700 1 -- 172.21.15.173:0/5547728 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d01a0ff0 con 0x7f14d0105ed0 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.684+0000 7f14bf7fe700 1 -- 172.21.15.173:0/5547728 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f14c400f460 con 0x7f14d0105ed0 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.685+0000 7f14d4d7c700 1 -- 172.21.15.173:0/5547728 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d0069730 con 0x7f14d0105ed0 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.685+0000 7f14bf7fe700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f14b8039890 0x7f14b803bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.685+0000 7f14bf7fe700 1 -- 172.21.15.173:0/5547728 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f14c404d810 con 0x7f14d0105ed0 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.685+0000 7f14ce59c700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f14b8039890 0x7f14b803bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:37.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.685+0000 7f14ce59c700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f14b8039890 0x7f14b803bd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f14d019a140 tx=0x7f14c00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:37.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:37 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:37.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:37 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:37.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:37 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:37.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:37 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:37.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:37 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:37.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:37 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:37.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.690+0000 7f14bf7fe700 1 -- 172.21.15.173:0/5547728 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14c4013070 con 0x7f14d0105ed0 2024-04-21T08:31:37.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.898+0000 7f14d4d7c700 1 -- 172.21.15.173:0/5547728 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi094=x;smithi111=y;smithi173=z", "target": ["mon-mgr", ""]}) -- 0x7f14d0062e80 con 0x7f14b8039890 2024-04-21T08:31:37.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.904+0000 7f14bf7fe700 1 -- 172.21.15.173:0/5547728 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f14d0062e80 con 0x7f14b8039890 2024-04-21T08:31:37.904 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-04-21T08:31:37.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 -- 172.21.15.173:0/5547728 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f14b8039890 msgr2=0x7f14b803bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:37.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f14b8039890 0x7f14b803bd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f14d019a140 tx=0x7f14c00073d0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 -- 172.21.15.173:0/5547728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14d0105ed0 msgr2=0x7f14d01991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:37.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14d0105ed0 0x7f14d01991a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f14c4006010 tx=0x7f14c40047b0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 -- 172.21.15.173:0/5547728 shutdown_connections 2024-04-21T08:31:37.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f14b8039890 0x7f14b803bd50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14d01035a0 0x7f14d0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d0108800 0x7f14d01a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 --2- 172.21.15.173:0/5547728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14d0105ed0 0x7f14d01991a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:37.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 -- 172.21.15.173:0/5547728 >> 172.21.15.173:0/5547728 conn(0x7f14d00fcf70 msgr2=0x7f14d0105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:37.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 -- 172.21.15.173:0/5547728 shutdown_connections 2024-04-21T08:31:37.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:31:37.907+0000 7f14d4d7c700 1 -- 172.21.15.173:0/5547728 wait complete. 2024-04-21T08:31:38.502 DEBUG:teuthology.orchestra.run.smithi111:mgr.y> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.y.service 2024-04-21T08:31:38.505 DEBUG:teuthology.orchestra.run.smithi173:mgr.z> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.z.service 2024-04-21T08:31:38.527 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:38 smithi111 ceph-mon[31493]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:38.528 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:38.528 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:38.528 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:38.528 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:38.528 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:38.528 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:31:38.528 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:31:38.528 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:31:38.528 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:38.545 INFO:tasks.cephadm:Deploying OSDs... 2024-04-21T08:31:38.546 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:31:38.546 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:31:38.564 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:31:38.565 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-04-21T08:31:38.566 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:38 smithi173 ceph-mon[31428]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:38.566 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:38.566 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:38.566 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:38.566 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:38.566 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:38.566 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:31:38.566 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:31:38.567 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:31:38.567 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:38.622 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:31:38.622 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:38.622 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-21T08:31:38.622 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:38.622 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:38.623 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-21 08:30:42.231479117 +0000 2024-04-21T08:31:38.623 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-21 08:25:27.465263904 +0000 2024-04-21T08:31:38.623 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-21 08:25:27.465263904 +0000 2024-04-21T08:31:38.623 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-21 08:25:27.465263904 +0000 2024-04-21T08:31:38.623 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:31:38.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:38 smithi094 ceph-mon[27988]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:38.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:38.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:38.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:38.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:38.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:38.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:31:38.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:31:38.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:31:38.688 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:38.690 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-21T08:31:38.690 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-21T08:31:38.690 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000110086 s, 4.7 MB/s 2024-04-21T08:31:38.692 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:31:38.751 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-04-21T08:31:38.809 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:31:38.809 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:38.809 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-21T08:31:38.809 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:38.809 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:38.809 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-21 08:30:42.268478488 +0000 2024-04-21T08:31:38.809 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-21 08:25:27.807257973 +0000 2024-04-21T08:31:38.809 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-21 08:25:27.807257973 +0000 2024-04-21T08:31:38.809 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-21 08:25:27.807257973 +0000 2024-04-21T08:31:38.810 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:31:38.875 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-21T08:31:38.875 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-21T08:31:38.875 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000122373 s, 4.2 MB/s 2024-04-21T08:31:38.876 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:31:38.934 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-04-21T08:31:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:31:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:38.991 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-21T08:31:38.991 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:38.991 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:38.991 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-21 08:30:42.298477978 +0000 2024-04-21T08:31:38.991 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-21 08:25:28.136252267 +0000 2024-04-21T08:31:38.991 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-21 08:25:28.136252267 +0000 2024-04-21T08:31:38.991 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-21 08:25:28.136252267 +0000 2024-04-21T08:31:38.991 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:31:39.056 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-21T08:31:39.056 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-21T08:31:39.056 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.00012063 s, 4.2 MB/s 2024-04-21T08:31:39.057 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:31:39.114 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-04-21T08:31:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:31:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:39.171 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-21T08:31:39.171 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:39.171 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:39.171 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-21 08:30:42.323477553 +0000 2024-04-21T08:31:39.171 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-21 08:25:28.460246648 +0000 2024-04-21T08:31:39.171 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-21 08:25:28.460246648 +0000 2024-04-21T08:31:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-21 08:25:28.460246648 +0000 2024-04-21T08:31:39.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:31:39.233 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-21T08:31:39.233 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-21T08:31:39.234 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000116498 s, 4.4 MB/s 2024-04-21T08:31:39.235 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:31:39.292 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T08:31:39.292 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:31:39.311 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:31:39.311 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-04-21T08:31:39.368 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:31:39.368 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:39.368 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-21T08:31:39.368 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:39.368 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:39.368 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 08:31:23.641853052 +0000 2024-04-21T08:31:39.368 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 08:26:21.362242034 +0000 2024-04-21T08:31:39.368 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 08:26:21.362242034 +0000 2024-04-21T08:31:39.368 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 08:26:21.362242034 +0000 2024-04-21T08:31:39.369 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:31:39.433 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T08:31:39.434 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T08:31:39.434 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000279599 s, 1.8 MB/s 2024-04-21T08:31:39.435 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:31:39.493 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-04-21T08:31:39.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:39 smithi094 ceph-mon[27988]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi094=x;smithi111=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:31:39.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:39 smithi094 ceph-mon[27988]: Saving service mgr spec with placement smithi094=x;smithi111=y;smithi173=z;count:3 2024-04-21T08:31:39.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:39 smithi094 ceph-mon[27988]: Deploying daemon mgr.z on smithi173 2024-04-21T08:31:39.529 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:39 smithi173 ceph-mon[31428]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi094=x;smithi111=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:31:39.529 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:39 smithi173 ceph-mon[31428]: Saving service mgr spec with placement smithi094=x;smithi111=y;smithi173=z;count:3 2024-04-21T08:31:39.529 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:39 smithi173 ceph-mon[31428]: Deploying daemon mgr.z on smithi173 2024-04-21T08:31:39.549 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:31:39.549 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:39.549 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-21T08:31:39.549 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:39.549 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:39.549 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 08:31:23.642853035 +0000 2024-04-21T08:31:39.549 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 08:26:21.701236053 +0000 2024-04-21T08:31:39.550 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 08:26:21.701236053 +0000 2024-04-21T08:31:39.550 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 08:26:21.701236053 +0000 2024-04-21T08:31:39.550 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:31:39.611 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:39 smithi111 ceph-mon[31493]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi094=x;smithi111=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:31:39.611 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:39 smithi111 ceph-mon[31493]: Saving service mgr spec with placement smithi094=x;smithi111=y;smithi173=z;count:3 2024-04-21T08:31:39.611 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:39 smithi111 ceph-mon[31493]: Deploying daemon mgr.z on smithi173 2024-04-21T08:31:39.613 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T08:31:39.613 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T08:31:39.613 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000222053 s, 2.3 MB/s 2024-04-21T08:31:39.614 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:31:39.671 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-04-21T08:31:39.727 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:31:39.727 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:39.727 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-21T08:31:39.727 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:39.727 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:39.727 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 08:31:23.642853035 +0000 2024-04-21T08:31:39.727 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 08:26:22.030230248 +0000 2024-04-21T08:31:39.727 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 08:26:22.030230248 +0000 2024-04-21T08:31:39.727 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 08:26:22.030230248 +0000 2024-04-21T08:31:39.728 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:31:39.791 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T08:31:39.791 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T08:31:39.791 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000202124 s, 2.5 MB/s 2024-04-21T08:31:39.793 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:31:39.849 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-04-21T08:31:39.904 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:31:39.904 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:39.904 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-21T08:31:39.904 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:39.904 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:39.904 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 08:31:23.642853035 +0000 2024-04-21T08:31:39.904 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 08:26:22.383224020 +0000 2024-04-21T08:31:39.905 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 08:26:22.383224020 +0000 2024-04-21T08:31:39.905 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 08:26:22.383224020 +0000 2024-04-21T08:31:39.905 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:31:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T08:31:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T08:31:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000189998 s, 2.7 MB/s 2024-04-21T08:31:39.968 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:31:40.023 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-21T08:31:40.024 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:31:40.041 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:39 smithi173 systemd[1]: Starting Ceph mgr.z for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:31:40.041 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:31:40.041 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-04-21T08:31:40.098 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:31:40.099 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:40.099 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-21T08:31:40.099 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:40.099 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:40.099 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-21 08:31:39.895897207 +0000 2024-04-21T08:31:40.099 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-21 08:25:44.926693963 +0000 2024-04-21T08:31:40.099 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-21 08:25:44.926693963 +0000 2024-04-21T08:31:40.099 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-21 08:25:44.926693963 +0000 2024-04-21T08:31:40.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:31:40.164 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-21T08:31:40.164 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-21T08:31:40.164 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000250909 s, 2.0 MB/s 2024-04-21T08:31:40.165 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:31:40.221 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-04-21T08:31:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:31:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:40.277 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-21T08:31:40.277 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:40.277 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:40.277 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-21 08:31:39.895897207 +0000 2024-04-21T08:31:40.278 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-21 08:25:45.272685916 +0000 2024-04-21T08:31:40.278 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-21 08:25:45.272685916 +0000 2024-04-21T08:31:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-21 08:25:45.272685916 +0000 2024-04-21T08:31:40.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:31:40.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:40 smithi173 ceph-mon[31428]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:40.306 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:40 smithi173 podman[33094]: 2024-04-21 08:31:40.084464389 +0000 UTC m=+0.018376738 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:31:40.306 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:40 smithi173 podman[33094]: 2024-04-21 08:31:40.24995303 +0000 UTC m=+0.183865372 container create d3f8281e890fa13cd845924d4f8bb26f2844e5587ee1415195190727327988a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-04-21T08:31:40.328 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-21T08:31:40.328 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-21T08:31:40.328 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000207454 s, 2.5 MB/s 2024-04-21T08:31:40.329 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:31:40.386 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-04-21T08:31:40.456 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:31:40.456 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:40.456 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-21T08:31:40.457 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:40.457 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:40.457 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-21 08:31:39.896897184 +0000 2024-04-21T08:31:40.457 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-21 08:25:45.629677613 +0000 2024-04-21T08:31:40.457 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-21 08:25:45.629677613 +0000 2024-04-21T08:31:40.457 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-21 08:25:45.629677613 +0000 2024-04-21T08:31:40.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:31:40.522 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-21T08:31:40.522 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-21T08:31:40.522 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.00020494 s, 2.5 MB/s 2024-04-21T08:31:40.524 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:31:40.526 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:40 smithi111 ceph-mon[31493]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:40.580 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-04-21T08:31:40.584 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:40 smithi173 podman[33094]: 2024-04-21 08:31:40.489037805 +0000 UTC m=+0.422950149 container init d3f8281e890fa13cd845924d4f8bb26f2844e5587ee1415195190727327988a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:31:40.585 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:40 smithi173 podman[33094]: 2024-04-21 08:31:40.492018459 +0000 UTC m=+0.425930812 container start d3f8281e890fa13cd845924d4f8bb26f2844e5587ee1415195190727327988a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-04-21T08:31:40.585 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:40 smithi173 ceph-mgr[33243]: -- 172.21.15.173:0/69272664 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5653bf5865a0 con 0x5653bf55a800 2024-04-21T08:31:40.599 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:31:40.600 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:31:40.600 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-21T08:31:40.600 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:31:40.600 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:31:40.600 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-21 08:31:39.896897184 +0000 2024-04-21T08:31:40.600 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-21 08:25:45.961669892 +0000 2024-04-21T08:31:40.600 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-21 08:25:45.961669892 +0000 2024-04-21T08:31:40.600 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-21 08:25:45.961669892 +0000 2024-04-21T08:31:40.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:31:40.663 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-21T08:31:40.663 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-21T08:31:40.663 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000224586 s, 2.3 MB/s 2024-04-21T08:31:40.665 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:31:40.721 INFO:tasks.cephadm:Deploying osd.0 on smithi094 with /dev/vg_nvme/lv_4... 2024-04-21T08:31:40.721 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:31:40.745 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:40 smithi094 ceph-mon[27988]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:40.850 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:40 smithi173 bash[33094]: d3f8281e890fa13cd845924d4f8bb26f2844e5587ee1415195190727327988a0 2024-04-21T08:31:40.850 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:40 smithi173 systemd[1]: Started Ceph mgr.z for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:31:40.850 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:40.658+0000 7f4d4dce8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:31:40.851 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:40.851+0000 7f4d4dce8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:31:40.966 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:31:41.260 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:40.955+0000 7f4d4dce8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:31:41.677 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:41.273+0000 7f4d4dce8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:31:41.677 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:41.360+0000 7f4d4dce8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:31:41.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:41.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:41.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:41.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:31:41.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:31:41.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:31:41.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:41.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:41 smithi111 ceph-mon[31493]: Deploying daemon mgr.y on smithi111 2024-04-21T08:31:41.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:41 smithi111 ceph-mon[31493]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:31:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:31:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:31:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:41 smithi094 ceph-mon[27988]: Deploying daemon mgr.y on smithi111 2024-04-21T08:31:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:41 smithi094 ceph-mon[27988]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:42.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:42.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:42.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:42.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:31:42.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:31:42.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:31:42.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:42.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-mon[31428]: Deploying daemon mgr.y on smithi111 2024-04-21T08:31:42.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:41 smithi173 ceph-mon[31428]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:42.842 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:31:42.842 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:31:42.842 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-04-21T08:31:42.842 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-04-21T08:31:42.843 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00946285 s, 1.1 GB/s 2024-04-21T08:31:42.843 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-04-21T08:31:42.859 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:42 smithi111 systemd[1]: Starting Ceph mgr.y for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:31:42.867 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:42.530+0000 7f4d4dce8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:31:42.867 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:42.698+0000 7f4d4dce8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:31:42.867 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:42.868+0000 7f4d4dce8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:31:43.178 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:42 smithi111 podman[32901]: 2024-04-21 08:31:42.763345118 +0000 UTC m=+0.017657206 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:31:43.178 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:42 smithi111 podman[32901]: 2024-04-21 08:31:42.885216918 +0000 UTC m=+0.139529035 container create bfe91950e324280a2936b8d3b6490adde9c8c43fa8f24b96fa2885809fbb9b15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:31:43.178 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:43 smithi111 podman[32901]: 2024-04-21 08:31:43.078608204 +0000 UTC m=+0.332920275 container init bfe91950e324280a2936b8d3b6490adde9c8c43fa8f24b96fa2885809fbb9b15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-21T08:31:43.178 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:43 smithi111 podman[32901]: 2024-04-21 08:31:43.08121468 +0000 UTC m=+0.335526761 container start bfe91950e324280a2936b8d3b6490adde9c8c43fa8f24b96fa2885809fbb9b15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T08:31:43.178 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:43 smithi111 ceph-mgr[32920]: -- 172.21.15.111:0/3022767392 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a217ee45a0 con 0x55a217eb8800 2024-04-21T08:31:43.178 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:43 smithi111 bash[32901]: bfe91950e324280a2936b8d3b6490adde9c8c43fa8f24b96fa2885809fbb9b15 2024-04-21T08:31:43.437 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:43 smithi111 systemd[1]: Started Ceph mgr.y for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:31:43.437 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:43 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:43.231+0000 7fce4c1f0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:31:43.437 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:43 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:43.417+0000 7fce4c1f0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:31:43.452 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:43.152+0000 7f4d4dce8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:31:43.452 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:43.245+0000 7f4d4dce8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:31:43.452 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:43.452+0000 7f4d4dce8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:31:43.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_4 2024-04-21T08:31:43.749 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:31:43.760 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:43.652+0000 7f4d4dce8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:31:43.845 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:43 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:43.524+0000 7fce4c1f0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:31:43.845 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:43 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:43.846+0000 7fce4c1f0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:31:44.187 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:43 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:43.944+0000 7fce4c1f0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:31:44.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:44.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:44 smithi094 ceph-mon[27988]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:44.511 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:44 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:44.286+0000 7f4d4dce8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:31:44.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:44.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:44 smithi173 ceph-mon[31428]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:44.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:44.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:44.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:44 smithi111 ceph-mon[31493]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:45.010 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:44 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:44.723+0000 7f4d4dce8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:31:45.010 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:44 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:44.822+0000 7f4d4dce8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:31:45.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.340+0000 7fc496dfe700 1 -- 172.21.15.94:0/585390928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc490101c40 msgr2=0x7fc4901020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:45.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.340+0000 7fc496dfe700 1 --2- 172.21.15.94:0/585390928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc490101c40 0x7fc4901020c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009990 tx=0x7fc47c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:31:45.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.340+0000 7fc496dfe700 1 -- 172.21.15.94:0/585390928 shutdown_connections 2024-04-21T08:31:45.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.340+0000 7fc496dfe700 1 --2- 172.21.15.94:0/585390928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc490103070 0x7fc490103490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:45.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.340+0000 7fc496dfe700 1 --2- 172.21.15.94:0/585390928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc490102600 0x7fc490109ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:45.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.340+0000 7fc496dfe700 1 --2- 172.21.15.94:0/585390928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc490101c40 0x7fc4901020c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:45.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.340+0000 7fc496dfe700 1 -- 172.21.15.94:0/585390928 >> 172.21.15.94:0/585390928 conn(0x7fc4900fc710 msgr2=0x7fc4900feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:31:45.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.340+0000 7fc496dfe700 1 -- 172.21.15.94:0/585390928 shutdown_connections 2024-04-21T08:31:45.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.340+0000 7fc496dfe700 1 -- 172.21.15.94:0/585390928 wait complete. 2024-04-21T08:31:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc496dfe700 1 Processor -- start 2024-04-21T08:31:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc496dfe700 1 -- start start 2024-04-21T08:31:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc496dfe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc490101c40 0x7fc490101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc496dfe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc490102600 0x7fc49019d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc496dfe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc490103070 0x7fc49019ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc496dfe700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc49010d460 con 0x7fc490103070 2024-04-21T08:31:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc496dfe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc49010d2e0 con 0x7fc490102600 2024-04-21T08:31:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc496dfe700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc49010d5e0 con 0x7fc490101c40 2024-04-21T08:31:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc494b9a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc490101c40 0x7fc490101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc494b9a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc490101c40 0x7fc490101690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:43522/0 (socket says 172.21.15.94:43522) 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc494b9a700 1 -- 172.21.15.94:0/2834898822 learned_addr learned my addr 172.21.15.94:0/2834898822 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc48ffff700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc490102600 0x7fc49019d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc49539b700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc490103070 0x7fc49019ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc494b9a700 1 -- 172.21.15.94:0/2834898822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc490102600 msgr2=0x7fc49019d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc494b9a700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc490102600 0x7fc49019d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc494b9a700 1 -- 172.21.15.94:0/2834898822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc490103070 msgr2=0x7fc49019ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc494b9a700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc490103070 0x7fc49019ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.341+0000 7fc494b9a700 1 -- 172.21.15.94:0/2834898822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4901a0810 con 0x7fc490101c40 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.342+0000 7fc494b9a700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc490101c40 0x7fc490101690 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc48400d980 tx=0x7fc48400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.342+0000 7fc48dffb700 1 -- 172.21.15.94:0/2834898822 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc484016070 con 0x7fc490101c40 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.342+0000 7fc496dfe700 1 -- 172.21.15.94:0/2834898822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4901a0b00 con 0x7fc490101c40 2024-04-21T08:31:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.342+0000 7fc48dffb700 1 -- 172.21.15.94:0/2834898822 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc484004830 con 0x7fc490101c40 2024-04-21T08:31:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.342+0000 7fc48dffb700 1 -- 172.21.15.94:0/2834898822 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc484005540 con 0x7fc490101c40 2024-04-21T08:31:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.342+0000 7fc496dfe700 1 -- 172.21.15.94:0/2834898822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4901a1090 con 0x7fc490101c40 2024-04-21T08:31:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.343+0000 7fc48dffb700 1 -- 172.21.15.94:0/2834898822 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc48400f690 con 0x7fc490101c40 2024-04-21T08:31:45.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.343+0000 7fc48dffb700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc480039890 0x7fc48003bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:31:45.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.343+0000 7fc48dffb700 1 -- 172.21.15.94:0/2834898822 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc48404c820 con 0x7fc490101c40 2024-04-21T08:31:45.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.343+0000 7fc48ffff700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc480039890 0x7fc48003bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:31:45.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.343+0000 7fc496dfe700 1 -- 172.21.15.94:0/2834898822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc490069730 con 0x7fc490101c40 2024-04-21T08:31:45.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.344+0000 7fc48ffff700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc480039890 0x7fc48003bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc490101c10 tx=0x7fc47c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:31:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.349+0000 7fc48dffb700 1 -- 172.21.15.94:0/2834898822 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc48401f020 con 0x7fc490101c40 2024-04-21T08:31:45.437 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:45.170+0000 7fce4c1f0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:31:45.438 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:45.336+0000 7fce4c1f0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:31:45.510 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:45.023+0000 7f4d4dce8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:31:45.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:45.576+0000 7fc496dfe700 1 -- 172.21.15.94:0/2834898822 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc490062e80 con 0x7fc480039890 2024-04-21T08:31:45.727 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.727 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.727 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.727 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.727 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.727 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:45.727 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:45.727 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.727 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:31:45.727 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:31:45.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:45 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:45.788 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:45.505+0000 7fce4c1f0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:31:45.788 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:45.789+0000 7fce4c1f0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:31:45.789 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:45.866 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:45.540+0000 7f4d4dce8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:31:45.866 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:45.631+0000 7f4d4dce8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:31:45.867 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:46.076 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:45 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:45.879+0000 7fce4c1f0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:31:46.076 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:46.077+0000 7fce4c1f0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:31:46.201 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:45.867+0000 7f4d4dce8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:31:46.202 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:45 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:45.997+0000 7f4d4dce8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:31:46.202 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:46.089+0000 7f4d4dce8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:31:46.202 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:46.202+0000 7f4d4dce8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:31:46.437 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:46.266+0000 7fce4c1f0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:31:46.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:46 smithi094 ceph-mon[27988]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:31:46.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:46 smithi094 ceph-mon[27988]: Reconfiguring daemon mgr.x on smithi094 2024-04-21T08:31:46.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:46 smithi094 ceph-mon[27988]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:46.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:46 smithi094 ceph-mon[27988]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:31:46.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:31:46.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:31:46.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:46.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:46.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:46.762 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:46.503+0000 7f4d4dce8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:31:46.763 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-mon[31428]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:31:46.763 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-mon[31428]: Reconfiguring daemon mgr.x on smithi094 2024-04-21T08:31:46.763 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-mon[31428]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:46.763 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-mon[31428]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:31:46.763 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:31:46.764 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:31:46.764 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:46.764 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:46.764 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:46 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:46.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-mon[31493]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:31:46.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-mon[31493]: Reconfiguring daemon mgr.x on smithi094 2024-04-21T08:31:46.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-mon[31493]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:46.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-mon[31493]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:31:46.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:31:46.859 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:31:46.859 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:46.859 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:46.859 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:47.187 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:46 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:46.858+0000 7fce4c1f0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:31:47.510 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:47.097+0000 7f4d4dce8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:31:47.511 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:31:47.307+0000 7f4d4dce8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:31:47.547 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:47.272+0000 7fce4c1f0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:31:47.548 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:47.366+0000 7fce4c1f0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:31:47.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:47.590+0000 7fc48dffb700 1 -- 172.21.15.94:0/2834898822 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7fc484003a00 con 0x7fc490101c40 2024-04-21T08:31:47.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:47 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:47.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:47 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:47.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:47 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:47.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:47 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:47.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:47 smithi094 ceph-mon[27988]: Standby manager daemon z started 2024-04-21T08:31:47.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:47 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:31:47.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:47 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:31:47.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:47 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:31:47.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:47 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:31:47.937 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:47.567+0000 7fce4c1f0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:31:47.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:47.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:47.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:47.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:47.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-mon[31493]: Standby manager daemon z started 2024-04-21T08:31:47.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:31:47.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:31:47.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:31:47.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:47 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:31:48.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:31:48.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:48.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:31:48.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:31:48.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-mon[31428]: Standby manager daemon z started 2024-04-21T08:31:48.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:31:48.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:31:48.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:31:48.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:47 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.173:0/2195917503' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:31:48.404 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:48 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:48.078+0000 7fce4c1f0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:31:48.405 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:48 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:48.167+0000 7fce4c1f0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:31:48.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:48 smithi111 ceph-mon[31493]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:48.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:48 smithi111 ceph-mon[31493]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-21T08:31:48.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:48 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:31:48.688 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:48 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:48.405+0000 7fce4c1f0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:31:48.688 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:48 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:48.538+0000 7fce4c1f0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:31:48.688 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:48 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:48.632+0000 7fce4c1f0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:31:48.953 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:48 smithi094 ceph-mon[27988]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:48.953 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:48 smithi094 ceph-mon[27988]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-21T08:31:48.953 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:48 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:31:49.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:48 smithi173 ceph-mon[31428]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:49.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:48 smithi173 ceph-mon[31428]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-21T08:31:49.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:48 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:31:49.050 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:48 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:48.746+0000 7fce4c1f0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:31:49.050 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:49 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:49.051+0000 7fce4c1f0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:31:49.855 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:49 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3937438363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec26f622-360a-48df-bc52-c85842709d88"}]: dispatch 2024-04-21T08:31:49.855 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:49 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3937438363' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec26f622-360a-48df-bc52-c85842709d88"}]': finished 2024-04-21T08:31:49.855 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:49 smithi111 ceph-mon[31493]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:31:49.855 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:49 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:31:49.855 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:49 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4265269679' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:31:49.856 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:49 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:49.644+0000 7fce4c1f0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:31:49.856 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:31:49 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:31:49.855+0000 7fce4c1f0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:31:50.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:49 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3937438363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec26f622-360a-48df-bc52-c85842709d88"}]: dispatch 2024-04-21T08:31:50.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:49 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3937438363' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec26f622-360a-48df-bc52-c85842709d88"}]': finished 2024-04-21T08:31:50.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:49 smithi094 ceph-mon[27988]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:31:50.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:49 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:31:50.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:49 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4265269679' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:31:50.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:49 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3937438363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec26f622-360a-48df-bc52-c85842709d88"}]: dispatch 2024-04-21T08:31:50.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:49 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3937438363' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec26f622-360a-48df-bc52-c85842709d88"}]': finished 2024-04-21T08:31:50.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:49 smithi173 ceph-mon[31428]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:31:50.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:49 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:31:50.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:49 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4265269679' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:31:50.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:31:50.660+0000 7fc48dffb700 1 -- 172.21.15.94:0/2834898822 <== mon.1 v2:172.21.15.173:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fc48400b5d0 con 0x7fc490101c40 2024-04-21T08:31:50.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:50 smithi111 ceph-mon[31493]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:50.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:50 smithi111 ceph-mon[31493]: Standby manager daemon y started 2024-04-21T08:31:50.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:50 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:31:50.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:50 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:31:50.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:50 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:31:50.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:50 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:31:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:50 smithi094 ceph-mon[27988]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:50 smithi094 ceph-mon[27988]: Standby manager daemon y started 2024-04-21T08:31:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:50 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:31:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:50 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:31:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:50 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:31:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:50 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:31:51.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:50 smithi173 ceph-mon[31428]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:51.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:50 smithi173 ceph-mon[31428]: Standby manager daemon y started 2024-04-21T08:31:51.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:50 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:31:51.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:50 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:31:51.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:50 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:31:51.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:50 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.111:0/1281504609' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:31:51.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:51 smithi111 ceph-mon[31493]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-21T08:31:51.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:31:51.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:51 smithi111 ceph-mon[31493]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:52.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:51 smithi094 ceph-mon[27988]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-21T08:31:52.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:31:52.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:51 smithi094 ceph-mon[27988]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:52.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:51 smithi173 ceph-mon[31428]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-21T08:31:52.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:31:52.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:51 smithi173 ceph-mon[31428]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:54.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:54 smithi094 ceph-mon[27988]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:54 smithi111 ceph-mon[31493]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:54 smithi173 ceph-mon[31428]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:56 smithi111 ceph-mon[31493]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:31:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:56.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:56 smithi111 ceph-mon[31493]: Deploying daemon osd.0 on smithi094 2024-04-21T08:31:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:56 smithi094 ceph-mon[27988]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:31:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:56 smithi094 ceph-mon[27988]: Deploying daemon osd.0 on smithi094 2024-04-21T08:31:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:56 smithi173 ceph-mon[31428]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:31:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:31:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:56 smithi173 ceph-mon[31428]: Deploying daemon osd.0 on smithi094 2024-04-21T08:31:58.820 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:31:58 smithi094 ceph-mon[27988]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:31:58 smithi111 ceph-mon[31493]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:31:59.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:31:58 smithi173 ceph-mon[31428]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:00.860 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:00 smithi094 ceph-mon[27988]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:00 smithi111 ceph-mon[31493]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:01.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:00 smithi173 ceph-mon[31428]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:02.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:02.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:02.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:02.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:02 smithi094 ceph-mon[27988]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:02.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:02 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:02.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:02 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:02.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:02 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:02.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:02 smithi111 ceph-mon[31493]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:02.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:02 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:02.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:02 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:02.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:02 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:02.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:02 smithi173 ceph-mon[31428]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:04.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.463+0000 7fc48dffb700 1 -- 172.21.15.94:0/2834898822 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc490062e80 con 0x7fc480039890 2024-04-21T08:32:04.463 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 0 on host 'smithi094' 2024-04-21T08:32:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.465+0000 7fc496dfe700 1 -- 172.21.15.94:0/2834898822 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc480039890 msgr2=0x7fc48003bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.465+0000 7fc496dfe700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc480039890 0x7fc48003bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc490101c10 tx=0x7fc47c005fb0 comp rx=0 tx=0).stop 2024-04-21T08:32:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.465+0000 7fc496dfe700 1 -- 172.21.15.94:0/2834898822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc490101c40 msgr2=0x7fc490101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.465+0000 7fc496dfe700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc490101c40 0x7fc490101690 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc48400d980 tx=0x7fc48400dc90 comp rx=0 tx=0).stop 2024-04-21T08:32:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.465+0000 7fc496dfe700 1 -- 172.21.15.94:0/2834898822 shutdown_connections 2024-04-21T08:32:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.465+0000 7fc496dfe700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc480039890 0x7fc48003bd50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.465+0000 7fc496dfe700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc490103070 0x7fc49019ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.465+0000 7fc496dfe700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc490101c40 0x7fc490101690 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.465+0000 7fc496dfe700 1 --2- 172.21.15.94:0/2834898822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc490102600 0x7fc49019d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.465+0000 7fc496dfe700 1 -- 172.21.15.94:0/2834898822 >> 172.21.15.94:0/2834898822 conn(0x7fc4900fc710 msgr2=0x7fc490105f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:32:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.466+0000 7fc496dfe700 1 -- 172.21.15.94:0/2834898822 shutdown_connections 2024-04-21T08:32:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:04.466+0000 7fc496dfe700 1 -- 172.21.15.94:0/2834898822 wait complete. 2024-04-21T08:32:04.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:04 smithi094 ceph-mon[27988]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:04.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:04.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:04.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:04.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:04 smithi111 ceph-mon[31493]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:05.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:04 smithi173 ceph-mon[31428]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:05.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:05.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:05.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:05.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:05.112 DEBUG:teuthology.orchestra.run.smithi094:osd.0> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.0.service 2024-04-21T08:32:05.114 INFO:tasks.cephadm:Deploying osd.1 on smithi111 with /dev/vg_nvme/lv_4... 2024-04-21T08:32:05.114 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:32:05.367 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:32:05.748 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:05 smithi111 ceph-mon[31493]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:32:05.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:05 smithi094 ceph-mon[27988]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:32:06.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:05 smithi173 ceph-mon[31428]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:32:06.634 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:06 smithi111 ceph-mon[31493]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:06.634 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:06 smithi111 ceph-mon[31493]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:32:06.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:06 smithi111 ceph-mon[31493]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:32:06.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:06 smithi111 ceph-mon[31493]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:32:06.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:06 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:06.848 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:06 smithi094 ceph-mon[27988]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:06.848 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:06 smithi094 ceph-mon[27988]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:32:06.848 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:06 smithi094 ceph-mon[27988]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:32:06.848 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:06 smithi094 ceph-mon[27988]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:32:06.848 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:06 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:07.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:06 smithi173 ceph-mon[31428]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:07.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:06 smithi173 ceph-mon[31428]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:32:07.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:06 smithi173 ceph-mon[31428]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:32:07.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:06 smithi173 ceph-mon[31428]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:32:07.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:06 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:07.182 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:32:07.182 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:32:07.182 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T08:32:07.182 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T08:32:07.182 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114954 s, 912 MB/s 2024-04-21T08:32:07.182 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T08:32:07.260 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:32:07 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0[42388]: 2024-04-21T08:32:07.066+0000 7fa22d972700 -1 osd.0 0 waiting for initial osdmap 2024-04-21T08:32:07.260 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:32:07 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0[42388]: 2024-04-21T08:32:07.069+0000 7fa22575d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:32:07.750 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:07 smithi094 ceph-mon[27988]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:32:07.750 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:07 smithi094 ceph-mon[27988]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:32:07.750 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:07.750 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:07.750 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:07 smithi094 ceph-mon[27988]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' 2024-04-21T08:32:07.753 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:07 smithi111 ceph-mon[31493]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:32:07.753 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:07 smithi111 ceph-mon[31493]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:32:07.754 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:07.754 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:07.754 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:07 smithi111 ceph-mon[31493]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' 2024-04-21T08:32:07.783 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_4 2024-04-21T08:32:08.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:07 smithi173 ceph-mon[31428]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:32:08.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:07 smithi173 ceph-mon[31428]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:32:08.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:07 smithi173 ceph-mon[31428]: from='osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749]' entity='osd.0' 2024-04-21T08:32:08.022 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:32:08.835 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:32:08.835 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:32:08.835 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] boot 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:08.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:32:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:32:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] boot 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:09.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: osd.0 [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] boot 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:32:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:09.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:09.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:09.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:09.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:09.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.632+0000 7f4dfbece700 1 -- 172.21.15.111:0/1261204191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df4104160 msgr2=0x7f4df4104580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.632+0000 7f4dfbece700 1 --2- 172.21.15.111:0/1261204191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df4104160 0x7f4df4104580 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4df00099c0 tx=0x7f4df0009cd0 comp rx=0 tx=0).stop 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.632+0000 7f4dfbece700 1 -- 172.21.15.111:0/1261204191 shutdown_connections 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.632+0000 7f4dfbece700 1 --2- 172.21.15.111:0/1261204191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df4104160 0x7f4df4104580 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.632+0000 7f4dfbece700 1 --2- 172.21.15.111:0/1261204191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4df41015a0 0x7f4df410b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.632+0000 7f4dfbece700 1 --2- 172.21.15.111:0/1261204191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4df4100be0 0x7f4df4101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.632+0000 7f4dfbece700 1 -- 172.21.15.111:0/1261204191 >> 172.21.15.111:0/1261204191 conn(0x7f4df40fc780 msgr2=0x7f4df40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.632+0000 7f4dfbece700 1 -- 172.21.15.111:0/1261204191 shutdown_connections 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.632+0000 7f4dfbece700 1 -- 172.21.15.111:0/1261204191 wait complete. 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4dfbece700 1 Processor -- start 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4dfbece700 1 -- start start 2024-04-21T08:32:09.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4dfbece700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df4100be0 0x7f4df4103860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4dfbece700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4df41015a0 0x7f4df4101eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4dfbece700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4df4104160 0x7f4df41023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4dfbece700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4df410e640 con 0x7f4df41015a0 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4dfbece700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4df410e4c0 con 0x7f4df4100be0 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4dfbece700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4df410e7c0 con 0x7f4df4104160 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4df9c6a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df4100be0 0x7f4df4103860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4df9c6a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df4100be0 0x7f4df4103860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32966/0 (socket says 172.21.15.111:32966) 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4df9c6a700 1 -- 172.21.15.111:0/3255170146 learned_addr learned my addr 172.21.15.111:0/3255170146 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4dfa46b700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4df4104160 0x7f4df41023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.633+0000 7f4df9469700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4df41015a0 0x7f4df4101eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4df9c6a700 1 -- 172.21.15.111:0/3255170146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4df4104160 msgr2=0x7f4df41023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4df9c6a700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4df4104160 0x7f4df41023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4df9c6a700 1 -- 172.21.15.111:0/3255170146 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4df41015a0 msgr2=0x7f4df4101eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4df9c6a700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4df41015a0 0x7f4df4101eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:09.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4df9c6a700 1 -- 172.21.15.111:0/3255170146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df4102ca0 con 0x7f4df4100be0 2024-04-21T08:32:09.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4df9c6a700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df4100be0 0x7f4df4103860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4df0006010 tx=0x7f4df00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:32:09.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4deaffd700 1 -- 172.21.15.111:0/3255170146 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df0027070 con 0x7f4df4100be0 2024-04-21T08:32:09.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4dfbece700 1 -- 172.21.15.111:0/3255170146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df41a2de0 con 0x7f4df4100be0 2024-04-21T08:32:09.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4deaffd700 1 -- 172.21.15.111:0/3255170146 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4df00040e0 con 0x7f4df4100be0 2024-04-21T08:32:09.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4deaffd700 1 -- 172.21.15.111:0/3255170146 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df000f460 con 0x7f4df4100be0 2024-04-21T08:32:09.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.634+0000 7f4dfbece700 1 -- 172.21.15.111:0/3255170146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df41a3180 con 0x7f4df4100be0 2024-04-21T08:32:09.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.635+0000 7f4dfbece700 1 -- 172.21.15.111:0/3255170146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df4069730 con 0x7f4df4100be0 2024-04-21T08:32:09.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.636+0000 7f4deaffd700 1 -- 172.21.15.111:0/3255170146 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f4df002a070 con 0x7f4df4100be0 2024-04-21T08:32:09.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.636+0000 7f4deaffd700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4de00a3bb0 0x7f4de00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:09.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.637+0000 7f4deaffd700 1 -- 172.21.15.111:0/3255170146 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f4df00cf7c0 con 0x7f4df4100be0 2024-04-21T08:32:09.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.637+0000 7f4df9469700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4de00a3bb0 0x7f4de00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:09.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.637+0000 7f4df9469700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4de00a3bb0 0x7f4de00a6070 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4df406b140 tx=0x7f4de400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:32:09.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.641+0000 7f4deaffd700 1 -- 172.21.15.111:0/3255170146 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4df005eff0 con 0x7f4df4100be0 2024-04-21T08:32:09.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:09 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi094 2024-04-21T08:32:09.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:09 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi094 to 17088M 2024-04-21T08:32:09.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:09.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:09.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:09.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:09.850+0000 7f4dfbece700 1 -- 172.21.15.111:0/3255170146 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4df4062e80 con 0x7f4de00a3bb0 2024-04-21T08:32:09.912 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:09 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi094 2024-04-21T08:32:09.912 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:09 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi094 to 17088M 2024-04-21T08:32:09.912 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:09.912 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:09.912 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:10.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:09 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi094 2024-04-21T08:32:10.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:09 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi094 to 17088M 2024-04-21T08:32:10.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:10.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:10.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:10.769 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:10 smithi111 ceph-mon[31493]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:10.769 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:10 smithi111 ceph-mon[31493]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:32:10.769 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:32:10.769 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:32:10.769 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:11.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:10 smithi094 ceph-mon[27988]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:11.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:10 smithi094 ceph-mon[27988]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:32:11.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:32:11.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:32:11.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:11.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:10 smithi173 ceph-mon[31428]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:11.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:10 smithi173 ceph-mon[31428]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:32:11.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:32:11.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:32:11.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:11.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:11 smithi111 ceph-mon[31493]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:32:12.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:11 smithi094 ceph-mon[27988]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:32:12.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:11 smithi173 ceph-mon[31428]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:32:12.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:12 smithi111 ceph-mon[31493]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:12.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:12 smithi111 ceph-mon[31493]: from='client.? 172.21.15.111:0/87527204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "188e7b8f-e006-4b64-8136-9e71097e107f"}]: dispatch 2024-04-21T08:32:12.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:12 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "188e7b8f-e006-4b64-8136-9e71097e107f"}]: dispatch 2024-04-21T08:32:12.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:12 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "188e7b8f-e006-4b64-8136-9e71097e107f"}]': finished 2024-04-21T08:32:12.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:12 smithi111 ceph-mon[31493]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:32:12.858 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:12 smithi094 ceph-mon[27988]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:12 smithi094 ceph-mon[27988]: from='client.? 172.21.15.111:0/87527204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "188e7b8f-e006-4b64-8136-9e71097e107f"}]: dispatch 2024-04-21T08:32:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:12 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "188e7b8f-e006-4b64-8136-9e71097e107f"}]: dispatch 2024-04-21T08:32:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:12 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "188e7b8f-e006-4b64-8136-9e71097e107f"}]': finished 2024-04-21T08:32:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:12 smithi094 ceph-mon[27988]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:32:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:12 smithi173 ceph-mon[31428]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:12 smithi173 ceph-mon[31428]: from='client.? 172.21.15.111:0/87527204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "188e7b8f-e006-4b64-8136-9e71097e107f"}]: dispatch 2024-04-21T08:32:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:12 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "188e7b8f-e006-4b64-8136-9e71097e107f"}]: dispatch 2024-04-21T08:32:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:12 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "188e7b8f-e006-4b64-8136-9e71097e107f"}]': finished 2024-04-21T08:32:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:12 smithi173 ceph-mon[31428]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:32:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:13.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:13 smithi111 ceph-mon[31493]: from='client.? 172.21.15.111:0/3766162487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:32:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:13 smithi094 ceph-mon[27988]: from='client.? 172.21.15.111:0/3766162487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:32:14.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:13 smithi173 ceph-mon[31428]: from='client.? 172.21.15.111:0/3766162487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:32:14.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:14 smithi111 ceph-mon[31493]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:15.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:14 smithi094 ceph-mon[27988]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:15.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:14 smithi173 ceph-mon[31428]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:16.747 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:16 smithi111 ceph-mon[31493]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:17.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:16 smithi094 ceph-mon[27988]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:17.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:16 smithi173 ceph-mon[31428]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:18.868 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:18 smithi111 ceph-mon[31493]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:18 smithi094 ceph-mon[27988]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:19.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:18 smithi173 ceph-mon[31428]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:19.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:19 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:32:19.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:19 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:20.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:19 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:32:20.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:19 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:19 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:32:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:19 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:20.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:20 smithi111 ceph-mon[31493]: Deploying daemon osd.1 on smithi111 2024-04-21T08:32:20.798 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:20 smithi111 ceph-mon[31493]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:20.911 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:20 smithi094 ceph-mon[27988]: Deploying daemon osd.1 on smithi111 2024-04-21T08:32:20.911 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:20 smithi094 ceph-mon[27988]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:21.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:20 smithi173 ceph-mon[31428]: Deploying daemon osd.1 on smithi111 2024-04-21T08:32:21.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:20 smithi173 ceph-mon[31428]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:22.776 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:22 smithi111 ceph-mon[31493]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:23.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:22 smithi094 ceph-mon[27988]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:23.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:22 smithi173 ceph-mon[31428]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:24.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:24 smithi111 ceph-mon[31493]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:24.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:24.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:24.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:25.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:24 smithi094 ceph-mon[27988]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:25.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:25.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:25.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:25.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:24 smithi173 ceph-mon[31428]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:25.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:25.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:25.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:26 smithi111 ceph-mon[31493]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:26 smithi094 ceph-mon[27988]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:27.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:26 smithi173 ceph-mon[31428]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:28.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.062+0000 7f4deaffd700 1 -- 172.21.15.111:0/3255170146 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4df4062e80 con 0x7f4de00a3bb0 2024-04-21T08:32:28.062 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 1 on host 'smithi111' 2024-04-21T08:32:28.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.064+0000 7f4dfbece700 1 -- 172.21.15.111:0/3255170146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4de00a3bb0 msgr2=0x7f4de00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:28.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.064+0000 7f4dfbece700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4de00a3bb0 0x7f4de00a6070 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4df406b140 tx=0x7f4de400b410 comp rx=0 tx=0).stop 2024-04-21T08:32:28.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.064+0000 7f4dfbece700 1 -- 172.21.15.111:0/3255170146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df4100be0 msgr2=0x7f4df4103860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:28.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.064+0000 7f4dfbece700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df4100be0 0x7f4df4103860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4df0006010 tx=0x7f4df00047b0 comp rx=0 tx=0).stop 2024-04-21T08:32:28.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.064+0000 7f4dfbece700 1 -- 172.21.15.111:0/3255170146 shutdown_connections 2024-04-21T08:32:28.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.065+0000 7f4dfbece700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4de00a3bb0 0x7f4de00a6070 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:28.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.065+0000 7f4dfbece700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df4100be0 0x7f4df4103860 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:28.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.065+0000 7f4dfbece700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4df4104160 0x7f4df41023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:28.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.065+0000 7f4dfbece700 1 --2- 172.21.15.111:0/3255170146 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4df41015a0 0x7f4df4101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:28.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.065+0000 7f4dfbece700 1 -- 172.21.15.111:0/3255170146 >> 172.21.15.111:0/3255170146 conn(0x7f4df40fc780 msgr2=0x7f4df4107200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:32:28.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.065+0000 7f4dfbece700 1 -- 172.21.15.111:0/3255170146 shutdown_connections 2024-04-21T08:32:28.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:32:28.065+0000 7f4dfbece700 1 -- 172.21.15.111:0/3255170146 wait complete. 2024-04-21T08:32:28.365 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:28.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:28.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:28 smithi111 ceph-mon[31493]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:28.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:28.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:28.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:28.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:28 smithi094 ceph-mon[27988]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:28.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:28.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:28.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:28.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:28 smithi173 ceph-mon[31428]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:28.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:28.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:28.700 DEBUG:teuthology.orchestra.run.smithi111:osd.1> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.1.service 2024-04-21T08:32:28.702 INFO:tasks.cephadm:Deploying osd.2 on smithi173 with /dev/vg_nvme/lv_4... 2024-04-21T08:32:28.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:32:28.946 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:32:29.388 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:29 smithi111 ceph-mon[31493]: from='osd.1 [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:32:29.388 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:29 smithi111 ceph-mon[31493]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:32:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:29 smithi173 ceph-mon[31428]: from='osd.1 [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:32:29.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:29 smithi173 ceph-mon[31428]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:32:29.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:29 smithi094 ceph-mon[27988]: from='osd.1 [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:32:29.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:29 smithi094 ceph-mon[27988]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:32:30.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:30 smithi111 ceph-mon[31493]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:32:30.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:30 smithi111 ceph-mon[31493]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:32:30.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:30.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:30 smithi111 ceph-mon[31493]: from='osd.1 [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:32:30.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:30 smithi111 ceph-mon[31493]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:32:30.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:30 smithi111 ceph-mon[31493]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:30.438 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:32:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1[36106]: 2024-04-21T08:32:30.212+0000 7f90ceec4700 -1 osd.1 0 waiting for initial osdmap 2024-04-21T08:32:30.438 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:32:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1[36106]: 2024-04-21T08:32:30.217+0000 7f90cacb7700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:32:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:30 smithi094 ceph-mon[27988]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:32:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:30 smithi094 ceph-mon[27988]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:32:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:30 smithi094 ceph-mon[27988]: from='osd.1 [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:32:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:30 smithi094 ceph-mon[27988]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:32:30.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:30 smithi094 ceph-mon[27988]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:30 smithi173 ceph-mon[31428]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:32:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:30 smithi173 ceph-mon[31428]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:32:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:30 smithi173 ceph-mon[31428]: from='osd.1 [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:32:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:30 smithi173 ceph-mon[31428]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:32:30.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:30 smithi173 ceph-mon[31428]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:30.811 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:32:30.812 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:32:30.812 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-21T08:32:30.812 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-21T08:32:30.812 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114596 s, 915 MB/s 2024-04-21T08:32:30.812 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-21T08:32:31.405 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:31 smithi173 ceph-mon[31428]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:32:31.405 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:31 smithi173 ceph-mon[31428]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:32:31.405 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:31.405 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:31.420 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:31 smithi111 ceph-mon[31493]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:32:31.420 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:31 smithi111 ceph-mon[31493]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:32:31.421 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:31.421 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:31.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-04-21T08:32:31.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:31 smithi094 ceph-mon[27988]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:32:31.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:31 smithi094 ceph-mon[27988]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:32:31.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:31.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:31.671 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: osd.1 [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] boot 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi111 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-21T08:32:32.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:32.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:32.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:32 smithi173 ceph-mon[31428]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:32.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:32:32.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:32:32.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: osd.1 [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] boot 2024-04-21T08:32:32.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:32:32.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:32.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi111 2024-04-21T08:32:32.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:32:32.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-21T08:32:32.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:32.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:32.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:32 smithi094 ceph-mon[27988]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:32:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: osd.1 [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] boot 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi111 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:32 smithi111 ceph-mon[31493]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:32:33.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.166+0000 7fa60875c700 1 -- 172.21.15.173:0/1089788068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6001019a0 msgr2=0x7fa600101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:33.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.166+0000 7fa60875c700 1 --2- 172.21.15.173:0/1089788068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6001019a0 0x7fa600101dc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000b0a0 tx=0x7fa5f000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:32:33.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.166+0000 7fa60875c700 1 -- 172.21.15.173:0/1089788068 shutdown_connections 2024-04-21T08:32:33.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.166+0000 7fa60875c700 1 --2- 172.21.15.173:0/1089788068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa600103540 0x7fa600109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:33.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.166+0000 7fa60875c700 1 --2- 172.21.15.173:0/1089788068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6001019a0 0x7fa600101dc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:33.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.166+0000 7fa60875c700 1 --2- 172.21.15.173:0/1089788068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa600102ba0 0x7fa600103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:33.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.166+0000 7fa60875c700 1 -- 172.21.15.173:0/1089788068 >> 172.21.15.173:0/1089788068 conn(0x7fa6000fcf40 msgr2=0x7fa6000ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:32:33.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.166+0000 7fa60875c700 1 -- 172.21.15.173:0/1089788068 shutdown_connections 2024-04-21T08:32:33.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.166+0000 7fa60875c700 1 -- 172.21.15.173:0/1089788068 wait complete. 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa60875c700 1 Processor -- start 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa60875c700 1 -- start start 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa60875c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6001019a0 0x7fa600198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa60875c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa600102ba0 0x7fa6001992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa60875c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa600103540 0x7fa6001a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa60875c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa60010d4e0 con 0x7fa6001019a0 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa60875c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa60010d360 con 0x7fa600102ba0 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa60875c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa60010d660 con 0x7fa600103540 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa606cf9700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa600103540 0x7fa6001a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa606cf9700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa600103540 0x7fa6001a0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:42448/0 (socket says 172.21.15.173:42448) 2024-04-21T08:32:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa606cf9700 1 -- 172.21.15.173:0/3064740770 learned_addr learned my addr 172.21.15.173:0/3064740770 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa6064f8700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6001019a0 0x7fa600198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa605cf7700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa600102ba0 0x7fa6001992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa606cf9700 1 -- 172.21.15.173:0/3064740770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa600102ba0 msgr2=0x7fa6001992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa606cf9700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa600102ba0 0x7fa6001992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa606cf9700 1 -- 172.21.15.173:0/3064740770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6001019a0 msgr2=0x7fa600198d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa606cf9700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6001019a0 0x7fa600198d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.167+0000 7fa606cf9700 1 -- 172.21.15.173:0/3064740770 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6001a09a0 con 0x7fa600103540 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.168+0000 7fa606cf9700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa600103540 0x7fa6001a0300 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800d9b0 tx=0x7fa5f800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.168+0000 7fa5ff7fe700 1 -- 172.21.15.173:0/3064740770 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8017070 con 0x7fa600103540 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.168+0000 7fa60875c700 1 -- 172.21.15.173:0/3064740770 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6001a0c90 con 0x7fa600103540 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.168+0000 7fa5ff7fe700 1 -- 172.21.15.173:0/3064740770 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5f8005020 con 0x7fa600103540 2024-04-21T08:32:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.168+0000 7fa5ff7fe700 1 -- 172.21.15.173:0/3064740770 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8009990 con 0x7fa600103540 2024-04-21T08:32:33.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.168+0000 7fa60875c700 1 -- 172.21.15.173:0/3064740770 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6001a1200 con 0x7fa600103540 2024-04-21T08:32:33.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.169+0000 7fa60875c700 1 -- 172.21.15.173:0/3064740770 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa600069730 con 0x7fa600103540 2024-04-21T08:32:33.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.170+0000 7fa5ff7fe700 1 -- 172.21.15.173:0/3064740770 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fa5f8004a30 con 0x7fa600103540 2024-04-21T08:32:33.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.170+0000 7fa5ff7fe700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa5e80a3bb0 0x7fa5e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:33.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.170+0000 7fa5ff7fe700 1 -- 172.21.15.173:0/3064740770 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fa5f80ce8a0 con 0x7fa600103540 2024-04-21T08:32:33.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.171+0000 7fa6064f8700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa5e80a3bb0 0x7fa5e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:33.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.171+0000 7fa6064f8700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa5e80a3bb0 0x7fa5e80a6070 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009fd0 tx=0x7fa5f000b7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:32:33.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.175+0000 7fa5ff7fe700 1 -- 172.21.15.173:0/3064740770 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5f8020020 con 0x7fa600103540 2024-04-21T08:32:33.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:33 smithi173 ceph-mon[31428]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:32:33.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:33.384+0000 7fa60875c700 1 -- 172.21.15.173:0/3064740770 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa60006ae30 con 0x7fa5e80a3bb0 2024-04-21T08:32:33.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:33 smithi094 ceph-mon[27988]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:32:33.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:33 smithi111 ceph-mon[31493]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:32:34.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:34 smithi173 ceph-mon[31428]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:32:34.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:34 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:32:34.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:34 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:32:34.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:34 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:34.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:34 smithi173 ceph-mon[31428]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:34.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:34 smithi094 ceph-mon[27988]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:32:34.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:32:34.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:32:34.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:34.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:34 smithi094 ceph-mon[27988]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:34.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:34 smithi111 ceph-mon[31493]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:32:34.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:34 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:32:34.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:34 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:32:34.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:34 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:34.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:34 smithi111 ceph-mon[31493]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:36 smithi111 ceph-mon[31493]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:36 smithi111 ceph-mon[31493]: from='client.? 172.21.15.173:0/1723203000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f3dff21-6860-4c3b-9c15-3f120e0f3120"}]: dispatch 2024-04-21T08:32:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:36 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f3dff21-6860-4c3b-9c15-3f120e0f3120"}]: dispatch 2024-04-21T08:32:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:36 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f3dff21-6860-4c3b-9c15-3f120e0f3120"}]': finished 2024-04-21T08:32:36.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:36 smithi111 ceph-mon[31493]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:32:36.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:36.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:36 smithi111 ceph-mon[31493]: from='client.? 172.21.15.173:0/3123883589' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:32:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:36 smithi094 ceph-mon[27988]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:36 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/1723203000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f3dff21-6860-4c3b-9c15-3f120e0f3120"}]: dispatch 2024-04-21T08:32:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:36 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f3dff21-6860-4c3b-9c15-3f120e0f3120"}]: dispatch 2024-04-21T08:32:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:36 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f3dff21-6860-4c3b-9c15-3f120e0f3120"}]': finished 2024-04-21T08:32:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:36 smithi094 ceph-mon[27988]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:32:37.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:37.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:36 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/3123883589' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:32:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:36 smithi173 ceph-mon[31428]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:36 smithi173 ceph-mon[31428]: from='client.? 172.21.15.173:0/1723203000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f3dff21-6860-4c3b-9c15-3f120e0f3120"}]: dispatch 2024-04-21T08:32:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:36 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f3dff21-6860-4c3b-9c15-3f120e0f3120"}]: dispatch 2024-04-21T08:32:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:36 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f3dff21-6860-4c3b-9c15-3f120e0f3120"}]': finished 2024-04-21T08:32:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:36 smithi173 ceph-mon[31428]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:32:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:36 smithi173 ceph-mon[31428]: from='client.? 172.21.15.173:0/3123883589' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:32:38.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:38 smithi111 ceph-mon[31493]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:39.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:38 smithi094 ceph-mon[27988]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:39.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:38 smithi173 ceph-mon[31428]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:40.860 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:40 smithi173 ceph-mon[31428]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:40.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:40 smithi111 ceph-mon[31493]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:41.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:40 smithi094 ceph-mon[27988]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:42.864 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:42 smithi173 ceph-mon[31428]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:42.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:42 smithi111 ceph-mon[31493]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:43.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:42 smithi094 ceph-mon[27988]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:43.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:43 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:32:43.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:43 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:43.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:43 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:32:43.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:43 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:44.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:43 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:32:44.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:43 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:44.739 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:44 smithi173 ceph-mon[31428]: Deploying daemon osd.2 on smithi173 2024-04-21T08:32:44.739 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:44 smithi173 ceph-mon[31428]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:44.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:44 smithi111 ceph-mon[31493]: Deploying daemon osd.2 on smithi173 2024-04-21T08:32:44.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:44 smithi111 ceph-mon[31493]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:45.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:44 smithi094 ceph-mon[27988]: Deploying daemon osd.2 on smithi173 2024-04-21T08:32:45.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:44 smithi094 ceph-mon[27988]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:46.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:46 smithi111 ceph-mon[31493]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:47.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:46 smithi094 ceph-mon[27988]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:47.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:46 smithi173 ceph-mon[31428]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:48.916 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:48 smithi173 ceph-mon[31428]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:48.916 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:48 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:48.916 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:48 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:48.916 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:48 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:48.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:48 smithi111 ceph-mon[31493]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:48.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:48 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:48.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:48 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:48.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:48 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:49.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:48 smithi094 ceph-mon[27988]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:49.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:48 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:49.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:48 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:49.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:48 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:50.697 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:50 smithi173 ceph-mon[31428]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:50.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:50 smithi111 ceph-mon[31493]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:50 smithi094 ceph-mon[27988]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:51.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.849+0000 7fa5ff7fe700 1 -- 172.21.15.173:0/3064740770 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa60006ae30 con 0x7fa5e80a3bb0 2024-04-21T08:32:51.849 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 2 on host 'smithi173' 2024-04-21T08:32:51.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 -- 172.21.15.173:0/3064740770 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa5e80a3bb0 msgr2=0x7fa5e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:51.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa5e80a3bb0 0x7fa5e80a6070 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009fd0 tx=0x7fa5f000b7f0 comp rx=0 tx=0).stop 2024-04-21T08:32:51.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 -- 172.21.15.173:0/3064740770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa600103540 msgr2=0x7fa6001a0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:51.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa600103540 0x7fa6001a0300 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800d9b0 tx=0x7fa5f800dcc0 comp rx=0 tx=0).stop 2024-04-21T08:32:51.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 -- 172.21.15.173:0/3064740770 shutdown_connections 2024-04-21T08:32:51.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa5e80a3bb0 0x7fa5e80a6070 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:51.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa600103540 0x7fa6001a0300 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:51.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6001019a0 0x7fa600198d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:51.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 --2- 172.21.15.173:0/3064740770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa600102ba0 0x7fa6001992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:51.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 -- 172.21.15.173:0/3064740770 >> 172.21.15.173:0/3064740770 conn(0x7fa6000fcf40 msgr2=0x7fa60006a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:32:51.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 -- 172.21.15.173:0/3064740770 shutdown_connections 2024-04-21T08:32:51.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:32:51.852+0000 7fa60875c700 1 -- 172.21.15.173:0/3064740770 wait complete. 2024-04-21T08:32:52.139 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.139 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.139 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:52.139 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:52.139 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.139 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:51 smithi173 ceph-mon[31428]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:52.139 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.139 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.139 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:52.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:52.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:52.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:51 smithi094 ceph-mon[27988]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:52.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:52.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:52.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:52.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:51 smithi111 ceph-mon[31493]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:52.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:52.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:32:52.452 DEBUG:teuthology.orchestra.run.smithi173:osd.2> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.2.service 2024-04-21T08:32:52.454 INFO:tasks.cephadm:Deploying osd.3 on smithi094 with /dev/vg_nvme/lv_3... 2024-04-21T08:32:52.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:32:52.702 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:32:53.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:52 smithi173 ceph-mon[31428]: from='osd.2 [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:32:53.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:52 smithi173 ceph-mon[31428]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:32:53.097 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:52 smithi094 ceph-mon[27988]: from='osd.2 [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:32:53.097 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:52 smithi094 ceph-mon[27988]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:32:53.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:52 smithi111 ceph-mon[31493]: from='osd.2 [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:32:53.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:52 smithi111 ceph-mon[31493]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:32:54.127 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:54 smithi094 ceph-mon[27988]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:32:54.127 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:54 smithi094 ceph-mon[27988]: from='osd.2 [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:32:54.127 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:54 smithi094 ceph-mon[27988]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:32:54.127 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:54 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:54.127 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:54 smithi094 ceph-mon[27988]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:32:54.127 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:54 smithi094 ceph-mon[27988]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:54.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:54 smithi173 ceph-mon[31428]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:32:54.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:54 smithi173 ceph-mon[31428]: from='osd.2 [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:32:54.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:54 smithi173 ceph-mon[31428]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:32:54.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:54 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:54.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:54 smithi173 ceph-mon[31428]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:32:54.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:54 smithi173 ceph-mon[31428]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:54.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:54 smithi111 ceph-mon[31493]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:32:54.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:54 smithi111 ceph-mon[31493]: from='osd.2 [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:32:54.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:54 smithi111 ceph-mon[31493]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:32:54.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:54 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:54.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:54 smithi111 ceph-mon[31493]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:32:54.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:54 smithi111 ceph-mon[31493]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:54.654 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:32:54.654 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:32:54.655 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-04-21T08:32:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-04-21T08:32:54.655 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0101196 s, 1.0 GB/s 2024-04-21T08:32:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-04-21T08:32:54.759 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:32:54 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2[36601]: 2024-04-21T08:32:54.423+0000 7fd3c719f700 -1 osd.2 0 waiting for initial osdmap 2024-04-21T08:32:54.759 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:32:54 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2[36601]: 2024-04-21T08:32:54.427+0000 7fd3be789700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:32:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:55 smithi173 ceph-mon[31428]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:32:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:55 smithi173 ceph-mon[31428]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:32:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:55 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:55.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:55 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:55.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:55 smithi173 ceph-mon[31428]: from='osd.2 ' entity='osd.2' 2024-04-21T08:32:55.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:55 smithi094 ceph-mon[27988]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:32:55.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:55 smithi094 ceph-mon[27988]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:32:55.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:55 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:55.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:55 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:55.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:55 smithi094 ceph-mon[27988]: from='osd.2 ' entity='osd.2' 2024-04-21T08:32:55.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_3 2024-04-21T08:32:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:55 smithi111 ceph-mon[31493]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:32:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:55 smithi111 ceph-mon[31493]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:32:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:55 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:55 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:55 smithi111 ceph-mon[31493]: from='osd.2 ' entity='osd.2' 2024-04-21T08:32:55.572 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:32:56.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:32:56.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: osd.2 [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] boot 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi173 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-21T08:32:56.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:56.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:56.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:32:56.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:32:56.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:32:56.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:56.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: osd.2 [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] boot 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi173 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:32:56.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: osd.2 [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] boot 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi173 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-21T08:32:56.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.439 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:56.439 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:32:56.439 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:56.439 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.165+0000 7fa08368b700 1 -- 172.21.15.94:0/656267984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa07c102aa0 msgr2=0x7fa07c102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.165+0000 7fa08368b700 1 --2- 172.21.15.94:0/656267984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa07c102aa0 0x7fa07c102ec0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa0780099c0 tx=0x7fa078009cd0 comp rx=0 tx=0).stop 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 -- 172.21.15.94:0/656267984 shutdown_connections 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 --2- 172.21.15.94:0/656267984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa07c104640 0x7fa07c10aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 --2- 172.21.15.94:0/656267984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa07c102aa0 0x7fa07c102ec0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 --2- 172.21.15.94:0/656267984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07c103ca0 0x7fa07c104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 -- 172.21.15.94:0/656267984 >> 172.21.15.94:0/656267984 conn(0x7fa07c0fe040 msgr2=0x7fa07c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 -- 172.21.15.94:0/656267984 shutdown_connections 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 -- 172.21.15.94:0/656267984 wait complete. 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 Processor -- start 2024-04-21T08:32:57.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 -- start start 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa07c102aa0 0x7fa07c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa07c103ca0 0x7fa07c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07c104640 0x7fa07c1a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.166+0000 7fa08368b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa07c10e560 con 0x7fa07c103ca0 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa08368b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa07c10e3e0 con 0x7fa07c104640 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa08368b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa07c10e6e0 con 0x7fa07c102aa0 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa080c26700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa07c103ca0 0x7fa07c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa080c26700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa07c103ca0 0x7fa07c1991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51218/0 (socket says 172.21.15.94:51218) 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa080c26700 1 -- 172.21.15.94:0/4119153683 learned_addr learned my addr 172.21.15.94:0/4119153683 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa081427700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa07c102aa0 0x7fa07c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa080c26700 1 -- 172.21.15.94:0/4119153683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa07c102aa0 msgr2=0x7fa07c198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa080c26700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa07c102aa0 0x7fa07c198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa080c26700 1 -- 172.21.15.94:0/4119153683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07c104640 msgr2=0x7fa07c1a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:32:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa081c28700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07c104640 0x7fa07c1a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:57.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa080c26700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07c104640 0x7fa07c1a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:32:57.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa080c26700 1 -- 172.21.15.94:0/4119153683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07c1a08b0 con 0x7fa07c103ca0 2024-04-21T08:32:57.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa080c26700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa07c103ca0 0x7fa07c1991a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00c920 tx=0x7fa06c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:32:57.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa0727fc700 1 -- 172.21.15.94:0/4119153683 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06c007b00 con 0x7fa07c103ca0 2024-04-21T08:32:57.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa08368b700 1 -- 172.21.15.94:0/4119153683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa07c1a0ba0 con 0x7fa07c103ca0 2024-04-21T08:32:57.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa0727fc700 1 -- 172.21.15.94:0/4119153683 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa06c004ce0 con 0x7fa07c103ca0 2024-04-21T08:32:57.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa0727fc700 1 -- 172.21.15.94:0/4119153683 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06c005500 con 0x7fa07c103ca0 2024-04-21T08:32:57.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.167+0000 7fa08368b700 1 -- 172.21.15.94:0/4119153683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa07c1a1100 con 0x7fa07c103ca0 2024-04-21T08:32:57.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.168+0000 7fa08368b700 1 -- 172.21.15.94:0/4119153683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa07c069730 con 0x7fa07c103ca0 2024-04-21T08:32:57.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.169+0000 7fa0727fc700 1 -- 172.21.15.94:0/4119153683 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fa06c007ca0 con 0x7fa07c103ca0 2024-04-21T08:32:57.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.170+0000 7fa0727fc700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0680a3bb0 0x7fa0680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:32:57.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.170+0000 7fa081427700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0680a3bb0 0x7fa0680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:32:57.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.170+0000 7fa0727fc700 1 -- 172.21.15.94:0/4119153683 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7fa06c0cee80 con 0x7fa07c103ca0 2024-04-21T08:32:57.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.170+0000 7fa081427700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0680a3bb0 0x7fa0680a6070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa078009f50 tx=0x7fa07800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:32:57.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.174+0000 7fa0727fc700 1 -- 172.21.15.94:0/4119153683 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa06c014030 con 0x7fa07c103ca0 2024-04-21T08:32:57.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:57.384+0000 7fa08368b700 1 -- 172.21.15.94:0/4119153683 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa07c062e80 con 0x7fa0680a3bb0 2024-04-21T08:32:57.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:32:57.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 ceph-mon[27988]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:32:57.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:32:57.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:32:57.676 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:32:57.676 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:57.813 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40665]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:32:57.813 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40665]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:57.813 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40665]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:57.813 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40665]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:57.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:32:57.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 ceph-mon[31428]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:32:57.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:32:57.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:32:57.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:32:57.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:57.932 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:32:57.932 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 ceph-mon[31493]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:32:57.932 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:32:57.932 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:32:57.932 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:32:57.932 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:32:57.933 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40231]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:32:57.933 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40231]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:57.933 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40231]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:57.933 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40231]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:57.933 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40234]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:32:57.933 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40234]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:57.933 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40234]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:57.933 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40234]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:32:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47035]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47035]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47050]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:32:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47050]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47050]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47050]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:58.010 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47014]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:32:58.011 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47014]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:58.011 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47014]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:58.011 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47014]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:58.011 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47017]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:32:58.011 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:58.011 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47017]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:58.011 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:32:57 smithi094 sudo[47017]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:58.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40237]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:32:58.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40237]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:58.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40237]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:58.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40237]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:58.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40240]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:32:58.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40240]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:58.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40240]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:58.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:57 smithi111 sudo[40240]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:58.260 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40668]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:32:58.260 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40668]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:58.260 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40668]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:58.260 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40668]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:58.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40671]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:32:58.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40671]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:58.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40671]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:58.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40671]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:58.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40674]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:32:58.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40674]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:32:58.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40674]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:32:58.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:57 smithi173 sudo[40674]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:32:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:32:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:32:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:32:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:32:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:32:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:32:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:32:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:32:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:32:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:32:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:58 smithi094 ceph-mon[27988]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:32:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:32:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:58 smithi111 ceph-mon[31493]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:32:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:32:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:32:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:32:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:32:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:32:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:32:59.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:32:59.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:32:59.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:58 smithi173 ceph-mon[31428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:32:59.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:32:59.572+0000 7fa0727fc700 1 -- 172.21.15.94:0/4119153683 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa06c098c90 con 0x7fa07c103ca0 2024-04-21T08:32:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:32:59 smithi111 ceph-mon[31493]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:33:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:32:59 smithi094 ceph-mon[27988]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:33:00.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:32:59 smithi173 ceph-mon[31428]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:33:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:00 smithi094 ceph-mon[27988]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:00 smithi094 ceph-mon[27988]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:33:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:00 smithi094 ceph-mon[27988]: Cluster is now healthy 2024-04-21T08:33:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:00 smithi094 ceph-mon[27988]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:33:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/966927744' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b30afbd0-4fe9-432a-ac54-7db53d055e37"}]: dispatch 2024-04-21T08:33:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/966927744' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b30afbd0-4fe9-432a-ac54-7db53d055e37"}]': finished 2024-04-21T08:33:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:00 smithi094 ceph-mon[27988]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T08:33:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:00 smithi111 ceph-mon[31493]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:00 smithi111 ceph-mon[31493]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:33:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:00 smithi111 ceph-mon[31493]: Cluster is now healthy 2024-04-21T08:33:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:00 smithi111 ceph-mon[31493]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:33:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/966927744' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b30afbd0-4fe9-432a-ac54-7db53d055e37"}]: dispatch 2024-04-21T08:33:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/966927744' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b30afbd0-4fe9-432a-ac54-7db53d055e37"}]': finished 2024-04-21T08:33:00.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:00 smithi111 ceph-mon[31493]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T08:33:00.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:01.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:00 smithi173 ceph-mon[31428]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:01.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:00 smithi173 ceph-mon[31428]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:33:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:00 smithi173 ceph-mon[31428]: Cluster is now healthy 2024-04-21T08:33:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:00 smithi173 ceph-mon[31428]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:33:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/966927744' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b30afbd0-4fe9-432a-ac54-7db53d055e37"}]: dispatch 2024-04-21T08:33:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/966927744' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b30afbd0-4fe9-432a-ac54-7db53d055e37"}]': finished 2024-04-21T08:33:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:00 smithi173 ceph-mon[31428]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T08:33:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:01 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4595314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:33:02.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:01 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4595314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:33:02.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:01 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4595314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:33:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:02 smithi111 ceph-mon[31493]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:03.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:02 smithi094 ceph-mon[27988]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:03.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:02 smithi173 ceph-mon[31428]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:04 smithi111 ceph-mon[31493]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:04 smithi094 ceph-mon[27988]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:05.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:04 smithi173 ceph-mon[31428]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:06.830 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:06 smithi094 ceph-mon[27988]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:06.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:06 smithi111 ceph-mon[31493]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:07.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:06 smithi173 ceph-mon[31428]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:07.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:33:07.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:07.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:33:07.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:08.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:33:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:08.748 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:08 smithi094 ceph-mon[27988]: Deploying daemon osd.3 on smithi094 2024-04-21T08:33:08.748 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:08 smithi094 ceph-mon[27988]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:08.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:08 smithi111 ceph-mon[31493]: Deploying daemon osd.3 on smithi094 2024-04-21T08:33:08.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:08 smithi111 ceph-mon[31493]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:09.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:08 smithi173 ceph-mon[31428]: Deploying daemon osd.3 on smithi094 2024-04-21T08:33:09.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:08 smithi173 ceph-mon[31428]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:10.847 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:10 smithi094 ceph-mon[27988]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:10.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:10 smithi111 ceph-mon[31493]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:11.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:10 smithi173 ceph-mon[31428]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:12.735 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:12 smithi094 ceph-mon[27988]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:12 smithi111 ceph-mon[31493]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:13.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:12 smithi173 ceph-mon[31428]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:14 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:14 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:14 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:14 smithi094 ceph-mon[27988]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:14.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:14 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:14.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:14 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:14 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:14 smithi173 ceph-mon[31428]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:14.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:14 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:14.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:14 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:14.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:14 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:14.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:14 smithi111 ceph-mon[31493]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:16.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.447+0000 7fa0727fc700 1 -- 172.21.15.94:0/4119153683 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa07c062e80 con 0x7fa0680a3bb0 2024-04-21T08:33:16.447 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 3 on host 'smithi094' 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 -- 172.21.15.94:0/4119153683 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0680a3bb0 msgr2=0x7fa0680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0680a3bb0 0x7fa0680a6070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa078009f50 tx=0x7fa07800b540 comp rx=0 tx=0).stop 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 -- 172.21.15.94:0/4119153683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa07c103ca0 msgr2=0x7fa07c1991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa07c103ca0 0x7fa07c1991a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00c920 tx=0x7fa06c00cc30 comp rx=0 tx=0).stop 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 -- 172.21.15.94:0/4119153683 shutdown_connections 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa0680a3bb0 0x7fa0680a6070 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07c104640 0x7fa07c1a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa07c102aa0 0x7fa07c198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 --2- 172.21.15.94:0/4119153683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa07c103ca0 0x7fa07c1991a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 -- 172.21.15.94:0/4119153683 >> 172.21.15.94:0/4119153683 conn(0x7fa07c0fe040 msgr2=0x7fa07c107110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 -- 172.21.15.94:0/4119153683 shutdown_connections 2024-04-21T08:33:16.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:33:16.449+0000 7fa08368b700 1 -- 172.21.15.94:0/4119153683 wait complete. 2024-04-21T08:33:16.755 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:16 smithi094 ceph-mon[27988]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:16.756 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:16.756 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:16.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:16 smithi111 ceph-mon[31493]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:16.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:16.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:17.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:16 smithi173 ceph-mon[31428]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:17.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:17.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:17.083 DEBUG:teuthology.orchestra.run.smithi094:osd.3> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.3.service 2024-04-21T08:33:17.086 INFO:tasks.cephadm:Deploying osd.4 on smithi111 with /dev/vg_nvme/lv_3... 2024-04-21T08:33:17.086 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:33:17.332 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:33:17.753 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:17 smithi111 ceph-mon[31493]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:33:17.753 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:17.753 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:17.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:17 smithi094 ceph-mon[27988]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:33:17.834 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:17.834 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:18.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:17 smithi173 ceph-mon[31428]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:33:18.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:18.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:18.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:18 smithi111 ceph-mon[31493]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:18.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:18 smithi111 ceph-mon[31493]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:33:18.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:18 smithi111 ceph-mon[31493]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:33:18.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:18 smithi111 ceph-mon[31493]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:33:18.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:18 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:18.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:18 smithi094 ceph-mon[27988]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:18.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:18 smithi094 ceph-mon[27988]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:33:18.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:18 smithi094 ceph-mon[27988]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:33:18.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:18 smithi094 ceph-mon[27988]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:33:18.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:18 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:18.964 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:33:18 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3[49208]: 2024-04-21T08:33:18.591+0000 7f2497002700 -1 osd.3 0 waiting for initial osdmap 2024-04-21T08:33:18.964 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:33:18 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3[49208]: 2024-04-21T08:33:18.596+0000 7f2492df5700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:33:19.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:18 smithi173 ceph-mon[31428]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:18 smithi173 ceph-mon[31428]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:33:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:18 smithi173 ceph-mon[31428]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:33:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:18 smithi173 ceph-mon[31428]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:33:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:18 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:19.313 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:33:19.313 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:33:19.313 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T08:33:19.313 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T08:33:19.313 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011273 s, 930 MB/s 2024-04-21T08:33:19.313 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T08:33:19.905 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:19 smithi111 ceph-mon[31493]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:33:19.905 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:19 smithi111 ceph-mon[31493]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:33:19.905 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:19 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:19.905 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:19 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:19.932 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_3 2024-04-21T08:33:19.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:19 smithi094 ceph-mon[27988]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:33:19.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:19 smithi094 ceph-mon[27988]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:33:19.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:19 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:19.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:19 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:19 smithi173 ceph-mon[31428]: from='osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:33:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:19 smithi173 ceph-mon[31428]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:33:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:19 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:20.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:19 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:20.164 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:33:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:20 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:33:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:20 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:33:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:20 smithi111 ceph-mon[31493]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:20 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:20 smithi111 ceph-mon[31493]: osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] boot 2024-04-21T08:33:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:20 smithi111 ceph-mon[31493]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T08:33:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:20 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:20 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:33:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:20 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:33:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:20 smithi094 ceph-mon[27988]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:20 smithi094 ceph-mon[27988]: osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] boot 2024-04-21T08:33:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:20 smithi094 ceph-mon[27988]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T08:33:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:20 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:33:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:20 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:33:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:20 smithi173 ceph-mon[31428]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:33:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:20 smithi173 ceph-mon[31428]: osd.3 [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] boot 2024-04-21T08:33:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:20 smithi173 ceph-mon[31428]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T08:33:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:33:21.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.804+0000 7f22fc70a700 1 -- 172.21.15.111:0/1636671258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f41087a0 msgr2=0x7f22f410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:33:21.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.804+0000 7f22fc70a700 1 --2- 172.21.15.111:0/1636671258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f41087a0 0x7f22f410ac10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00b0a0 tx=0x7f22ec00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:33:21.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.804+0000 7f22fc70a700 1 -- 172.21.15.111:0/1636671258 shutdown_connections 2024-04-21T08:33:21.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.804+0000 7f22fc70a700 1 --2- 172.21.15.111:0/1636671258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f41087a0 0x7f22f410ac10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:21.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.804+0000 7f22fc70a700 1 --2- 172.21.15.111:0/1636671258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22f4103540 0x7f22f4105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:21.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.804+0000 7f22fc70a700 1 --2- 172.21.15.111:0/1636671258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22f4105e70 0x7f22f4108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:21.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.804+0000 7f22fc70a700 1 -- 172.21.15.111:0/1636671258 >> 172.21.15.111:0/1636671258 conn(0x7f22f40fcf70 msgr2=0x7f22f40ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.804+0000 7f22fc70a700 1 -- 172.21.15.111:0/1636671258 shutdown_connections 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.804+0000 7f22fc70a700 1 -- 172.21.15.111:0/1636671258 wait complete. 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22fc70a700 1 Processor -- start 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22fc70a700 1 -- start start 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22fc70a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22f4103540 0x7f22f4198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22fc70a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22f4105e70 0x7f22f4199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22fc70a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f41087a0 0x7f22f41a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22fc70a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410e540 con 0x7f22f4105e70 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22fc70a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410e3c0 con 0x7f22f41087a0 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22fc70a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410e6c0 con 0x7f22f4103540 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22faca7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f41087a0 0x7f22f41a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:33:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22faca7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f41087a0 0x7f22f41a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54958/0 (socket says 172.21.15.111:54958) 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22faca7700 1 -- 172.21.15.111:0/308791892 learned_addr learned my addr 172.21.15.111:0/308791892 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22fa4a6700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22f4103540 0x7f22f4198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22f9ca5700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22f4105e70 0x7f22f4199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22faca7700 1 -- 172.21.15.111:0/308791892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22f4103540 msgr2=0x7f22f4198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22faca7700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22f4103540 0x7f22f4198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22faca7700 1 -- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22f4105e70 msgr2=0x7f22f4199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22faca7700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22f4105e70 0x7f22f4199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22faca7700 1 -- 172.21.15.111:0/308791892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f41a0890 con 0x7f22f41087a0 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22f9ca5700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22f4105e70 0x7f22f4199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22faca7700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f41087a0 0x7f22f41a01c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0062a0 tx=0x7f22ec004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.805+0000 7f22eb7fe700 1 -- 172.21.15.111:0/308791892 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec021070 con 0x7f22f41087a0 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.806+0000 7f22eb7fe700 1 -- 172.21.15.111:0/308791892 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22ec007c80 con 0x7f22f41087a0 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.806+0000 7f22fc70a700 1 -- 172.21.15.111:0/308791892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f41a0b20 con 0x7f22f41087a0 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.806+0000 7f22eb7fe700 1 -- 172.21.15.111:0/308791892 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec01d450 con 0x7f22f41087a0 2024-04-21T08:33:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.806+0000 7f22fc70a700 1 -- 172.21.15.111:0/308791892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f41a0fd0 con 0x7f22f41087a0 2024-04-21T08:33:21.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.806+0000 7f22fc70a700 1 -- 172.21.15.111:0/308791892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f4069730 con 0x7f22f41087a0 2024-04-21T08:33:21.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.808+0000 7f22eb7fe700 1 -- 172.21.15.111:0/308791892 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f22ec004000 con 0x7f22f41087a0 2024-04-21T08:33:21.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.808+0000 7f22eb7fe700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22e00a3cc0 0x7f22e00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:33:21.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.808+0000 7f22eb7fe700 1 -- 172.21.15.111:0/308791892 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f22ec0d05f0 con 0x7f22f41087a0 2024-04-21T08:33:21.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.808+0000 7f22fa4a6700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22e00a3cc0 0x7f22e00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:33:21.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.809+0000 7f22fa4a6700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22e00a3cc0 0x7f22e00a6180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f22f419a120 tx=0x7f22f00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:33:21.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:21.812+0000 7f22eb7fe700 1 -- 172.21.15.111:0/308791892 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22ec09e260 con 0x7f22f41087a0 2024-04-21T08:33:21.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:33:21.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi094 2024-04-21T08:33:21.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:21.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:21.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:21.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:21.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:21.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:21.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:21.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:21.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:21.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:21.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:21.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:33:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi094 2024-04-21T08:33:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:21.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:21.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:21.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:21.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:21.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:21.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:22.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:33:22.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi094 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:22.018 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:22.019+0000 7f22fc70a700 1 -- 172.21.15.111:0/308791892 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f22f4062e80 con 0x7f22e00a3cc0 2024-04-21T08:33:22.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:22 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi094 to 8544M 2024-04-21T08:33:22.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:22 smithi111 ceph-mon[31493]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:22.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:22 smithi111 ceph-mon[31493]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:33:22.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:33:22.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:33:22.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:23.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:22 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi094 to 8544M 2024-04-21T08:33:23.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:22 smithi094 ceph-mon[27988]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:23.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:22 smithi094 ceph-mon[27988]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:33:23.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:33:23.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:33:23.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:23.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:22 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi094 to 8544M 2024-04-21T08:33:23.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:22 smithi173 ceph-mon[31428]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:23.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:22 smithi173 ceph-mon[31428]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:33:23.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:33:23.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:33:23.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:23.872 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:23 smithi111 ceph-mon[31493]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:33:23.872 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:23 smithi111 ceph-mon[31493]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:33:24.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:23 smithi094 ceph-mon[27988]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:33:24.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:23 smithi094 ceph-mon[27988]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:33:24.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:23 smithi173 ceph-mon[31428]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:33:24.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:23 smithi173 ceph-mon[31428]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:33:24.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:24 smithi111 ceph-mon[31493]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:33:24.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:24 smithi111 ceph-mon[31493]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:33:25.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:24 smithi094 ceph-mon[27988]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:33:25.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:24 smithi094 ceph-mon[27988]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:33:25.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:24 smithi173 ceph-mon[31428]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:33:25.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:24 smithi173 ceph-mon[31428]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:33:25.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:25 smithi111 ceph-mon[31493]: from='client.? 172.21.15.111:0/44931316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "125dfc8c-d9ca-4ea4-a06f-605e4c76c919"}]: dispatch 2024-04-21T08:33:25.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:25 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "125dfc8c-d9ca-4ea4-a06f-605e4c76c919"}]: dispatch 2024-04-21T08:33:25.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:25 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "125dfc8c-d9ca-4ea4-a06f-605e4c76c919"}]': finished 2024-04-21T08:33:25.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:25 smithi111 ceph-mon[31493]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T08:33:25.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:25 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:25.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:25 smithi111 ceph-mon[31493]: from='client.? 172.21.15.111:0/1498590494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:33:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:25 smithi094 ceph-mon[27988]: from='client.? 172.21.15.111:0/44931316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "125dfc8c-d9ca-4ea4-a06f-605e4c76c919"}]: dispatch 2024-04-21T08:33:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:25 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "125dfc8c-d9ca-4ea4-a06f-605e4c76c919"}]: dispatch 2024-04-21T08:33:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:25 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "125dfc8c-d9ca-4ea4-a06f-605e4c76c919"}]': finished 2024-04-21T08:33:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:25 smithi094 ceph-mon[27988]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T08:33:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:25 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:25 smithi094 ceph-mon[27988]: from='client.? 172.21.15.111:0/1498590494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:33:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:25 smithi173 ceph-mon[31428]: from='client.? 172.21.15.111:0/44931316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "125dfc8c-d9ca-4ea4-a06f-605e4c76c919"}]: dispatch 2024-04-21T08:33:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:25 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "125dfc8c-d9ca-4ea4-a06f-605e4c76c919"}]: dispatch 2024-04-21T08:33:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:25 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "125dfc8c-d9ca-4ea4-a06f-605e4c76c919"}]': finished 2024-04-21T08:33:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:25 smithi173 ceph-mon[31428]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T08:33:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:25 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:25 smithi173 ceph-mon[31428]: from='client.? 172.21.15.111:0/1498590494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:33:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:26 smithi111 ceph-mon[31493]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:33:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:26 smithi094 ceph-mon[27988]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:33:27.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:26 smithi173 ceph-mon[31428]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:33:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:28 smithi111 ceph-mon[31493]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-21T08:33:29.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:28 smithi094 ceph-mon[27988]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-21T08:33:29.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:28 smithi173 ceph-mon[31428]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-21T08:33:30.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:30 smithi111 ceph-mon[31493]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:31.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:30 smithi094 ceph-mon[27988]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:31.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:30 smithi173 ceph-mon[31428]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:32.804 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:32 smithi111 ceph-mon[31493]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:32.804 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:33:32.805 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:32 smithi094 ceph-mon[27988]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:33:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:33.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:32 smithi173 ceph-mon[31428]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:33:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:33.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:33 smithi111 ceph-mon[31493]: Deploying daemon osd.4 on smithi111 2024-04-21T08:33:34.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:33 smithi094 ceph-mon[27988]: Deploying daemon osd.4 on smithi111 2024-04-21T08:33:34.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:33 smithi173 ceph-mon[31428]: Deploying daemon osd.4 on smithi111 2024-04-21T08:33:34.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:34 smithi111 ceph-mon[31493]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:35.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:34 smithi094 ceph-mon[27988]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:35.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:34 smithi173 ceph-mon[31428]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:36.764 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:36 smithi111 ceph-mon[31493]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:36 smithi094 ceph-mon[27988]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:37.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:36 smithi173 ceph-mon[31428]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:38.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:38.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:38.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:38.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:38 smithi111 ceph-mon[31493]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:38 smithi094 ceph-mon[27988]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:38.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:38.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:38.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:38.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:38 smithi173 ceph-mon[31428]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:40.800 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:40 smithi111 ceph-mon[31493]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:40.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.883+0000 7f22eb7fe700 1 -- 172.21.15.111:0/308791892 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f22f4062e80 con 0x7f22e00a3cc0 2024-04-21T08:33:40.883 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 4 on host 'smithi111' 2024-04-21T08:33:40.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.886+0000 7f22fc70a700 1 -- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22e00a3cc0 msgr2=0x7f22e00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:33:40.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.886+0000 7f22fc70a700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22e00a3cc0 0x7f22e00a6180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f22f419a120 tx=0x7f22f00073d0 comp rx=0 tx=0).stop 2024-04-21T08:33:40.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.886+0000 7f22fc70a700 1 -- 172.21.15.111:0/308791892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f41087a0 msgr2=0x7f22f41a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:33:40.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.886+0000 7f22fc70a700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f41087a0 0x7f22f41a01c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0062a0 tx=0x7f22ec004e40 comp rx=0 tx=0).stop 2024-04-21T08:33:40.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.886+0000 7f22fc70a700 1 -- 172.21.15.111:0/308791892 shutdown_connections 2024-04-21T08:33:40.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.886+0000 7f22fc70a700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22e00a3cc0 0x7f22e00a6180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:40.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.886+0000 7f22fc70a700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f41087a0 0x7f22f41a01c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:40.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.886+0000 7f22fc70a700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22f4103540 0x7f22f4198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:40.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.886+0000 7f22fc70a700 1 --2- 172.21.15.111:0/308791892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22f4105e70 0x7f22f4199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:40.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.886+0000 7f22fc70a700 1 -- 172.21.15.111:0/308791892 >> 172.21.15.111:0/308791892 conn(0x7f22f40fcf70 msgr2=0x7f22f4103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:33:40.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.887+0000 7f22fc70a700 1 -- 172.21.15.111:0/308791892 shutdown_connections 2024-04-21T08:33:40.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:33:40.887+0000 7f22fc70a700 1 -- 172.21.15.111:0/308791892 wait complete. 2024-04-21T08:33:41.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:40 smithi094 ceph-mon[27988]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:41.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:40 smithi173 ceph-mon[31428]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:41.532 DEBUG:teuthology.orchestra.run.smithi111:osd.4> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.4.service 2024-04-21T08:33:41.535 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_3... 2024-04-21T08:33:41.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:33:41.771 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:33:42.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:41 smithi111 ceph-mon[31493]: from='osd.4 [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:33:42.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:41 smithi111 ceph-mon[31493]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:33:42.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:41 smithi111 ceph-mon[31493]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:41 smithi173 ceph-mon[31428]: from='osd.4 [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:33:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:41 smithi173 ceph-mon[31428]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:33:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:41 smithi173 ceph-mon[31428]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:41 smithi094 ceph-mon[27988]: from='osd.4 [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:33:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:41 smithi094 ceph-mon[27988]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:33:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:41 smithi094 ceph-mon[27988]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:43.016 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:42 smithi111 ceph-mon[31493]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:33:43.016 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:42 smithi111 ceph-mon[31493]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:33:43.016 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:42 smithi111 ceph-mon[31493]: from='osd.4 [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:33:43.016 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:42 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:43.016 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:42 smithi111 ceph-mon[31493]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:33:43.016 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:33:42 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4[43045]: 2024-04-21T08:33:42.960+0000 7fb623509700 -1 osd.4 0 waiting for initial osdmap 2024-04-21T08:33:43.016 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:33:42 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4[43045]: 2024-04-21T08:33:42.967+0000 7fb61f2fc700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:33:43.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:42 smithi173 ceph-mon[31428]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:33:43.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:42 smithi173 ceph-mon[31428]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:33:43.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:42 smithi173 ceph-mon[31428]: from='osd.4 [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:33:43.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:42 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:43.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:42 smithi173 ceph-mon[31428]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:33:43.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:42 smithi094 ceph-mon[27988]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:33:43.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:42 smithi094 ceph-mon[27988]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:33:43.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:42 smithi094 ceph-mon[27988]: from='osd.4 [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:33:43.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:42 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:43.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:42 smithi094 ceph-mon[27988]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:33:43.792 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:33:43.792 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:33:43.792 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-21T08:33:43.792 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-21T08:33:43.792 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114039 s, 919 MB/s 2024-04-21T08:33:43.792 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 2024-04-21T08:33:43.792 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-21T08:33:44.151 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:43 smithi111 ceph-mon[31493]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:33:44.151 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:43 smithi111 ceph-mon[31493]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:33:44.151 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:43 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:44.151 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:43 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:44.151 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:43 smithi111 ceph-mon[31493]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:44.162 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:43 smithi173 ceph-mon[31428]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:33:44.162 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:43 smithi173 ceph-mon[31428]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:33:44.162 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:43 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:44.162 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:43 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:44.162 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:43 smithi173 ceph-mon[31428]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:43 smithi094 ceph-mon[27988]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:33:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:43 smithi094 ceph-mon[27988]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:33:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:43 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:43 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:43 smithi094 ceph-mon[27988]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:33:44.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-04-21T08:33:44.702 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:33:45.133 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:33:45.133 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:33:45.133 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: osd.4 [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] boot 2024-04-21T08:33:45.133 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T08:33:45.133 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:45.133 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi111 2024-04-21T08:33:45.133 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:45.134 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:33:45.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:33:45.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: osd.4 [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] boot 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi111 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:45.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:45.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:45.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:45.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:33:45.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:33:45.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: osd.4 [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] boot 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi111 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:33:45.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:45.439 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:33:45.439 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:33:46.153 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:45 smithi173 ceph-mon[31428]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:33:46.154 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:45 smithi173 ceph-mon[31428]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:46.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:45 smithi094 ceph-mon[27988]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:33:46.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:45 smithi094 ceph-mon[27988]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:46.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.384+0000 7efc7c242700 1 -- 172.21.15.173:0/2024604217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc74101d30 msgr2=0x7efc74102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:33:46.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.384+0000 7efc7c242700 1 --2- 172.21.15.173:0/2024604217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc74101d30 0x7efc74102190 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efc700099c0 tx=0x7efc70009cd0 comp rx=0 tx=0).stop 2024-04-21T08:33:46.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.384+0000 7efc7c242700 1 -- 172.21.15.173:0/2024604217 shutdown_connections 2024-04-21T08:33:46.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.384+0000 7efc7c242700 1 --2- 172.21.15.173:0/2024604217 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc741026d0 0x7efc74109f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:46.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.384+0000 7efc7c242700 1 --2- 172.21.15.173:0/2024604217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc74102ff0 0x7efc74103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:46.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.384+0000 7efc7c242700 1 --2- 172.21.15.173:0/2024604217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc74101d30 0x7efc74102190 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:46.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.384+0000 7efc7c242700 1 -- 172.21.15.173:0/2024604217 >> 172.21.15.173:0/2024604217 conn(0x7efc740fc780 msgr2=0x7efc740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:33:46.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.384+0000 7efc7c242700 1 -- 172.21.15.173:0/2024604217 shutdown_connections 2024-04-21T08:33:46.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc7c242700 1 -- 172.21.15.173:0/2024604217 wait complete. 2024-04-21T08:33:46.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc7c242700 1 Processor -- start 2024-04-21T08:33:46.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc7c242700 1 -- start start 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc7c242700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc74101d30 0x7efc7410e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc7c242700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc741026d0 0x7efc7410cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc7c242700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc74102ff0 0x7efc7410d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc7c242700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efc741017b0 con 0x7efc741026d0 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc7c242700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc74101630 con 0x7efc74102ff0 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc7c242700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc74101930 con 0x7efc74101d30 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc79fde700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc74101d30 0x7efc7410e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc79fde700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc74101d30 0x7efc7410e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:45768/0 (socket says 172.21.15.173:45768) 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.385+0000 7efc79fde700 1 -- 172.21.15.173:0/890069091 learned_addr learned my addr 172.21.15.173:0/890069091 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc7a7df700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc74102ff0 0x7efc7410d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc797dd700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc741026d0 0x7efc7410cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:33:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc79fde700 1 -- 172.21.15.173:0/890069091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc74102ff0 msgr2=0x7efc7410d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc79fde700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc74102ff0 0x7efc7410d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc79fde700 1 -- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc741026d0 msgr2=0x7efc7410cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc79fde700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc741026d0 0x7efc7410cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc79fde700 1 -- 172.21.15.173:0/890069091 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7410d920 con 0x7efc74101d30 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc797dd700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc741026d0 0x7efc7410cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc79fde700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc74101d30 0x7efc7410e4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efc6400c970 tx=0x7efc6400cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc6affd700 1 -- 172.21.15.173:0/890069091 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc64004220 con 0x7efc74101d30 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc7c242700 1 -- 172.21.15.173:0/890069091 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc741a2dd0 con 0x7efc74101d30 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc6affd700 1 -- 172.21.15.173:0/890069091 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc64007d50 con 0x7efc74101d30 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc6affd700 1 -- 172.21.15.173:0/890069091 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc64003cc0 con 0x7efc74101d30 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.386+0000 7efc7c242700 1 -- 172.21.15.173:0/890069091 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc741a3220 con 0x7efc74101d30 2024-04-21T08:33:46.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.387+0000 7efc7c242700 1 -- 172.21.15.173:0/890069091 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc74069730 con 0x7efc74101d30 2024-04-21T08:33:46.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.388+0000 7efc6affd700 1 -- 172.21.15.173:0/890069091 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efc640043c0 con 0x7efc74101d30 2024-04-21T08:33:46.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.389+0000 7efc6affd700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7efc600a3c40 0x7efc600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:33:46.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.389+0000 7efc6affd700 1 -- 172.21.15.173:0/890069091 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7efc640cf2d0 con 0x7efc74101d30 2024-04-21T08:33:46.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.389+0000 7efc797dd700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7efc600a3c40 0x7efc600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:33:46.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.390+0000 7efc797dd700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7efc600a3c40 0x7efc600a6100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efc70009fd0 tx=0x7efc70005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:33:46.392 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.393+0000 7efc6affd700 1 -- 172.21.15.173:0/890069091 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc64014030 con 0x7efc74101d30 2024-04-21T08:33:46.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:45 smithi111 ceph-mon[31493]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:33:46.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:45 smithi111 ceph-mon[31493]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:46.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:33:46.598+0000 7efc7c242700 1 -- 172.21.15.173:0/890069091 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7efc74062e80 con 0x7efc600a3c40 2024-04-21T08:33:47.248 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:46 smithi173 ceph-mon[31428]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:33:47.249 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:46 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:33:47.249 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:46 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:33:47.249 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:46 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:47.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:46 smithi094 ceph-mon[27988]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:33:47.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:33:47.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:33:47.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:46 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:46 smithi111 ceph-mon[31493]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:33:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:46 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:33:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:46 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:33:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:46 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:48.251 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:47 smithi173 ceph-mon[31428]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:48.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:47 smithi094 ceph-mon[27988]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:48.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:47 smithi111 ceph-mon[31493]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:49.660 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:49 smithi173 ceph-mon[31428]: from='client.? 172.21.15.173:0/3304973120' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01402189-deb7-49ad-bda8-0c35038a3348"}]: dispatch 2024-04-21T08:33:49.660 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:49 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01402189-deb7-49ad-bda8-0c35038a3348"}]: dispatch 2024-04-21T08:33:49.660 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:49 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01402189-deb7-49ad-bda8-0c35038a3348"}]': finished 2024-04-21T08:33:49.660 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:49 smithi173 ceph-mon[31428]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T08:33:49.660 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:49 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:33:49.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:49 smithi111 ceph-mon[31493]: from='client.? 172.21.15.173:0/3304973120' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01402189-deb7-49ad-bda8-0c35038a3348"}]: dispatch 2024-04-21T08:33:49.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:49 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01402189-deb7-49ad-bda8-0c35038a3348"}]: dispatch 2024-04-21T08:33:49.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:49 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01402189-deb7-49ad-bda8-0c35038a3348"}]': finished 2024-04-21T08:33:49.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:49 smithi111 ceph-mon[31493]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T08:33:49.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:49 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:33:49.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:49 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/3304973120' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01402189-deb7-49ad-bda8-0c35038a3348"}]: dispatch 2024-04-21T08:33:49.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:49 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01402189-deb7-49ad-bda8-0c35038a3348"}]: dispatch 2024-04-21T08:33:49.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:49 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01402189-deb7-49ad-bda8-0c35038a3348"}]': finished 2024-04-21T08:33:49.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:49 smithi094 ceph-mon[27988]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T08:33:49.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:49 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:33:50.661 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:50 smithi094 ceph-mon[27988]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:50.662 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:50 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/1527596608' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:33:50.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:50 smithi111 ceph-mon[31493]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:50.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:50 smithi111 ceph-mon[31493]: from='client.? 172.21.15.173:0/1527596608' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:33:50.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:50 smithi173 ceph-mon[31428]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:50.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:50 smithi173 ceph-mon[31428]: from='client.? 172.21.15.173:0/1527596608' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:33:52.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:52 smithi111 ceph-mon[31493]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:52.947 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:52 smithi173 ceph-mon[31428]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:53.009 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:52 smithi094 ceph-mon[27988]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:54.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:54 smithi173 ceph-mon[31428]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:54 smithi111 ceph-mon[31493]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:54 smithi094 ceph-mon[27988]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:56.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:56 smithi173 ceph-mon[31428]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:56.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:33:56.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:56 smithi111 ceph-mon[31493]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:33:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:56 smithi094 ceph-mon[27988]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:33:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:33:57.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:57 smithi173 ceph-mon[31428]: Deploying daemon osd.5 on smithi173 2024-04-21T08:33:57.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:57 smithi111 ceph-mon[31493]: Deploying daemon osd.5 on smithi173 2024-04-21T08:33:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:57 smithi094 ceph-mon[27988]: Deploying daemon osd.5 on smithi173 2024-04-21T08:33:58.866 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:33:58 smithi173 ceph-mon[31428]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:33:58 smithi111 ceph-mon[31493]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:33:59.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:33:58 smithi094 ceph-mon[27988]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:00 smithi111 ceph-mon[31493]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:00.941 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:00 smithi173 ceph-mon[31428]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:01.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:00 smithi094 ceph-mon[27988]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:02.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:02 smithi173 ceph-mon[31428]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:02 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:02 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:02 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:02 smithi111 ceph-mon[31493]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:02 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:02 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:02 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:03.009 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:02 smithi094 ceph-mon[27988]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:03.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:03.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:03.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:04.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:04 smithi173 ceph-mon[31428]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:04 smithi111 ceph-mon[31493]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:04 smithi094 ceph-mon[27988]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:05.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.863+0000 7efc6affd700 1 -- 172.21.15.173:0/890069091 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7efc74062e80 con 0x7efc600a3c40 2024-04-21T08:34:05.863 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-04-21T08:34:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 -- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7efc600a3c40 msgr2=0x7efc600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7efc600a3c40 0x7efc600a6100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efc70009fd0 tx=0x7efc70005e80 comp rx=0 tx=0).stop 2024-04-21T08:34:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 -- 172.21.15.173:0/890069091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc74101d30 msgr2=0x7efc7410e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc74101d30 0x7efc7410e4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efc6400c970 tx=0x7efc6400cd30 comp rx=0 tx=0).stop 2024-04-21T08:34:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 -- 172.21.15.173:0/890069091 shutdown_connections 2024-04-21T08:34:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7efc600a3c40 0x7efc600a6100 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc74102ff0 0x7efc7410d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc74101d30 0x7efc7410e4e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 --2- 172.21.15.173:0/890069091 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc741026d0 0x7efc7410cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 -- 172.21.15.173:0/890069091 >> 172.21.15.173:0/890069091 conn(0x7efc740fc780 msgr2=0x7efc74106040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 -- 172.21.15.173:0/890069091 shutdown_connections 2024-04-21T08:34:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:34:05.866+0000 7efc7c242700 1 -- 172.21.15.173:0/890069091 wait complete. 2024-04-21T08:34:06.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:06.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:06.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:06.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:06.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: from='osd.5 [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:34:06.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:06 smithi111 ceph-mon[31493]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:34:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: from='osd.5 [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:06 smithi094 ceph-mon[27988]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:06.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:06.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: from='osd.5 [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:34:06.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:06 smithi173 ceph-mon[31428]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:34:06.589 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.5.service 2024-04-21T08:34:06.591 INFO:tasks.cephadm:Deploying osd.6 on smithi094 with /dev/vg_nvme/lv_2... 2024-04-21T08:34:06.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:34:06.838 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:34:07.963 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:07 smithi173 ceph-mon[31428]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:34:07.963 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:07 smithi173 ceph-mon[31428]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:34:07.963 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:07.963 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:07 smithi173 ceph-mon[31428]: from='osd.5 [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:34:07.963 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:07 smithi173 ceph-mon[31428]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:34:07.963 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:07 smithi173 ceph-mon[31428]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:08.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:07 smithi111 ceph-mon[31493]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:34:08.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:07 smithi111 ceph-mon[31493]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:34:08.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:08.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:07 smithi111 ceph-mon[31493]: from='osd.5 [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:34:08.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:07 smithi111 ceph-mon[31493]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:34:08.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:07 smithi111 ceph-mon[31493]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:08.221 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:07 smithi094 ceph-mon[27988]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:34:08.221 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:07 smithi094 ceph-mon[27988]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:34:08.221 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:08.222 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:07 smithi094 ceph-mon[27988]: from='osd.5 [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:34:08.222 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:07 smithi094 ceph-mon[27988]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:34:08.222 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:07 smithi094 ceph-mon[27988]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:08.760 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:34:08 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:34:08.311+0000 7f818de53700 -1 osd.5 0 waiting for initial osdmap 2024-04-21T08:34:08.761 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:34:08 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:34:08.317+0000 7f8185c3e700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:34:08.827 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:34:08.827 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:34:08.827 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-04-21T08:34:08.827 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-04-21T08:34:08.827 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00928033 s, 1.1 GB/s 2024-04-21T08:34:08.828 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-04-21T08:34:09.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:08 smithi111 ceph-mon[31493]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:34:09.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:08 smithi111 ceph-mon[31493]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:34:09.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:09.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:09.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:08 smithi111 ceph-mon[31493]: from='osd.5 ' entity='osd.5' 2024-04-21T08:34:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:08 smithi094 ceph-mon[27988]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:34:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:08 smithi094 ceph-mon[27988]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:34:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:09.210 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:08 smithi094 ceph-mon[27988]: from='osd.5 ' entity='osd.5' 2024-04-21T08:34:09.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:08 smithi173 ceph-mon[31428]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:34:09.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:08 smithi173 ceph-mon[31428]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:34:09.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:09.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:09.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:08 smithi173 ceph-mon[31428]: from='osd.5 ' entity='osd.5' 2024-04-21T08:34:09.559 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_2 2024-04-21T08:34:09.817 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:34:10.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:34:10.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: osd.5 [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] boot 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:34:10.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:34:10.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: osd.5 [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] boot 2024-04-21T08:34:10.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T08:34:10.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:10.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:10.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.103 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.103 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:10.103 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:10.103 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.103 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:10.103 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:10.103 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:34:10.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:34:10.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: osd.5 [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] boot 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:11.004 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:10 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi173 2024-04-21T08:34:11.004 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:10 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-21T08:34:11.004 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:10 smithi094 ceph-mon[27988]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:34:11.005 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:10 smithi094 ceph-mon[27988]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:34:11.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:10 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi173 2024-04-21T08:34:11.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:10 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-21T08:34:11.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:10 smithi111 ceph-mon[31493]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:34:11.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:10 smithi111 ceph-mon[31493]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:34:11.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:10 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi173 2024-04-21T08:34:11.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:10 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-21T08:34:11.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:10 smithi173 ceph-mon[31428]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:34:11.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:10 smithi173 ceph-mon[31428]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:34:11.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.521+0000 7f70cba54700 1 -- 172.21.15.94:0/1661847751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c41087d0 msgr2=0x7f70c410ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:11.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.521+0000 7f70cba54700 1 --2- 172.21.15.94:0/1661847751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c41087d0 0x7f70c410ac00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00b0a0 tx=0x7f70bc00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:34:11.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.521+0000 7f70cba54700 1 -- 172.21.15.94:0/1661847751 shutdown_connections 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.521+0000 7f70cba54700 1 --2- 172.21.15.94:0/1661847751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c41087d0 0x7f70c410ac00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.521+0000 7f70cba54700 1 --2- 172.21.15.94:0/1661847751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70c4103570 0x7f70c4105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.521+0000 7f70cba54700 1 --2- 172.21.15.94:0/1661847751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70c4105ea0 0x7f70c4108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.521+0000 7f70cba54700 1 -- 172.21.15.94:0/1661847751 >> 172.21.15.94:0/1661847751 conn(0x7f70c40fcf60 msgr2=0x7f70c40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.521+0000 7f70cba54700 1 -- 172.21.15.94:0/1661847751 shutdown_connections 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.521+0000 7f70cba54700 1 -- 172.21.15.94:0/1661847751 wait complete. 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70cba54700 1 Processor -- start 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70cba54700 1 -- start start 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70cba54700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70c4103570 0x7f70c4198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70cba54700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4105ea0 0x7f70c41990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70cba54700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70c41087d0 0x7f70c41a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70cba54700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f70c410e580 con 0x7f70c4105ea0 2024-04-21T08:34:11.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70cba54700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f70c410e400 con 0x7f70c41087d0 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70cba54700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f70c410e700 con 0x7f70c4103570 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70c8fef700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4105ea0 0x7f70c41990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70c8fef700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4105ea0 0x7f70c41990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59024/0 (socket says 172.21.15.94:59024) 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70c8fef700 1 -- 172.21.15.94:0/94938317 learned_addr learned my addr 172.21.15.94:0/94938317 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70c97f0700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70c4103570 0x7f70c4198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.522+0000 7f70c9ff1700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70c41087d0 0x7f70c41a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70c8fef700 1 -- 172.21.15.94:0/94938317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70c4103570 msgr2=0x7f70c4198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70c8fef700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70c4103570 0x7f70c4198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70c8fef700 1 -- 172.21.15.94:0/94938317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70c41087d0 msgr2=0x7f70c41a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70c8fef700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70c41087d0 0x7f70c41a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70c8fef700 1 -- 172.21.15.94:0/94938317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c41a07f0 con 0x7f70c4105ea0 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70c97f0700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70c4103570 0x7f70c4198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:34:11.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70c9ff1700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70c41087d0 0x7f70c41a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:34:11.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70c8fef700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4105ea0 0x7f70c41990d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f70b400c9e0 tx=0x7f70b400cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:11.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70ba7fc700 1 -- 172.21.15.94:0/94938317 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4007ac0 con 0x7f70c4105ea0 2024-04-21T08:34:11.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70cba54700 1 -- 172.21.15.94:0/94938317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c41a0ae0 con 0x7f70c4105ea0 2024-04-21T08:34:11.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70ba7fc700 1 -- 172.21.15.94:0/94938317 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70b4007c60 con 0x7f70c4105ea0 2024-04-21T08:34:11.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70ba7fc700 1 -- 172.21.15.94:0/94938317 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4018660 con 0x7f70c4105ea0 2024-04-21T08:34:11.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.523+0000 7f70cba54700 1 -- 172.21.15.94:0/94938317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c41a1070 con 0x7f70c4105ea0 2024-04-21T08:34:11.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.524+0000 7f70cba54700 1 -- 172.21.15.94:0/94938317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c4069730 con 0x7f70c4105ea0 2024-04-21T08:34:11.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.525+0000 7f70ba7fc700 1 -- 172.21.15.94:0/94938317 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f70b400f450 con 0x7f70c4105ea0 2024-04-21T08:34:11.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.525+0000 7f70ba7fc700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f70b00a3cc0 0x7f70b00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:11.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.526+0000 7f70c97f0700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f70b00a3cc0 0x7f70b00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:11.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.526+0000 7f70ba7fc700 1 -- 172.21.15.94:0/94938317 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f70b40cf4a0 con 0x7f70c4105ea0 2024-04-21T08:34:11.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.526+0000 7f70c97f0700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f70b00a3cc0 0x7f70b00a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f70c00099d0 tx=0x7f70c0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:11.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.530+0000 7f70ba7fc700 1 -- 172.21.15.94:0/94938317 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f70b4014030 con 0x7f70c4105ea0 2024-04-21T08:34:11.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:11.740+0000 7f70cba54700 1 -- 172.21.15.94:0/94938317 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f70c4062e80 con 0x7f70b00a3cc0 2024-04-21T08:34:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:12 smithi111 ceph-mon[31493]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:12 smithi111 ceph-mon[31493]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:34:12.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:34:12.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:34:12.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:12 smithi094 ceph-mon[27988]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:12 smithi094 ceph-mon[27988]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:34:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:34:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:34:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:12 smithi173 ceph-mon[31428]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:12 smithi173 ceph-mon[31428]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:34:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:34:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:34:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:13.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:13 smithi111 ceph-mon[31493]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:13.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:13 smithi111 ceph-mon[31493]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:34:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:13 smithi094 ceph-mon[27988]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:13 smithi094 ceph-mon[27988]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:34:14.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:13 smithi173 ceph-mon[31428]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:14.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:13 smithi173 ceph-mon[31428]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:34:14.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:14 smithi111 ceph-mon[31493]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:34:14.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:14 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2647873604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a09d9c1-e8fb-4ff8-a349-8d5aba865099"}]: dispatch 2024-04-21T08:34:14.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:14 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2647873604' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a09d9c1-e8fb-4ff8-a349-8d5aba865099"}]': finished 2024-04-21T08:34:14.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:14 smithi111 ceph-mon[31493]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T08:34:14.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:14 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:15.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:14 smithi094 ceph-mon[27988]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:34:15.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:14 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2647873604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a09d9c1-e8fb-4ff8-a349-8d5aba865099"}]: dispatch 2024-04-21T08:34:15.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:14 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2647873604' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a09d9c1-e8fb-4ff8-a349-8d5aba865099"}]': finished 2024-04-21T08:34:15.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:14 smithi094 ceph-mon[27988]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T08:34:15.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:14 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:15.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:14 smithi173 ceph-mon[31428]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:34:15.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:14 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2647873604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a09d9c1-e8fb-4ff8-a349-8d5aba865099"}]: dispatch 2024-04-21T08:34:15.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:14 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2647873604' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a09d9c1-e8fb-4ff8-a349-8d5aba865099"}]': finished 2024-04-21T08:34:15.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:14 smithi173 ceph-mon[31428]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T08:34:15.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:14 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:15.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:15 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4053301092' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:34:15.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:15 smithi111 ceph-mon[31493]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-21T08:34:16.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:15 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4053301092' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:34:16.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:15 smithi094 ceph-mon[27988]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-21T08:34:16.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:15 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4053301092' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:34:16.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:15 smithi173 ceph-mon[31428]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-21T08:34:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:18 smithi111 ceph-mon[31493]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:34:18.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:18 smithi094 ceph-mon[27988]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:34:19.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:18 smithi173 ceph-mon[31428]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:34:20.863 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:20 smithi094 ceph-mon[27988]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-21T08:34:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:20 smithi111 ceph-mon[31493]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-21T08:34:21.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:20 smithi173 ceph-mon[31428]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-21T08:34:22.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:22 smithi094 ceph-mon[27988]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:22.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:34:22.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:22.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:22 smithi111 ceph-mon[31493]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:22.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:34:22.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:23.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:22 smithi173 ceph-mon[31428]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:23.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:34:23.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:23.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:23 smithi111 ceph-mon[31493]: Deploying daemon osd.6 on smithi094 2024-04-21T08:34:23.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:23 smithi094 ceph-mon[27988]: Deploying daemon osd.6 on smithi094 2024-04-21T08:34:24.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:23 smithi173 ceph-mon[31428]: Deploying daemon osd.6 on smithi094 2024-04-21T08:34:24.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:24 smithi111 ceph-mon[31493]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:25.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:24 smithi094 ceph-mon[27988]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:25.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:24 smithi173 ceph-mon[31428]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:26.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:26 smithi094 ceph-mon[27988]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:26 smithi111 ceph-mon[31493]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:27.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:26 smithi173 ceph-mon[31428]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:28.740 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:28 smithi094 ceph-mon[27988]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:28.740 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:28.740 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:28.740 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:28 smithi111 ceph-mon[31493]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:29.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:28 smithi173 ceph-mon[31428]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:29.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:29.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:29.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:30.751 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:30 smithi094 ceph-mon[27988]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:30.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:30 smithi111 ceph-mon[31493]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:31.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:30 smithi173 ceph-mon[31428]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:31.758 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 6 on host 'smithi094' 2024-04-21T08:34:31.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.758+0000 7f70ba7fc700 1 -- 172.21.15.94:0/94938317 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f70c4062e80 con 0x7f70b00a3cc0 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 -- 172.21.15.94:0/94938317 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f70b00a3cc0 msgr2=0x7f70b00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f70b00a3cc0 0x7f70b00a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f70c00099d0 tx=0x7f70c0008040 comp rx=0 tx=0).stop 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 -- 172.21.15.94:0/94938317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4105ea0 msgr2=0x7f70c41990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4105ea0 0x7f70c41990d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f70b400c9e0 tx=0x7f70b400cda0 comp rx=0 tx=0).stop 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 -- 172.21.15.94:0/94938317 shutdown_connections 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f70b00a3cc0 0x7f70b00a6180 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70c41087d0 0x7f70c41a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70c4103570 0x7f70c4198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 --2- 172.21.15.94:0/94938317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4105ea0 0x7f70c41990d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 -- 172.21.15.94:0/94938317 >> 172.21.15.94:0/94938317 conn(0x7f70c40fcf60 msgr2=0x7f70c4103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 -- 172.21.15.94:0/94938317 shutdown_connections 2024-04-21T08:34:31.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:34:31.761+0000 7f70cba54700 1 -- 172.21.15.94:0/94938317 wait complete. 2024-04-21T08:34:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:31 smithi094 ceph-mon[27988]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:31 smithi094 ceph-mon[27988]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:34:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:32.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:32.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:32.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:31 smithi111 ceph-mon[31493]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:32.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:31 smithi111 ceph-mon[31493]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:34:32.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:32.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:32.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:32.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:31 smithi173 ceph-mon[31428]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:32.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:31 smithi173 ceph-mon[31428]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:34:32.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:32.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:32.455 DEBUG:teuthology.orchestra.run.smithi094:osd.6> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.6.service 2024-04-21T08:34:32.458 INFO:tasks.cephadm:Deploying osd.7 on smithi111 with /dev/vg_nvme/lv_2... 2024-04-21T08:34:32.458 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:34:32.703 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:34:33.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:32 smithi094 ceph-mon[27988]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:34:33.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:32 smithi094 ceph-mon[27988]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:34:33.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:32 smithi094 ceph-mon[27988]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:34:33.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:33.126 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:32 smithi111 ceph-mon[31493]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:34:33.127 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:32 smithi111 ceph-mon[31493]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:34:33.127 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:32 smithi111 ceph-mon[31493]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:34:33.127 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:33.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:32 smithi173 ceph-mon[31428]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:34:33.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:32 smithi173 ceph-mon[31428]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:34:33.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:32 smithi173 ceph-mon[31428]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:34:33.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:33.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:34:33 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:34:33.332+0000 7f1bd4c89700 -1 osd.6 0 waiting for initial osdmap 2024-04-21T08:34:33.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:34:33 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:34:33.340+0000 7f1bcc273700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:34:34.088 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:33 smithi094 ceph-mon[27988]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:34:34.088 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:33 smithi094 ceph-mon[27988]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:34:34.088 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:33 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:34.088 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:33 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:34.088 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:33 smithi094 ceph-mon[27988]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' 2024-04-21T08:34:34.088 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:33 smithi094 ceph-mon[27988]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:34.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:33 smithi111 ceph-mon[31493]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:34:34.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:33 smithi111 ceph-mon[31493]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:34:34.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:33 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:34.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:33 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:34.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:33 smithi111 ceph-mon[31493]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' 2024-04-21T08:34:34.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:33 smithi111 ceph-mon[31493]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:34.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:33 smithi173 ceph-mon[31428]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:34:34.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:33 smithi173 ceph-mon[31428]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:34:34.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:33 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:34.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:33 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:34.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:33 smithi173 ceph-mon[31428]: from='osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342]' entity='osd.6' 2024-04-21T08:34:34.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:33 smithi173 ceph-mon[31428]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:34:34.787 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:34:34.787 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:34:34.787 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T08:34:34.788 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T08:34:34.788 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112176 s, 935 MB/s 2024-04-21T08:34:34.788 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T08:34:35.112 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:34 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:34:35.112 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:34 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:34:35.112 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:34 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:35.112 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:34 smithi111 ceph-mon[31493]: osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] boot 2024-04-21T08:34:35.112 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:34 smithi111 ceph-mon[31493]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T08:34:35.112 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:34 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:35.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:34 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:34:35.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:34 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:34:35.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:35.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:34 smithi094 ceph-mon[27988]: osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] boot 2024-04-21T08:34:35.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:34 smithi094 ceph-mon[27988]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T08:34:35.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:34 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:35.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:34 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:34:35.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:34 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:34:35.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:34 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:35.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:34 smithi173 ceph-mon[31428]: osd.6 [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] boot 2024-04-21T08:34:35.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:34 smithi173 ceph-mon[31428]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T08:34:35.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:34 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:34:35.443 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_2 2024-04-21T08:34:35.684 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:34:36.662 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:34:36.662 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:36.662 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.662 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.662 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:36.662 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:36.662 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:36.662 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:36.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:36.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:34:36.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:36.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:36.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:36.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:36.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:36.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:36.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:34:36.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:36.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:36.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:36.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:34:36.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:36.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:36.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:36.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:37.194 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 -- 172.21.15.111:0/3806656266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f847c102ba0 msgr2=0x7f847c103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:37.194 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 --2- 172.21.15.111:0/3806656266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f847c102ba0 0x7f847c103000 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f84700099c0 tx=0x7f8470009cd0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.194 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 -- 172.21.15.111:0/3806656266 shutdown_connections 2024-04-21T08:34:37.194 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 --2- 172.21.15.111:0/3806656266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f847c103540 0x7f847c109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.194 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 --2- 172.21.15.111:0/3806656266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847c1019a0 0x7f847c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.194 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 --2- 172.21.15.111:0/3806656266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f847c102ba0 0x7f847c103000 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.194 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 -- 172.21.15.111:0/3806656266 >> 172.21.15.111:0/3806656266 conn(0x7f847c0fcf40 msgr2=0x7f847c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 -- 172.21.15.111:0/3806656266 shutdown_connections 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 -- 172.21.15.111:0/3806656266 wait complete. 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 Processor -- start 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.193+0000 7f8481925700 1 -- start start 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8481925700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f847c1019a0 0x7f847c06b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8481925700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847c102ba0 0x7f847c19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8481925700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f847c103540 0x7f847c19ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8481925700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f847c10d440 con 0x7f847c102ba0 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8481925700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f847c10d2c0 con 0x7f847c1019a0 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8481925700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f847c10d5c0 con 0x7f847c103540 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f847affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f847c1019a0 0x7f847c06b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f847affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f847c1019a0 0x7f847c06b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47060/0 (socket says 172.21.15.111:47060) 2024-04-21T08:34:37.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f847affd700 1 -- 172.21.15.111:0/192964601 learned_addr learned my addr 172.21.15.111:0/192964601 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f847b7fe700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f847c103540 0x7f847c19ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f847affd700 1 -- 172.21.15.111:0/192964601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f847c103540 msgr2=0x7f847c19ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f847affd700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f847c103540 0x7f847c19ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f847affd700 1 -- 172.21.15.111:0/192964601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847c102ba0 msgr2=0x7f847c19d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f847affd700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847c102ba0 0x7f847c19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f847affd700 1 -- 172.21.15.111:0/192964601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847c1a08b0 con 0x7f847c1019a0 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f847affd700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f847c1019a0 0x7f847c06b280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f846c00d980 tx=0x7f846c00dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8480923700 1 -- 172.21.15.111:0/192964601 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c017030 con 0x7f847c1019a0 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8480923700 1 -- 172.21.15.111:0/192964601 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f846c0044d0 con 0x7f847c1019a0 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8481925700 1 -- 172.21.15.111:0/192964601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f847c1a0ba0 con 0x7f847c1019a0 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8480923700 1 -- 172.21.15.111:0/192964601 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c003aa0 con 0x7f847c1019a0 2024-04-21T08:34:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.194+0000 7f8481925700 1 -- 172.21.15.111:0/192964601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f847c1a10e0 con 0x7f847c1019a0 2024-04-21T08:34:37.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.195+0000 7f8481925700 1 -- 172.21.15.111:0/192964601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f847c069730 con 0x7f847c1019a0 2024-04-21T08:34:37.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.196+0000 7f8480923700 1 -- 172.21.15.111:0/192964601 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f846c01a070 con 0x7f847c1019a0 2024-04-21T08:34:37.198 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.197+0000 7f8480923700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f84640a3c40 0x7f84640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:37.198 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.197+0000 7f8480923700 1 -- 172.21.15.111:0/192964601 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f846c0cf490 con 0x7f847c1019a0 2024-04-21T08:34:37.198 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.197+0000 7f847a7fc700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f84640a3c40 0x7f84640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:37.198 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.198+0000 7f847a7fc700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f84640a3c40 0x7f84640a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f84700098e0 tx=0x7f8470000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:37.200 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.201+0000 7f8480923700 1 -- 172.21.15.111:0/192964601 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f846c09cb80 con 0x7f847c1019a0 2024-04-21T08:34:37.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:37.406+0000 7f8481925700 1 -- 172.21.15.111:0/192964601 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f847c062e80 con 0x7f84640a3c40 2024-04-21T08:34:37.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:37 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi094 2024-04-21T08:34:37.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:37 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi094 to 5696M 2024-04-21T08:34:37.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:37 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi094 2024-04-21T08:34:37.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:37 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi094 to 5696M 2024-04-21T08:34:37.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:37 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi094 2024-04-21T08:34:37.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:37 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi094 to 5696M 2024-04-21T08:34:38.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:38 smithi111 ceph-mon[31493]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:38.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:34:38.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:34:38.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:38.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:38 smithi111 ceph-mon[31493]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:38 smithi094 ceph-mon[27988]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:34:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:34:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:38 smithi094 ceph-mon[27988]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:38.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:38 smithi173 ceph-mon[31428]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:38.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:34:38.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:34:38.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:38.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:38 smithi173 ceph-mon[31428]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:40.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:40 smithi111 ceph-mon[31493]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:40.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:40 smithi111 ceph-mon[31493]: from='client.? 172.21.15.111:0/929552180' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed806162-485b-4c84-9687-78fcc1f8cc59"}]: dispatch 2024-04-21T08:34:40.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:40 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed806162-485b-4c84-9687-78fcc1f8cc59"}]: dispatch 2024-04-21T08:34:40.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:40 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed806162-485b-4c84-9687-78fcc1f8cc59"}]': finished 2024-04-21T08:34:40.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:40 smithi111 ceph-mon[31493]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T08:34:40.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:40 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:41.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:40 smithi094 ceph-mon[27988]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:41.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:40 smithi094 ceph-mon[27988]: from='client.? 172.21.15.111:0/929552180' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed806162-485b-4c84-9687-78fcc1f8cc59"}]: dispatch 2024-04-21T08:34:41.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:40 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed806162-485b-4c84-9687-78fcc1f8cc59"}]: dispatch 2024-04-21T08:34:41.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:40 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed806162-485b-4c84-9687-78fcc1f8cc59"}]': finished 2024-04-21T08:34:41.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:40 smithi094 ceph-mon[27988]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T08:34:41.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:40 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:41.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:40 smithi173 ceph-mon[31428]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:41.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:40 smithi173 ceph-mon[31428]: from='client.? 172.21.15.111:0/929552180' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed806162-485b-4c84-9687-78fcc1f8cc59"}]: dispatch 2024-04-21T08:34:41.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:40 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed806162-485b-4c84-9687-78fcc1f8cc59"}]: dispatch 2024-04-21T08:34:41.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:40 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed806162-485b-4c84-9687-78fcc1f8cc59"}]': finished 2024-04-21T08:34:41.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:40 smithi173 ceph-mon[31428]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T08:34:41.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:40 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:41.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:41 smithi111 ceph-mon[31493]: from='client.? 172.21.15.111:0/460799083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:34:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:41 smithi094 ceph-mon[27988]: from='client.? 172.21.15.111:0/460799083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:34:42.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:41 smithi173 ceph-mon[31428]: from='client.? 172.21.15.111:0/460799083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:34:42.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:42 smithi111 ceph-mon[31493]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:43.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:42 smithi094 ceph-mon[27988]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:43.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:42 smithi173 ceph-mon[31428]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:44.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:44 smithi111 ceph-mon[31493]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:45.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:44 smithi094 ceph-mon[27988]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:45.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:44 smithi173 ceph-mon[31428]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:46.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:46 smithi111 ceph-mon[31493]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:47.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:46 smithi094 ceph-mon[27988]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:47.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:46 smithi173 ceph-mon[31428]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:47.892 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:47 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:34:47.892 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:47 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:48.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:47 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:34:48.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:47 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:48.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:47 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:34:48.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:47 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:48.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:48 smithi111 ceph-mon[31493]: Deploying daemon osd.7 on smithi111 2024-04-21T08:34:48.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:48 smithi111 ceph-mon[31493]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:49.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:48 smithi094 ceph-mon[27988]: Deploying daemon osd.7 on smithi111 2024-04-21T08:34:49.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:48 smithi094 ceph-mon[27988]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:49.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:48 smithi173 ceph-mon[31428]: Deploying daemon osd.7 on smithi111 2024-04-21T08:34:49.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:48 smithi173 ceph-mon[31428]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:50.869 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:50 smithi111 ceph-mon[31493]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:50 smithi094 ceph-mon[27988]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:51.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:50 smithi173 ceph-mon[31428]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:52.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:52 smithi111 ceph-mon[31493]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:53.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:52 smithi094 ceph-mon[27988]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:53.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:52 smithi173 ceph-mon[31428]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:53 smithi111 ceph-mon[31493]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:54.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:54.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:54.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:54.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:53 smithi094 ceph-mon[27988]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:54.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:53 smithi173 ceph-mon[31428]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:56.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.318+0000 7f8480923700 1 -- 172.21.15.111:0/192964601 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f847c062e80 con 0x7f84640a3c40 2024-04-21T08:34:56.318 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 7 on host 'smithi111' 2024-04-21T08:34:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.320+0000 7f8481925700 1 -- 172.21.15.111:0/192964601 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f84640a3c40 msgr2=0x7f84640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.320+0000 7f8481925700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f84640a3c40 0x7f84640a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f84700098e0 tx=0x7f8470000bc0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.320+0000 7f8481925700 1 -- 172.21.15.111:0/192964601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f847c1019a0 msgr2=0x7f847c06b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.320+0000 7f8481925700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f847c1019a0 0x7f847c06b280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f846c00d980 tx=0x7f846c00dd40 comp rx=0 tx=0).stop 2024-04-21T08:34:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.321+0000 7f8481925700 1 -- 172.21.15.111:0/192964601 shutdown_connections 2024-04-21T08:34:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.321+0000 7f8481925700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f84640a3c40 0x7f84640a6100 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.321+0000 7f8481925700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f847c1019a0 0x7f847c06b280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.321+0000 7f8481925700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f847c103540 0x7f847c19ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.321+0000 7f8481925700 1 --2- 172.21.15.111:0/192964601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847c102ba0 0x7f847c19d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.321+0000 7f8481925700 1 -- 172.21.15.111:0/192964601 >> 172.21.15.111:0/192964601 conn(0x7f847c0fcf40 msgr2=0x7f847c105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.321+0000 7f8481925700 1 -- 172.21.15.111:0/192964601 shutdown_connections 2024-04-21T08:34:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:34:56.321+0000 7f8481925700 1 -- 172.21.15.111:0/192964601 wait complete. 2024-04-21T08:34:56.605 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.605 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.605 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:56.605 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:56.605 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.605 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:56 smithi111 ceph-mon[31493]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:56.605 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.605 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.605 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:56 smithi094 ceph-mon[27988]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:56.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:56.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:34:56.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:56 smithi173 ceph-mon[31428]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:56.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:34:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:57.001 DEBUG:teuthology.orchestra.run.smithi111:osd.7> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.7.service 2024-04-21T08:34:57.003 INFO:tasks.cephadm:Deploying osd.8 on smithi173 with /dev/vg_nvme/lv_2... 2024-04-21T08:34:57.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:34:57.248 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:34:57.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:57 smithi111 ceph-mon[31493]: from='osd.7 [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:34:57.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:57 smithi111 ceph-mon[31493]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:34:57.689 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:57 smithi173 ceph-mon[31428]: from='osd.7 [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:34:57.689 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:57 smithi173 ceph-mon[31428]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:34:57.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:57 smithi094 ceph-mon[27988]: from='osd.7 [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:34:57.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:57 smithi094 ceph-mon[27988]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:34:58.662 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:58 smithi173 ceph-mon[31428]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:34:58.663 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:58 smithi173 ceph-mon[31428]: from='osd.7 [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:34:58.663 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:58 smithi173 ceph-mon[31428]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:34:58.663 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:58.663 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:58 smithi173 ceph-mon[31428]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:34:58.663 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:58 smithi173 ceph-mon[31428]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:58.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:58 smithi111 ceph-mon[31493]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:34:58.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:58 smithi111 ceph-mon[31493]: from='osd.7 [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:34:58.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:58 smithi111 ceph-mon[31493]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:34:58.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:58.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:58 smithi111 ceph-mon[31493]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:34:58.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:58 smithi111 ceph-mon[31493]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:58.688 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:34:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:34:58.363+0000 7fc42c838700 -1 osd.7 0 waiting for initial osdmap 2024-04-21T08:34:58.688 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:34:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:34:58.372+0000 7fc42862b700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:34:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:58 smithi094 ceph-mon[27988]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:34:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:58 smithi094 ceph-mon[27988]: from='osd.7 [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:34:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:58 smithi094 ceph-mon[27988]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:34:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:58 smithi094 ceph-mon[27988]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:34:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:58 smithi094 ceph-mon[27988]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:34:59.247 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:34:59.247 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:34:59.247 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-21T08:34:59.247 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-21T08:34:59.247 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112499 s, 932 MB/s 2024-04-21T08:34:59.247 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-21T08:34:59.369 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:59 smithi111 ceph-mon[31493]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:34:59.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:59 smithi111 ceph-mon[31493]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:34:59.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:59 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:59.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:34:59 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:59.543 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:59 smithi173 ceph-mon[31428]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:34:59.543 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:59 smithi173 ceph-mon[31428]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:34:59.544 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:59 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:59.544 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:34:59 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:59.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:59 smithi094 ceph-mon[27988]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:34:59.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:59 smithi094 ceph-mon[27988]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:34:59.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:59 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:59.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:34:59 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:34:59.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-04-21T08:35:00.148 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:35:00.639 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: osd.7 [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] boot 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:00.640 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.641 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:00.641 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:00.641 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:00 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: osd.7 [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] boot 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:00 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:35:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:35:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: osd.7 [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] boot 2024-04-21T08:35:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T08:35:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:35:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:35:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:00.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:00.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:00.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:00.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:00.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:00.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:00 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:01.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:01 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi111 2024-04-21T08:35:01.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:01 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-21T08:35:01.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:01 smithi173 ceph-mon[31428]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:35:01.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:01 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi111 2024-04-21T08:35:01.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:01 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-21T08:35:01.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:01 smithi111 ceph-mon[31493]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:35:01.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:01 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi111 2024-04-21T08:35:01.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:01 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-21T08:35:01.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:01 smithi094 ceph-mon[27988]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:35:01.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.800+0000 7f6b1dafc700 1 -- 172.21.15.173:0/3339449471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b18103540 msgr2=0x7f6b18109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:01.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.800+0000 7f6b1dafc700 1 --2- 172.21.15.173:0/3339449471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b18103540 0x7f6b18109df0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000b0a0 tx=0x7f6b1000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:35:01.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.800+0000 7f6b1dafc700 1 -- 172.21.15.173:0/3339449471 shutdown_connections 2024-04-21T08:35:01.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.800+0000 7f6b1dafc700 1 --2- 172.21.15.173:0/3339449471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b18103540 0x7f6b18109df0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:01.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.800+0000 7f6b1dafc700 1 --2- 172.21.15.173:0/3339449471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b181019a0 0x7f6b18101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:01.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.800+0000 7f6b1dafc700 1 --2- 172.21.15.173:0/3339449471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b18102ba0 0x7f6b18103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:01.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.800+0000 7f6b1dafc700 1 -- 172.21.15.173:0/3339449471 >> 172.21.15.173:0/3339449471 conn(0x7f6b180fcf40 msgr2=0x7f6b180ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.800+0000 7f6b1dafc700 1 -- 172.21.15.173:0/3339449471 shutdown_connections 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.800+0000 7f6b1dafc700 1 -- 172.21.15.173:0/3339449471 wait complete. 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b1dafc700 1 Processor -- start 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b1dafc700 1 -- start start 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b1dafc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b181019a0 0x7f6b1806b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b1dafc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b18102ba0 0x7f6b1819d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b1dafc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b18103540 0x7f6b1819ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b1dafc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1810d4e0 con 0x7f6b181019a0 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b1dafc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1810d360 con 0x7f6b18103540 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b1dafc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1810d660 con 0x7f6b18102ba0 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b0effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b18102ba0 0x7f6b1819d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b0effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b18102ba0 0x7f6b1819d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:38978/0 (socket says 172.21.15.173:38978) 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b0effd700 1 -- 172.21.15.173:0/2624595045 learned_addr learned my addr 172.21.15.173:0/2624595045 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-21T08:35:01.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b177fe700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b181019a0 0x7f6b1806b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b17fff700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b18103540 0x7f6b1819ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b0effd700 1 -- 172.21.15.173:0/2624595045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b18103540 msgr2=0x7f6b1819ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b0effd700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b18103540 0x7f6b1819ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b0effd700 1 -- 172.21.15.173:0/2624595045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b181019a0 msgr2=0x7f6b1806b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b0effd700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b181019a0 0x7f6b1806b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.801+0000 7f6b0effd700 1 -- 172.21.15.173:0/2624595045 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b181a0850 con 0x7f6b18102ba0 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.802+0000 7f6b0effd700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b18102ba0 0x7f6b1819d930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000b530 tx=0x7f6b0000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.802+0000 7f6b157fa700 1 -- 172.21.15.173:0/2624595045 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00015020 con 0x7f6b18102ba0 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.802+0000 7f6b157fa700 1 -- 172.21.15.173:0/2624595045 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b00011ce0 con 0x7f6b18102ba0 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.802+0000 7f6b1dafc700 1 -- 172.21.15.173:0/2624595045 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b181a0b40 con 0x7f6b18102ba0 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.802+0000 7f6b157fa700 1 -- 172.21.15.173:0/2624595045 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00010d20 con 0x7f6b18102ba0 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.802+0000 7f6b1dafc700 1 -- 172.21.15.173:0/2624595045 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b181a1030 con 0x7f6b18102ba0 2024-04-21T08:35:01.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.803+0000 7f6b1dafc700 1 -- 172.21.15.173:0/2624595045 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b18069730 con 0x7f6b18102ba0 2024-04-21T08:35:01.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.804+0000 7f6b157fa700 1 -- 172.21.15.173:0/2624595045 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6b00018020 con 0x7f6b18102ba0 2024-04-21T08:35:01.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.804+0000 7f6b157fa700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6b040a3c40 0x7f6b040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:01.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.804+0000 7f6b157fa700 1 -- 172.21.15.173:0/2624595045 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7f6b000cf120 con 0x7f6b18102ba0 2024-04-21T08:35:01.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.805+0000 7f6b177fe700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6b040a3c40 0x7f6b040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:01.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.805+0000 7f6b177fe700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6b040a3c40 0x7f6b040a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6b08005fd0 tx=0x7f6b08005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:01.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:01.809+0000 7f6b157fa700 1 -- 172.21.15.173:0/2624595045 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b0001f020 con 0x7f6b18102ba0 2024-04-21T08:35:02.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:02.015+0000 7f6b1dafc700 1 -- 172.21.15.173:0/2624595045 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6b18062e80 con 0x7f6b040a3c40 2024-04-21T08:35:02.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:02 smithi173 ceph-mon[31428]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:35:02.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:02 smithi173 ceph-mon[31428]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:02.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:02 smithi173 ceph-mon[31428]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:35:02.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:02 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:35:02.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:02 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:35:02.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:02 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:02.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:02 smithi111 ceph-mon[31493]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:35:02.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:02 smithi111 ceph-mon[31493]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:02.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:02 smithi111 ceph-mon[31493]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:35:02.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:02 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:35:02.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:02 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:35:02.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:02 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:02.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:02 smithi094 ceph-mon[27988]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:35:02.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:02 smithi094 ceph-mon[27988]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:02.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:02 smithi094 ceph-mon[27988]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:35:02.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:35:02.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:35:02.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:03.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:03 smithi111 ceph-mon[31493]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:03.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:03 smithi111 ceph-mon[31493]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:35:03.999 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:03 smithi173 ceph-mon[31428]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:04.000 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:03 smithi173 ceph-mon[31428]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:35:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:03 smithi094 ceph-mon[27988]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:03 smithi094 ceph-mon[27988]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:35:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:04 smithi111 ceph-mon[31493]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:04 smithi094 ceph-mon[27988]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:05.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:04 smithi173 ceph-mon[31428]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:05 smithi111 ceph-mon[31493]: from='client.? 172.21.15.173:0/3600812558' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b61b68b-048f-4904-923a-df40dda39925"}]: dispatch 2024-04-21T08:35:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:05 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b61b68b-048f-4904-923a-df40dda39925"}]: dispatch 2024-04-21T08:35:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:05 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b61b68b-048f-4904-923a-df40dda39925"}]': finished 2024-04-21T08:35:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:05 smithi111 ceph-mon[31493]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T08:35:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:05 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:05.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:05 smithi111 ceph-mon[31493]: from='client.? 172.21.15.173:0/3140324591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:35:06.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:05 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/3600812558' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b61b68b-048f-4904-923a-df40dda39925"}]: dispatch 2024-04-21T08:35:06.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:05 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b61b68b-048f-4904-923a-df40dda39925"}]: dispatch 2024-04-21T08:35:06.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:05 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b61b68b-048f-4904-923a-df40dda39925"}]': finished 2024-04-21T08:35:06.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:05 smithi094 ceph-mon[27988]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T08:35:06.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:05 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:06.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:05 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/3140324591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:35:06.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:05 smithi173 ceph-mon[31428]: from='client.? 172.21.15.173:0/3600812558' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b61b68b-048f-4904-923a-df40dda39925"}]: dispatch 2024-04-21T08:35:06.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:05 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b61b68b-048f-4904-923a-df40dda39925"}]: dispatch 2024-04-21T08:35:06.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:05 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b61b68b-048f-4904-923a-df40dda39925"}]': finished 2024-04-21T08:35:06.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:05 smithi173 ceph-mon[31428]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T08:35:06.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:05 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:06.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:05 smithi173 ceph-mon[31428]: from='client.? 172.21.15.173:0/3140324591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:35:06.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:06 smithi111 ceph-mon[31493]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:07.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:06 smithi094 ceph-mon[27988]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:07.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:06 smithi173 ceph-mon[31428]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:08.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:07 smithi094 ceph-mon[27988]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:08.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:07 smithi173 ceph-mon[31428]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:08.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:07 smithi111 ceph-mon[31493]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:10.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:10 smithi173 ceph-mon[31428]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:10.890 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:10 smithi111 ceph-mon[31493]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:11.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:10 smithi094 ceph-mon[27988]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:12.904 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:12 smithi173 ceph-mon[31428]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:12.904 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:35:12.904 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:12 smithi111 ceph-mon[31493]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:35:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:12 smithi094 ceph-mon[27988]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:35:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:13.911 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:13 smithi173 ceph-mon[31428]: Deploying daemon osd.8 on smithi173 2024-04-21T08:35:13.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:13 smithi111 ceph-mon[31493]: Deploying daemon osd.8 on smithi173 2024-04-21T08:35:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:13 smithi094 ceph-mon[27988]: Deploying daemon osd.8 on smithi173 2024-04-21T08:35:14.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:14 smithi173 ceph-mon[31428]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:14.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:14 smithi111 ceph-mon[31493]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:15.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:14 smithi094 ceph-mon[27988]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:16.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:16 smithi111 ceph-mon[31493]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:17.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:16 smithi094 ceph-mon[27988]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:17.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:16 smithi173 ceph-mon[31428]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:18.726 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:18 smithi173 ceph-mon[31428]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:18.727 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:18 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:18.727 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:18 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:18.727 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:18 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:18 smithi111 ceph-mon[31493]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:18 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:18 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:18 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:18 smithi094 ceph-mon[27988]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:18 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:18 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:18 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:20.887 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:20 smithi173 ceph-mon[31428]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:20 smithi111 ceph-mon[31493]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:20 smithi094 ceph-mon[27988]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:22.037 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 8 on host 'smithi173' 2024-04-21T08:35:22.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.037+0000 7f6b157fa700 1 -- 172.21.15.173:0/2624595045 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6b18062e80 con 0x7f6b040a3c40 2024-04-21T08:35:22.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.039+0000 7f6b1dafc700 1 -- 172.21.15.173:0/2624595045 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6b040a3c40 msgr2=0x7f6b040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:22.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6b040a3c40 0x7f6b040a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6b08005fd0 tx=0x7f6b08005e50 comp rx=0 tx=0).stop 2024-04-21T08:35:22.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 -- 172.21.15.173:0/2624595045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b18102ba0 msgr2=0x7f6b1819d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:22.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b18102ba0 0x7f6b1819d930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000b530 tx=0x7f6b0000b8f0 comp rx=0 tx=0).stop 2024-04-21T08:35:22.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 -- 172.21.15.173:0/2624595045 shutdown_connections 2024-04-21T08:35:22.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6b040a3c40 0x7f6b040a6100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:22.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b18103540 0x7f6b1819ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:22.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b181019a0 0x7f6b1806b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:22.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 --2- 172.21.15.173:0/2624595045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b18102ba0 0x7f6b1819d930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:22.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 -- 172.21.15.173:0/2624595045 >> 172.21.15.173:0/2624595045 conn(0x7f6b180fcf40 msgr2=0x7f6b18105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:22.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 -- 172.21.15.173:0/2624595045 shutdown_connections 2024-04-21T08:35:22.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:35:22.040+0000 7f6b1dafc700 1 -- 172.21.15.173:0/2624595045 wait complete. 2024-04-21T08:35:22.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:22.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:22.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:22.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: from='osd.8 [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:35:22.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:35:22.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:22 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:22.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:22.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:22.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:22.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: from='osd.8 [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:35:22.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:35:22.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:22 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:22.599 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.599 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.599 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:22.600 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:22.600 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.600 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:22.600 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: from='osd.8 [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:35:22.600 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:35:22.600 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.600 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:22.600 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:22 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:22.708 DEBUG:teuthology.orchestra.run.smithi173:osd.8> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.8.service 2024-04-21T08:35:22.710 INFO:tasks.cephadm:Deploying osd.9 on smithi094 with /dev/vg_nvme/lv_1... 2024-04-21T08:35:22.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:35:22.943 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:23.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:23 smithi173 ceph-mon[31428]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:35:23.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:23 smithi173 ceph-mon[31428]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:35:23.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:23 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:23.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:23 smithi173 ceph-mon[31428]: from='osd.8 [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:35:23.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:23 smithi173 ceph-mon[31428]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:35:23.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:35:23 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:35:23.232+0000 7fd08e1b6700 -1 osd.8 0 waiting for initial osdmap 2024-04-21T08:35:23.262 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:35:23 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:35:23.241+0000 7fd089fa9700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:35:23.340 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:23 smithi094 ceph-mon[27988]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:35:23.340 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:23 smithi094 ceph-mon[27988]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:35:23.340 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:23 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:23.340 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:23 smithi094 ceph-mon[27988]: from='osd.8 [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:35:23.340 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:23 smithi094 ceph-mon[27988]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:35:23.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:23 smithi111 ceph-mon[31493]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:35:23.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:23 smithi111 ceph-mon[31493]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:35:23.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:23 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:23.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:23 smithi111 ceph-mon[31493]: from='osd.8 [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:35:23.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:23 smithi111 ceph-mon[31493]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:35:24.362 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:24 smithi094 ceph-mon[27988]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:35:24.362 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:24 smithi094 ceph-mon[27988]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:35:24.362 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:24.362 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:24.362 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:24 smithi094 ceph-mon[27988]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:24.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:24 smithi173 ceph-mon[31428]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:35:24.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:24 smithi173 ceph-mon[31428]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:35:24.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:24.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:24.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:24 smithi173 ceph-mon[31428]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:24.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:24 smithi111 ceph-mon[31493]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:35:24.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:24 smithi111 ceph-mon[31493]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:35:24.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:24.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:24.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:24 smithi111 ceph-mon[31493]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:35:25.040 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:35:25.040 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:35:25.040 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-04-21T08:35:25.040 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-04-21T08:35:25.040 INFO:teuthology.orchestra.run.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00952057 s, 1.1 GB/s 2024-04-21T08:35:25.040 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-04-21T08:35:25.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:25 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:35:25.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:25 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:35:25.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:25 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:25.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:25 smithi094 ceph-mon[27988]: osd.8 [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] boot 2024-04-21T08:35:25.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:25 smithi094 ceph-mon[27988]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T08:35:25.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:25 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:25.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:25 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:35:25.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:25 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:35:25.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:25 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:25.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:25 smithi173 ceph-mon[31428]: osd.8 [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] boot 2024-04-21T08:35:25.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:25 smithi173 ceph-mon[31428]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T08:35:25.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:25 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:25.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:25 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:35:25.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:25 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:35:25.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:25 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:25.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:25 smithi111 ceph-mon[31493]: osd.8 [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] boot 2024-04-21T08:35:25.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:25 smithi111 ceph-mon[31493]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T08:35:25.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:25 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:35:25.833 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_1 2024-04-21T08:35:26.071 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:26.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:35:26.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:26.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:26.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:26.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:26.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:26.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:26.465 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:26.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:35:26.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:26.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:26.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:26.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:26.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:26.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:26.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:26.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:27.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:27 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi173 2024-04-21T08:35:27.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:27 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-21T08:35:27.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:27 smithi094 ceph-mon[27988]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:35:27.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:27 smithi094 ceph-mon[27988]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:35:27.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:27 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi173 2024-04-21T08:35:27.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:27 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-21T08:35:27.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:27 smithi173 ceph-mon[31428]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:35:27.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:27 smithi173 ceph-mon[31428]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:35:27.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:27 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi173 2024-04-21T08:35:27.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:27 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-21T08:35:27.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:27 smithi111 ceph-mon[31493]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:35:27.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:27 smithi111 ceph-mon[31493]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:35:27.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.736+0000 7fb9ad742700 1 -- 172.21.15.94:0/473870093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a8108780 msgr2=0x7fb9a810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:27.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.736+0000 7fb9ad742700 1 --2- 172.21.15.94:0/473870093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a8108780 0x7fb9a810abf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009990 tx=0x7fb9a0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:35:27.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.736+0000 7fb9ad742700 1 -- 172.21.15.94:0/473870093 shutdown_connections 2024-04-21T08:35:27.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.736+0000 7fb9ad742700 1 --2- 172.21.15.94:0/473870093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a8108780 0x7fb9a810abf0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:27.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.736+0000 7fb9ad742700 1 --2- 172.21.15.94:0/473870093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a8103520 0x7fb9a8105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:27.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.736+0000 7fb9ad742700 1 --2- 172.21.15.94:0/473870093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8105e50 0x7fb9a8108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:27.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.736+0000 7fb9ad742700 1 -- 172.21.15.94:0/473870093 >> 172.21.15.94:0/473870093 conn(0x7fb9a80fcf30 msgr2=0x7fb9a80ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:27.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.736+0000 7fb9ad742700 1 -- 172.21.15.94:0/473870093 shutdown_connections 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.736+0000 7fb9ad742700 1 -- 172.21.15.94:0/473870093 wait complete. 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9ad742700 1 Processor -- start 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9ad742700 1 -- start start 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9ad742700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8103520 0x7fb9a8198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9ad742700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a8105e50 0x7fb9a8198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9ad742700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a8108780 0x7fb9a81a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9ad742700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a810e4f0 con 0x7fb9a8108780 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9ad742700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a810e370 con 0x7fb9a8103520 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9ad742700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a810e670 con 0x7fb9a8105e50 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9a77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a8108780 0x7fb9a81a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9a77fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a8108780 0x7fb9a81a0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56972/0 (socket says 172.21.15.94:56972) 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9a77fe700 1 -- 172.21.15.94:0/26167811 learned_addr learned my addr 172.21.15.94:0/26167811 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9a6ffd700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8103520 0x7fb9a8198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9a67fc700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a8105e50 0x7fb9a8198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9a77fe700 1 -- 172.21.15.94:0/26167811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a8105e50 msgr2=0x7fb9a8198ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9a77fe700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a8105e50 0x7fb9a8198ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9a77fe700 1 -- 172.21.15.94:0/26167811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8103520 msgr2=0x7fb9a8198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9a77fe700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8103520 0x7fb9a8198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.737+0000 7fb9a77fe700 1 -- 172.21.15.94:0/26167811 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a81a0710 con 0x7fb9a8108780 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.738+0000 7fb9a77fe700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a8108780 0x7fb9a81a0070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb9a00057f0 tx=0x7fb9a0009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.738+0000 7fb99ffff700 1 -- 172.21.15.94:0/26167811 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0027070 con 0x7fb9a8108780 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.738+0000 7fb9ad742700 1 -- 172.21.15.94:0/26167811 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a81a09a0 con 0x7fb9a8108780 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.738+0000 7fb99ffff700 1 -- 172.21.15.94:0/26167811 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9a0004140 con 0x7fb9a8108780 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.738+0000 7fb99ffff700 1 -- 172.21.15.94:0/26167811 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0022d60 con 0x7fb9a8108780 2024-04-21T08:35:27.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.738+0000 7fb9ad742700 1 -- 172.21.15.94:0/26167811 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a81a0e50 con 0x7fb9a8108780 2024-04-21T08:35:27.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.739+0000 7fb9ad742700 1 -- 172.21.15.94:0/26167811 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a8069730 con 0x7fb9a8108780 2024-04-21T08:35:27.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.740+0000 7fb99ffff700 1 -- 172.21.15.94:0/26167811 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb9a002a070 con 0x7fb9a8108780 2024-04-21T08:35:27.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.740+0000 7fb99ffff700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9940a3c40 0x7fb9940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:27.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.740+0000 7fb9a6ffd700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9940a3c40 0x7fb9940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:27.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.740+0000 7fb99ffff700 1 -- 172.21.15.94:0/26167811 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fb9a00d0d20 con 0x7fb9a8108780 2024-04-21T08:35:27.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.741+0000 7fb9a6ffd700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9940a3c40 0x7fb9940a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb99800f440 tx=0x7fb9980079e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:27.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.744+0000 7fb99ffff700 1 -- 172.21.15.94:0/26167811 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9a0013070 con 0x7fb9a8108780 2024-04-21T08:35:27.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:27.953+0000 7fb9ad742700 1 -- 172.21.15.94:0/26167811 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb9a8062e80 con 0x7fb9940a3c40 2024-04-21T08:35:28.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:28 smithi094 ceph-mon[27988]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:28.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:28 smithi094 ceph-mon[27988]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:35:28.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:35:28.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:35:28.834 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:28 smithi111 ceph-mon[31493]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:28 smithi111 ceph-mon[31493]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:35:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:35:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:35:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:29.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:28 smithi173 ceph-mon[31428]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:29.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:28 smithi173 ceph-mon[31428]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:35:29.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:35:29.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:35:29.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:29.916 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:29 smithi094 ceph-mon[27988]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:29 smithi111 ceph-mon[31493]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:29 smithi173 ceph-mon[31428]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:30.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:30 smithi111 ceph-mon[31493]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:31.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:30 smithi094 ceph-mon[27988]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:31.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:30 smithi173 ceph-mon[31428]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:31 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1301688051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f384696-fd22-4f83-b99c-e55623ce934e"}]: dispatch 2024-04-21T08:35:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:31 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1301688051' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f384696-fd22-4f83-b99c-e55623ce934e"}]': finished 2024-04-21T08:35:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:31 smithi111 ceph-mon[31493]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T08:35:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:31 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3260702412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:35:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:31 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1301688051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f384696-fd22-4f83-b99c-e55623ce934e"}]: dispatch 2024-04-21T08:35:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:31 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1301688051' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f384696-fd22-4f83-b99c-e55623ce934e"}]': finished 2024-04-21T08:35:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:31 smithi094 ceph-mon[27988]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T08:35:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:31 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3260702412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:35:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:31 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1301688051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f384696-fd22-4f83-b99c-e55623ce934e"}]: dispatch 2024-04-21T08:35:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:31 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1301688051' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f384696-fd22-4f83-b99c-e55623ce934e"}]': finished 2024-04-21T08:35:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:31 smithi173 ceph-mon[31428]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T08:35:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:31 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3260702412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:35:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:32 smithi111 ceph-mon[31493]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-21T08:35:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:32 smithi094 ceph-mon[27988]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-21T08:35:33.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:32 smithi173 ceph-mon[31428]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-21T08:35:33.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:33 smithi111 ceph-mon[31493]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-21T08:35:34.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:33 smithi094 ceph-mon[27988]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-21T08:35:34.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:33 smithi173 ceph-mon[31428]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-21T08:35:36.886 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:36 smithi094 ceph-mon[27988]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:35:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:36 smithi111 ceph-mon[31493]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:35:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:36 smithi173 ceph-mon[31428]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:35:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:38 smithi094 ceph-mon[27988]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:35:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:35:38.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:38 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:38.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:38 smithi111 ceph-mon[31493]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:35:38.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:35:38.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:38 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:39.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:38 smithi173 ceph-mon[31428]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:35:39.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:35:39.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:38 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:39.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:39 smithi111 ceph-mon[31493]: Deploying daemon osd.9 on smithi094 2024-04-21T08:35:40.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:39 smithi094 ceph-mon[27988]: Deploying daemon osd.9 on smithi094 2024-04-21T08:35:40.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:39 smithi173 ceph-mon[31428]: Deploying daemon osd.9 on smithi094 2024-04-21T08:35:40.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:40 smithi094 ceph-mon[27988]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:35:40.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:40 smithi111 ceph-mon[31493]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:35:41.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:40 smithi173 ceph-mon[31428]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:35:42.715 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:42 smithi094 ceph-mon[27988]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:42.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:42 smithi111 ceph-mon[31493]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:43.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:42 smithi173 ceph-mon[31428]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:44.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:44 smithi094 ceph-mon[27988]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:44.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:44.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:44.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:44.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:44 smithi111 ceph-mon[31493]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:44.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:44.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:44.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:45.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:44 smithi173 ceph-mon[31428]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:45.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:45.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:45.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:46.742 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:46 smithi094 ceph-mon[27988]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:46.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:46 smithi111 ceph-mon[31493]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:47.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:46 smithi173 ceph-mon[31428]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:48.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.083+0000 7fb99ffff700 1 -- 172.21.15.94:0/26167811 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb9a8062e80 con 0x7fb9940a3c40 2024-04-21T08:35:48.082 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 9 on host 'smithi094' 2024-04-21T08:35:48.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.084+0000 7fb9ad742700 1 -- 172.21.15.94:0/26167811 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9940a3c40 msgr2=0x7fb9940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:48.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.084+0000 7fb9ad742700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9940a3c40 0x7fb9940a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb99800f440 tx=0x7fb9980079e0 comp rx=0 tx=0).stop 2024-04-21T08:35:48.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.084+0000 7fb9ad742700 1 -- 172.21.15.94:0/26167811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a8108780 msgr2=0x7fb9a81a0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:48.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.084+0000 7fb9ad742700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a8108780 0x7fb9a81a0070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb9a00057f0 tx=0x7fb9a0009510 comp rx=0 tx=0).stop 2024-04-21T08:35:48.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.084+0000 7fb9ad742700 1 -- 172.21.15.94:0/26167811 shutdown_connections 2024-04-21T08:35:48.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.084+0000 7fb9ad742700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9940a3c40 0x7fb9940a6100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:48.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.084+0000 7fb9ad742700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8103520 0x7fb9a8198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:48.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.085+0000 7fb9ad742700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a8108780 0x7fb9a81a0070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:48.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.085+0000 7fb9ad742700 1 --2- 172.21.15.94:0/26167811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a8105e50 0x7fb9a8198ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:48.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.085+0000 7fb9ad742700 1 -- 172.21.15.94:0/26167811 >> 172.21.15.94:0/26167811 conn(0x7fb9a80fcf30 msgr2=0x7fb9a8103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:48.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.085+0000 7fb9ad742700 1 -- 172.21.15.94:0/26167811 shutdown_connections 2024-04-21T08:35:48.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:35:48.085+0000 7fb9ad742700 1 -- 172.21.15.94:0/26167811 wait complete. 2024-04-21T08:35:48.682 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:48 smithi094 ceph-mon[27988]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:48.682 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:48 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:48.683 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:48 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:48.683 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:48 smithi094 ceph-mon[27988]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:35:48.708 DEBUG:teuthology.orchestra.run.smithi094:osd.9> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.9.service 2024-04-21T08:35:48.711 INFO:tasks.cephadm:Deploying osd.10 on smithi111 with /dev/vg_nvme/lv_1... 2024-04-21T08:35:48.711 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:35:48.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:48 smithi111 ceph-mon[31493]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:48.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:48 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:48.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:48 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:48.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:48 smithi111 ceph-mon[31493]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:35:48.957 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:35:49.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:48 smithi173 ceph-mon[31428]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:49.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:48 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:49.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:48 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:49.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:48 smithi173 ceph-mon[31428]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:35:50.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:49 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:50.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:49 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:50.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:49 smithi111 ceph-mon[31493]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:35:50.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:49 smithi111 ceph-mon[31493]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:35:50.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:49 smithi111 ceph-mon[31493]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:35:50.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:49 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:50.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:49 smithi111 ceph-mon[31493]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:50.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:49 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:50.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:49 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:50.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:49 smithi094 ceph-mon[27988]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:35:50.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:49 smithi094 ceph-mon[27988]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:35:50.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:49 smithi094 ceph-mon[27988]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:35:50.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:49 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:50.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:49 smithi094 ceph-mon[27988]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:50.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:49 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:50.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:49 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:50.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:49 smithi173 ceph-mon[31428]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:35:50.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:49 smithi173 ceph-mon[31428]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:35:50.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:49 smithi173 ceph-mon[31428]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-21T08:35:50.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:49 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:50.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:49 smithi173 ceph-mon[31428]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:50.517 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:35:50 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:35:50.506+0000 7f8100624700 -1 osd.9 0 waiting for initial osdmap 2024-04-21T08:35:50.517 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:35:50 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:35:50.518+0000 7f80f840f700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:35:50.965 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:35:50.965 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:35:50.965 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T08:35:50.965 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T08:35:50.965 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111803 s, 938 MB/s 2024-04-21T08:35:50.965 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T08:35:51.207 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:51 smithi094 ceph-mon[27988]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:35:51.207 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:51 smithi094 ceph-mon[27988]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:35:51.207 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:51.207 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:51 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:51.207 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:51 smithi094 ceph-mon[27988]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' 2024-04-21T08:35:51.293 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:51 smithi111 ceph-mon[31493]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:35:51.293 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:51 smithi111 ceph-mon[31493]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:35:51.293 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:51.293 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:51 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:51.294 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:51 smithi111 ceph-mon[31493]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' 2024-04-21T08:35:51.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:51 smithi173 ceph-mon[31428]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-21T08:35:51.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:51 smithi173 ceph-mon[31428]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:35:51.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:51.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:51 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:51.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:51 smithi173 ceph-mon[31428]: from='osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749]' entity='osd.9' 2024-04-21T08:35:51.614 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_1 2024-04-21T08:35:51.854 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:35:52.121 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:52 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:35:52.121 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:52 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:35:52.121 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:52 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:52.121 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:52 smithi094 ceph-mon[27988]: osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] boot 2024-04-21T08:35:52.121 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:52 smithi094 ceph-mon[27988]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T08:35:52.121 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:52 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:52.121 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:52 smithi094 ceph-mon[27988]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:52.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:52 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:35:52.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:52 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:35:52.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:52 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:52.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:52 smithi111 ceph-mon[31493]: osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] boot 2024-04-21T08:35:52.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:52 smithi111 ceph-mon[31493]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T08:35:52.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:52 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:52.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:52 smithi111 ceph-mon[31493]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:52.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:52 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:35:52.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:52 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:35:52.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:52 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:52.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:52 smithi173 ceph-mon[31428]: osd.9 [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] boot 2024-04-21T08:35:52.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:52 smithi173 ceph-mon[31428]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T08:35:52.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:52 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:35:52.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:52 smithi173 ceph-mon[31428]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.501+0000 7f7617100700 1 -- 172.21.15.111:0/1243720968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7610100b70 msgr2=0x7f7610100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.501+0000 7f7617100700 1 --2- 172.21.15.111:0/1243720968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7610100b70 0x7f7610100fd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009990 tx=0x7f75f8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.501+0000 7f7617100700 1 -- 172.21.15.111:0/1243720968 shutdown_connections 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.501+0000 7f7617100700 1 --2- 172.21.15.111:0/1243720968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7610101510 0x7f761010b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.501+0000 7f7617100700 1 --2- 172.21.15.111:0/1243720968 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f76101040f0 0x7f7610104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.501+0000 7f7617100700 1 --2- 172.21.15.111:0/1243720968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7610100b70 0x7f7610100fd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.501+0000 7f7617100700 1 -- 172.21.15.111:0/1243720968 >> 172.21.15.111:0/1243720968 conn(0x7f76100fc6f0 msgr2=0x7f76100feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.501+0000 7f7617100700 1 -- 172.21.15.111:0/1243720968 shutdown_connections 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.501+0000 7f7617100700 1 -- 172.21.15.111:0/1243720968 wait complete. 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f7617100700 1 Processor -- start 2024-04-21T08:35:53.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f7617100700 1 -- start start 2024-04-21T08:35:53.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f7617100700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7610100b70 0x7f7610198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f7617100700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7610101510 0x7f76101990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f7617100700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76101040f0 0x7f76101a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f7617100700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f761010e560 con 0x7f7610100b70 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f7617100700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f761010e3e0 con 0x7f7610101510 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f7617100700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f761010e6e0 con 0x7f76101040f0 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f760ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7610101510 0x7f76101990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f760ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7610101510 0x7f76101990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33578/0 (socket says 172.21.15.111:33578) 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f760ffff700 1 -- 172.21.15.111:0/2867323356 learned_addr learned my addr 172.21.15.111:0/2867323356 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f7614e9c700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7610100b70 0x7f7610198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f761569d700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76101040f0 0x7f76101a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f760ffff700 1 -- 172.21.15.111:0/2867323356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76101040f0 msgr2=0x7f76101a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f760ffff700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76101040f0 0x7f76101a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f760ffff700 1 -- 172.21.15.111:0/2867323356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7610100b70 msgr2=0x7f7610198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f760ffff700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7610100b70 0x7f7610198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.502+0000 7f760ffff700 1 -- 172.21.15.111:0/2867323356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76101a0800 con 0x7f7610101510 2024-04-21T08:35:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.503+0000 7f760ffff700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7610101510 0x7f76101990e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f75f8006010 tx=0x7f75f80047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.503+0000 7f760dffb700 1 -- 172.21.15.111:0/2867323356 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f8027070 con 0x7f7610101510 2024-04-21T08:35:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.503+0000 7f760dffb700 1 -- 172.21.15.111:0/2867323356 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75f8004080 con 0x7f7610101510 2024-04-21T08:35:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.503+0000 7f7617100700 1 -- 172.21.15.111:0/2867323356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76101a0a90 con 0x7f7610101510 2024-04-21T08:35:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.503+0000 7f760dffb700 1 -- 172.21.15.111:0/2867323356 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f8022dc0 con 0x7f7610101510 2024-04-21T08:35:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.503+0000 7f7617100700 1 -- 172.21.15.111:0/2867323356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76101a0f40 con 0x7f7610101510 2024-04-21T08:35:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.504+0000 7f7617100700 1 -- 172.21.15.111:0/2867323356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7610069730 con 0x7f7610101510 2024-04-21T08:35:53.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.505+0000 7f760dffb700 1 -- 172.21.15.111:0/2867323356 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f75f802a070 con 0x7f7610101510 2024-04-21T08:35:53.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.506+0000 7f760dffb700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f75fc0a3cc0 0x7f75fc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:53.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.506+0000 7f760dffb700 1 -- 172.21.15.111:0/2867323356 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f75f80d1210 con 0x7f7610101510 2024-04-21T08:35:53.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.506+0000 7f7614e9c700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f75fc0a3cc0 0x7f75fc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:53.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.507+0000 7f7614e9c700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f75fc0a3cc0 0x7f75fc0a6180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f761006ae90 tx=0x7f760000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:53.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.510+0000 7f760dffb700 1 -- 172.21.15.111:0/2867323356 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75f8031070 con 0x7f7610101510 2024-04-21T08:35:53.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:35:53.722+0000 7f7617100700 1 -- 172.21.15.111:0/2867323356 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7610062e80 con 0x7f75fc0a3cc0 2024-04-21T08:35:54.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi094 2024-04-21T08:35:54.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi094 to 4272M 2024-04-21T08:35:54.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:54.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:54.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:54.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:54.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:54.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:35:54.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:53 smithi094 ceph-mon[27988]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:35:54.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi094 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi094 to 4272M 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:54.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:54.014 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:54.014 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.014 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:35:54.014 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:53 smithi173 ceph-mon[31428]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:35:54.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi094 2024-04-21T08:35:54.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi094 to 4272M 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:35:54.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:53 smithi111 ceph-mon[31493]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:35:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:54 smithi111 ceph-mon[31493]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:54 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:35:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:54 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:35:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:54 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:54 smithi094 ceph-mon[27988]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:54 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:35:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:54 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:35:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:54 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:54 smithi173 ceph-mon[31428]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:55.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:54 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:35:55.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:54 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:35:55.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:54 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:55.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:55 smithi111 ceph-mon[31493]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:35:56.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:55 smithi094 ceph-mon[27988]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:35:56.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:55 smithi173 ceph-mon[31428]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:35:56.997 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:56 smithi111 ceph-mon[31493]: from='client.? 172.21.15.111:0/1661077000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f6aaa6b-83b7-4f4e-8626-6bde0352caf9"}]: dispatch 2024-04-21T08:35:56.997 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:56 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f6aaa6b-83b7-4f4e-8626-6bde0352caf9"}]: dispatch 2024-04-21T08:35:56.997 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:56 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f6aaa6b-83b7-4f4e-8626-6bde0352caf9"}]': finished 2024-04-21T08:35:56.997 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:56 smithi111 ceph-mon[31493]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T08:35:56.997 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:35:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:56 smithi094 ceph-mon[27988]: from='client.? 172.21.15.111:0/1661077000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f6aaa6b-83b7-4f4e-8626-6bde0352caf9"}]: dispatch 2024-04-21T08:35:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:56 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f6aaa6b-83b7-4f4e-8626-6bde0352caf9"}]: dispatch 2024-04-21T08:35:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:56 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f6aaa6b-83b7-4f4e-8626-6bde0352caf9"}]': finished 2024-04-21T08:35:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:56 smithi094 ceph-mon[27988]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T08:35:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:35:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:56 smithi173 ceph-mon[31428]: from='client.? 172.21.15.111:0/1661077000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f6aaa6b-83b7-4f4e-8626-6bde0352caf9"}]: dispatch 2024-04-21T08:35:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:56 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f6aaa6b-83b7-4f4e-8626-6bde0352caf9"}]: dispatch 2024-04-21T08:35:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:56 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f6aaa6b-83b7-4f4e-8626-6bde0352caf9"}]': finished 2024-04-21T08:35:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:56 smithi173 ceph-mon[31428]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T08:35:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:35:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:57 smithi094 ceph-mon[27988]: from='client.? 172.21.15.111:0/2399487259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:35:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:35:57 smithi094 ceph-mon[27988]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:35:58.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:57 smithi173 ceph-mon[31428]: from='client.? 172.21.15.111:0/2399487259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:35:58.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:35:57 smithi173 ceph-mon[31428]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:35:58.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:57 smithi111 ceph-mon[31493]: from='client.? 172.21.15.111:0/2399487259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:35:58.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:35:57 smithi111 ceph-mon[31493]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:00 smithi111 ceph-mon[31493]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:01.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:00 smithi094 ceph-mon[27988]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:01.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:00 smithi173 ceph-mon[31428]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:02.874 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:02 smithi111 ceph-mon[31493]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:03.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:02 smithi094 ceph-mon[27988]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:03.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:02 smithi173 ceph-mon[31428]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:04.934 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:04 smithi111 ceph-mon[31493]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:04.935 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:36:04.935 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:04 smithi094 ceph-mon[27988]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:36:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:05.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:04 smithi173 ceph-mon[31428]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:05.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:36:05.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:05.680 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:05 smithi111 ceph-mon[31493]: Deploying daemon osd.10 on smithi111 2024-04-21T08:36:06.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:05 smithi094 ceph-mon[27988]: Deploying daemon osd.10 on smithi111 2024-04-21T08:36:06.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:05 smithi173 ceph-mon[31428]: Deploying daemon osd.10 on smithi111 2024-04-21T08:36:06.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:06 smithi111 ceph-mon[31493]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:07.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:06 smithi094 ceph-mon[27988]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:07.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:06 smithi173 ceph-mon[31428]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:08.855 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:08 smithi111 ceph-mon[31493]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:08 smithi094 ceph-mon[27988]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:09.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:08 smithi173 ceph-mon[31428]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:10.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:10 smithi111 ceph-mon[31493]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:10.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:10.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:10.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:11.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:10 smithi094 ceph-mon[27988]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:11.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:11.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:11.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:11.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:10 smithi173 ceph-mon[31428]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:11.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:11.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:11.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:12 smithi111 ceph-mon[31493]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:12 smithi094 ceph-mon[27988]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:13.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:12 smithi173 ceph-mon[31428]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:13.637 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 10 on host 'smithi111' 2024-04-21T08:36:13.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.638+0000 7f760dffb700 1 -- 172.21.15.111:0/2867323356 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f7610062e80 con 0x7f75fc0a3cc0 2024-04-21T08:36:13.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.640+0000 7f7617100700 1 -- 172.21.15.111:0/2867323356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f75fc0a3cc0 msgr2=0x7f75fc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:13.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.640+0000 7f7617100700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f75fc0a3cc0 0x7f75fc0a6180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f761006ae90 tx=0x7f760000b410 comp rx=0 tx=0).stop 2024-04-21T08:36:13.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.641+0000 7f7617100700 1 -- 172.21.15.111:0/2867323356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7610101510 msgr2=0x7f76101990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:13.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.641+0000 7f7617100700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7610101510 0x7f76101990e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f75f8006010 tx=0x7f75f80047b0 comp rx=0 tx=0).stop 2024-04-21T08:36:13.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.641+0000 7f7617100700 1 -- 172.21.15.111:0/2867323356 shutdown_connections 2024-04-21T08:36:13.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.641+0000 7f7617100700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f75fc0a3cc0 0x7f75fc0a6180 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:13.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.641+0000 7f7617100700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76101040f0 0x7f76101a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:13.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.641+0000 7f7617100700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7610100b70 0x7f7610198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:13.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.641+0000 7f7617100700 1 --2- 172.21.15.111:0/2867323356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7610101510 0x7f76101990e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:13.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.641+0000 7f7617100700 1 -- 172.21.15.111:0/2867323356 >> 172.21.15.111:0/2867323356 conn(0x7f76100fc6f0 msgr2=0x7f76101071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:13.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.641+0000 7f7617100700 1 -- 172.21.15.111:0/2867323356 shutdown_connections 2024-04-21T08:36:13.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T08:36:13.641+0000 7f7617100700 1 -- 172.21.15.111:0/2867323356 wait complete. 2024-04-21T08:36:13.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:13.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:13.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:13.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:13.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: from='osd.10 [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:36:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:13 smithi111 ceph-mon[31493]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:36:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:14.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:14.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: from='osd.10 [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:36:14.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:13 smithi094 ceph-mon[27988]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:36:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:14.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:14.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:14.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:14.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: from='osd.10 [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:36:14.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:13 smithi173 ceph-mon[31428]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:36:14.313 DEBUG:teuthology.orchestra.run.smithi111:osd.10> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.10.service 2024-04-21T08:36:14.315 INFO:tasks.cephadm:Deploying osd.11 on smithi173 with /dev/vg_nvme/lv_1... 2024-04-21T08:36:14.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:36:14.571 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:36:15.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:15 smithi111 ceph-mon[31493]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:36:15.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:15 smithi111 ceph-mon[31493]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:36:15.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:15 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:15.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:15 smithi111 ceph-mon[31493]: from='osd.10 [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:36:15.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:15 smithi111 ceph-mon[31493]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:36:15.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:15 smithi173 ceph-mon[31428]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:36:15.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:15 smithi173 ceph-mon[31428]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:36:15.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:15 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:15.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:15 smithi173 ceph-mon[31428]: from='osd.10 [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:36:15.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:15 smithi173 ceph-mon[31428]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:36:16.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:15 smithi094 ceph-mon[27988]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:36:16.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:15 smithi094 ceph-mon[27988]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:36:16.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:15 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:16.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:15 smithi094 ceph-mon[27988]: from='osd.10 [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:36:16.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:15 smithi094 ceph-mon[27988]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T08:36:16.438 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:36:16 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:36:16.065+0000 7fba18b65700 -1 osd.10 0 waiting for initial osdmap 2024-04-21T08:36:16.438 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:36:16 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:36:16.080+0000 7fba1014f700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:36:16.664 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:36:16.665 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:36:16.665 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-21T08:36:16.665 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-21T08:36:16.665 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114135 s, 919 MB/s 2024-04-21T08:36:16.665 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-21T08:36:16.869 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:16 smithi111 ceph-mon[31493]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:16.869 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:16 smithi111 ceph-mon[31493]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:36:16.870 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:16 smithi111 ceph-mon[31493]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:36:16.870 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:16.870 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:16.870 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:16 smithi111 ceph-mon[31493]: from='osd.10 ' entity='osd.10' 2024-04-21T08:36:17.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:16 smithi094 ceph-mon[27988]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:17.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:16 smithi094 ceph-mon[27988]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:36:17.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:16 smithi094 ceph-mon[27988]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:36:17.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:17.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:17.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:16 smithi094 ceph-mon[27988]: from='osd.10 ' entity='osd.10' 2024-04-21T08:36:17.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:16 smithi173 ceph-mon[31428]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:17.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:16 smithi173 ceph-mon[31428]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T08:36:17.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:16 smithi173 ceph-mon[31428]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:36:17.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:17.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:17.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:16 smithi173 ceph-mon[31428]: from='osd.10 ' entity='osd.10' 2024-04-21T08:36:17.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-04-21T08:36:17.805 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:36:17.907 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: osd.10 [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] boot 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:17.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:17.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:17.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:17.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:17 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:17.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: osd.10 [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] boot 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:17.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:17.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:17.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:17.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:17.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:17 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:18.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:36:18.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:36:18.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:18.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: osd.10 [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] boot 2024-04-21T08:36:18.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T08:36:18.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:36:18.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:36:18.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:18.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:18.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:18.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:18.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:18.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:18.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:18.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:18.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:18.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:17 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:18 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi111 2024-04-21T08:36:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:18 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-21T08:36:18.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:18 smithi111 ceph-mon[31493]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:18.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:18 smithi111 ceph-mon[31493]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:36:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:18 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi111 2024-04-21T08:36:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:18 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-21T08:36:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:18 smithi094 ceph-mon[27988]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:18 smithi094 ceph-mon[27988]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:36:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:18 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi111 2024-04-21T08:36:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:18 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-21T08:36:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:18 smithi173 ceph-mon[31428]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:36:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:18 smithi173 ceph-mon[31428]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:36:19.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.406+0000 7fe40c5a8700 1 -- 172.21.15.173:0/1505673836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe404102aa0 msgr2=0x7fe404102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:19.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.406+0000 7fe40c5a8700 1 --2- 172.21.15.173:0/1505673836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe404102aa0 0x7fe404102ec0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe400009990 tx=0x7fe400009ca0 comp rx=0 tx=0).stop 2024-04-21T08:36:19.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 -- 172.21.15.173:0/1505673836 shutdown_connections 2024-04-21T08:36:19.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 --2- 172.21.15.173:0/1505673836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404104640 0x7fe40410aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:19.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 --2- 172.21.15.173:0/1505673836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe404102aa0 0x7fe404102ec0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:19.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 --2- 172.21.15.173:0/1505673836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe404103ca0 0x7fe404104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:19.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 -- 172.21.15.173:0/1505673836 >> 172.21.15.173:0/1505673836 conn(0x7fe4040fe040 msgr2=0x7fe404100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:19.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 -- 172.21.15.173:0/1505673836 shutdown_connections 2024-04-21T08:36:19.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 -- 172.21.15.173:0/1505673836 wait complete. 2024-04-21T08:36:19.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 Processor -- start 2024-04-21T08:36:19.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 -- start start 2024-04-21T08:36:19.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404102aa0 0x7fe4040767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:19.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe404103ca0 0x7fe404074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:19.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe404104640 0x7fe404075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:19.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe40410e590 con 0x7fe404104640 2024-04-21T08:36:19.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe40410e410 con 0x7fe404102aa0 2024-04-21T08:36:19.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.407+0000 7fe40c5a8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe40410e710 con 0x7fe404103ca0 2024-04-21T08:36:19.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe409b43700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe404103ca0 0x7fe404074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:19.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe409b43700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe404103ca0 0x7fe404074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:48090/0 (socket says 172.21.15.173:48090) 2024-04-21T08:36:19.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe409b43700 1 -- 172.21.15.173:0/828214355 learned_addr learned my addr 172.21.15.173:0/828214355 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-21T08:36:19.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe40a344700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404102aa0 0x7fe4040767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:19.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe40ab45700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe404104640 0x7fe404075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:19.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe409b43700 1 -- 172.21.15.173:0/828214355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404102aa0 msgr2=0x7fe4040767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:19.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe409b43700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404102aa0 0x7fe4040767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:19.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe409b43700 1 -- 172.21.15.173:0/828214355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe404104640 msgr2=0x7fe404075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:19.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe409b43700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe404104640 0x7fe404075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:19.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe409b43700 1 -- 172.21.15.173:0/828214355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe404075c20 con 0x7fe404103ca0 2024-04-21T08:36:19.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe409b43700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe404103ca0 0x7fe404074e30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400c920 tx=0x7fe3f400cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:19.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe3fb7fe700 1 -- 172.21.15.173:0/828214355 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4004a30 con 0x7fe404103ca0 2024-04-21T08:36:19.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe3fb7fe700 1 -- 172.21.15.173:0/828214355 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3f40054d0 con 0x7fe404103ca0 2024-04-21T08:36:19.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe40c5a8700 1 -- 172.21.15.173:0/828214355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4041a2df0 con 0x7fe404103ca0 2024-04-21T08:36:19.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe3fb7fe700 1 -- 172.21.15.173:0/828214355 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4007c70 con 0x7fe404103ca0 2024-04-21T08:36:19.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.408+0000 7fe40c5a8700 1 -- 172.21.15.173:0/828214355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4041a3250 con 0x7fe404103ca0 2024-04-21T08:36:19.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.409+0000 7fe40c5a8700 1 -- 172.21.15.173:0/828214355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe404069730 con 0x7fe404103ca0 2024-04-21T08:36:19.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.410+0000 7fe3fb7fe700 1 -- 172.21.15.173:0/828214355 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe3f4004c60 con 0x7fe404103ca0 2024-04-21T08:36:19.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.411+0000 7fe3fb7fe700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe3f00a3c40 0x7fe3f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:19.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.411+0000 7fe3fb7fe700 1 -- 172.21.15.173:0/828214355 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7fe3f40cfc30 con 0x7fe404103ca0 2024-04-21T08:36:19.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.411+0000 7fe40a344700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe3f00a3c40 0x7fe3f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:19.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.412+0000 7fe40a344700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe3f00a3c40 0x7fe3f00a6100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe4000038a0 tx=0x7fe400005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:19.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.416+0000 7fe3fb7fe700 1 -- 172.21.15.173:0/828214355 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3f4014030 con 0x7fe404103ca0 2024-04-21T08:36:19.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:19.624+0000 7fe40c5a8700 1 -- 172.21.15.173:0/828214355 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe404062e80 con 0x7fe3f00a3c40 2024-04-21T08:36:20.489 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:20 smithi173 ceph-mon[31428]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:36:20.489 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:20 smithi173 ceph-mon[31428]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:20.489 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:20 smithi173 ceph-mon[31428]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:20.489 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:20.489 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:20.489 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:20 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:20.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:20 smithi094 ceph-mon[27988]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:36:20.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:20 smithi094 ceph-mon[27988]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:20.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:20 smithi094 ceph-mon[27988]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:20.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:20.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:20.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:20 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:20.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:20 smithi111 ceph-mon[31493]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:36:20.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:20 smithi111 ceph-mon[31493]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:20.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:20 smithi111 ceph-mon[31493]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:20.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:20 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:20.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:20 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:20.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:20 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:21.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:21 smithi094 ceph-mon[27988]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:36:21.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:21 smithi173 ceph-mon[31428]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:36:21.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:21 smithi111 ceph-mon[31493]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:36:22.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:22 smithi094 ceph-mon[27988]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-21T08:36:22.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:22 smithi111 ceph-mon[31493]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-21T08:36:22.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:22 smithi173 ceph-mon[31428]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-21T08:36:23.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:23 smithi173 ceph-mon[31428]: from='client.? 172.21.15.173:0/1909091282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7529c3d-ab5c-4963-9b8a-40aa40bf97dc"}]: dispatch 2024-04-21T08:36:23.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:23 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7529c3d-ab5c-4963-9b8a-40aa40bf97dc"}]: dispatch 2024-04-21T08:36:23.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:23 smithi173 ceph-mon[31428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7529c3d-ab5c-4963-9b8a-40aa40bf97dc"}]': finished 2024-04-21T08:36:23.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:23 smithi173 ceph-mon[31428]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T08:36:23.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:23 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:23.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:23 smithi173 ceph-mon[31428]: from='client.? 172.21.15.173:0/2592403710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:36:23.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:23 smithi111 ceph-mon[31493]: from='client.? 172.21.15.173:0/1909091282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7529c3d-ab5c-4963-9b8a-40aa40bf97dc"}]: dispatch 2024-04-21T08:36:23.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:23 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7529c3d-ab5c-4963-9b8a-40aa40bf97dc"}]: dispatch 2024-04-21T08:36:23.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:23 smithi111 ceph-mon[31493]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7529c3d-ab5c-4963-9b8a-40aa40bf97dc"}]': finished 2024-04-21T08:36:23.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:23 smithi111 ceph-mon[31493]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T08:36:23.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:23 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:23.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:23 smithi111 ceph-mon[31493]: from='client.? 172.21.15.173:0/2592403710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:36:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:23 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/1909091282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7529c3d-ab5c-4963-9b8a-40aa40bf97dc"}]: dispatch 2024-04-21T08:36:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:23 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7529c3d-ab5c-4963-9b8a-40aa40bf97dc"}]: dispatch 2024-04-21T08:36:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:23 smithi094 ceph-mon[27988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7529c3d-ab5c-4963-9b8a-40aa40bf97dc"}]': finished 2024-04-21T08:36:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:23 smithi094 ceph-mon[27988]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T08:36:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:23 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:23 smithi094 ceph-mon[27988]: from='client.? 172.21.15.173:0/2592403710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:36:24.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:24 smithi111 ceph-mon[31493]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-21T08:36:24.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:24 smithi094 ceph-mon[27988]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-21T08:36:24.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:24 smithi173 ceph-mon[31428]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-21T08:36:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:26 smithi111 ceph-mon[31493]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-21T08:36:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:26 smithi094 ceph-mon[27988]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-21T08:36:27.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:26 smithi173 ceph-mon[31428]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-21T08:36:28.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:28 smithi173 ceph-mon[31428]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:36:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:28 smithi111 ceph-mon[31493]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:36:29.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:28 smithi094 ceph-mon[27988]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:36:30.888 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:30 smithi173 ceph-mon[31428]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-21T08:36:30.888 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:36:30.888 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:30.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:30 smithi111 ceph-mon[31493]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-21T08:36:30.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:36:30.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:31.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:30 smithi094 ceph-mon[27988]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-21T08:36:31.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:36:31.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:31 smithi111 ceph-mon[31493]: Deploying daemon osd.11 on smithi173 2024-04-21T08:36:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:31 smithi094 ceph-mon[27988]: Deploying daemon osd.11 on smithi173 2024-04-21T08:36:32.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:31 smithi173 ceph-mon[31428]: Deploying daemon osd.11 on smithi173 2024-04-21T08:36:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:32 smithi111 ceph-mon[31493]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:32 smithi094 ceph-mon[27988]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:33.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:32 smithi173 ceph-mon[31428]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:34.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:34 smithi111 ceph-mon[31493]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:35.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:34 smithi094 ceph-mon[27988]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:35.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:34 smithi173 ceph-mon[31428]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:36.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:36 smithi173 ceph-mon[31428]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:36.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:36.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:36.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:36 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:36 smithi111 ceph-mon[31493]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:36 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:36 smithi094 ceph-mon[27988]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:36 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:38.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:38 smithi111 ceph-mon[31493]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:38.961 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:38 smithi173 ceph-mon[31428]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:39.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:38 smithi094 ceph-mon[27988]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:40.192 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.192+0000 7fe3fb7fe700 1 -- 172.21.15.173:0/828214355 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe404062e80 con 0x7fe3f00a3c40 2024-04-21T08:36:40.192 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 11 on host 'smithi173' 2024-04-21T08:36:40.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 -- 172.21.15.173:0/828214355 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe3f00a3c40 msgr2=0x7fe3f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:40.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe3f00a3c40 0x7fe3f00a6100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe4000038a0 tx=0x7fe400005fb0 comp rx=0 tx=0).stop 2024-04-21T08:36:40.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 -- 172.21.15.173:0/828214355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe404103ca0 msgr2=0x7fe404074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:40.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe404103ca0 0x7fe404074e30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400c920 tx=0x7fe3f400cc30 comp rx=0 tx=0).stop 2024-04-21T08:36:40.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 -- 172.21.15.173:0/828214355 shutdown_connections 2024-04-21T08:36:40.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe3f00a3c40 0x7fe3f00a6100 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:40.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404102aa0 0x7fe4040767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:40.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe404104640 0x7fe404075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:40.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 --2- 172.21.15.173:0/828214355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe404103ca0 0x7fe404074e30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:40.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 -- 172.21.15.173:0/828214355 >> 172.21.15.173:0/828214355 conn(0x7fe4040fe040 msgr2=0x7fe4041070e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:40.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 -- 172.21.15.173:0/828214355 shutdown_connections 2024-04-21T08:36:40.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:40.195+0000 7fe40c5a8700 1 -- 172.21.15.173:0/828214355 wait complete. 2024-04-21T08:36:40.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:40.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:40.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:40.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: from='osd.11 [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:36:40.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:36:40.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:40 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:40.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:40.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:40.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:40.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: from='osd.11 [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:36:40.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:36:40.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:40 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:40.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:40.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:40.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:40.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: from='osd.11 [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:36:40.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:36:40.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:40.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:40 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:40.871 DEBUG:teuthology.orchestra.run.smithi173:osd.11> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.11.service 2024-04-21T08:36:40.873 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-21T08:36:40.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-21T08:36:41.120 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:41.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:41 smithi094 ceph-mon[27988]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:36:41.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:41 smithi094 ceph-mon[27988]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:36:41.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:41 smithi094 ceph-mon[27988]: from='osd.11 [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:36:41.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:41 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:41.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:41 smithi094 ceph-mon[27988]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:36:41.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:41 smithi173 ceph-mon[31428]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:36:41.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:41 smithi173 ceph-mon[31428]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:36:41.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:41 smithi173 ceph-mon[31428]: from='osd.11 [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:36:41.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:41 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:41.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:41 smithi173 ceph-mon[31428]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:36:41.511 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:36:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:36:41.257+0000 7fa409124700 -1 osd.11 0 waiting for initial osdmap 2024-04-21T08:36:41.511 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:36:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:36:41.272+0000 7fa404f17700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:36:41.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:41 smithi111 ceph-mon[31493]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:36:41.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:41 smithi111 ceph-mon[31493]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:36:41.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:41 smithi111 ceph-mon[31493]: from='osd.11 [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:36:41.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:41 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:41.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:41 smithi111 ceph-mon[31493]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-21T08:36:42.304 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:42 smithi173 ceph-mon[31428]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:36:42.304 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:42 smithi173 ceph-mon[31428]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:36:42.304 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:42 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:42.304 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:42 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:42.304 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:42 smithi173 ceph-mon[31428]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:42.570 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:42 smithi094 ceph-mon[27988]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:36:42.570 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:42 smithi094 ceph-mon[27988]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:36:42.570 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:42 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:42.571 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:42 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:42.571 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:42 smithi094 ceph-mon[27988]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:42.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:42 smithi111 ceph-mon[31493]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-21T08:36:42.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:42 smithi111 ceph-mon[31493]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:36:42.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:42 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:42.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:42 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:42.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:42 smithi111 ceph-mon[31493]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:36:42.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.969+0000 7fde2a7e1700 1 -- 172.21.15.94:0/3303997985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde24108790 msgr2=0x7fde2410ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:42.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.969+0000 7fde2a7e1700 1 --2- 172.21.15.94:0/3303997985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde24108790 0x7fde2410ac00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00b0a0 tx=0x7fde1c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:36:42.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.969+0000 7fde2a7e1700 1 -- 172.21.15.94:0/3303997985 shutdown_connections 2024-04-21T08:36:42.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.969+0000 7fde2a7e1700 1 --2- 172.21.15.94:0/3303997985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde24103530 0x7fde24105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:42.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.969+0000 7fde2a7e1700 1 --2- 172.21.15.94:0/3303997985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde24108790 0x7fde2410ac00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:42.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.969+0000 7fde2a7e1700 1 --2- 172.21.15.94:0/3303997985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde24105e60 0x7fde24108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:42.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.969+0000 7fde2a7e1700 1 -- 172.21.15.94:0/3303997985 >> 172.21.15.94:0/3303997985 conn(0x7fde240fcf40 msgr2=0x7fde240ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:42.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.969+0000 7fde2a7e1700 1 -- 172.21.15.94:0/3303997985 shutdown_connections 2024-04-21T08:36:42.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.969+0000 7fde2a7e1700 1 -- 172.21.15.94:0/3303997985 wait complete. 2024-04-21T08:36:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.970+0000 7fde2a7e1700 1 Processor -- start 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.970+0000 7fde2a7e1700 1 -- start start 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde2a7e1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde24103530 0x7fde24198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde2a7e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde24105e60 0x7fde24199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde2a7e1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde24108790 0x7fde241a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde2a7e1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fde2410e620 con 0x7fde24103530 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde2a7e1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fde2410e4a0 con 0x7fde24105e60 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde2a7e1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fde2410e7a0 con 0x7fde24108790 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde23fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde24103530 0x7fde24198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde23fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde24103530 0x7fde24198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48468/0 (socket says 172.21.15.94:48468) 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde23fff700 1 -- 172.21.15.94:0/1403564674 learned_addr learned my addr 172.21.15.94:0/1403564674 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde28d7e700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde24108790 0x7fde241a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde237fe700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde24105e60 0x7fde24199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde23fff700 1 -- 172.21.15.94:0/1403564674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde24108790 msgr2=0x7fde241a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde23fff700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde24108790 0x7fde241a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde23fff700 1 -- 172.21.15.94:0/1403564674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde24105e60 msgr2=0x7fde24199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde23fff700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde24105e60 0x7fde24199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde23fff700 1 -- 172.21.15.94:0/1403564674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde241a0940 con 0x7fde24103530 2024-04-21T08:36:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde237fe700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde24105e60 0x7fde24199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:36:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde23fff700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde24103530 0x7fde24198cf0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fde1400b530 tx=0x7fde1400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde217fa700 1 -- 172.21.15.94:0/1403564674 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14015020 con 0x7fde24103530 2024-04-21T08:36:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde217fa700 1 -- 172.21.15.94:0/1403564674 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde14011ce0 con 0x7fde24103530 2024-04-21T08:36:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde217fa700 1 -- 172.21.15.94:0/1403564674 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14010d20 con 0x7fde24103530 2024-04-21T08:36:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde2a7e1700 1 -- 172.21.15.94:0/1403564674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde241a0c30 con 0x7fde24103530 2024-04-21T08:36:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.971+0000 7fde2a7e1700 1 -- 172.21.15.94:0/1403564674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde241a11c0 con 0x7fde24103530 2024-04-21T08:36:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.972+0000 7fde2a7e1700 1 -- 172.21.15.94:0/1403564674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde24069730 con 0x7fde24103530 2024-04-21T08:36:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.973+0000 7fde217fa700 1 -- 172.21.15.94:0/1403564674 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fde14018020 con 0x7fde24103530 2024-04-21T08:36:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.974+0000 7fde217fa700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde0c0a3cc0 0x7fde0c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.974+0000 7fde237fe700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde0c0a3cc0 0x7fde0c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.974+0000 7fde217fa700 1 -- 172.21.15.94:0/1403564674 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7fde140cf740 con 0x7fde24103530 2024-04-21T08:36:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.974+0000 7fde237fe700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde0c0a3cc0 0x7fde0c0a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde18006fd0 tx=0x7fde18008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:42.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:42.979+0000 7fde217fa700 1 -- 172.21.15.94:0/1403564674 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde1401f020 con 0x7fde24103530 2024-04-21T08:36:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.176+0000 7fde2a7e1700 1 -- 172.21.15.94:0/1403564674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fde2402e190 con 0x7fde24103530 2024-04-21T08:36:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.177+0000 7fde217fa700 1 -- 172.21.15.94:0/1403564674 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7fde1409c5c0 con 0x7fde24103530 2024-04-21T08:36:43.177 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 -- 172.21.15.94:0/1403564674 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde0c0a3cc0 msgr2=0x7fde0c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde0c0a3cc0 0x7fde0c0a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde18006fd0 tx=0x7fde18008040 comp rx=0 tx=0).stop 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 -- 172.21.15.94:0/1403564674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde24103530 msgr2=0x7fde24198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde24103530 0x7fde24198cf0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fde1400b530 tx=0x7fde1400b8f0 comp rx=0 tx=0).stop 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 -- 172.21.15.94:0/1403564674 shutdown_connections 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde0c0a3cc0 0x7fde0c0a6180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde24108790 0x7fde241a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde24103530 0x7fde24198cf0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 --2- 172.21.15.94:0/1403564674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde24105e60 0x7fde24199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 -- 172.21.15.94:0/1403564674 >> 172.21.15.94:0/1403564674 conn(0x7fde240fcf40 msgr2=0x7fde24103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 -- 172.21.15.94:0/1403564674 shutdown_connections 2024-04-21T08:36:43.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:43.179+0000 7fde2a7e1700 1 -- 172.21.15.94:0/1403564674 wait complete. 2024-04-21T08:36:43.468 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:43 smithi094 ceph-mon[27988]: purged_snaps scrub starts 2024-04-21T08:36:43.469 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:43 smithi094 ceph-mon[27988]: purged_snaps scrub ok 2024-04-21T08:36:43.469 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:43 smithi094 ceph-mon[27988]: osd.11 [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] boot 2024-04-21T08:36:43.469 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:43 smithi094 ceph-mon[27988]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T08:36:43.469 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:43 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:43.469 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:43 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1403564674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:36:43.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:43 smithi173 ceph-mon[31428]: purged_snaps scrub starts 2024-04-21T08:36:43.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:43 smithi173 ceph-mon[31428]: purged_snaps scrub ok 2024-04-21T08:36:43.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:43 smithi173 ceph-mon[31428]: osd.11 [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] boot 2024-04-21T08:36:43.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:43 smithi173 ceph-mon[31428]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T08:36:43.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:43 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:43.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:43 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1403564674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:36:43.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:43 smithi111 ceph-mon[31493]: purged_snaps scrub starts 2024-04-21T08:36:43.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:43 smithi111 ceph-mon[31493]: purged_snaps scrub ok 2024-04-21T08:36:43.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:43 smithi111 ceph-mon[31493]: osd.11 [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] boot 2024-04-21T08:36:43.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:43 smithi111 ceph-mon[31493]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T08:36:43.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:43 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:36:43.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:43 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1403564674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:36:43.851 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1713688602,"num_in_osds":12,"osd_in_since":1713688582,"num_remapped_pgs":0} 2024-04-21T08:36:43.852 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:36:44.113 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:44.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:44.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:44 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:36:44.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:44.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:44.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:44.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:44 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:36:44.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:44.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:44.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:44.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:44.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:44.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:44 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:45.425 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:45 smithi094 ceph-mon[27988]: Detected new or changed devices on smithi173 2024-04-21T08:36:45.425 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:45 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-21T08:36:45.535 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:45 smithi173 ceph-mon[31428]: Detected new or changed devices on smithi173 2024-04-21T08:36:45.535 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:45 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-21T08:36:45.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:45 smithi111 ceph-mon[31493]: Detected new or changed devices on smithi173 2024-04-21T08:36:45.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:45 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.961+0000 7fa844081700 1 -- 172.21.15.94:0/3562779198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa83c102300 msgr2=0x7fa83c10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.961+0000 7fa844081700 1 --2- 172.21.15.94:0/3562779198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa83c102300 0x7fa83c10e7f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa83400b0a0 tx=0x7fa83400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.962+0000 7fa844081700 1 -- 172.21.15.94:0/3562779198 shutdown_connections 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.962+0000 7fa844081700 1 --2- 172.21.15.94:0/3562779198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa83c102300 0x7fa83c10e7f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.962+0000 7fa844081700 1 --2- 172.21.15.94:0/3562779198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa83c107990 0x7fa83c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.962+0000 7fa844081700 1 --2- 172.21.15.94:0/3562779198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa83c101940 0x7fa83c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.962+0000 7fa844081700 1 -- 172.21.15.94:0/3562779198 >> 172.21.15.94:0/3562779198 conn(0x7fa83c0788f0 msgr2=0x7fa83c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.962+0000 7fa844081700 1 -- 172.21.15.94:0/3562779198 shutdown_connections 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.962+0000 7fa844081700 1 -- 172.21.15.94:0/3562779198 wait complete. 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.962+0000 7fa844081700 1 Processor -- start 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.962+0000 7fa844081700 1 -- start start 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa844081700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa83c101940 0x7fa83c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa844081700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa83c102300 0x7fa83c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa844081700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa83c107990 0x7fa83c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa844081700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c111b60 con 0x7fa83c107990 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa844081700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c1119e0 con 0x7fa83c101940 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa844081700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c111ce0 con 0x7fa83c102300 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa84261e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa83c107990 0x7fa83c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa84261e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa83c107990 0x7fa83c19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48082/0 (socket says 172.21.15.94:48082) 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa84261e700 1 -- 172.21.15.94:0/3704123519 learned_addr learned my addr 172.21.15.94:0/3704123519 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa841e1d700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa83c101940 0x7fa83c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa84161c700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa83c102300 0x7fa83c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa84261e700 1 -- 172.21.15.94:0/3704123519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa83c102300 msgr2=0x7fa83c1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa84261e700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa83c102300 0x7fa83c1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa84261e700 1 -- 172.21.15.94:0/3704123519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa83c101940 msgr2=0x7fa83c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa84261e700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa83c101940 0x7fa83c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa84261e700 1 -- 172.21.15.94:0/3704123519 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83c19cf80 con 0x7fa83c107990 2024-04-21T08:36:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa84261e700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa83c107990 0x7fa83c19c840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa8340062a0 tx=0x7fa834004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa832ffd700 1 -- 172.21.15.94:0/3704123519 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834021070 con 0x7fa83c107990 2024-04-21T08:36:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa832ffd700 1 -- 172.21.15.94:0/3704123519 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa834003c00 con 0x7fa83c107990 2024-04-21T08:36:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa844081700 1 -- 172.21.15.94:0/3704123519 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83c19d210 con 0x7fa83c107990 2024-04-21T08:36:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa832ffd700 1 -- 172.21.15.94:0/3704123519 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834004830 con 0x7fa83c107990 2024-04-21T08:36:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.963+0000 7fa844081700 1 -- 172.21.15.94:0/3704123519 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa83c1a4af0 con 0x7fa83c107990 2024-04-21T08:36:45.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.965+0000 7fa844081700 1 -- 172.21.15.94:0/3704123519 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa83c109eb0 con 0x7fa83c107990 2024-04-21T08:36:45.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.965+0000 7fa832ffd700 1 -- 172.21.15.94:0/3704123519 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa834007a00 con 0x7fa83c107990 2024-04-21T08:36:45.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.966+0000 7fa832ffd700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8280a3cc0 0x7fa8280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:45.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.966+0000 7fa841e1d700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8280a3cc0 0x7fa8280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:45.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.966+0000 7fa832ffd700 1 -- 172.21.15.94:0/3704123519 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa83400f020 con 0x7fa83c107990 2024-04-21T08:36:45.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.966+0000 7fa841e1d700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8280a3cc0 0x7fa8280a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa82c009a10 tx=0x7fa82c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:45.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:45.970+0000 7fa832ffd700 1 -- 172.21.15.94:0/3704123519 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa83402a020 con 0x7fa83c107990 2024-04-21T08:36:46.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.168+0000 7fa844081700 1 -- 172.21.15.94:0/3704123519 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa83c1992d0 con 0x7fa83c107990 2024-04-21T08:36:46.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.169+0000 7fa832ffd700 1 -- 172.21.15.94:0/3704123519 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18420 (secure 0 0 0) 0x7fa83409e3c0 con 0x7fa83c107990 2024-04-21T08:36:46.170 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:36:46.170 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":81,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:36:43.263319+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:36:46.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 -- 172.21.15.94:0/3704123519 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8280a3cc0 msgr2=0x7fa8280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:46.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8280a3cc0 0x7fa8280a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa82c009a10 tx=0x7fa82c009190 comp rx=0 tx=0).stop 2024-04-21T08:36:46.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 -- 172.21.15.94:0/3704123519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa83c107990 msgr2=0x7fa83c19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:46.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa83c107990 0x7fa83c19c840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa8340062a0 tx=0x7fa834004e20 comp rx=0 tx=0).stop 2024-04-21T08:36:46.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 -- 172.21.15.94:0/3704123519 shutdown_connections 2024-04-21T08:36:46.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8280a3cc0 0x7fa8280a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa83c101940 0x7fa83c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa83c107990 0x7fa83c19c840 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 --2- 172.21.15.94:0/3704123519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa83c102300 0x7fa83c1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 -- 172.21.15.94:0/3704123519 >> 172.21.15.94:0/3704123519 conn(0x7fa83c0788f0 msgr2=0x7fa83c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:46.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 -- 172.21.15.94:0/3704123519 shutdown_connections 2024-04-21T08:36:46.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:46.171+0000 7fa844081700 1 -- 172.21.15.94:0/3704123519 wait complete. 2024-04-21T08:36:46.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:46 smithi094 ceph-mon[27988]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:46.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:46 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3704123519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:36:46.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:46 smithi111 ceph-mon[31493]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:46.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:46 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3704123519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:36:46.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:46 smithi173 ceph-mon[31428]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:46.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:46 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3704123519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:36:47.056 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:36:47.056 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-21T08:36:47.293 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.879+0000 7f2314298700 1 -- 172.21.15.94:0/2084920230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f230c108a10 msgr2=0x7f230c108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.879+0000 7f2314298700 1 --2- 172.21.15.94:0/2084920230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f230c108a10 0x7f230c108df0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2308009990 tx=0x7f2308009ca0 comp rx=0 tx=0).stop 2024-04-21T08:36:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.879+0000 7f2314298700 1 -- 172.21.15.94:0/2084920230 shutdown_connections 2024-04-21T08:36:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.879+0000 7f2314298700 1 --2- 172.21.15.94:0/2084920230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f230c1034c0 0x7f230c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:48.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.879+0000 7f2314298700 1 --2- 172.21.15.94:0/2084920230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f230c108a10 0x7f230c108df0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:48.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.879+0000 7f2314298700 1 --2- 172.21.15.94:0/2084920230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f230c102b00 0x7f230c102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:48.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.879+0000 7f2314298700 1 -- 172.21.15.94:0/2084920230 >> 172.21.15.94:0/2084920230 conn(0x7f230c079c00 msgr2=0x7f230c07a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:48.880 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:48 smithi094 ceph-mon[27988]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.879+0000 7f2314298700 1 -- 172.21.15.94:0/2084920230 shutdown_connections 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.879+0000 7f2314298700 1 -- 172.21.15.94:0/2084920230 wait complete. 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.880+0000 7f2314298700 1 Processor -- start 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.880+0000 7f2314298700 1 -- start start 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.880+0000 7f2314298700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f230c102b00 0x7f230c199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.880+0000 7f2314298700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f230c1034c0 0x7f230c1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.880+0000 7f2314298700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f230c108a10 0x7f230c19da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.880+0000 7f2314298700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f230c112c50 con 0x7f230c1034c0 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2314298700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f230c112ad0 con 0x7f230c102b00 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2314298700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f230c112dd0 con 0x7f230c108a10 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2311833700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f230c1034c0 0x7f230c1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2311833700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f230c1034c0 0x7f230c1996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48086/0 (socket says 172.21.15.94:48086) 2024-04-21T08:36:48.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2311833700 1 -- 172.21.15.94:0/4022831333 learned_addr learned my addr 172.21.15.94:0/4022831333 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2312034700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f230c102b00 0x7f230c199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2311833700 1 -- 172.21.15.94:0/4022831333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f230c108a10 msgr2=0x7f230c19da50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2311833700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f230c108a10 0x7f230c19da50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2311833700 1 -- 172.21.15.94:0/4022831333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f230c102b00 msgr2=0x7f230c199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2311833700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f230c102b00 0x7f230c199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2311833700 1 -- 172.21.15.94:0/4022831333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230c19e190 con 0x7f230c1034c0 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2311833700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f230c1034c0 0x7f230c1996c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00c920 tx=0x7f22fc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f23037fe700 1 -- 172.21.15.94:0/4022831333 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc007b00 con 0x7f230c1034c0 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f2314298700 1 -- 172.21.15.94:0/4022831333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230c19e480 con 0x7f230c1034c0 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f23037fe700 1 -- 172.21.15.94:0/4022831333 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22fc004ce0 con 0x7f230c1034c0 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.881+0000 7f23037fe700 1 -- 172.21.15.94:0/4022831333 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc005650 con 0x7f230c1034c0 2024-04-21T08:36:48.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.882+0000 7f2314298700 1 -- 172.21.15.94:0/4022831333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f230c1a5d00 con 0x7f230c1034c0 2024-04-21T08:36:48.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.882+0000 7f2314298700 1 -- 172.21.15.94:0/4022831333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f230c104150 con 0x7f230c1034c0 2024-04-21T08:36:48.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.883+0000 7f23037fe700 1 -- 172.21.15.94:0/4022831333 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f22fc007ca0 con 0x7f230c1034c0 2024-04-21T08:36:48.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.884+0000 7f23037fe700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22f80a3c40 0x7f22f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:48.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.884+0000 7f2312034700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22f80a3c40 0x7f22f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.884+0000 7f23037fe700 1 -- 172.21.15.94:0/4022831333 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f22fc0cfef0 con 0x7f230c1034c0 2024-04-21T08:36:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.885+0000 7f2312034700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22f80a3c40 0x7f22f80a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f230800b5c0 tx=0x7f2308005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:48.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:48.890+0000 7f23037fe700 1 -- 172.21.15.94:0/4022831333 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22fc014030 con 0x7f230c1034c0 2024-04-21T08:36:48.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:48 smithi111 ceph-mon[31493]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:49.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:48 smithi173 ceph-mon[31428]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:49.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.082+0000 7f2314298700 1 -- 172.21.15.94:0/4022831333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f230c02df80 con 0x7f230c1034c0 2024-04-21T08:36:49.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.082+0000 7f23037fe700 1 -- 172.21.15.94:0/4022831333 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f22fc09cd70 con 0x7f230c1034c0 2024-04-21T08:36:49.082 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 -- 172.21.15.94:0/4022831333 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22f80a3c40 msgr2=0x7f22f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22f80a3c40 0x7f22f80a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f230800b5c0 tx=0x7f2308005e80 comp rx=0 tx=0).stop 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 -- 172.21.15.94:0/4022831333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f230c1034c0 msgr2=0x7f230c1996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f230c1034c0 0x7f230c1996c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00c920 tx=0x7f22fc00cce0 comp rx=0 tx=0).stop 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 -- 172.21.15.94:0/4022831333 shutdown_connections 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f22f80a3c40 0x7f22f80a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f230c102b00 0x7f230c199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f230c108a10 0x7f230c19da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 --2- 172.21.15.94:0/4022831333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f230c1034c0 0x7f230c1996c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 -- 172.21.15.94:0/4022831333 >> 172.21.15.94:0/4022831333 conn(0x7f230c079c00 msgr2=0x7f230c100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:49.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 -- 172.21.15.94:0/4022831333 shutdown_connections 2024-04-21T08:36:49.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:49.085+0000 7f2314298700 1 -- 172.21.15.94:0/4022831333 wait complete. 2024-04-21T08:36:49.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:49 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4022831333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:36:49.802 INFO:tasks.cephadm:Adding mds.a on smithi094 2024-04-21T08:36:49.802 INFO:tasks.cephadm:Adding mds.d on smithi094 2024-04-21T08:36:49.802 INFO:tasks.cephadm:Adding mds.g on smithi094 2024-04-21T08:36:49.802 INFO:tasks.cephadm:Adding mds.j on smithi094 2024-04-21T08:36:49.802 INFO:tasks.cephadm:Adding mds.b on smithi111 2024-04-21T08:36:49.802 INFO:tasks.cephadm:Adding mds.e on smithi111 2024-04-21T08:36:49.802 INFO:tasks.cephadm:Adding mds.h on smithi111 2024-04-21T08:36:49.802 INFO:tasks.cephadm:Adding mds.k on smithi111 2024-04-21T08:36:49.803 INFO:tasks.cephadm:Adding mds.c on smithi173 2024-04-21T08:36:49.803 INFO:tasks.cephadm:Adding mds.f on smithi173 2024-04-21T08:36:49.803 INFO:tasks.cephadm:Adding mds.i on smithi173 2024-04-21T08:36:49.803 INFO:tasks.cephadm:Adding mds.l on smithi173 2024-04-21T08:36:49.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l' 2024-04-21T08:36:49.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:49 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4022831333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:36:50.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:49 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4022831333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:36:50.053 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:36:50.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:50 smithi173 ceph-mon[31428]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:50.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:50 smithi111 ceph-mon[31493]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:50 smithi094 ceph-mon[27988]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:51.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.697+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2648500280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e0102f80 msgr2=0x7fd0e01033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:51.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.697+0000 7fd0e6c09700 1 --2- 172.21.15.173:0/2648500280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e0102f80 0x7fd0e01033a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40099c0 tx=0x7fd0d4009cd0 comp rx=0 tx=0).stop 2024-04-21T08:36:51.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.697+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2648500280 shutdown_connections 2024-04-21T08:36:51.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.697+0000 7fd0e6c09700 1 --2- 172.21.15.173:0/2648500280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0e0102660 0x7fd0e0109f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:51.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.697+0000 7fd0e6c09700 1 --2- 172.21.15.173:0/2648500280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e0102f80 0x7fd0e01033a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:51.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.697+0000 7fd0e6c09700 1 --2- 172.21.15.173:0/2648500280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0101cc0 0x7fd0e0102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:51.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.697+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2648500280 >> 172.21.15.173:0/2648500280 conn(0x7fd0e00fc710 msgr2=0x7fd0e00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:51.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.697+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2648500280 shutdown_connections 2024-04-21T08:36:51.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.697+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2648500280 wait complete. 2024-04-21T08:36:51.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0e6c09700 1 Processor -- start 2024-04-21T08:36:51.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0e6c09700 1 -- start start 2024-04-21T08:36:51.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0e6c09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0101cc0 0x7fd0e0101710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0e6c09700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e0102660 0x7fd0e019d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0e6c09700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0e0102f80 0x7fd0e01a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0e6c09700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e010d4e0 con 0x7fd0e0102f80 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0e6c09700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e010d360 con 0x7fd0e0101cc0 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0e6c09700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e010d660 con 0x7fd0e0102660 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0dffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e0102660 0x7fd0e019d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0dffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e0102660 0x7fd0e019d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:56982/0 (socket says 172.21.15.173:56982) 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.698+0000 7fd0dffff700 1 -- 172.21.15.173:0/2295565296 learned_addr learned my addr 172.21.15.173:0/2295565296 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0e49a5700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0101cc0 0x7fd0e0101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0dffff700 1 -- 172.21.15.173:0/2295565296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0101cc0 msgr2=0x7fd0e0101710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0e51a6700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0e0102f80 0x7fd0e01a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0dffff700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0101cc0 0x7fd0e0101710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:51.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0dffff700 1 -- 172.21.15.173:0/2295565296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0e0102f80 msgr2=0x7fd0e01a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:51.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0dffff700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0e0102f80 0x7fd0e01a0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:51.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0dffff700 1 -- 172.21.15.173:0/2295565296 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e01a08e0 con 0x7fd0e0102660 2024-04-21T08:36:51.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0dffff700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e0102660 0x7fd0e019d9d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00b530 tx=0x7fd0cc00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:51.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0ddffb700 1 -- 172.21.15.173:0/2295565296 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc015020 con 0x7fd0e0102660 2024-04-21T08:36:51.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0ddffb700 1 -- 172.21.15.173:0/2295565296 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0cc011ce0 con 0x7fd0e0102660 2024-04-21T08:36:51.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2295565296 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e01a0bd0 con 0x7fd0e0102660 2024-04-21T08:36:51.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0ddffb700 1 -- 172.21.15.173:0/2295565296 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc010e40 con 0x7fd0e0102660 2024-04-21T08:36:51.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.699+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2295565296 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e01a10c0 con 0x7fd0e0102660 2024-04-21T08:36:51.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.700+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2295565296 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e0069730 con 0x7fd0e0102660 2024-04-21T08:36:51.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.701+0000 7fd0ddffb700 1 -- 172.21.15.173:0/2295565296 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd0cc018020 con 0x7fd0e0102660 2024-04-21T08:36:51.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.702+0000 7fd0ddffb700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd0d00a3c40 0x7fd0d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:51.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.702+0000 7fd0e49a5700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd0d00a3c40 0x7fd0d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:51.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.702+0000 7fd0ddffb700 1 -- 172.21.15.173:0/2295565296 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd0cc0cf7f0 con 0x7fd0e0102660 2024-04-21T08:36:51.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.703+0000 7fd0e49a5700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd0d00a3c40 0x7fd0d00a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40038c0 tx=0x7fd0d400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:51.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.707+0000 7fd0ddffb700 1 -- 172.21.15.173:0/2295565296 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0cc01f020 con 0x7fd0e0102660 2024-04-21T08:36:51.916 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.916+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2295565296 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}) -- 0x7fd0e0062f80 con 0x7fd0d00a3c40 2024-04-21T08:36:51.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.924+0000 7fd0ddffb700 1 -- 172.21.15.173:0/2295565296 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd0e0062f80 con 0x7fd0d00a3c40 2024-04-21T08:36:51.924 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mds.all update... 2024-04-21T08:36:51.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2295565296 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd0d00a3c40 msgr2=0x7fd0d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:51.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd0d00a3c40 0x7fd0d00a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40038c0 tx=0x7fd0d400b540 comp rx=0 tx=0).stop 2024-04-21T08:36:51.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2295565296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e0102660 msgr2=0x7fd0e019d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:51.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e0102660 0x7fd0e019d9d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00b530 tx=0x7fd0cc00b840 comp rx=0 tx=0).stop 2024-04-21T08:36:51.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2295565296 shutdown_connections 2024-04-21T08:36:51.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd0d00a3c40 0x7fd0d00a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:51.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0101cc0 0x7fd0e0101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:51.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0e0102f80 0x7fd0e01a0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:51.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 --2- 172.21.15.173:0/2295565296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e0102660 0x7fd0e019d9d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:51.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2295565296 >> 172.21.15.173:0/2295565296 conn(0x7fd0e00fc710 msgr2=0x7fd0e01060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:51.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2295565296 shutdown_connections 2024-04-21T08:36:51.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-21T08:36:51.927+0000 7fd0e6c09700 1 -- 172.21.15.173:0/2295565296 wait complete. 2024-04-21T08:36:52.629 DEBUG:teuthology.orchestra.run.smithi094:mds.a> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.a.service 2024-04-21T08:36:52.632 DEBUG:teuthology.orchestra.run.smithi094:mds.d> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.d.service 2024-04-21T08:36:52.675 DEBUG:teuthology.orchestra.run.smithi094:mds.g> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.g.service 2024-04-21T08:36:52.717 DEBUG:teuthology.orchestra.run.smithi094:mds.j> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.j.service 2024-04-21T08:36:52.760 DEBUG:teuthology.orchestra.run.smithi111:mds.b> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.b.service 2024-04-21T08:36:52.763 DEBUG:teuthology.orchestra.run.smithi111:mds.e> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.e.service 2024-04-21T08:36:52.808 DEBUG:teuthology.orchestra.run.smithi111:mds.h> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.h.service 2024-04-21T08:36:52.851 DEBUG:teuthology.orchestra.run.smithi111:mds.k> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.k.service 2024-04-21T08:36:52.895 DEBUG:teuthology.orchestra.run.smithi173:mds.c> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.c.service 2024-04-21T08:36:52.897 DEBUG:teuthology.orchestra.run.smithi173:mds.f> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.f.service 2024-04-21T08:36:52.911 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:52 smithi111 ceph-mon[31493]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:52.911 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:52 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:52.911 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:52 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:52.916 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:52 smithi173 ceph-mon[31428]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:52.917 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:52 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:52.917 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:52 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:52.918 DEBUG:teuthology.orchestra.run.smithi173:mds.i> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.i.service 2024-04-21T08:36:52.961 DEBUG:teuthology.orchestra.run.smithi173:mds.l> sudo journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.l.service 2024-04-21T08:36:53.004 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-21T08:36:53.005 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:36:53.009 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:52 smithi094 ceph-mon[27988]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:53.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:52 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:53.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:52 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:53.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.115+0000 7f06f3340640 1 -- 172.21.15.94:0/24059484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06ec100e70 msgr2=0x7f06ec101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.115+0000 7f06f3340640 1 --2- 172.21.15.94:0/24059484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06ec100e70 0x7f06ec101290 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f06e8009900 tx=0x7f06e802bc30 comp rx=0 tx=0).stop 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.115+0000 7f06f3340640 1 -- 172.21.15.94:0/24059484 shutdown_connections 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.115+0000 7f06f3340640 1 --2- 172.21.15.94:0/24059484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec101fc0 0x7f06ec102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.115+0000 7f06f3340640 1 --2- 172.21.15.94:0/24059484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec102980 0x7f06ec1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.115+0000 7f06f3340640 1 --2- 172.21.15.94:0/24059484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06ec100e70 0x7f06ec101290 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.115+0000 7f06f3340640 1 -- 172.21.15.94:0/24059484 >> 172.21.15.94:0/24059484 conn(0x7f06ec0fc3e0 msgr2=0x7f06ec0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.115+0000 7f06f3340640 1 -- 172.21.15.94:0/24059484 shutdown_connections 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.115+0000 7f06f3340640 1 -- 172.21.15.94:0/24059484 wait complete. 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f3340640 1 Processor -- start 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f3340640 1 -- start start 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f3340640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec100e70 0x7f06ec069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f3340640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06ec101fc0 0x7f06ec19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f3340640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec102980 0x7f06ec19f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f3340640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec10ca40 con 0x7f06ec101fc0 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f3340640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec10c8c0 con 0x7f06ec100e70 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f3340640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec10cbc0 con 0x7f06ec102980 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f08db640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06ec101fc0 0x7f06ec19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f08db640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06ec101fc0 0x7f06ec19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48102/0 (socket says 172.21.15.94:48102) 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f08db640 1 -- 172.21.15.94:0/2870749833 learned_addr learned my addr 172.21.15.94:0/2870749833 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f10dc640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec100e70 0x7f06ec069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f18dd640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec102980 0x7f06ec19f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f08db640 1 -- 172.21.15.94:0/2870749833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec102980 msgr2=0x7f06ec19f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f08db640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec102980 0x7f06ec19f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f08db640 1 -- 172.21.15.94:0/2870749833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec100e70 msgr2=0x7f06ec069cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f08db640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec100e70 0x7f06ec069cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f08db640 1 -- 172.21.15.94:0/2870749833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06ec19fe80 con 0x7f06ec101fc0 2024-04-21T08:36:53.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.116+0000 7f06f18dd640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec102980 0x7f06ec19f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:36:53.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.117+0000 7f06f08db640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06ec101fc0 0x7f06ec19cf50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00b450 tx=0x7f06dc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:53.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.117+0000 7f06e27fc640 1 -- 172.21.15.94:0/2870749833 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc015020 con 0x7f06ec101fc0 2024-04-21T08:36:53.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.117+0000 7f06e27fc640 1 -- 172.21.15.94:0/2870749833 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06dc011d40 con 0x7f06ec101fc0 2024-04-21T08:36:53.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.117+0000 7f06e27fc640 1 -- 172.21.15.94:0/2870749833 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc00f340 con 0x7f06ec101fc0 2024-04-21T08:36:53.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.117+0000 7f06f3340640 1 -- 172.21.15.94:0/2870749833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06ec1a0170 con 0x7f06ec101fc0 2024-04-21T08:36:53.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.117+0000 7f06f3340640 1 -- 172.21.15.94:0/2870749833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06ec1a0670 con 0x7f06ec101fc0 2024-04-21T08:36:53.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.118+0000 7f06f3340640 1 -- 172.21.15.94:0/2870749833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06ec1082a0 con 0x7f06ec101fc0 2024-04-21T08:36:53.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.119+0000 7f06e27fc640 1 -- 172.21.15.94:0/2870749833 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f06dc018020 con 0x7f06ec101fc0 2024-04-21T08:36:53.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.120+0000 7f06e27fc640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f06d80a3bb0 0x7f06d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.120+0000 7f06f10dc640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f06d80a3bb0 0x7f06d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.120+0000 7f06e27fc640 1 -- 172.21.15.94:0/2870749833 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f06dc0d00c0 con 0x7f06ec101fc0 2024-04-21T08:36:53.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.120+0000 7f06f10dc640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f06d80a3bb0 0x7f06d80a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f06e8000c00 tx=0x7f06e8005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:53.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.123+0000 7f06e27fc640 1 -- 172.21.15.94:0/2870749833 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f06dc01f020 con 0x7f06ec101fc0 2024-04-21T08:36:53.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.302+0000 7f06f3340640 1 -- 172.21.15.94:0/2870749833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06ec02d810 con 0x7f06ec101fc0 2024-04-21T08:36:53.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.303+0000 7f06e27fc640 1 -- 172.21.15.94:0/2870749833 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18420 (secure 0 0 0) 0x7f06dc00d040 con 0x7f06ec101fc0 2024-04-21T08:36:53.302 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:36:53.302 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":81,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:36:43.263319+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:36:53.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 -- 172.21.15.94:0/2870749833 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f06d80a3bb0 msgr2=0x7f06d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f06d80a3bb0 0x7f06d80a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f06e8000c00 tx=0x7f06e8005be0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 -- 172.21.15.94:0/2870749833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06ec101fc0 msgr2=0x7f06ec19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06ec101fc0 0x7f06ec19cf50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00b450 tx=0x7f06dc00b910 comp rx=0 tx=0).stop 2024-04-21T08:36:53.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 -- 172.21.15.94:0/2870749833 shutdown_connections 2024-04-21T08:36:53.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f06d80a3bb0 0x7f06d80a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec102980 0x7f06ec19f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06ec101fc0 0x7f06ec19cf50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 --2- 172.21.15.94:0/2870749833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec100e70 0x7f06ec069cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 -- 172.21.15.94:0/2870749833 >> 172.21.15.94:0/2870749833 conn(0x7f06ec0fc3e0 msgr2=0x7f06ec105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:53.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 -- 172.21.15.94:0/2870749833 shutdown_connections 2024-04-21T08:36:53.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.305+0000 7f06f3340640 1 -- 172.21.15.94:0/2870749833 wait complete. 2024-04-21T08:36:53.315 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:36:53.316 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:36:53.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.458+0000 7fb187ee4640 1 -- 172.21.15.94:0/448581157 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1801029a0 msgr2=0x7fb1801092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.458+0000 7fb187ee4640 1 --2- 172.21.15.94:0/448581157 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1801029a0 0x7fb1801092c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb178009ca0 tx=0x7fb17802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.458+0000 7fb187ee4640 1 -- 172.21.15.94:0/448581157 shutdown_connections 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.458+0000 7fb187ee4640 1 --2- 172.21.15.94:0/448581157 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1801029a0 0x7fb1801092c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.458+0000 7fb187ee4640 1 --2- 172.21.15.94:0/448581157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb180101fe0 0x7fb180102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.458+0000 7fb187ee4640 1 --2- 172.21.15.94:0/448581157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb180100e90 0x7fb1801012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.458+0000 7fb187ee4640 1 -- 172.21.15.94:0/448581157 >> 172.21.15.94:0/448581157 conn(0x7fb1800fc400 msgr2=0x7fb1800fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb187ee4640 1 -- 172.21.15.94:0/448581157 shutdown_connections 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb187ee4640 1 -- 172.21.15.94:0/448581157 wait complete. 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb187ee4640 1 Processor -- start 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb187ee4640 1 -- start start 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb187ee4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb180100e90 0x7fb180069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb187ee4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb180101fe0 0x7fb18019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb187ee4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1801029a0 0x7fb18019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb187ee4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb18010c9f0 con 0x7fb180100e90 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb187ee4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb18010c870 con 0x7fb180101fe0 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb187ee4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb18010cb70 con 0x7fb1801029a0 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb185c80640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb180100e90 0x7fb180069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb185c80640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb180100e90 0x7fb180069c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48114/0 (socket says 172.21.15.94:48114) 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb185c80640 1 -- 172.21.15.94:0/1954211123 learned_addr learned my addr 172.21.15.94:0/1954211123 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb186481640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1801029a0 0x7fb18019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.459+0000 7fb18547f640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb180101fe0 0x7fb18019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb185c80640 1 -- 172.21.15.94:0/1954211123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1801029a0 msgr2=0x7fb18019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb185c80640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1801029a0 0x7fb18019f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb185c80640 1 -- 172.21.15.94:0/1954211123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb180101fe0 msgr2=0x7fb18019cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb185c80640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb180101fe0 0x7fb18019cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb185c80640 1 -- 172.21.15.94:0/1954211123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18019fe50 con 0x7fb180100e90 2024-04-21T08:36:53.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb185c80640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb180100e90 0x7fb180069c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00d770 tx=0x7fb17c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:53.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb176ffd640 1 -- 172.21.15.94:0/1954211123 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c017070 con 0x7fb180100e90 2024-04-21T08:36:53.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb176ffd640 1 -- 172.21.15.94:0/1954211123 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17c0054d0 con 0x7fb180100e90 2024-04-21T08:36:53.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb187ee4640 1 -- 172.21.15.94:0/1954211123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1801a00e0 con 0x7fb180100e90 2024-04-21T08:36:53.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb176ffd640 1 -- 172.21.15.94:0/1954211123 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c004c30 con 0x7fb180100e90 2024-04-21T08:36:53.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.460+0000 7fb187ee4640 1 -- 172.21.15.94:0/1954211123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1801a0630 con 0x7fb180100e90 2024-04-21T08:36:53.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.461+0000 7fb187ee4640 1 -- 172.21.15.94:0/1954211123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1801081c0 con 0x7fb180100e90 2024-04-21T08:36:53.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.462+0000 7fb176ffd640 1 -- 172.21.15.94:0/1954211123 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb17c004dd0 con 0x7fb180100e90 2024-04-21T08:36:53.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.463+0000 7fb176ffd640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb16c0a3c30 0x7fb16c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.463+0000 7fb18547f640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb16c0a3c30 0x7fb16c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.463+0000 7fb176ffd640 1 -- 172.21.15.94:0/1954211123 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb17c0cfd40 con 0x7fb180100e90 2024-04-21T08:36:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.463+0000 7fb18547f640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb16c0a3c30 0x7fb16c0a60f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb1800699c0 tx=0x7fb17000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.466+0000 7fb176ffd640 1 -- 172.21.15.94:0/1954211123 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb17c09c5d0 con 0x7fb180100e90 2024-04-21T08:36:53.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.645+0000 7fb187ee4640 1 -- 172.21.15.94:0/1954211123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb18002d830 con 0x7fb180100e90 2024-04-21T08:36:53.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.645+0000 7fb176ffd640 1 -- 172.21.15.94:0/1954211123 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fb17c0a08d0 con 0x7fb180100e90 2024-04-21T08:36:53.644 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:36:53.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.647+0000 7fb187ee4640 1 -- 172.21.15.94:0/1954211123 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb16c0a3c30 msgr2=0x7fb16c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.647+0000 7fb187ee4640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb16c0a3c30 0x7fb16c0a60f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb1800699c0 tx=0x7fb17000b390 comp rx=0 tx=0).stop 2024-04-21T08:36:53.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.647+0000 7fb187ee4640 1 -- 172.21.15.94:0/1954211123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb180100e90 msgr2=0x7fb180069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.647+0000 7fb187ee4640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb180100e90 0x7fb180069c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00d770 tx=0x7fb17c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:36:53.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.647+0000 7fb187ee4640 1 -- 172.21.15.94:0/1954211123 shutdown_connections 2024-04-21T08:36:53.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.647+0000 7fb187ee4640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb16c0a3c30 0x7fb16c0a60f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.647+0000 7fb187ee4640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1801029a0 0x7fb18019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.647+0000 7fb187ee4640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb180101fe0 0x7fb18019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.647+0000 7fb187ee4640 1 --2- 172.21.15.94:0/1954211123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb180100e90 0x7fb180069c50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.647+0000 7fb187ee4640 1 -- 172.21.15.94:0/1954211123 >> 172.21.15.94:0/1954211123 conn(0x7fb1800fc400 msgr2=0x7fb180105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:53.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.648+0000 7fb187ee4640 1 -- 172.21.15.94:0/1954211123 shutdown_connections 2024-04-21T08:36:53.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.648+0000 7fb187ee4640 1 -- 172.21.15.94:0/1954211123 wait complete. 2024-04-21T08:36:53.657 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:36:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.756+0000 7fe4683b2640 1 -- 172.21.15.94:0/2336793859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe460100e70 msgr2=0x7fe460101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.756+0000 7fe4683b2640 1 --2- 172.21.15.94:0/2336793859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe460100e70 0x7fe460101290 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe458009ca0 tx=0x7fe45802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.756+0000 7fe4683b2640 1 -- 172.21.15.94:0/2336793859 shutdown_connections 2024-04-21T08:36:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.756+0000 7fe4683b2640 1 --2- 172.21.15.94:0/2336793859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe460101fc0 0x7fe460102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.756+0000 7fe4683b2640 1 --2- 172.21.15.94:0/2336793859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe460102980 0x7fe4601092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.756+0000 7fe4683b2640 1 --2- 172.21.15.94:0/2336793859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe460100e70 0x7fe460101290 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 -- 172.21.15.94:0/2336793859 >> 172.21.15.94:0/2336793859 conn(0x7fe4600fc3e0 msgr2=0x7fe4600fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 -- 172.21.15.94:0/2336793859 shutdown_connections 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 -- 172.21.15.94:0/2336793859 wait complete. 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 Processor -- start 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 -- start start 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe460100e70 0x7fe46010b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe460101fc0 0x7fe460109ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe460102980 0x7fe46010a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe46010dae0 con 0x7fe460100e70 2024-04-21T08:36:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe46010d960 con 0x7fe460102980 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe4683b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe46010dc60 con 0x7fe460101fc0 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe46614e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe460100e70 0x7fe46010b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe46614e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe460100e70 0x7fe46010b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58140/0 (socket says 172.21.15.94:58140) 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe46614e640 1 -- 172.21.15.94:0/428290095 learned_addr learned my addr 172.21.15.94:0/428290095 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe46694f640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe460102980 0x7fe46010a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.757+0000 7fe46594d640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe460101fc0 0x7fe460109ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe46614e640 1 -- 172.21.15.94:0/428290095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe460101fc0 msgr2=0x7fe460109ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe46614e640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe460101fc0 0x7fe460109ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe46614e640 1 -- 172.21.15.94:0/428290095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe460102980 msgr2=0x7fe46010a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe46614e640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe460102980 0x7fe46010a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe46614e640 1 -- 172.21.15.94:0/428290095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46010aa70 con 0x7fe460100e70 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe46614e640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe460100e70 0x7fe46010b660 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe458005e30 tx=0x7fe458034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe4577fe640 1 -- 172.21.15.94:0/428290095 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458039040 con 0x7fe460100e70 2024-04-21T08:36:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe4683b2640 1 -- 172.21.15.94:0/428290095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4601a2290 con 0x7fe460100e70 2024-04-21T08:36:53.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe4577fe640 1 -- 172.21.15.94:0/428290095 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4580325c0 con 0x7fe460100e70 2024-04-21T08:36:53.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe4577fe640 1 -- 172.21.15.94:0/428290095 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4580042a0 con 0x7fe460100e70 2024-04-21T08:36:53.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.758+0000 7fe4683b2640 1 -- 172.21.15.94:0/428290095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4601a27b0 con 0x7fe460100e70 2024-04-21T08:36:53.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.759+0000 7fe4683b2640 1 -- 172.21.15.94:0/428290095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4601081d0 con 0x7fe460100e70 2024-04-21T08:36:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.760+0000 7fe4577fe640 1 -- 172.21.15.94:0/428290095 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe458046030 con 0x7fe460100e70 2024-04-21T08:36:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.760+0000 7fe4577fe640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe44c0a3bb0 0x7fe44c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.760+0000 7fe46594d640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe44c0a3bb0 0x7fe44c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.760+0000 7fe4577fe640 1 -- 172.21.15.94:0/428290095 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe4580f1e70 con 0x7fe460100e70 2024-04-21T08:36:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.761+0000 7fe46594d640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe44c0a3bb0 0x7fe44c0a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe450005fd0 tx=0x7fe450005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:53.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.764+0000 7fe4577fe640 1 -- 172.21.15.94:0/428290095 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe45804d060 con 0x7fe460100e70 2024-04-21T08:36:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:53 smithi111 ceph-mon[31493]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:53 smithi111 ceph-mon[31493]: Saving service mds.all spec with placement smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-04-21T08:36:53.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:53.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:53.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:53.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:36:53.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:36:53.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:53 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:53.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:53 smithi111 ceph-mon[31493]: Deploying daemon mds.l on smithi173 2024-04-21T08:36:53.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:53 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2870749833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:36:53.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.977+0000 7fe4683b2640 1 -- 172.21.15.94:0/428290095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe46002d5c0 con 0x7fe460100e70 2024-04-21T08:36:53.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.978+0000 7fe4577fe640 1 -- 172.21.15.94:0/428290095 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fe4580bed80 con 0x7fe460100e70 2024-04-21T08:36:53.977 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:36:53.977 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-21T08:36:53.977 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 1 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 -- 172.21.15.94:0/428290095 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe44c0a3bb0 msgr2=0x7fe44c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe44c0a3bb0 0x7fe44c0a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe450005fd0 tx=0x7fe450005850 comp rx=0 tx=0).stop 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 -- 172.21.15.94:0/428290095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe460100e70 msgr2=0x7fe46010b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe460100e70 0x7fe46010b660 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe458005e30 tx=0x7fe458034760 comp rx=0 tx=0).stop 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 -- 172.21.15.94:0/428290095 shutdown_connections 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe44c0a3bb0 0x7fe44c0a6070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe460101fc0 0x7fe460109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe460102980 0x7fe46010a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 --2- 172.21.15.94:0/428290095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe460100e70 0x7fe46010b660 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 -- 172.21.15.94:0/428290095 >> 172.21.15.94:0/428290095 conn(0x7fe4600fc3e0 msgr2=0x7fe4601053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 -- 172.21.15.94:0/428290095 shutdown_connections 2024-04-21T08:36:53.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:53.980+0000 7fe4683b2640 1 -- 172.21.15.94:0/428290095 wait complete. 2024-04-21T08:36:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:53 smithi094 ceph-mon[27988]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:53 smithi094 ceph-mon[27988]: Saving service mds.all spec with placement smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-04-21T08:36:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:36:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:36:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:53 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:53 smithi094 ceph-mon[27988]: Deploying daemon mds.l on smithi173 2024-04-21T08:36:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:53 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2870749833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:36:54.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:53 smithi173 ceph-mon[31428]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:53 smithi173 ceph-mon[31428]: Saving service mds.all spec with placement smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-04-21T08:36:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:36:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:36:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:53 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:53 smithi173 ceph-mon[31428]: Deploying daemon mds.l on smithi173 2024-04-21T08:36:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:53 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2870749833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:36:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:54 smithi111 ceph-mon[31493]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:54 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1954211123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:36:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:54 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/428290095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:36:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:54 smithi094 ceph-mon[27988]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:54 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1954211123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:36:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:54 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/428290095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:36:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:54 smithi173 ceph-mon[31428]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:54 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1954211123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:36:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:54 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/428290095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:36:55.011 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:36:54 smithi173 systemd[1]: Starting Ceph mds.l for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:36:55.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:36:55 smithi173 podman[61939]: 2024-04-21 08:36:54.970292394 +0000 UTC m=+0.017883393 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:36:55.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:36:55 smithi173 podman[61939]: 2024-04-21 08:36:55.114481611 +0000 UTC m=+0.162072624 container create ed554a819eb2b7acfa1c2245be2d476bc2d95071b4c6786b92485e3fd2a12243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T08:36:55.761 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:36:55 smithi173 podman[61939]: 2024-04-21 08:36:55.366897235 +0000 UTC m=+0.414488239 container init ed554a819eb2b7acfa1c2245be2d476bc2d95071b4c6786b92485e3fd2a12243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-21T08:36:55.761 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:36:55 smithi173 podman[61939]: 2024-04-21 08:36:55.369466905 +0000 UTC m=+0.417057877 container start ed554a819eb2b7acfa1c2245be2d476bc2d95071b4c6786b92485e3fd2a12243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-l, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-21T08:36:55.761 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:36:55 smithi173 ceph-mds[61958]: -- 172.21.15.173:0/4139218741 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c84e35a40 con 0x559c84e09000 2024-04-21T08:36:55.761 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:36:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-l[61954]: starting mds.l at 2024-04-21T08:36:55.762 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:36:55 smithi173 bash[61939]: ed554a819eb2b7acfa1c2245be2d476bc2d95071b4c6786b92485e3fd2a12243 2024-04-21T08:36:55.762 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:36:55 smithi173 systemd[1]: Started Ceph mds.l for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:36:55.990 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 -- 172.21.15.94:0/2397777951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b54100e70 msgr2=0x7f3b54101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 --2- 172.21.15.94:0/2397777951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b54100e70 0x7f3b54101290 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009900 tx=0x7f3b3c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 -- 172.21.15.94:0/2397777951 shutdown_connections 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 --2- 172.21.15.94:0/2397777951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b54101fc0 0x7f3b54102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 --2- 172.21.15.94:0/2397777951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b54102980 0x7f3b541092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 --2- 172.21.15.94:0/2397777951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b54100e70 0x7f3b54101290 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 -- 172.21.15.94:0/2397777951 >> 172.21.15.94:0/2397777951 conn(0x7f3b540fc3e0 msgr2=0x7f3b540fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 -- 172.21.15.94:0/2397777951 shutdown_connections 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 -- 172.21.15.94:0/2397777951 wait complete. 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 Processor -- start 2024-04-21T08:36:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.100+0000 7f3b5a754640 1 -- start start 2024-04-21T08:36:56.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b5a754640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b54100e70 0x7f3b54069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:56.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b5a754640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b54101fc0 0x7f3b5419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:56.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b5a754640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b54102980 0x7f3b5419f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:56.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b5a754640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5410c9c0 con 0x7f3b54101fc0 2024-04-21T08:36:56.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b5a754640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5410c840 con 0x7f3b54102980 2024-04-21T08:36:56.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b5a754640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5410cb40 con 0x7f3b54100e70 2024-04-21T08:36:56.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b54101fc0 0x7f3b5419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:56.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b54101fc0 0x7f3b5419ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58164/0 (socket says 172.21.15.94:58164) 2024-04-21T08:36:56.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b537fe640 1 -- 172.21.15.94:0/1420449568 learned_addr learned my addr 172.21.15.94:0/1420449568 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:36:56.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b58cf1640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b54102980 0x7f3b5419f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b537fe640 1 -- 172.21.15.94:0/1420449568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b54100e70 msgr2=0x7f3b54069cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b537fe640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b54100e70 0x7f3b54069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b537fe640 1 -- 172.21.15.94:0/1420449568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b54102980 msgr2=0x7f3b5419f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b537fe640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b54102980 0x7f3b5419f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b537fe640 1 -- 172.21.15.94:0/1420449568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b5419fda0 con 0x7f3b54101fc0 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b58cf1640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b54102980 0x7f3b5419f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b537fe640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b54101fc0 0x7f3b5419ce70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400d950 tx=0x7f3b4400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b517fa640 1 -- 172.21.15.94:0/1420449568 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44017070 con 0x7f3b54101fc0 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b517fa640 1 -- 172.21.15.94:0/1420449568 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b44009c30 con 0x7f3b54101fc0 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.101+0000 7f3b5a754640 1 -- 172.21.15.94:0/1420449568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b541a0090 con 0x7f3b54101fc0 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.102+0000 7f3b517fa640 1 -- 172.21.15.94:0/1420449568 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4400f620 con 0x7f3b54101fc0 2024-04-21T08:36:56.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.102+0000 7f3b5a754640 1 -- 172.21.15.94:0/1420449568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b541a0590 con 0x7f3b54101fc0 2024-04-21T08:36:56.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.103+0000 7f3b5a754640 1 -- 172.21.15.94:0/1420449568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b54108250 con 0x7f3b54101fc0 2024-04-21T08:36:56.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.104+0000 7f3b517fa640 1 -- 172.21.15.94:0/1420449568 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3b4401a070 con 0x7f3b54101fc0 2024-04-21T08:36:56.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.104+0000 7f3b517fa640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b400a3bb0 0x7f3b400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:56.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.104+0000 7f3b53fff640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b400a3bb0 0x7f3b400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:56.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.104+0000 7f3b517fa640 1 -- 172.21.15.94:0/1420449568 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3b440cf9a0 con 0x7f3b54101fc0 2024-04-21T08:36:56.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.105+0000 7f3b53fff640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b400a3bb0 0x7f3b400a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c036040 tx=0x7f3b3c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:56.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.109+0000 7f3b517fa640 1 -- 172.21.15.94:0/1420449568 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b44009de0 con 0x7f3b54101fc0 2024-04-21T08:36:56.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.326+0000 7f3b5a754640 1 -- 172.21.15.94:0/1420449568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b5402d5c0 con 0x7f3b54101fc0 2024-04-21T08:36:56.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.326+0000 7f3b517fa640 1 -- 172.21.15.94:0/1420449568 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f3b4409c820 con 0x7f3b54101fc0 2024-04-21T08:36:56.325 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:36:56.325 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:36:56.325 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 2 2024-04-21T08:36:56.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.328+0000 7f3b5a754640 1 -- 172.21.15.94:0/1420449568 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b400a3bb0 msgr2=0x7f3b400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.328+0000 7f3b5a754640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b400a3bb0 0x7f3b400a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c036040 tx=0x7f3b3c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.328+0000 7f3b5a754640 1 -- 172.21.15.94:0/1420449568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b54101fc0 msgr2=0x7f3b5419ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.329+0000 7f3b5a754640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b54101fc0 0x7f3b5419ce70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400d950 tx=0x7f3b4400de10 comp rx=0 tx=0).stop 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.329+0000 7f3b5a754640 1 -- 172.21.15.94:0/1420449568 shutdown_connections 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.329+0000 7f3b5a754640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b400a3bb0 0x7f3b400a6070 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.329+0000 7f3b5a754640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b54102980 0x7f3b5419f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.329+0000 7f3b5a754640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b54101fc0 0x7f3b5419ce70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.329+0000 7f3b5a754640 1 --2- 172.21.15.94:0/1420449568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b54100e70 0x7f3b54069cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.329+0000 7f3b5a754640 1 -- 172.21.15.94:0/1420449568 >> 172.21.15.94:0/1420449568 conn(0x7f3b540fc3e0 msgr2=0x7f3b54105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.329+0000 7f3b5a754640 1 -- 172.21.15.94:0/1420449568 shutdown_connections 2024-04-21T08:36:56.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:56.329+0000 7f3b5a754640 1 -- 172.21.15.94:0/1420449568 wait complete. 2024-04-21T08:36:56.431 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.173:6832/2495824461,v1:172.21.15.173:6833/2495824461] up:boot 2024-04-21T08:36:56.431 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: fsmap 1 up:standby 2024-04-21T08:36:56.431 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:36:56.431 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:56.431 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:56.431 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:56.431 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:36:56.431 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:36:56.432 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:56.432 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: Deploying daemon mds.k on smithi111 2024-04-21T08:36:56.432 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:56.432 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:56 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1420449568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:36:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.173:6832/2495824461,v1:172.21.15.173:6833/2495824461] up:boot 2024-04-21T08:36:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: fsmap 1 up:standby 2024-04-21T08:36:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:36:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:36:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:36:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: Deploying daemon mds.k on smithi111 2024-04-21T08:36:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:56.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:56 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1420449568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:36:56.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.173:6832/2495824461,v1:172.21.15.173:6833/2495824461] up:boot 2024-04-21T08:36:56.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: fsmap 1 up:standby 2024-04-21T08:36:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:36:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:36:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:36:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: Deploying daemon mds.k on smithi111 2024-04-21T08:36:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:56.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:56 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1420449568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:36:57.860 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:36:57 smithi111 systemd[1]: Starting Ceph mds.k for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:36:58.115 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:36:57 smithi111 podman[61038]: 2024-04-21 08:36:57.76441783 +0000 UTC m=+0.018138662 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:36:58.116 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:36:57 smithi111 podman[61038]: 2024-04-21 08:36:57.897959899 +0000 UTC m=+0.151680704 container create 85af0acb4c4a92af54a3cca7cd68e45205c643c2be86d053f3c8f7489a15c46d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-21T08:36:58.116 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:36:58 smithi111 podman[61038]: 2024-04-21 08:36:58.117071915 +0000 UTC m=+0.370792728 container init 85af0acb4c4a92af54a3cca7cd68e45205c643c2be86d053f3c8f7489a15c46d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-k, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-21T08:36:58.343 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:36:58.437 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:36:58 smithi111 podman[61038]: 2024-04-21 08:36:58.119971505 +0000 UTC m=+0.373692302 container start 85af0acb4c4a92af54a3cca7cd68e45205c643c2be86d053f3c8f7489a15c46d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:36:58.437 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:36:58 smithi111 ceph-mds[61057]: -- 172.21.15.111:0/922428404 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e19da65a40 con 0x55e19da38800 2024-04-21T08:36:58.437 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:36:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-k[61053]: starting mds.k at 2024-04-21T08:36:58.437 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:36:58 smithi111 bash[61038]: 85af0acb4c4a92af54a3cca7cd68e45205c643c2be86d053f3c8f7489a15c46d 2024-04-21T08:36:58.437 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:36:58 smithi111 systemd[1]: Started Ceph mds.k for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:36:58.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.464+0000 7f3970cac640 1 -- 172.21.15.94:0/2480535857 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c105380 msgr2=0x7f396c107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:58.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.464+0000 7f3970cac640 1 --2- 172.21.15.94:0/2480535857 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c105380 0x7f396c107770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3960009900 tx=0x7f396002bc30 comp rx=0 tx=0).stop 2024-04-21T08:36:58.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.464+0000 7f3970cac640 1 -- 172.21.15.94:0/2480535857 shutdown_connections 2024-04-21T08:36:58.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.464+0000 7f3970cac640 1 --2- 172.21.15.94:0/2480535857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f396c107cb0 0x7f396c10a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:58.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.464+0000 7f3970cac640 1 --2- 172.21.15.94:0/2480535857 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c105380 0x7f396c107770 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:58.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.464+0000 7f3970cac640 1 --2- 172.21.15.94:0/2480535857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f396c102a50 0x7f396c104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:58.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.464+0000 7f3970cac640 1 -- 172.21.15.94:0/2480535857 >> 172.21.15.94:0/2480535857 conn(0x7f396c0fc3e0 msgr2=0x7f396c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:58.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.464+0000 7f3970cac640 1 -- 172.21.15.94:0/2480535857 shutdown_connections 2024-04-21T08:36:58.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.464+0000 7f3970cac640 1 -- 172.21.15.94:0/2480535857 wait complete. 2024-04-21T08:36:58.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3970cac640 1 Processor -- start 2024-04-21T08:36:58.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3970cac640 1 -- start start 2024-04-21T08:36:58.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3970cac640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f396c102a50 0x7f396c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:58.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3970cac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c105380 0x7f396c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:58.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3970cac640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f396c107cb0 0x7f396c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:58.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3970cac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f396c10daa0 con 0x7f396c105380 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3970cac640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f396c10d920 con 0x7f396c102a50 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3970cac640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f396c10dc20 con 0x7f396c107cb0 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3969d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c105380 0x7f396c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3969d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c105380 0x7f396c198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58190/0 (socket says 172.21.15.94:58190) 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.465+0000 7f3969d9b640 1 -- 172.21.15.94:0/2004906255 learned_addr learned my addr 172.21.15.94:0/2004906255 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f396ad9d640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f396c107cb0 0x7f396c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f396a59c640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f396c102a50 0x7f396c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f3969d9b640 1 -- 172.21.15.94:0/2004906255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f396c107cb0 msgr2=0x7f396c19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f3969d9b640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f396c107cb0 0x7f396c19f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f3969d9b640 1 -- 172.21.15.94:0/2004906255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f396c102a50 msgr2=0x7f396c198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f3969d9b640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f396c102a50 0x7f396c198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f3969d9b640 1 -- 172.21.15.94:0/2004906255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c19fe40 con 0x7f396c105380 2024-04-21T08:36:58.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f396ad9d640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f396c107cb0 0x7f396c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:36:58.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f3969d9b640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c105380 0x7f396c198760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3960000c00 tx=0x7f3960004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:58.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f395b7fe640 1 -- 172.21.15.94:0/2004906255 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960043020 con 0x7f396c105380 2024-04-21T08:36:58.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f395b7fe640 1 -- 172.21.15.94:0/2004906255 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3960004aa0 con 0x7f396c105380 2024-04-21T08:36:58.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f395b7fe640 1 -- 172.21.15.94:0/2004906255 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960003d00 con 0x7f396c105380 2024-04-21T08:36:58.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f3970cac640 1 -- 172.21.15.94:0/2004906255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c1a00d0 con 0x7f396c105380 2024-04-21T08:36:58.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.466+0000 7f3970cac640 1 -- 172.21.15.94:0/2004906255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c1a05d0 con 0x7f396c105380 2024-04-21T08:36:58.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.467+0000 7f3970cac640 1 -- 172.21.15.94:0/2004906255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396c0fded0 con 0x7f396c105380 2024-04-21T08:36:58.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.468+0000 7f395b7fe640 1 -- 172.21.15.94:0/2004906255 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3960046020 con 0x7f396c105380 2024-04-21T08:36:58.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.469+0000 7f395b7fe640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f39540a3bb0 0x7f39540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:58.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.469+0000 7f395b7fe640 1 -- 172.21.15.94:0/2004906255 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f39600f25a0 con 0x7f396c105380 2024-04-21T08:36:58.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.469+0000 7f396a59c640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f39540a3bb0 0x7f39540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:58.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.470+0000 7f396a59c640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f39540a3bb0 0x7f39540a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f395c005fd0 tx=0x7f395c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:58.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.474+0000 7f395b7fe640 1 -- 172.21.15.94:0/2004906255 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f396004d020 con 0x7f396c105380 2024-04-21T08:36:58.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.697+0000 7f3970cac640 1 -- 172.21.15.94:0/2004906255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f396c02d790 con 0x7f396c105380 2024-04-21T08:36:58.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.697+0000 7f395b7fe640 1 -- 172.21.15.94:0/2004906255 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f39600bf420 con 0x7f396c105380 2024-04-21T08:36:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:36:58.696 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-21T08:36:58.697 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 3 2024-04-21T08:36:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 -- 172.21.15.94:0/2004906255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f39540a3bb0 msgr2=0x7f39540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f39540a3bb0 0x7f39540a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f395c005fd0 tx=0x7f395c005e20 comp rx=0 tx=0).stop 2024-04-21T08:36:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 -- 172.21.15.94:0/2004906255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c105380 msgr2=0x7f396c198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c105380 0x7f396c198760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3960000c00 tx=0x7f3960004270 comp rx=0 tx=0).stop 2024-04-21T08:36:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 -- 172.21.15.94:0/2004906255 shutdown_connections 2024-04-21T08:36:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f39540a3bb0 0x7f39540a6070 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f396c107cb0 0x7f396c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c105380 0x7f396c198760 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 --2- 172.21.15.94:0/2004906255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f396c102a50 0x7f396c198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 -- 172.21.15.94:0/2004906255 >> 172.21.15.94:0/2004906255 conn(0x7f396c0fc3e0 msgr2=0x7f396c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:58.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 -- 172.21.15.94:0/2004906255 shutdown_connections 2024-04-21T08:36:58.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:36:58.700+0000 7f3970cac640 1 -- 172.21.15.94:0/2004906255 wait complete. 2024-04-21T08:36:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:58 smithi111 ceph-mon[31493]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:36:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:36:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:58 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:58.971 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:58 smithi094 ceph-mon[27988]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:58.971 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:58.972 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:58.972 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:58.972 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:36:58.972 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:36:58.972 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:58 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:59.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:58 smithi173 ceph-mon[31428]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:36:59.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:59.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:36:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:36:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:36:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:58 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:59 smithi111 ceph-mon[31493]: Deploying daemon mds.j on smithi094 2024-04-21T08:36:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:59 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.111:6832/4021460222,v1:172.21.15.111:6833/4021460222] up:boot 2024-04-21T08:36:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:59 smithi111 ceph-mon[31493]: fsmap 2 up:standby 2024-04-21T08:36:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:59 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:36:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:36:59 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2004906255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:59 smithi094 ceph-mon[27988]: Deploying daemon mds.j on smithi094 2024-04-21T08:37:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:59 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.111:6832/4021460222,v1:172.21.15.111:6833/4021460222] up:boot 2024-04-21T08:37:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:59 smithi094 ceph-mon[27988]: fsmap 2 up:standby 2024-04-21T08:37:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:59 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:37:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:36:59 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2004906255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:00.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:59 smithi173 ceph-mon[31428]: Deploying daemon mds.j on smithi094 2024-04-21T08:37:00.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:59 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.111:6832/4021460222,v1:172.21.15.111:6833/4021460222] up:boot 2024-04-21T08:37:00.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:59 smithi173 ceph-mon[31428]: fsmap 2 up:standby 2024-04-21T08:37:00.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:59 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:37:00.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:36:59 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2004906255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:00.635 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:37:00 smithi094 systemd[1]: Starting Ceph mds.j for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:37:00.710 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:00.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.814+0000 7f16139b6640 1 -- 172.21.15.94:0/4226443527 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f160c102a10 msgr2=0x7f160c109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:00.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.814+0000 7f16139b6640 1 --2- 172.21.15.94:0/4226443527 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f160c102a10 0x7f160c109330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1604009ca0 tx=0x7f160402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:00.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.814+0000 7f16139b6640 1 -- 172.21.15.94:0/4226443527 shutdown_connections 2024-04-21T08:37:00.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.814+0000 7f16139b6640 1 --2- 172.21.15.94:0/4226443527 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f160c102a10 0x7f160c109330 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:00.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.814+0000 7f16139b6640 1 --2- 172.21.15.94:0/4226443527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160c102050 0x7f160c1024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:00.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.814+0000 7f16139b6640 1 --2- 172.21.15.94:0/4226443527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f160c100e70 0x7f160c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:00.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.814+0000 7f16139b6640 1 -- 172.21.15.94:0/4226443527 >> 172.21.15.94:0/4226443527 conn(0x7f160c0fc470 msgr2=0x7f160c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:00.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.814+0000 7f16139b6640 1 -- 172.21.15.94:0/4226443527 shutdown_connections 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.814+0000 7f16139b6640 1 -- 172.21.15.94:0/4226443527 wait complete. 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f16139b6640 1 Processor -- start 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f16139b6640 1 -- start start 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f16139b6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f160c100e70 0x7f160c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f16139b6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160c102050 0x7f160c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f16139b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f160c102a10 0x7f160c19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f16139b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f160c10db40 con 0x7f160c102a10 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f16139b6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f160c10d9c0 con 0x7f160c102050 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f16139b6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f160c10dcc0 con 0x7f160c100e70 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611f53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f160c102a10 0x7f160c19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611f53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f160c102a10 0x7f160c19f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58210/0 (socket says 172.21.15.94:58210) 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611f53640 1 -- 172.21.15.94:0/4012063088 learned_addr learned my addr 172.21.15.94:0/4012063088 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1610f51640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160c102050 0x7f160c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:00.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611752640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f160c100e70 0x7f160c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611f53640 1 -- 172.21.15.94:0/4012063088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f160c100e70 msgr2=0x7f160c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611f53640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f160c100e70 0x7f160c1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611f53640 1 -- 172.21.15.94:0/4012063088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160c102050 msgr2=0x7f160c198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611f53640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160c102050 0x7f160c198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611f53640 1 -- 172.21.15.94:0/4012063088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160c19fcf0 con 0x7f160c102a10 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611752640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f160c100e70 0x7f160c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f1611f53640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f160c102a10 0x7f160c19f680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f160402c700 tx=0x7f1604034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.815+0000 7f16027fc640 1 -- 172.21.15.94:0/4012063088 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1604039040 con 0x7f160c102a10 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.816+0000 7f16027fc640 1 -- 172.21.15.94:0/4012063088 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1604044070 con 0x7f160c102a10 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.816+0000 7f16139b6640 1 -- 172.21.15.94:0/4012063088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f160c19ff80 con 0x7f160c102a10 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.816+0000 7f16027fc640 1 -- 172.21.15.94:0/4012063088 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16040042a0 con 0x7f160c102a10 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.816+0000 7f16139b6640 1 -- 172.21.15.94:0/4012063088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f160c1a03e0 con 0x7f160c102a10 2024-04-21T08:37:00.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.816+0000 7f16139b6640 1 -- 172.21.15.94:0/4012063088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f160c1081b0 con 0x7f160c102a10 2024-04-21T08:37:00.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.818+0000 7f16027fc640 1 -- 172.21.15.94:0/4012063088 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f16040325c0 con 0x7f160c102a10 2024-04-21T08:37:00.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.818+0000 7f16027fc640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f15f80a3bb0 0x7f15f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:00.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.818+0000 7f1611752640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f15f80a3bb0 0x7f15f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:00.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.818+0000 7f16027fc640 1 -- 172.21.15.94:0/4012063088 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f16040f2dd0 con 0x7f160c102a10 2024-04-21T08:37:00.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.819+0000 7f1611752640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f15f80a3bb0 0x7f15f80a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0058c0 tx=0x7f15fc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:00.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:00.822+0000 7f16027fc640 1 -- 172.21.15.94:0/4012063088 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16040bfc50 con 0x7f160c102a10 2024-04-21T08:37:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:00 smithi111 ceph-mon[31493]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:00.967 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:00 smithi094 ceph-mon[27988]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:00.967 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:37:00 smithi094 podman[68096]: 2024-04-21 08:37:00.627777466 +0000 UTC m=+0.018167612 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:37:00.967 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:37:00 smithi094 podman[68096]: 2024-04-21 08:37:00.748217284 +0000 UTC m=+0.138607413 container create fc928bce4bb99a605fb4b2cb3617b38b6e2a93e71d234c9bb08b1f4fa01067b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-j, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-21T08:37:00.967 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:37:00 smithi094 podman[68096]: 2024-04-21 08:37:00.968238469 +0000 UTC m=+0.358628608 container init fc928bce4bb99a605fb4b2cb3617b38b6e2a93e71d234c9bb08b1f4fa01067b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-21T08:37:01.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:00 smithi173 ceph-mon[31428]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.048+0000 7f16139b6640 1 -- 172.21.15.94:0/4012063088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f160c02d5c0 con 0x7f160c102a10 2024-04-21T08:37:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.049+0000 7f16027fc640 1 -- 172.21.15.94:0/4012063088 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f16040bf550 con 0x7f160c102a10 2024-04-21T08:37:01.048 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:01.048 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-21T08:37:01.048 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 3 2024-04-21T08:37:01.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 -- 172.21.15.94:0/4012063088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f15f80a3bb0 msgr2=0x7f15f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:01.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f15f80a3bb0 0x7f15f80a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0058c0 tx=0x7f15fc005850 comp rx=0 tx=0).stop 2024-04-21T08:37:01.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 -- 172.21.15.94:0/4012063088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f160c102a10 msgr2=0x7f160c19f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:01.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f160c102a10 0x7f160c19f680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f160402c700 tx=0x7f1604034760 comp rx=0 tx=0).stop 2024-04-21T08:37:01.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 -- 172.21.15.94:0/4012063088 shutdown_connections 2024-04-21T08:37:01.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f15f80a3bb0 0x7f15f80a6070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:01.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f160c102a10 0x7f160c19f680 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:01.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160c102050 0x7f160c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:01.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 --2- 172.21.15.94:0/4012063088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f160c100e70 0x7f160c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 -- 172.21.15.94:0/4012063088 >> 172.21.15.94:0/4012063088 conn(0x7f160c0fc470 msgr2=0x7f160c105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 -- 172.21.15.94:0/4012063088 shutdown_connections 2024-04-21T08:37:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:01.051+0000 7f16139b6640 1 -- 172.21.15.94:0/4012063088 wait complete. 2024-04-21T08:37:01.260 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:37:00 smithi094 podman[68096]: 2024-04-21 08:37:00.972567922 +0000 UTC m=+0.362958053 container start fc928bce4bb99a605fb4b2cb3617b38b6e2a93e71d234c9bb08b1f4fa01067b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD) 2024-04-21T08:37:01.260 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:37:00 smithi094 ceph-mds[68154]: -- 172.21.15.94:0/1269672334 <== mon.0 v2:172.21.15.94:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557762547a40 con 0x55776251b000 2024-04-21T08:37:01.260 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:37:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-j[68150]: starting mds.j at 2024-04-21T08:37:01.260 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:37:01 smithi094 bash[68096]: fc928bce4bb99a605fb4b2cb3617b38b6e2a93e71d234c9bb08b1f4fa01067b0 2024-04-21T08:37:01.260 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:37:01 smithi094 systemd[1]: Started Ceph mds.j for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:37:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:01 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4012063088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:01 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:01 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:01 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:01 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:01 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:01.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:01 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:01.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:01 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4012063088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:01.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:01 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:01.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:01 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:01.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:01 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:01.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:01 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:01.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:01 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:01.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:01 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:02.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:01 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4012063088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:02.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:01 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:02.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:01 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:02.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:01 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:02.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:01 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:02.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:01 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:02.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:01 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:02.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:02 smithi094 ceph-mon[27988]: Deploying daemon mds.d on smithi094 2024-04-21T08:37:02.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:02 smithi094 ceph-mon[27988]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:02.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:02 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.94:6834/1080139605,v1:172.21.15.94:6835/1080139605] up:boot 2024-04-21T08:37:02.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:02 smithi094 ceph-mon[27988]: fsmap 3 up:standby 2024-04-21T08:37:02.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:02 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:37:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:02 smithi111 ceph-mon[31493]: Deploying daemon mds.d on smithi094 2024-04-21T08:37:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:02 smithi111 ceph-mon[31493]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:02 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.94:6834/1080139605,v1:172.21.15.94:6835/1080139605] up:boot 2024-04-21T08:37:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:02 smithi111 ceph-mon[31493]: fsmap 3 up:standby 2024-04-21T08:37:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:02 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:37:03.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:02 smithi173 ceph-mon[31428]: Deploying daemon mds.d on smithi094 2024-04-21T08:37:03.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:02 smithi173 ceph-mon[31428]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:03.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:02 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.94:6834/1080139605,v1:172.21.15.94:6835/1080139605] up:boot 2024-04-21T08:37:03.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:02 smithi173 ceph-mon[31428]: fsmap 3 up:standby 2024-04-21T08:37:03.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:02 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:37:03.062 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.282+0000 7fb7c908c640 1 -- 172.21.15.94:0/2933183204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c4100e70 msgr2=0x7fb7c4101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.282+0000 7fb7c908c640 1 --2- 172.21.15.94:0/2933183204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c4100e70 0x7fb7c4101290 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009900 tx=0x7fb7bc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.282+0000 7fb7c908c640 1 -- 172.21.15.94:0/2933183204 shutdown_connections 2024-04-21T08:37:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.282+0000 7fb7c908c640 1 --2- 172.21.15.94:0/2933183204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4101fc0 0x7fb7c4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.282+0000 7fb7c908c640 1 --2- 172.21.15.94:0/2933183204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7c4102980 0x7fb7c41092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.282+0000 7fb7c908c640 1 --2- 172.21.15.94:0/2933183204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c4100e70 0x7fb7c4101290 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.282+0000 7fb7c908c640 1 -- 172.21.15.94:0/2933183204 >> 172.21.15.94:0/2933183204 conn(0x7fb7c40fc3e0 msgr2=0x7fb7c40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.282+0000 7fb7c908c640 1 -- 172.21.15.94:0/2933183204 shutdown_connections 2024-04-21T08:37:03.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.282+0000 7fb7c908c640 1 -- 172.21.15.94:0/2933183204 wait complete. 2024-04-21T08:37:03.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.283+0000 7fb7c908c640 1 Processor -- start 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.283+0000 7fb7c908c640 1 -- start start 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.283+0000 7fb7c908c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4100e70 0x7fb7c4069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.283+0000 7fb7c908c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7c4101fc0 0x7fb7c419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.283+0000 7fb7c908c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c4102980 0x7fb7c419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.283+0000 7fb7c908c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c410c9c0 con 0x7fb7c4102980 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.283+0000 7fb7c908c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c410c840 con 0x7fb7c4101fc0 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.283+0000 7fb7c908c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c410cb40 con 0x7fb7c4100e70 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.283+0000 7fb7c359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c4102980 0x7fb7c419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c4102980 0x7fb7c419f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58252/0 (socket says 172.21.15.94:58252) 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c359e640 1 -- 172.21.15.94:0/758750445 learned_addr learned my addr 172.21.15.94:0/758750445 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c259c640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7c4101fc0 0x7fb7c419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c2d9d640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4100e70 0x7fb7c4069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c359e640 1 -- 172.21.15.94:0/758750445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4100e70 msgr2=0x7fb7c4069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c359e640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4100e70 0x7fb7c4069c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c359e640 1 -- 172.21.15.94:0/758750445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7c4101fc0 msgr2=0x7fb7c419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c359e640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7c4101fc0 0x7fb7c419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c359e640 1 -- 172.21.15.94:0/758750445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c419fd90 con 0x7fb7c4102980 2024-04-21T08:37:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c359e640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c4102980 0x7fb7c419f4d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400cce0 tx=0x7fb7b400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7abfff640 1 -- 172.21.15.94:0/758750445 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b400f450 con 0x7fb7c4102980 2024-04-21T08:37:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7abfff640 1 -- 172.21.15.94:0/758750445 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7b400faa0 con 0x7fb7c4102980 2024-04-21T08:37:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c908c640 1 -- 172.21.15.94:0/758750445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c41a0080 con 0x7fb7c4102980 2024-04-21T08:37:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7abfff640 1 -- 172.21.15.94:0/758750445 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4018b70 con 0x7fb7c4102980 2024-04-21T08:37:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.284+0000 7fb7c908c640 1 -- 172.21.15.94:0/758750445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c41a0580 con 0x7fb7c4102980 2024-04-21T08:37:03.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.285+0000 7fb7c908c640 1 -- 172.21.15.94:0/758750445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c4108180 con 0x7fb7c4102980 2024-04-21T08:37:03.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.286+0000 7fb7abfff640 1 -- 172.21.15.94:0/758750445 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb7b4018d10 con 0x7fb7c4102980 2024-04-21T08:37:03.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.287+0000 7fb7abfff640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7ac0a3bb0 0x7fb7ac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:03.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.287+0000 7fb7c2d9d640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7ac0a3bb0 0x7fb7ac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:03.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.287+0000 7fb7abfff640 1 -- 172.21.15.94:0/758750445 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb7b40d03e0 con 0x7fb7c4102980 2024-04-21T08:37:03.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.288+0000 7fb7c2d9d640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7ac0a3bb0 0x7fb7ac0a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc000c00 tx=0x7fb7bc02c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:03.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.291+0000 7fb7abfff640 1 -- 172.21.15.94:0/758750445 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7b4014030 con 0x7fb7c4102980 2024-04-21T08:37:03.510 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:37:03 smithi094 systemd[1]: Starting Ceph mds.d for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:37:03.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.532+0000 7fb7c908c640 1 -- 172.21.15.94:0/758750445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7c402d770 con 0x7fb7c4102980 2024-04-21T08:37:03.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.533+0000 7fb7abfff640 1 -- 172.21.15.94:0/758750445 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fb7b409d260 con 0x7fb7c4102980 2024-04-21T08:37:03.532 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:03.532 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-21T08:37:03.532 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 4 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 -- 172.21.15.94:0/758750445 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7ac0a3bb0 msgr2=0x7fb7ac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7ac0a3bb0 0x7fb7ac0a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc000c00 tx=0x7fb7bc02c800 comp rx=0 tx=0).stop 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 -- 172.21.15.94:0/758750445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c4102980 msgr2=0x7fb7c419f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c4102980 0x7fb7c419f4d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400cce0 tx=0x7fb7b400a620 comp rx=0 tx=0).stop 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 -- 172.21.15.94:0/758750445 shutdown_connections 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7ac0a3bb0 0x7fb7ac0a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c4102980 0x7fb7c419f4d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7c4101fc0 0x7fb7c419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 --2- 172.21.15.94:0/758750445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4100e70 0x7fb7c4069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 -- 172.21.15.94:0/758750445 >> 172.21.15.94:0/758750445 conn(0x7fb7c40fc3e0 msgr2=0x7fb7c4105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 -- 172.21.15.94:0/758750445 shutdown_connections 2024-04-21T08:37:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:03.536+0000 7fb7c908c640 1 -- 172.21.15.94:0/758750445 wait complete. 2024-04-21T08:37:03.800 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:03 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/758750445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:03.800 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:37:03 smithi094 podman[68489]: 2024-04-21 08:37:03.547720762 +0000 UTC m=+0.018195603 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:37:03.800 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:37:03 smithi094 podman[68489]: 2024-04-21 08:37:03.664436612 +0000 UTC m=+0.134911427 container create a9482c9c542f18a93cbb731de240e387928ecc804a66153a638a17f0805fc25b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-d, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:37:03.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:03 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/758750445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:04.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:03 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/758750445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:04.260 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:37:03 smithi094 podman[68489]: 2024-04-21 08:37:03.909212399 +0000 UTC m=+0.379687221 container init a9482c9c542f18a93cbb731de240e387928ecc804a66153a638a17f0805fc25b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T08:37:04.260 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:37:03 smithi094 podman[68489]: 2024-04-21 08:37:03.911846201 +0000 UTC m=+0.382321023 container start a9482c9c542f18a93cbb731de240e387928ecc804a66153a638a17f0805fc25b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True) 2024-04-21T08:37:04.260 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:37:03 smithi094 ceph-mds[68510]: -- 172.21.15.94:0/1944158260 <== mon.0 v2:172.21.15.94:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bcea229a40 con 0x55bcea1fc800 2024-04-21T08:37:04.260 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:37:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-d[68506]: starting mds.d at 2024-04-21T08:37:04.261 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:37:04 smithi094 bash[68489]: a9482c9c542f18a93cbb731de240e387928ecc804a66153a638a17f0805fc25b 2024-04-21T08:37:04.261 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:37:04 smithi094 systemd[1]: Started Ceph mds.d for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:37:04.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:04 smithi111 ceph-mon[31493]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:04.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:04.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:04.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:04.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:04.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:04.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:04 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:04 smithi094 ceph-mon[27988]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:05.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:05.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:04 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:05.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:04 smithi173 ceph-mon[31428]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:05.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:05.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:05.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:05.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:05.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:05.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:04 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:05.549 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:05.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.673+0000 7fd8cc322640 1 -- 172.21.15.94:0/1887330820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8c4105320 msgr2=0x7fd8c4107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:05.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.673+0000 7fd8cc322640 1 --2- 172.21.15.94:0/1887330820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8c4105320 0x7fd8c4107710 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009900 tx=0x7fd8b402bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:05.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.673+0000 7fd8cc322640 1 -- 172.21.15.94:0/1887330820 shutdown_connections 2024-04-21T08:37:05.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.673+0000 7fd8cc322640 1 --2- 172.21.15.94:0/1887330820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c4107c50 0x7fd8c410a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:05.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.673+0000 7fd8cc322640 1 --2- 172.21.15.94:0/1887330820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8c4105320 0x7fd8c4107710 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:05.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.673+0000 7fd8cc322640 1 --2- 172.21.15.94:0/1887330820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c41029f0 0x7fd8c4104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:05.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.673+0000 7fd8cc322640 1 -- 172.21.15.94:0/1887330820 >> 172.21.15.94:0/1887330820 conn(0x7fd8c40fc420 msgr2=0x7fd8c40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:05.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.673+0000 7fd8cc322640 1 -- 172.21.15.94:0/1887330820 shutdown_connections 2024-04-21T08:37:05.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.673+0000 7fd8cc322640 1 -- 172.21.15.94:0/1887330820 wait complete. 2024-04-21T08:37:05.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8cc322640 1 Processor -- start 2024-04-21T08:37:05.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8cc322640 1 -- start start 2024-04-21T08:37:05.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8cc322640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c41029f0 0x7fd8c4198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:05.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8cc322640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8c4105320 0x7fd8c4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8cc322640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c4107c50 0x7fd8c419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8cc322640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c410da80 con 0x7fd8c4105320 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8cc322640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c410d900 con 0x7fd8c41029f0 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8cc322640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c410dc00 con 0x7fd8c4107c50 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8c98bd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8c4105320 0x7fd8c4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8c98bd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8c4105320 0x7fd8c4198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34960/0 (socket says 172.21.15.94:34960) 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8c98bd640 1 -- 172.21.15.94:0/1720315507 learned_addr learned my addr 172.21.15.94:0/1720315507 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.674+0000 7fd8ca8bf640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c4107c50 0x7fd8c419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8ca0be640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c41029f0 0x7fd8c4198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8c98bd640 1 -- 172.21.15.94:0/1720315507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c4107c50 msgr2=0x7fd8c419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8c98bd640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c4107c50 0x7fd8c419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8c98bd640 1 -- 172.21.15.94:0/1720315507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c41029f0 msgr2=0x7fd8c4198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8c98bd640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c41029f0 0x7fd8c4198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:05.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8c98bd640 1 -- 172.21.15.94:0/1720315507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c419fdf0 con 0x7fd8c4105320 2024-04-21T08:37:05.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8ca0be640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c41029f0 0x7fd8c4198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:05.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8c98bd640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8c4105320 0x7fd8c4198760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4000c00 tx=0x7fd8b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:05.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/1720315507 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4043020 con 0x7fd8c4105320 2024-04-21T08:37:05.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/1720315507 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b4004aa0 con 0x7fd8c4105320 2024-04-21T08:37:05.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8cc322640 1 -- 172.21.15.94:0/1720315507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c41a0080 con 0x7fd8c4105320 2024-04-21T08:37:05.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/1720315507 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4003d00 con 0x7fd8c4105320 2024-04-21T08:37:05.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.675+0000 7fd8cc322640 1 -- 172.21.15.94:0/1720315507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c41a05d0 con 0x7fd8c4105320 2024-04-21T08:37:05.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.676+0000 7fd8cc322640 1 -- 172.21.15.94:0/1720315507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c40fdf10 con 0x7fd8c4105320 2024-04-21T08:37:05.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.678+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/1720315507 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd8b4046020 con 0x7fd8c4105320 2024-04-21T08:37:05.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.678+0000 7fd8bb7fe640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd8b00a3bb0 0x7fd8b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:05.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.679+0000 7fd8ca0be640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd8b00a3bb0 0x7fd8b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:05.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.679+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/1720315507 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd8b40f2980 con 0x7fd8c4105320 2024-04-21T08:37:05.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.679+0000 7fd8ca0be640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd8b00a3bb0 0x7fd8b00a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0005fd0 tx=0x7fd8c00093e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:05.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.682+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/1720315507 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8b404d020 con 0x7fd8c4105320 2024-04-21T08:37:05.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.903+0000 7fd8cc322640 1 -- 172.21.15.94:0/1720315507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8c402d5c0 con 0x7fd8c4105320 2024-04-21T08:37:05.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.903+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/1720315507 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7fd8b40bf800 con 0x7fd8c4105320 2024-04-21T08:37:05.903 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:05.903 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-21T08:37:05.904 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 5 2024-04-21T08:37:05.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.905+0000 7fd8cc322640 1 -- 172.21.15.94:0/1720315507 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd8b00a3bb0 msgr2=0x7fd8b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:05.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.905+0000 7fd8cc322640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd8b00a3bb0 0x7fd8b00a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0005fd0 tx=0x7fd8c00093e0 comp rx=0 tx=0).stop 2024-04-21T08:37:05.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.905+0000 7fd8cc322640 1 -- 172.21.15.94:0/1720315507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8c4105320 msgr2=0x7fd8c4198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:05.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.905+0000 7fd8cc322640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8c4105320 0x7fd8c4198760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4000c00 tx=0x7fd8b4004270 comp rx=0 tx=0).stop 2024-04-21T08:37:05.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.905+0000 7fd8cc322640 1 -- 172.21.15.94:0/1720315507 shutdown_connections 2024-04-21T08:37:05.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.905+0000 7fd8cc322640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd8b00a3bb0 0x7fd8b00a6070 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:05.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.906+0000 7fd8cc322640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c4107c50 0x7fd8c419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:05.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.906+0000 7fd8cc322640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8c4105320 0x7fd8c4198760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:05.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.906+0000 7fd8cc322640 1 --2- 172.21.15.94:0/1720315507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c41029f0 0x7fd8c4198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:05.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.906+0000 7fd8cc322640 1 -- 172.21.15.94:0/1720315507 >> 172.21.15.94:0/1720315507 conn(0x7fd8c40fc420 msgr2=0x7fd8c4103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:05.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.906+0000 7fd8cc322640 1 -- 172.21.15.94:0/1720315507 shutdown_connections 2024-04-21T08:37:05.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:05.906+0000 7fd8cc322640 1 -- 172.21.15.94:0/1720315507 wait complete. 2024-04-21T08:37:05.915 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:05 smithi094 ceph-mon[27988]: Deploying daemon mds.e on smithi111 2024-04-21T08:37:05.915 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:05 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.94:6836/4105566219,v1:172.21.15.94:6837/4105566219] up:boot 2024-04-21T08:37:05.915 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:05 smithi094 ceph-mon[27988]: fsmap 4 up:standby 2024-04-21T08:37:05.915 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:05 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:37:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:05 smithi111 ceph-mon[31493]: Deploying daemon mds.e on smithi111 2024-04-21T08:37:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:05 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.94:6836/4105566219,v1:172.21.15.94:6837/4105566219] up:boot 2024-04-21T08:37:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:05 smithi111 ceph-mon[31493]: fsmap 4 up:standby 2024-04-21T08:37:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:05 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:37:06.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:05 smithi173 ceph-mon[31428]: Deploying daemon mds.e on smithi111 2024-04-21T08:37:06.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:05 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.94:6836/4105566219,v1:172.21.15.94:6837/4105566219] up:boot 2024-04-21T08:37:06.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:05 smithi173 ceph-mon[31428]: fsmap 4 up:standby 2024-04-21T08:37:06.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:05 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:37:06.324 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:37:06 smithi111 systemd[1]: Starting Ceph mds.e for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:37:06.589 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:37:06 smithi111 podman[61357]: 2024-04-21 08:37:06.22806692 +0000 UTC m=+0.018080080 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:37:06.589 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:37:06 smithi111 podman[61357]: 2024-04-21 08:37:06.336955592 +0000 UTC m=+0.126968722 container create 53f3a3b8425cfa5fc05f3ede1c778bddfddac59f48e7066126f58e40e1f3ebc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-21T08:37:06.589 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:37:06 smithi111 podman[61357]: 2024-04-21 08:37:06.564419225 +0000 UTC m=+0.354432353 container init 53f3a3b8425cfa5fc05f3ede1c778bddfddac59f48e7066126f58e40e1f3ebc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T08:37:06.589 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:37:06 smithi111 podman[61357]: 2024-04-21 08:37:06.567009354 +0000 UTC m=+0.357022484 container start 53f3a3b8425cfa5fc05f3ede1c778bddfddac59f48e7066126f58e40e1f3ebc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-21T08:37:06.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:06 smithi111 ceph-mon[31493]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:06.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:06 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1720315507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:06.937 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:37:06 smithi111 ceph-mds[61376]: -- 172.21.15.111:0/4034807064 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a5bcc8fa40 con 0x55a5bcc63000 2024-04-21T08:37:06.937 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:37:06 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e[61372]: starting mds.e at 2024-04-21T08:37:06.938 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:37:06 smithi111 bash[61357]: 53f3a3b8425cfa5fc05f3ede1c778bddfddac59f48e7066126f58e40e1f3ebc4 2024-04-21T08:37:06.938 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:37:06 smithi111 systemd[1]: Started Ceph mds.e for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:37:07.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:06 smithi094 ceph-mon[27988]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:07.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:06 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1720315507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:07.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:06 smithi173 ceph-mon[31428]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:07.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:06 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1720315507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:07.917 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:07.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:07 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.111:6834/3967081988,v1:172.21.15.111:6835/3967081988] up:boot 2024-04-21T08:37:07.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:07 smithi111 ceph-mon[31493]: fsmap 5 up:standby 2024-04-21T08:37:07.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:37:07.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:07.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:07.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:07.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:07.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:07.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:07 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:07.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:07 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.111:6834/3967081988,v1:172.21.15.111:6835/3967081988] up:boot 2024-04-21T08:37:07.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:07 smithi094 ceph-mon[27988]: fsmap 5 up:standby 2024-04-21T08:37:07.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:37:07.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:07.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:07.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:07.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:07.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:07.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:07 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:08.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:07 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.111:6834/3967081988,v1:172.21.15.111:6835/3967081988] up:boot 2024-04-21T08:37:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:07 smithi173 ceph-mon[31428]: fsmap 5 up:standby 2024-04-21T08:37:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:37:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:07 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:08.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.032+0000 7fc9f2f15640 1 -- 172.21.15.94:0/889567764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9ec101fd0 msgr2=0x7fc9ec102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:08.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.032+0000 7fc9f2f15640 1 --2- 172.21.15.94:0/889567764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9ec101fd0 0x7fc9ec102450 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009900 tx=0x7fc9d802bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:08.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.032+0000 7fc9f2f15640 1 -- 172.21.15.94:0/889567764 shutdown_connections 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.032+0000 7fc9f2f15640 1 --2- 172.21.15.94:0/889567764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9ec102990 0x7fc9ec1092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.032+0000 7fc9f2f15640 1 --2- 172.21.15.94:0/889567764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9ec101fd0 0x7fc9ec102450 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.032+0000 7fc9f2f15640 1 --2- 172.21.15.94:0/889567764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ec100e80 0x7fc9ec1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.032+0000 7fc9f2f15640 1 -- 172.21.15.94:0/889567764 >> 172.21.15.94:0/889567764 conn(0x7fc9ec0fc410 msgr2=0x7fc9ec0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.032+0000 7fc9f2f15640 1 -- 172.21.15.94:0/889567764 shutdown_connections 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.032+0000 7fc9f2f15640 1 -- 172.21.15.94:0/889567764 wait complete. 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f2f15640 1 Processor -- start 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f2f15640 1 -- start start 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f2f15640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9ec100e80 0x7fc9ec06b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f2f15640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9ec101fd0 0x7fc9ec19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f2f15640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ec102990 0x7fc9ec19f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f2f15640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec10c9d0 con 0x7fc9ec100e80 2024-04-21T08:37:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f2f15640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec10c850 con 0x7fc9ec102990 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f2f15640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec10cb50 con 0x7fc9ec101fd0 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f0cb1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9ec100e80 0x7fc9ec06b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f0cb1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9ec100e80 0x7fc9ec06b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34978/0 (socket says 172.21.15.94:34978) 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f0cb1640 1 -- 172.21.15.94:0/2428991654 learned_addr learned my addr 172.21.15.94:0/2428991654 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f14b2640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ec102990 0x7fc9ec19f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9ebfff640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9ec101fd0 0x7fc9ec19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f0cb1640 1 -- 172.21.15.94:0/2428991654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9ec101fd0 msgr2=0x7fc9ec19cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f0cb1640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9ec101fd0 0x7fc9ec19cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f0cb1640 1 -- 172.21.15.94:0/2428991654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ec102990 msgr2=0x7fc9ec19f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f0cb1640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ec102990 0x7fc9ec19f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.033+0000 7fc9f0cb1640 1 -- 172.21.15.94:0/2428991654 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ec19fd10 con 0x7fc9ec100e80 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.034+0000 7fc9f0cb1640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9ec100e80 0x7fc9ec06b210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000c920 tx=0x7fc9e000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.034+0000 7fc9e9ffb640 1 -- 172.21.15.94:0/2428991654 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0007c70 con 0x7fc9ec100e80 2024-04-21T08:37:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.034+0000 7fc9e9ffb640 1 -- 172.21.15.94:0/2428991654 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e0007e10 con 0x7fc9ec100e80 2024-04-21T08:37:08.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.034+0000 7fc9f2f15640 1 -- 172.21.15.94:0/2428991654 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ec19ffa0 con 0x7fc9ec100e80 2024-04-21T08:37:08.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.034+0000 7fc9e9ffb640 1 -- 172.21.15.94:0/2428991654 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0018820 con 0x7fc9ec100e80 2024-04-21T08:37:08.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.034+0000 7fc9f2f15640 1 -- 172.21.15.94:0/2428991654 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ec1a0460 con 0x7fc9ec100e80 2024-04-21T08:37:08.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.035+0000 7fc9f2f15640 1 -- 172.21.15.94:0/2428991654 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9ec108190 con 0x7fc9ec100e80 2024-04-21T08:37:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.036+0000 7fc9e9ffb640 1 -- 172.21.15.94:0/2428991654 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc9e000f450 con 0x7fc9ec100e80 2024-04-21T08:37:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.036+0000 7fc9e9ffb640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc9dc0a3bb0 0x7fc9dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.037+0000 7fc9ebfff640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc9dc0a3bb0 0x7fc9dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.037+0000 7fc9e9ffb640 1 -- 172.21.15.94:0/2428991654 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc9e00d0080 con 0x7fc9ec100e80 2024-04-21T08:37:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.037+0000 7fc9ebfff640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc9dc0a3bb0 0x7fc9dc0a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8006010 tx=0x7fc9d8005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:08.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.041+0000 7fc9e9ffb640 1 -- 172.21.15.94:0/2428991654 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9e0014030 con 0x7fc9ec100e80 2024-04-21T08:37:08.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.258+0000 7fc9f2f15640 1 -- 172.21.15.94:0/2428991654 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9ec062b50 con 0x7fc9ec100e80 2024-04-21T08:37:08.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.259+0000 7fc9e9ffb640 1 -- 172.21.15.94:0/2428991654 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7fc9e009cf00 con 0x7fc9ec100e80 2024-04-21T08:37:08.259 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:08.259 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-21T08:37:08.259 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 6 2024-04-21T08:37:08.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 -- 172.21.15.94:0/2428991654 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc9dc0a3bb0 msgr2=0x7fc9dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:08.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc9dc0a3bb0 0x7fc9dc0a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8006010 tx=0x7fc9d8005be0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 -- 172.21.15.94:0/2428991654 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9ec100e80 msgr2=0x7fc9ec06b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:08.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9ec100e80 0x7fc9ec06b210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000c920 tx=0x7fc9e000cde0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 -- 172.21.15.94:0/2428991654 shutdown_connections 2024-04-21T08:37:08.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc9dc0a3bb0 0x7fc9dc0a6070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9ec101fd0 0x7fc9ec19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ec102990 0x7fc9ec19f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 --2- 172.21.15.94:0/2428991654 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9ec100e80 0x7fc9ec06b210 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:08.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 -- 172.21.15.94:0/2428991654 >> 172.21.15.94:0/2428991654 conn(0x7fc9ec0fc410 msgr2=0x7fc9ec105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:08.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 -- 172.21.15.94:0/2428991654 shutdown_connections 2024-04-21T08:37:08.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:08.261+0000 7fc9f2f15640 1 -- 172.21.15.94:0/2428991654 wait complete. 2024-04-21T08:37:08.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:08 smithi111 ceph-mon[31493]: Deploying daemon mds.f on smithi173 2024-04-21T08:37:08.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:08 smithi111 ceph-mon[31493]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:08.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:08 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2428991654' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:08 smithi094 ceph-mon[27988]: Deploying daemon mds.f on smithi173 2024-04-21T08:37:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:08 smithi094 ceph-mon[27988]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:08 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2428991654' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:09.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:08 smithi173 ceph-mon[31428]: Deploying daemon mds.f on smithi173 2024-04-21T08:37:09.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:08 smithi173 ceph-mon[31428]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:09.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:08 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2428991654' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:09.012 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:37:08 smithi173 systemd[1]: Starting Ceph mds.f for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:37:09.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:37:09 smithi173 podman[62248]: 2024-04-21 08:37:09.003357758 +0000 UTC m=+0.018576796 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:37:09.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:37:09 smithi173 podman[62248]: 2024-04-21 08:37:09.127388484 +0000 UTC m=+0.142607523 container create da02ff2d0f1c78a0e80c22c77fb04adca65d8c33896fd7d001802142cdb303c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-21T08:37:09.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:37:09 smithi173 podman[62248]: 2024-04-21 08:37:09.363388134 +0000 UTC m=+0.378607172 container init da02ff2d0f1c78a0e80c22c77fb04adca65d8c33896fd7d001802142cdb303c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-21T08:37:09.760 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:37:09 smithi173 podman[62248]: 2024-04-21 08:37:09.366027742 +0000 UTC m=+0.381246814 container start da02ff2d0f1c78a0e80c22c77fb04adca65d8c33896fd7d001802142cdb303c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-21T08:37:09.761 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:37:09 smithi173 ceph-mds[62267]: -- 172.21.15.173:0/3864772459 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d14b23da40 con 0x55d14b211000 2024-04-21T08:37:09.761 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:37:09 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f[62263]: starting mds.f at 2024-04-21T08:37:09.761 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:37:09 smithi173 bash[62248]: da02ff2d0f1c78a0e80c22c77fb04adca65d8c33896fd7d001802142cdb303c8 2024-04-21T08:37:09.761 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:37:09 smithi173 systemd[1]: Started Ceph mds.f for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:37:10.273 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:10.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.386+0000 7ff1ce048640 1 -- 172.21.15.94:0/4154513271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1c81032f0 msgr2=0x7ff1c8103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.386+0000 7ff1ce048640 1 --2- 172.21.15.94:0/4154513271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1c81032f0 0x7ff1c8103770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009930 tx=0x7ff1bc02bc20 comp rx=0 tx=0).stop 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.386+0000 7ff1ce048640 1 -- 172.21.15.94:0/4154513271 shutdown_connections 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.386+0000 7ff1ce048640 1 --2- 172.21.15.94:0/4154513271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c8103cb0 0x7ff1c810a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.386+0000 7ff1ce048640 1 --2- 172.21.15.94:0/4154513271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1c81032f0 0x7ff1c8103770 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.386+0000 7ff1ce048640 1 --2- 172.21.15.94:0/4154513271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1c8100280 0x7ff1c81006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.386+0000 7ff1ce048640 1 -- 172.21.15.94:0/4154513271 >> 172.21.15.94:0/4154513271 conn(0x7ff1c80fbb90 msgr2=0x7ff1c80fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.386+0000 7ff1ce048640 1 -- 172.21.15.94:0/4154513271 shutdown_connections 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.386+0000 7ff1ce048640 1 -- 172.21.15.94:0/4154513271 wait complete. 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1ce048640 1 Processor -- start 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1ce048640 1 -- start start 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1ce048640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1c8100280 0x7ff1c8198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:10.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1ce048640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c81032f0 0x7ff1c81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1ce048640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1c8103cb0 0x7ff1c819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1ce048640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c810daf0 con 0x7ff1c8100280 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1ce048640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c810d970 con 0x7ff1c81032f0 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1ce048640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c810dc70 con 0x7ff1c8103cb0 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1c77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1c8100280 0x7ff1c8198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1c77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1c8100280 0x7ff1c8198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34994/0 (socket says 172.21.15.94:34994) 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1c77fe640 1 -- 172.21.15.94:0/3500897693 learned_addr learned my addr 172.21.15.94:0/3500897693 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1c7fff640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1c8103cb0 0x7ff1c819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.387+0000 7ff1c6ffd640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c81032f0 0x7ff1c81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1c77fe640 1 -- 172.21.15.94:0/3500897693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1c8103cb0 msgr2=0x7ff1c819f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1c77fe640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1c8103cb0 0x7ff1c819f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:10.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1c77fe640 1 -- 172.21.15.94:0/3500897693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c81032f0 msgr2=0x7ff1c81986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:10.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1c77fe640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c81032f0 0x7ff1c81986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:10.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1c77fe640 1 -- 172.21.15.94:0/3500897693 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c819fdb0 con 0x7ff1c8100280 2024-04-21T08:37:10.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1c77fe640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1c8100280 0x7ff1c8198190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800e9c0 tx=0x7ff1b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:10.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1c4ff9640 1 -- 172.21.15.94:0/3500897693 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b800cd80 con 0x7ff1c8100280 2024-04-21T08:37:10.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1ce048640 1 -- 172.21.15.94:0/3500897693 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1c81a00a0 con 0x7ff1c8100280 2024-04-21T08:37:10.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1c4ff9640 1 -- 172.21.15.94:0/3500897693 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1b80035f0 con 0x7ff1c8100280 2024-04-21T08:37:10.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1c4ff9640 1 -- 172.21.15.94:0/3500897693 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b8010690 con 0x7ff1c8100280 2024-04-21T08:37:10.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.388+0000 7ff1ce048640 1 -- 172.21.15.94:0/3500897693 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1c81a05c0 con 0x7ff1c8100280 2024-04-21T08:37:10.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.389+0000 7ff1ce048640 1 -- 172.21.15.94:0/3500897693 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1c8109440 con 0x7ff1c8100280 2024-04-21T08:37:10.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.390+0000 7ff1c4ff9640 1 -- 172.21.15.94:0/3500897693 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff1b80037a0 con 0x7ff1c8100280 2024-04-21T08:37:10.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.391+0000 7ff1c4ff9640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff1b00a3bb0 0x7ff1b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:10.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.391+0000 7ff1c6ffd640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff1b00a3bb0 0x7ff1b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:10.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.391+0000 7ff1c4ff9640 1 -- 172.21.15.94:0/3500897693 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff1b80d02c0 con 0x7ff1c8100280 2024-04-21T08:37:10.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.391+0000 7ff1c6ffd640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff1b00a3bb0 0x7ff1b00a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009900 tx=0x7ff1bc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:10.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.395+0000 7ff1c4ff9640 1 -- 172.21.15.94:0/3500897693 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1b809d140 con 0x7ff1c8100280 2024-04-21T08:37:10.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.617+0000 7ff1ce048640 1 -- 172.21.15.94:0/3500897693 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1c802d5c0 con 0x7ff1c8100280 2024-04-21T08:37:10.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.617+0000 7ff1c4ff9640 1 -- 172.21.15.94:0/3500897693 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7ff1b809ca40 con 0x7ff1c8100280 2024-04-21T08:37:10.617 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:10.617 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T08:37:10.618 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 7 2024-04-21T08:37:10.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 -- 172.21.15.94:0/3500897693 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff1b00a3bb0 msgr2=0x7ff1b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:10.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff1b00a3bb0 0x7ff1b00a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009900 tx=0x7ff1bc005fd0 comp rx=0 tx=0).stop 2024-04-21T08:37:10.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 -- 172.21.15.94:0/3500897693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1c8100280 msgr2=0x7ff1c8198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:10.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1c8100280 0x7ff1c8198190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800e9c0 tx=0x7ff1b800ee80 comp rx=0 tx=0).stop 2024-04-21T08:37:10.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 -- 172.21.15.94:0/3500897693 shutdown_connections 2024-04-21T08:37:10.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff1b00a3bb0 0x7ff1b00a6070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:10.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1c8103cb0 0x7ff1c819f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:10.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c81032f0 0x7ff1c81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:10.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 --2- 172.21.15.94:0/3500897693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1c8100280 0x7ff1c8198190 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:10.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 -- 172.21.15.94:0/3500897693 >> 172.21.15.94:0/3500897693 conn(0x7ff1c80fbb90 msgr2=0x7ff1c8106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:10.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 -- 172.21.15.94:0/3500897693 shutdown_connections 2024-04-21T08:37:10.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:10.619+0000 7ff1ce048640 1 -- 172.21.15.94:0/3500897693 wait complete. 2024-04-21T08:37:10.629 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:10.629 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:10.630 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:10.630 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:10.630 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:10.630 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:10.630 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: Deploying daemon mds.c on smithi173 2024-04-21T08:37:10.630 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:10.630 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.173:6834/3922099450,v1:172.21.15.173:6835/3922099450] up:boot 2024-04-21T08:37:10.630 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: fsmap 6 up:standby 2024-04-21T08:37:10.630 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:10 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:37:10.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:10.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:10.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:10.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:10.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:10.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:10.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: Deploying daemon mds.c on smithi173 2024-04-21T08:37:10.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:10.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.173:6834/3922099450,v1:172.21.15.173:6835/3922099450] up:boot 2024-04-21T08:37:10.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: fsmap 6 up:standby 2024-04-21T08:37:10.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:10 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:37:10.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:10.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:10.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:10.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:10.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:10.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:10.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: Deploying daemon mds.c on smithi173 2024-04-21T08:37:10.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:10.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.173:6834/3922099450,v1:172.21.15.173:6835/3922099450] up:boot 2024-04-21T08:37:10.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: fsmap 6 up:standby 2024-04-21T08:37:10.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:10 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:37:11.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:11 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3500897693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:11.761 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:37:11 smithi173 systemd[1]: Starting Ceph mds.c for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:37:11.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:11 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3500897693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:12.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:11 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3500897693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:12.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:37:11 smithi173 podman[62556]: 2024-04-21 08:37:11.750699448 +0000 UTC m=+0.018083070 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:37:12.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:37:11 smithi173 podman[62556]: 2024-04-21 08:37:11.866030836 +0000 UTC m=+0.133414459 container create a35cf0a58fae7f0822f92a165df06a9ffd3ae50dc3d39a43261b65a435971473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-21T08:37:12.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:37:12 smithi173 podman[62556]: 2024-04-21 08:37:12.118875907 +0000 UTC m=+0.386259529 container init a35cf0a58fae7f0822f92a165df06a9ffd3ae50dc3d39a43261b65a435971473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-21T08:37:12.510 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:37:12 smithi173 podman[62556]: 2024-04-21 08:37:12.121649432 +0000 UTC m=+0.389033062 container start a35cf0a58fae7f0822f92a165df06a9ffd3ae50dc3d39a43261b65a435971473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-21T08:37:12.511 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:37:12 smithi173 ceph-mds[62575]: -- 172.21.15.173:0/43204901 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e9c7c6fa40 con 0x55e9c7c42800 2024-04-21T08:37:12.511 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:37:12 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c[62571]: starting mds.c at 2024-04-21T08:37:12.511 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:37:12 smithi173 bash[62556]: a35cf0a58fae7f0822f92a165df06a9ffd3ae50dc3d39a43261b65a435971473 2024-04-21T08:37:12.511 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:37:12 smithi173 systemd[1]: Started Ceph mds.c for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:37:12.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:12.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.746+0000 7f5b10aba640 1 -- 172.21.15.94:0/3946469103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b0c100860 msgr2=0x7f5b0c100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:12.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.746+0000 7f5b10aba640 1 --2- 172.21.15.94:0/3946469103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b0c100860 0x7f5b0c100c80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009900 tx=0x7f5af402bcd0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.746+0000 7f5b10aba640 1 -- 172.21.15.94:0/3946469103 shutdown_connections 2024-04-21T08:37:12.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.746+0000 7f5b10aba640 1 --2- 172.21.15.94:0/3946469103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b0c1011c0 0x7f5b0c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.746+0000 7f5b10aba640 1 --2- 172.21.15.94:0/3946469103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b0c101b60 0x7f5b0c101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.746+0000 7f5b10aba640 1 --2- 172.21.15.94:0/3946469103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b0c100860 0x7f5b0c100c80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.746+0000 7f5b10aba640 1 -- 172.21.15.94:0/3946469103 >> 172.21.15.94:0/3946469103 conn(0x7f5b0c0fc3e0 msgr2=0x7f5b0c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.746+0000 7f5b10aba640 1 -- 172.21.15.94:0/3946469103 shutdown_connections 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.746+0000 7f5b10aba640 1 -- 172.21.15.94:0/3946469103 wait complete. 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b10aba640 1 Processor -- start 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b10aba640 1 -- start start 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b10aba640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b0c100860 0x7f5b0c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b10aba640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b0c1011c0 0x7f5b0c19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b10aba640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b0c101b60 0x7f5b0c1a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b10aba640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0c10fc80 con 0x7f5b0c101b60 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b10aba640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0c10fb00 con 0x7f5b0c1011c0 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b10aba640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0c10fe00 con 0x7f5b0c100860 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b0ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b0c101b60 0x7f5b0c1a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b0ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b0c101b60 0x7f5b0c1a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35016/0 (socket says 172.21.15.94:35016) 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b0ad9d640 1 -- 172.21.15.94:0/1751963370 learned_addr learned my addr 172.21.15.94:0/1751963370 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b09d9b640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b0c1011c0 0x7f5b0c19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:12.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b0a59c640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b0c100860 0x7f5b0c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b0ad9d640 1 -- 172.21.15.94:0/1751963370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b0c100860 msgr2=0x7f5b0c19a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b0ad9d640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b0c100860 0x7f5b0c19a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b0ad9d640 1 -- 172.21.15.94:0/1751963370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b0c1011c0 msgr2=0x7f5b0c19a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b0ad9d640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b0c1011c0 0x7f5b0c19a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b0ad9d640 1 -- 172.21.15.94:0/1751963370 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0c1a1f40 con 0x7f5b0c101b60 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.747+0000 7f5b0ad9d640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b0c101b60 0x7f5b0c1a18d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b04009fd0 tx=0x7f5b0400def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.748+0000 7f5b037fe640 1 -- 172.21.15.94:0/1751963370 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04017070 con 0x7f5b0c101b60 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.748+0000 7f5b10aba640 1 -- 172.21.15.94:0/1751963370 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0c1a2230 con 0x7f5b0c101b60 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.748+0000 7f5b037fe640 1 -- 172.21.15.94:0/1751963370 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b04003e20 con 0x7f5b0c101b60 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.748+0000 7f5b037fe640 1 -- 172.21.15.94:0/1751963370 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04010460 con 0x7f5b0c101b60 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.748+0000 7f5b10aba640 1 -- 172.21.15.94:0/1751963370 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b0c1a2730 con 0x7f5b0c101b60 2024-04-21T08:37:12.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.749+0000 7f5b10aba640 1 -- 172.21.15.94:0/1751963370 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b0c10b7a0 con 0x7f5b0c101b60 2024-04-21T08:37:12.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.750+0000 7f5b037fe640 1 -- 172.21.15.94:0/1751963370 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5b04004020 con 0x7f5b0c101b60 2024-04-21T08:37:12.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.750+0000 7f5b037fe640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5af80a3bb0 0x7f5af80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:12.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.750+0000 7f5b0a59c640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5af80a3bb0 0x7f5af80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:12.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.751+0000 7f5b037fe640 1 -- 172.21.15.94:0/1751963370 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5b040cfc10 con 0x7f5b0c101b60 2024-04-21T08:37:12.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.751+0000 7f5b0a59c640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5af80a3bb0 0x7f5af80a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009230 tx=0x7f5af4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:12.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.755+0000 7f5b037fe640 1 -- 172.21.15.94:0/1751963370 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b0409ca90 con 0x7f5b0c101b60 2024-04-21T08:37:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:12 smithi111 ceph-mon[31493]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:12 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:12.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.975+0000 7f5b10aba640 1 -- 172.21.15.94:0/1751963370 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b0c02d5c0 con 0x7f5b0c101b60 2024-04-21T08:37:12.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.975+0000 7f5b037fe640 1 -- 172.21.15.94:0/1751963370 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7f5b0409c390 con 0x7f5b0c101b60 2024-04-21T08:37:12.975 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:12.975 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T08:37:12.975 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 8 2024-04-21T08:37:12.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 -- 172.21.15.94:0/1751963370 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5af80a3bb0 msgr2=0x7f5af80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:12.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5af80a3bb0 0x7f5af80a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009230 tx=0x7f5af4005fd0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 -- 172.21.15.94:0/1751963370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b0c101b60 msgr2=0x7f5b0c1a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:12.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b0c101b60 0x7f5b0c1a18d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b04009fd0 tx=0x7f5b0400def0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 -- 172.21.15.94:0/1751963370 shutdown_connections 2024-04-21T08:37:12.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5af80a3bb0 0x7f5af80a6070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b0c101b60 0x7f5b0c1a18d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b0c1011c0 0x7f5b0c19a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 --2- 172.21.15.94:0/1751963370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b0c100860 0x7f5b0c19a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:12.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 -- 172.21.15.94:0/1751963370 >> 172.21.15.94:0/1751963370 conn(0x7f5b0c0fc3e0 msgr2=0x7f5b0c0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:12.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 -- 172.21.15.94:0/1751963370 shutdown_connections 2024-04-21T08:37:12.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:12.977+0000 7f5b10aba640 1 -- 172.21.15.94:0/1751963370 wait complete. 2024-04-21T08:37:12.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:12 smithi094 ceph-mon[27988]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:12.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:12.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:12.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:12.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:12.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:12.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:12 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:13.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:12 smithi173 ceph-mon[31428]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:13.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:13.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:13.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:12 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:13.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:13 smithi111 ceph-mon[31493]: Deploying daemon mds.h on smithi111 2024-04-21T08:37:13.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:13 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.173:6836/1466520447,v1:172.21.15.173:6837/1466520447] up:boot 2024-04-21T08:37:13.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:13 smithi111 ceph-mon[31493]: fsmap 7 up:standby 2024-04-21T08:37:13.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:13 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:37:13.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:13 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1751963370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:13 smithi094 ceph-mon[27988]: Deploying daemon mds.h on smithi111 2024-04-21T08:37:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:13 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.173:6836/1466520447,v1:172.21.15.173:6837/1466520447] up:boot 2024-04-21T08:37:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:13 smithi094 ceph-mon[27988]: fsmap 7 up:standby 2024-04-21T08:37:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:13 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:37:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:13 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1751963370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:14.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:13 smithi173 ceph-mon[31428]: Deploying daemon mds.h on smithi111 2024-04-21T08:37:14.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:13 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.173:6836/1466520447,v1:172.21.15.173:6837/1466520447] up:boot 2024-04-21T08:37:14.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:13 smithi173 ceph-mon[31428]: fsmap 7 up:standby 2024-04-21T08:37:14.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:13 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:37:14.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:13 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1751963370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:14.550 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:37:14 smithi111 systemd[1]: Starting Ceph mds.h for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:37:14.816 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:14 smithi111 ceph-mon[31493]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:14.816 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:37:14 smithi111 podman[61669]: 2024-04-21 08:37:14.570097792 +0000 UTC m=+0.018582152 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:37:14.817 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:37:14 smithi111 podman[61669]: 2024-04-21 08:37:14.670693691 +0000 UTC m=+0.119178073 container create e79d7014fd23efce0643c28bd344793f6aa97bffcd9c6cc46ae75a509deb6605 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-04-21T08:37:14.912 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:14 smithi173 ceph-mon[31428]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:14.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:15.009 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:14 smithi094 ceph-mon[27988]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:15.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.101+0000 7fd3f6706640 1 -- 172.21.15.94:0/3308475947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f01032a0 msgr2=0x7fd3f0103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:15.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.101+0000 7fd3f6706640 1 --2- 172.21.15.94:0/3308475947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f01032a0 0x7fd3f0103720 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009930 tx=0x7fd3e402bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:15.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.101+0000 7fd3f6706640 1 -- 172.21.15.94:0/3308475947 shutdown_connections 2024-04-21T08:37:15.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.101+0000 7fd3f6706640 1 --2- 172.21.15.94:0/3308475947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f0103c60 0x7fd3f010a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:15.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.101+0000 7fd3f6706640 1 --2- 172.21.15.94:0/3308475947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f01032a0 0x7fd3f0103720 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:15.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.101+0000 7fd3f6706640 1 --2- 172.21.15.94:0/3308475947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3f01002d0 0x7fd3f01006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:15.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.101+0000 7fd3f6706640 1 -- 172.21.15.94:0/3308475947 >> 172.21.15.94:0/3308475947 conn(0x7fd3f00fbc00 msgr2=0x7fd3f00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.101+0000 7fd3f6706640 1 -- 172.21.15.94:0/3308475947 shutdown_connections 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.101+0000 7fd3f6706640 1 -- 172.21.15.94:0/3308475947 wait complete. 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f6706640 1 Processor -- start 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f6706640 1 -- start start 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f6706640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3f01002d0 0x7fd3f0198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f6706640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01032a0 0x7fd3f01986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f6706640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0103c60 0x7fd3f019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f6706640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f010db00 con 0x7fd3f0103c60 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f6706640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f010d980 con 0x7fd3f01002d0 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f6706640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f010dc80 con 0x7fd3f01032a0 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f4ca3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0103c60 0x7fd3f019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f4ca3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0103c60 0x7fd3f019f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34122/0 (socket says 172.21.15.94:34122) 2024-04-21T08:37:15.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f4ca3640 1 -- 172.21.15.94:0/850923295 learned_addr learned my addr 172.21.15.94:0/850923295 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3ef7fe640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01032a0 0x7fd3f01986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3effff640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3f01002d0 0x7fd3f0198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f4ca3640 1 -- 172.21.15.94:0/850923295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01032a0 msgr2=0x7fd3f01986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f4ca3640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01032a0 0x7fd3f01986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f4ca3640 1 -- 172.21.15.94:0/850923295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3f01002d0 msgr2=0x7fd3f0198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f4ca3640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3f01002d0 0x7fd3f0198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.102+0000 7fd3f4ca3640 1 -- 172.21.15.94:0/850923295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f019fdb0 con 0x7fd3f0103c60 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.103+0000 7fd3f4ca3640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0103c60 0x7fd3f019f740 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8009fd0 tx=0x7fd3e800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.103+0000 7fd3ed7fa640 1 -- 172.21.15.94:0/850923295 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8017070 con 0x7fd3f0103c60 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.103+0000 7fd3f6706640 1 -- 172.21.15.94:0/850923295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3f01a00a0 con 0x7fd3f0103c60 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.103+0000 7fd3ed7fa640 1 -- 172.21.15.94:0/850923295 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3e8003e20 con 0x7fd3f0103c60 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.103+0000 7fd3ed7fa640 1 -- 172.21.15.94:0/850923295 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8010460 con 0x7fd3f0103c60 2024-04-21T08:37:15.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.103+0000 7fd3f6706640 1 -- 172.21.15.94:0/850923295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3f01a05c0 con 0x7fd3f0103c60 2024-04-21T08:37:15.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.104+0000 7fd3f6706640 1 -- 172.21.15.94:0/850923295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3f01094a0 con 0x7fd3f0103c60 2024-04-21T08:37:15.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.105+0000 7fd3ed7fa640 1 -- 172.21.15.94:0/850923295 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd3e8003970 con 0x7fd3f0103c60 2024-04-21T08:37:15.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.105+0000 7fd3ed7fa640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd3d80a3bb0 0x7fd3d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:15.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.105+0000 7fd3effff640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd3d80a3bb0 0x7fd3d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:15.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.105+0000 7fd3ed7fa640 1 -- 172.21.15.94:0/850923295 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd3e80cfc40 con 0x7fd3f0103c60 2024-04-21T08:37:15.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.106+0000 7fd3effff640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd3d80a3bb0 0x7fd3d80a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd3f006a4a0 tx=0x7fd3e000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:15.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.110+0000 7fd3ed7fa640 1 -- 172.21.15.94:0/850923295 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd3e809c470 con 0x7fd3f0103c60 2024-04-21T08:37:15.187 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:37:14 smithi111 podman[61669]: 2024-04-21 08:37:14.931787179 +0000 UTC m=+0.380271515 container init e79d7014fd23efce0643c28bd344793f6aa97bffcd9c6cc46ae75a509deb6605 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-21T08:37:15.187 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:37:14 smithi111 podman[61669]: 2024-04-21 08:37:14.934284333 +0000 UTC m=+0.382768672 container start e79d7014fd23efce0643c28bd344793f6aa97bffcd9c6cc46ae75a509deb6605 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-21T08:37:15.187 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:37:14 smithi111 ceph-mds[61688]: -- 172.21.15.111:0/1069719076 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5578bfcc1a40 con 0x5578bfc94c00 2024-04-21T08:37:15.188 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:37:14 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h[61684]: starting mds.h at 2024-04-21T08:37:15.188 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:37:15 smithi111 bash[61669]: e79d7014fd23efce0643c28bd344793f6aa97bffcd9c6cc46ae75a509deb6605 2024-04-21T08:37:15.188 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:37:15 smithi111 systemd[1]: Started Ceph mds.h for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:37:15.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.343+0000 7fd3f6706640 1 -- 172.21.15.94:0/850923295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3f002d790 con 0x7fd3f0103c60 2024-04-21T08:37:15.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.343+0000 7fd3ed7fa640 1 -- 172.21.15.94:0/850923295 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7fd3e80a0770 con 0x7fd3f0103c60 2024-04-21T08:37:15.343 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:15.343 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T08:37:15.343 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 8 2024-04-21T08:37:15.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 -- 172.21.15.94:0/850923295 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd3d80a3bb0 msgr2=0x7fd3d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:15.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd3d80a3bb0 0x7fd3d80a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd3f006a4a0 tx=0x7fd3e000b390 comp rx=0 tx=0).stop 2024-04-21T08:37:15.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 -- 172.21.15.94:0/850923295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0103c60 msgr2=0x7fd3f019f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:15.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0103c60 0x7fd3f019f740 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8009fd0 tx=0x7fd3e800b260 comp rx=0 tx=0).stop 2024-04-21T08:37:15.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 -- 172.21.15.94:0/850923295 shutdown_connections 2024-04-21T08:37:15.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd3d80a3bb0 0x7fd3d80a6070 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:15.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f01032a0 0x7fd3f01986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:15.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0103c60 0x7fd3f019f740 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:15.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 --2- 172.21.15.94:0/850923295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3f01002d0 0x7fd3f0198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:15.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 -- 172.21.15.94:0/850923295 >> 172.21.15.94:0/850923295 conn(0x7fd3f00fbc00 msgr2=0x7fd3f0106670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:15.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.345+0000 7fd3f6706640 1 -- 172.21.15.94:0/850923295 shutdown_connections 2024-04-21T08:37:15.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:15.346+0000 7fd3f6706640 1 -- 172.21.15.94:0/850923295 wait complete. 2024-04-21T08:37:16.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:16.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:16.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:16.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:16.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:16.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:16.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: Deploying daemon mds.i on smithi173 2024-04-21T08:37:16.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/850923295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:16.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] up:boot 2024-04-21T08:37:16.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: fsmap 8 up:standby 2024-04-21T08:37:16.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:37:16.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:16 smithi111 ceph-mon[31493]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:16.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:16.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:16.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:16.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:16.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:16.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:16.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: Deploying daemon mds.i on smithi173 2024-04-21T08:37:16.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/850923295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:16.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] up:boot 2024-04-21T08:37:16.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: fsmap 8 up:standby 2024-04-21T08:37:16.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:37:16.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:16 smithi094 ceph-mon[27988]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:16.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:16.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:16.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:16.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:16.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:16.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:16.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: Deploying daemon mds.i on smithi173 2024-04-21T08:37:16.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/850923295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:16.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] up:boot 2024-04-21T08:37:16.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: fsmap 8 up:standby 2024-04-21T08:37:16.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:37:16.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:16 smithi173 ceph-mon[31428]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:17.357 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.467+0000 7ffb65775640 1 -- 172.21.15.94:0/197007202 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb60103bf0 msgr2=0x7ffb6010a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.467+0000 7ffb65775640 1 --2- 172.21.15.94:0/197007202 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb60103bf0 0x7ffb6010a480 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffb58009cd0 tx=0x7ffb5802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.467+0000 7ffb65775640 1 -- 172.21.15.94:0/197007202 shutdown_connections 2024-04-21T08:37:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.467+0000 7ffb65775640 1 --2- 172.21.15.94:0/197007202 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb60103bf0 0x7ffb6010a480 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.467+0000 7ffb65775640 1 --2- 172.21.15.94:0/197007202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb60103230 0x7ffb601036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.467+0000 7ffb65775640 1 --2- 172.21.15.94:0/197007202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb60100260 0x7ffb60100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.467+0000 7ffb65775640 1 -- 172.21.15.94:0/197007202 >> 172.21.15.94:0/197007202 conn(0x7ffb600fbb90 msgr2=0x7ffb600fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.467+0000 7ffb65775640 1 -- 172.21.15.94:0/197007202 shutdown_connections 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.467+0000 7ffb65775640 1 -- 172.21.15.94:0/197007202 wait complete. 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb65775640 1 Processor -- start 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb65775640 1 -- start start 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb65775640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb60100260 0x7ffb60198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb65775640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb60103230 0x7ffb60198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb65775640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb60103bf0 0x7ffb6019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb65775640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6010db00 con 0x7ffb60103bf0 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb65775640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6010d980 con 0x7ffb60103230 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb65775640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6010dc80 con 0x7ffb60100260 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb60103bf0 0x7ffb6019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb60103bf0 0x7ffb6019f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34136/0 (socket says 172.21.15.94:34136) 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5f7fe640 1 -- 172.21.15.94:0/2395497633 learned_addr learned my addr 172.21.15.94:0/2395497633 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5e7fc640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb60103230 0x7ffb60198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5effd640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb60100260 0x7ffb60198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5f7fe640 1 -- 172.21.15.94:0/2395497633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb60100260 msgr2=0x7ffb60198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5f7fe640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb60100260 0x7ffb60198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5f7fe640 1 -- 172.21.15.94:0/2395497633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb60103230 msgr2=0x7ffb60198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5f7fe640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb60103230 0x7ffb60198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5f7fe640 1 -- 172.21.15.94:0/2395497633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6019fd50 con 0x7ffb60103bf0 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.468+0000 7ffb5e7fc640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb60103230 0x7ffb60198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.469+0000 7ffb5f7fe640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb60103bf0 0x7ffb6019f6e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffb58005e30 tx=0x7ffb58034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.469+0000 7ffb47fff640 1 -- 172.21.15.94:0/2395497633 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58041020 con 0x7ffb60103bf0 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.469+0000 7ffb47fff640 1 -- 172.21.15.94:0/2395497633 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb58032840 con 0x7ffb60103bf0 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.469+0000 7ffb65775640 1 -- 172.21.15.94:0/2395497633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6019ffe0 con 0x7ffb60103bf0 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.469+0000 7ffb47fff640 1 -- 172.21.15.94:0/2395497633 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58039040 con 0x7ffb60103bf0 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.469+0000 7ffb65775640 1 -- 172.21.15.94:0/2395497633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb601a04a0 con 0x7ffb60103bf0 2024-04-21T08:37:17.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.470+0000 7ffb65775640 1 -- 172.21.15.94:0/2395497633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb60109360 con 0x7ffb60103bf0 2024-04-21T08:37:17.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.471+0000 7ffb47fff640 1 -- 172.21.15.94:0/2395497633 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffb58044020 con 0x7ffb60103bf0 2024-04-21T08:37:17.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.471+0000 7ffb47fff640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffb480a3c30 0x7ffb480a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:17.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.472+0000 7ffb5effd640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffb480a3c30 0x7ffb480a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:17.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.472+0000 7ffb47fff640 1 -- 172.21.15.94:0/2395497633 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ffb580f1fe0 con 0x7ffb60103bf0 2024-04-21T08:37:17.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.472+0000 7ffb5effd640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffb480a3c30 0x7ffb480a60f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffb50005fd0 tx=0x7ffb50005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:17.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.476+0000 7ffb47fff640 1 -- 172.21.15.94:0/2395497633 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb5804b020 con 0x7ffb60103bf0 2024-04-21T08:37:17.483 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:37:17 smithi173 systemd[1]: Starting Ceph mds.i for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:37:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.697+0000 7ffb65775640 1 -- 172.21.15.94:0/2395497633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb6002d810 con 0x7ffb60103bf0 2024-04-21T08:37:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.697+0000 7ffb47fff640 1 -- 172.21.15.94:0/2395497633 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7ffb580bee60 con 0x7ffb60103bf0 2024-04-21T08:37:17.697 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:17.697 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-21T08:37:17.698 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 9 2024-04-21T08:37:17.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.699+0000 7ffb65775640 1 -- 172.21.15.94:0/2395497633 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffb480a3c30 msgr2=0x7ffb480a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:17.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.699+0000 7ffb65775640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffb480a3c30 0x7ffb480a60f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffb50005fd0 tx=0x7ffb50005850 comp rx=0 tx=0).stop 2024-04-21T08:37:17.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.699+0000 7ffb65775640 1 -- 172.21.15.94:0/2395497633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb60103bf0 msgr2=0x7ffb6019f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:17.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.699+0000 7ffb65775640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb60103bf0 0x7ffb6019f6e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffb58005e30 tx=0x7ffb58034760 comp rx=0 tx=0).stop 2024-04-21T08:37:17.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.699+0000 7ffb65775640 1 -- 172.21.15.94:0/2395497633 shutdown_connections 2024-04-21T08:37:17.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.699+0000 7ffb65775640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffb480a3c30 0x7ffb480a60f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.699+0000 7ffb65775640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb60103bf0 0x7ffb6019f6e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.699+0000 7ffb65775640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb60103230 0x7ffb60198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.699+0000 7ffb65775640 1 --2- 172.21.15.94:0/2395497633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb60100260 0x7ffb60198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.699+0000 7ffb65775640 1 -- 172.21.15.94:0/2395497633 >> 172.21.15.94:0/2395497633 conn(0x7ffb600fbb90 msgr2=0x7ffb60106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:17.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.700+0000 7ffb65775640 1 -- 172.21.15.94:0/2395497633 shutdown_connections 2024-04-21T08:37:17.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:17.700+0000 7ffb65775640 1 -- 172.21.15.94:0/2395497633 wait complete. 2024-04-21T08:37:17.733 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:37:17 smithi173 podman[62876]: 2024-04-21 08:37:17.387504625 +0000 UTC m=+0.017796369 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:37:17.733 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:37:17 smithi173 podman[62876]: 2024-04-21 08:37:17.487441669 +0000 UTC m=+0.117733390 container create 63bb39d51f32adcf0f761bdbded302d32215fd8d9a4768aa917939bfbef7c096 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-21T08:37:17.733 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:37:17 smithi173 podman[62876]: 2024-04-21 08:37:17.732380261 +0000 UTC m=+0.362671987 container init 63bb39d51f32adcf0f761bdbded302d32215fd8d9a4768aa917939bfbef7c096 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:37:18.010 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:37:17 smithi173 podman[62876]: 2024-04-21 08:37:17.734928644 +0000 UTC m=+0.365220371 container start 63bb39d51f32adcf0f761bdbded302d32215fd8d9a4768aa917939bfbef7c096 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:37:18.011 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:37:17 smithi173 ceph-mds[62895]: -- 172.21.15.173:0/555892514 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559a3b149a40 con 0x559a3b11c800 2024-04-21T08:37:18.011 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:37:17 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i[62891]: starting mds.i at 2024-04-21T08:37:18.011 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:37:17 smithi173 bash[62876]: 63bb39d51f32adcf0f761bdbded302d32215fd8d9a4768aa917939bfbef7c096 2024-04-21T08:37:18.011 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:37:17 smithi173 systemd[1]: Started Ceph mds.i for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:37:18.803 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:18 smithi094 ceph-mon[27988]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:18.803 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:18 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2395497633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:18.804 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:18 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:18.804 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:18 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:18.804 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:18 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:18.804 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:18 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:18.804 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:18 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:18.804 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:18 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:18 smithi111 ceph-mon[31493]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:18 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2395497633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:18 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:18 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:18 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:18 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:18 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:18.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:18 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:19.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:18 smithi173 ceph-mon[31428]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:18 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2395497633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:18 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:18 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:18 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:18 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:18 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:18 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:19.712 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:19.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 -- 172.21.15.94:0/550398523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48cc10b8c0 msgr2=0x7f48cc10dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:19.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 --2- 172.21.15.94:0/550398523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48cc10b8c0 0x7f48cc10dd50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f48c0009900 tx=0x7f48c002bc50 comp rx=0 tx=0).stop 2024-04-21T08:37:19.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 -- 172.21.15.94:0/550398523 shutdown_connections 2024-04-21T08:37:19.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 --2- 172.21.15.94:0/550398523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48cc10b8c0 0x7f48cc10dd50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:19.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 --2- 172.21.15.94:0/550398523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48cc10e290 0x7f48cc10e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:19.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 --2- 172.21.15.94:0/550398523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48cc108f50 0x7f48cc10b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:19.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 -- 172.21.15.94:0/550398523 >> 172.21.15.94:0/550398523 conn(0x7f48cc06ede0 msgr2=0x7f48cc071220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:19.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 -- 172.21.15.94:0/550398523 shutdown_connections 2024-04-21T08:37:19.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 -- 172.21.15.94:0/550398523 wait complete. 2024-04-21T08:37:19.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 Processor -- start 2024-04-21T08:37:19.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.829+0000 7f48d27b2640 1 -- start start 2024-04-21T08:37:19.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d27b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48cc108f50 0x7f48cc1a6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:19.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d27b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48cc10b8c0 0x7f48cc1a7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:19.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d27b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48cc10e290 0x7f48cc1ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:19.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d27b2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc11c900 con 0x7f48cc10e290 2024-04-21T08:37:19.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d27b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc11c780 con 0x7f48cc108f50 2024-04-21T08:37:19.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d27b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc11ca80 con 0x7f48cc10b8c0 2024-04-21T08:37:19.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d0d4f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48cc10e290 0x7f48cc1ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:19.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d0d4f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48cc10e290 0x7f48cc1ae4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34164/0 (socket says 172.21.15.94:34164) 2024-04-21T08:37:19.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d0d4f640 1 -- 172.21.15.94:0/1612829340 learned_addr learned my addr 172.21.15.94:0/1612829340 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48cb7fe640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48cc10b8c0 0x7f48cc1a7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48cbfff640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48cc108f50 0x7f48cc1a6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d0d4f640 1 -- 172.21.15.94:0/1612829340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48cc10b8c0 msgr2=0x7f48cc1a7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d0d4f640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48cc10b8c0 0x7f48cc1a7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d0d4f640 1 -- 172.21.15.94:0/1612829340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48cc108f50 msgr2=0x7f48cc1a6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d0d4f640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48cc108f50 0x7f48cc1a6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d0d4f640 1 -- 172.21.15.94:0/1612829340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48cc1aeb10 con 0x7f48cc10e290 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.830+0000 7f48d0d4f640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48cc10e290 0x7f48cc1ae4a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f48c400eb10 tx=0x7f48c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.831+0000 7f48c97fa640 1 -- 172.21.15.94:0/1612829340 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c4004b20 con 0x7f48cc10e290 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.831+0000 7f48d27b2640 1 -- 172.21.15.94:0/1612829340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48cc1aee00 con 0x7f48cc10e290 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.831+0000 7f48d27b2640 1 -- 172.21.15.94:0/1612829340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48cc1af2c0 con 0x7f48cc10e290 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.831+0000 7f48c97fa640 1 -- 172.21.15.94:0/1612829340 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48c4004cc0 con 0x7f48cc10e290 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.831+0000 7f48c97fa640 1 -- 172.21.15.94:0/1612829340 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c4010670 con 0x7f48cc10e290 2024-04-21T08:37:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.832+0000 7f48d27b2640 1 -- 172.21.15.94:0/1612829340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48cc0718e0 con 0x7f48cc10e290 2024-04-21T08:37:19.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.833+0000 7f48c97fa640 1 -- 172.21.15.94:0/1612829340 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f48c4005020 con 0x7f48cc10e290 2024-04-21T08:37:19.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.833+0000 7f48c97fa640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f48b40a3c30 0x7f48b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:19.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.833+0000 7f48cbfff640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f48b40a3c30 0x7f48b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:19.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.834+0000 7f48c97fa640 1 -- 172.21.15.94:0/1612829340 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f48c40d0490 con 0x7f48cc10e290 2024-04-21T08:37:19.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.834+0000 7f48cbfff640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f48b40a3c30 0x7f48b40a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f48cc073040 tx=0x7f48bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:19.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:19.838+0000 7f48c97fa640 1 -- 172.21.15.94:0/1612829340 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f48c409d310 con 0x7f48cc10e290 2024-04-21T08:37:19.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:19 smithi111 ceph-mon[31493]: Deploying daemon mds.g on smithi094 2024-04-21T08:37:19.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:19 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] up:boot 2024-04-21T08:37:19.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:19 smithi111 ceph-mon[31493]: fsmap 9 up:standby 2024-04-21T08:37:19.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:19 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:37:19.977 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:19 smithi094 ceph-mon[27988]: Deploying daemon mds.g on smithi094 2024-04-21T08:37:19.978 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:19 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] up:boot 2024-04-21T08:37:19.978 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:19 smithi094 ceph-mon[27988]: fsmap 9 up:standby 2024-04-21T08:37:19.978 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:19 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:37:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:19 smithi173 ceph-mon[31428]: Deploying daemon mds.g on smithi094 2024-04-21T08:37:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:19 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] up:boot 2024-04-21T08:37:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:19 smithi173 ceph-mon[31428]: fsmap 9 up:standby 2024-04-21T08:37:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:19 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:37:20.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.071+0000 7f48d27b2640 1 -- 172.21.15.94:0/1612829340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48cc02d790 con 0x7f48cc10e290 2024-04-21T08:37:20.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.072+0000 7f48c97fa640 1 -- 172.21.15.94:0/1612829340 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7f48c409cc10 con 0x7f48cc10e290 2024-04-21T08:37:20.072 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:20.072 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:37:20.072 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 10 2024-04-21T08:37:20.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.074+0000 7f48d27b2640 1 -- 172.21.15.94:0/1612829340 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f48b40a3c30 msgr2=0x7f48b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:20.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.074+0000 7f48d27b2640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f48b40a3c30 0x7f48b40a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f48cc073040 tx=0x7f48bc005f90 comp rx=0 tx=0).stop 2024-04-21T08:37:20.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.074+0000 7f48d27b2640 1 -- 172.21.15.94:0/1612829340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48cc10e290 msgr2=0x7f48cc1ae4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:20.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.074+0000 7f48d27b2640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48cc10e290 0x7f48cc1ae4a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f48c400eb10 tx=0x7f48c400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:37:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.074+0000 7f48d27b2640 1 -- 172.21.15.94:0/1612829340 shutdown_connections 2024-04-21T08:37:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.074+0000 7f48d27b2640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f48b40a3c30 0x7f48b40a60f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.074+0000 7f48d27b2640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48cc10b8c0 0x7f48cc1a7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.074+0000 7f48d27b2640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48cc10e290 0x7f48cc1ae4a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.074+0000 7f48d27b2640 1 --2- 172.21.15.94:0/1612829340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48cc108f50 0x7f48cc1a6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.074+0000 7f48d27b2640 1 -- 172.21.15.94:0/1612829340 >> 172.21.15.94:0/1612829340 conn(0x7f48cc06ede0 msgr2=0x7f48cc109350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.075+0000 7f48d27b2640 1 -- 172.21.15.94:0/1612829340 shutdown_connections 2024-04-21T08:37:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:20.075+0000 7f48d27b2640 1 -- 172.21.15.94:0/1612829340 wait complete. 2024-04-21T08:37:20.260 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:37:20 smithi094 systemd[1]: Starting Ceph mds.g for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:37:20.527 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:37:20 smithi094 podman[69085]: 2024-04-21 08:37:20.27637081 +0000 UTC m=+0.018522475 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:37:20.528 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:37:20 smithi094 podman[69085]: 2024-04-21 08:37:20.380870009 +0000 UTC m=+0.123021721 container create b8094153e360ba799837f82256ceb499e7753ef39deadc50cacb3064ae425e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:37:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:20 smithi111 ceph-mon[31493]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:20 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1612829340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:20 smithi094 ceph-mon[27988]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:20 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1612829340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:21.010 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:37:20 smithi094 podman[69085]: 2024-04-21 08:37:20.667462498 +0000 UTC m=+0.409614145 container init b8094153e360ba799837f82256ceb499e7753ef39deadc50cacb3064ae425e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:37:21.010 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:37:20 smithi094 podman[69085]: 2024-04-21 08:37:20.670331916 +0000 UTC m=+0.412483563 container start b8094153e360ba799837f82256ceb499e7753ef39deadc50cacb3064ae425e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:37:21.010 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:37:20 smithi094 ceph-mds[69104]: -- 172.21.15.94:0/289896265 <== mon.0 v2:172.21.15.94:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571b7449a40 con 0x5571b741cc00 2024-04-21T08:37:21.010 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:37:20 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-g[69100]: starting mds.g at 2024-04-21T08:37:21.011 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:37:20 smithi094 bash[69085]: b8094153e360ba799837f82256ceb499e7753ef39deadc50cacb3064ae425e6a 2024-04-21T08:37:21.011 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:37:20 smithi094 systemd[1]: Started Ceph mds.g for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:37:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:20 smithi173 ceph-mon[31428]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:20 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1612829340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:22.086 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: Deploying daemon mds.a on smithi094 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.94:6838/4167052698,v1:172.21.15.94:6839/4167052698] up:boot 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: fsmap 10 up:standby 2024-04-21T08:37:22.102 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:21 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:37:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:22.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:22.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:22.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:22.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: Deploying daemon mds.a on smithi094 2024-04-21T08:37:22.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:22.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.94:6838/4167052698,v1:172.21.15.94:6839/4167052698] up:boot 2024-04-21T08:37:22.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: fsmap 10 up:standby 2024-04-21T08:37:22.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:21 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:37:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.192+0000 7f10d51ce640 1 -- 172.21.15.94:0/2768669018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10d0102980 msgr2=0x7f10d01092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.192+0000 7f10d51ce640 1 --2- 172.21.15.94:0/2768669018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10d0102980 0x7f10d01092a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009ca0 tx=0x7f10c802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.192+0000 7f10d51ce640 1 -- 172.21.15.94:0/2768669018 shutdown_connections 2024-04-21T08:37:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.192+0000 7f10d51ce640 1 --2- 172.21.15.94:0/2768669018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d0101fc0 0x7f10d0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.192+0000 7f10d51ce640 1 --2- 172.21.15.94:0/2768669018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10d0102980 0x7f10d01092a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.192+0000 7f10d51ce640 1 --2- 172.21.15.94:0/2768669018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d0100e70 0x7f10d0101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.192+0000 7f10d51ce640 1 -- 172.21.15.94:0/2768669018 >> 172.21.15.94:0/2768669018 conn(0x7f10d00fc3e0 msgr2=0x7f10d00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.192+0000 7f10d51ce640 1 -- 172.21.15.94:0/2768669018 shutdown_connections 2024-04-21T08:37:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.192+0000 7f10d51ce640 1 -- 172.21.15.94:0/2768669018 wait complete. 2024-04-21T08:37:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.193+0000 7f10d51ce640 1 Processor -- start 2024-04-21T08:37:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.193+0000 7f10d51ce640 1 -- start start 2024-04-21T08:37:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10d51ce640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d0100e70 0x7f10d0069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10d51ce640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10d0101fc0 0x7f10d019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10d51ce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d0102980 0x7f10d019f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10d51ce640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f10d010c9d0 con 0x7f10d0101fc0 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10d51ce640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f10d010c850 con 0x7f10d0100e70 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10d51ce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10d010cb50 con 0x7f10d0102980 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10ce59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10d0101fc0 0x7f10d019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10ce59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10d0101fc0 0x7f10d019cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34208/0 (socket says 172.21.15.94:34208) 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10ce59c640 1 -- 172.21.15.94:0/3954039308 learned_addr learned my addr 172.21.15.94:0/3954039308 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10ced9d640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d0100e70 0x7f10d0069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10cf59e640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d0102980 0x7f10d019f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10ce59c640 1 -- 172.21.15.94:0/3954039308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d0102980 msgr2=0x7f10d019f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10ce59c640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d0102980 0x7f10d019f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10ce59c640 1 -- 172.21.15.94:0/3954039308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d0100e70 msgr2=0x7f10d0069cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10ce59c640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d0100e70 0x7f10d0069cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10ce59c640 1 -- 172.21.15.94:0/3954039308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d019fd10 con 0x7f10d0101fc0 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.194+0000 7f10ce59c640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10d0101fc0 0x7f10d019cde0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f10c400d950 tx=0x7f10c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.195+0000 7f10b7fff640 1 -- 172.21.15.94:0/3954039308 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c4017070 con 0x7f10d0101fc0 2024-04-21T08:37:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.195+0000 7f10d51ce640 1 -- 172.21.15.94:0/3954039308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d019ffa0 con 0x7f10d0101fc0 2024-04-21T08:37:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.195+0000 7f10b7fff640 1 -- 172.21.15.94:0/3954039308 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10c4009c30 con 0x7f10d0101fc0 2024-04-21T08:37:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.195+0000 7f10b7fff640 1 -- 172.21.15.94:0/3954039308 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c400f620 con 0x7f10d0101fc0 2024-04-21T08:37:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.195+0000 7f10d51ce640 1 -- 172.21.15.94:0/3954039308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d01a04c0 con 0x7f10d0101fc0 2024-04-21T08:37:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.196+0000 7f10d51ce640 1 -- 172.21.15.94:0/3954039308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10d0108180 con 0x7f10d0101fc0 2024-04-21T08:37:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.197+0000 7f10b7fff640 1 -- 172.21.15.94:0/3954039308 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f10c401a070 con 0x7f10d0101fc0 2024-04-21T08:37:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.198+0000 7f10b7fff640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f10b80a3bb0 0x7f10b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.198+0000 7f10b7fff640 1 -- 172.21.15.94:0/3954039308 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f10c40cfa30 con 0x7f10d0101fc0 2024-04-21T08:37:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.198+0000 7f10ced9d640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f10b80a3bb0 0x7f10b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.199+0000 7f10ced9d640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f10b80a3bb0 0x7f10b80a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009b60 tx=0x7f10c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:22.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.202+0000 7f10b7fff640 1 -- 172.21.15.94:0/3954039308 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10c4009de0 con 0x7f10d0101fc0 2024-04-21T08:37:22.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: Deploying daemon mds.a on smithi094 2024-04-21T08:37:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.94:6838/4167052698,v1:172.21.15.94:6839/4167052698] up:boot 2024-04-21T08:37:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: fsmap 10 up:standby 2024-04-21T08:37:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:21 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:37:22.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.425+0000 7f10d51ce640 1 -- 172.21.15.94:0/3954039308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10d002d5c0 con 0x7f10d0101fc0 2024-04-21T08:37:22.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.425+0000 7f10b7fff640 1 -- 172.21.15.94:0/3954039308 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8839 (secure 0 0 0) 0x7f10c409c8b0 con 0x7f10d0101fc0 2024-04-21T08:37:22.425 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:22.425 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:37:22.426 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 11 2024-04-21T08:37:22.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.427+0000 7f10d51ce640 1 -- 172.21.15.94:0/3954039308 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f10b80a3bb0 msgr2=0x7f10b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:22.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.427+0000 7f10d51ce640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f10b80a3bb0 0x7f10b80a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009b60 tx=0x7f10c0005f90 comp rx=0 tx=0).stop 2024-04-21T08:37:22.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.427+0000 7f10d51ce640 1 -- 172.21.15.94:0/3954039308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10d0101fc0 msgr2=0x7f10d019cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:22.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.427+0000 7f10d51ce640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10d0101fc0 0x7f10d019cde0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f10c400d950 tx=0x7f10c400de10 comp rx=0 tx=0).stop 2024-04-21T08:37:22.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.427+0000 7f10d51ce640 1 -- 172.21.15.94:0/3954039308 shutdown_connections 2024-04-21T08:37:22.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.427+0000 7f10d51ce640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f10b80a3bb0 0x7f10b80a6070 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.427+0000 7f10d51ce640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d0102980 0x7f10d019f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.427+0000 7f10d51ce640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10d0101fc0 0x7f10d019cde0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.427+0000 7f10d51ce640 1 --2- 172.21.15.94:0/3954039308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d0100e70 0x7f10d0069cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.427+0000 7f10d51ce640 1 -- 172.21.15.94:0/3954039308 >> 172.21.15.94:0/3954039308 conn(0x7f10d00fc3e0 msgr2=0x7f10d0105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:22.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.428+0000 7f10d51ce640 1 -- 172.21.15.94:0/3954039308 shutdown_connections 2024-04-21T08:37:22.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:22.428+0000 7f10d51ce640 1 -- 172.21.15.94:0/3954039308 wait complete. 2024-04-21T08:37:23.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:22 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3954039308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:23.188 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:22 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3954039308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:23.188 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:37:22 smithi094 systemd[1]: Starting Ceph mds.a for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:37:23.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:22 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3954039308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:23.499 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:37:23 smithi094 podman[69446]: 2024-04-21 08:37:23.090900278 +0000 UTC m=+0.018250190 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:37:23.499 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:37:23 smithi094 podman[69446]: 2024-04-21 08:37:23.255852098 +0000 UTC m=+0.183201983 container create a445d8ba8f2d7ebde8efeceb722063104d91a272d220033adb711f9340a88511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-21T08:37:23.499 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:37:23 smithi094 podman[69446]: 2024-04-21 08:37:23.50041302 +0000 UTC m=+0.427762933 container init a445d8ba8f2d7ebde8efeceb722063104d91a272d220033adb711f9340a88511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T08:37:23.760 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:37:23 smithi094 podman[69446]: 2024-04-21 08:37:23.503003098 +0000 UTC m=+0.430352995 container start a445d8ba8f2d7ebde8efeceb722063104d91a272d220033adb711f9340a88511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:37:23.760 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:37:23 smithi094 ceph-mds[69467]: -- 172.21.15.94:0/4026571107 <== mon.0 v2:172.21.15.94:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fa157e1a40 con 0x55fa157b4800 2024-04-21T08:37:23.760 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:37:23 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-a[69463]: starting mds.a at 2024-04-21T08:37:23.760 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:37:23 smithi094 bash[69446]: a445d8ba8f2d7ebde8efeceb722063104d91a272d220033adb711f9340a88511 2024-04-21T08:37:23.760 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:37:23 smithi094 systemd[1]: Started Ceph mds.a for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:37:24.439 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:24.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.544+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2342865682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a28107c80 msgr2=0x7f3a2810a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.544+0000 7f3a2cd50640 1 --2- 172.21.15.94:0/2342865682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a28107c80 0x7f3a2810a0d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3a20009ca0 tx=0x7f3a2002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.544+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2342865682 shutdown_connections 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.544+0000 7f3a2cd50640 1 --2- 172.21.15.94:0/2342865682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a28105350 0x7f3a28107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.544+0000 7f3a2cd50640 1 --2- 172.21.15.94:0/2342865682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a28107c80 0x7f3a2810a0d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.544+0000 7f3a2cd50640 1 --2- 172.21.15.94:0/2342865682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28102a20 0x7f3a28104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.544+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2342865682 >> 172.21.15.94:0/2342865682 conn(0x7f3a280fc410 msgr2=0x7f3a280fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.544+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2342865682 shutdown_connections 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.544+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2342865682 wait complete. 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a2cd50640 1 Processor -- start 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a2cd50640 1 -- start start 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a2cd50640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28102a20 0x7f3a28102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:24.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a2cd50640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a28105350 0x7f3a281009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a2cd50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a28107c80 0x7f3a28100f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a2cd50640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2810dae0 con 0x7f3a28107c80 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a2cd50640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2810d960 con 0x7f3a28102a20 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a2cd50640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2810dc60 con 0x7f3a28105350 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a26d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a28107c80 0x7f3a28100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a26d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a28107c80 0x7f3a28100f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54006/0 (socket says 172.21.15.94:54006) 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a26d9d640 1 -- 172.21.15.94:0/2567612447 learned_addr learned my addr 172.21.15.94:0/2567612447 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a25d9b640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a28105350 0x7f3a281009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.545+0000 7f3a2659c640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28102a20 0x7f3a28102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a26d9d640 1 -- 172.21.15.94:0/2567612447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a28105350 msgr2=0x7f3a281009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a26d9d640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a28105350 0x7f3a281009d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a26d9d640 1 -- 172.21.15.94:0/2567612447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28102a20 msgr2=0x7f3a28102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:24.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a26d9d640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28102a20 0x7f3a28102390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a26d9d640 1 -- 172.21.15.94:0/2567612447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a281017a0 con 0x7f3a28107c80 2024-04-21T08:37:24.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a2659c640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28102a20 0x7f3a28102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:24.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a26d9d640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a28107c80 0x7f3a28100f10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3a2002c700 tx=0x7f3a20034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:24.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a177fe640 1 -- 172.21.15.94:0/2567612447 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20039040 con 0x7f3a28107c80 2024-04-21T08:37:24.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a177fe640 1 -- 172.21.15.94:0/2567612447 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a20044070 con 0x7f3a28107c80 2024-04-21T08:37:24.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2567612447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a281a21f0 con 0x7f3a28107c80 2024-04-21T08:37:24.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a177fe640 1 -- 172.21.15.94:0/2567612447 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a200042a0 con 0x7f3a28107c80 2024-04-21T08:37:24.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.546+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2567612447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a281a2790 con 0x7f3a28107c80 2024-04-21T08:37:24.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.547+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2567612447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a280fdf00 con 0x7f3a28107c80 2024-04-21T08:37:24.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.548+0000 7f3a177fe640 1 -- 172.21.15.94:0/2567612447 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3a20003970 con 0x7f3a28107c80 2024-04-21T08:37:24.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.548+0000 7f3a177fe640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a100a3c30 0x7f3a100a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:24.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.549+0000 7f3a2659c640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a100a3c30 0x7f3a100a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:24.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.549+0000 7f3a177fe640 1 -- 172.21.15.94:0/2567612447 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3a200f22c0 con 0x7f3a28107c80 2024-04-21T08:37:24.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.549+0000 7f3a2659c640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a100a3c30 0x7f3a100a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3a180058c0 tx=0x7f3a18005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:24.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.553+0000 7f3a177fe640 1 -- 172.21.15.94:0/2567612447 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a200bf140 con 0x7f3a28107c80 2024-04-21T08:37:24.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:24.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:24.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:24.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:24.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:24.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:24.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:24.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: Deploying daemon mds.b on smithi111 2024-04-21T08:37:24.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.94:6840/2997402360,v1:172.21.15.94:6841/2997402360] up:boot 2024-04-21T08:37:24.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: fsmap 11 up:standby 2024-04-21T08:37:24.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:24 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:37:24.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.772+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2567612447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a28062b50 con 0x7f3a28107c80 2024-04-21T08:37:24.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.772+0000 7f3a177fe640 1 -- 172.21.15.94:0/2567612447 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9665 (secure 0 0 0) 0x7f3a200bea40 con 0x7f3a28107c80 2024-04-21T08:37:24.772 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:24.772 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:37:24.772 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 12 2024-04-21T08:37:24.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2567612447 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a100a3c30 msgr2=0x7f3a100a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:24.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a100a3c30 0x7f3a100a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3a180058c0 tx=0x7f3a18005ec0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2567612447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a28107c80 msgr2=0x7f3a28100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a28107c80 0x7f3a28100f10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3a2002c700 tx=0x7f3a20034760 comp rx=0 tx=0).stop 2024-04-21T08:37:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2567612447 shutdown_connections 2024-04-21T08:37:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a100a3c30 0x7f3a100a60f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a28105350 0x7f3a281009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a28107c80 0x7f3a28100f10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 --2- 172.21.15.94:0/2567612447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28102a20 0x7f3a28102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2567612447 >> 172.21.15.94:0/2567612447 conn(0x7f3a280fc410 msgr2=0x7f3a28103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2567612447 shutdown_connections 2024-04-21T08:37:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:24.774+0000 7f3a2cd50640 1 -- 172.21.15.94:0/2567612447 wait complete. 2024-04-21T08:37:24.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:24.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:24.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:24.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:24.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:24.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:24.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:24.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: Deploying daemon mds.b on smithi111 2024-04-21T08:37:24.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.94:6840/2997402360,v1:172.21.15.94:6841/2997402360] up:boot 2024-04-21T08:37:24.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: fsmap 11 up:standby 2024-04-21T08:37:24.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:24 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:37:25.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:25.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:25.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:25.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:25.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:37:25.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:37:25.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:25.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: Deploying daemon mds.b on smithi111 2024-04-21T08:37:25.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.94:6840/2997402360,v1:172.21.15.94:6841/2997402360] up:boot 2024-04-21T08:37:25.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: fsmap 11 up:standby 2024-04-21T08:37:25.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:24 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:37:25.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:25 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2567612447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:25.938 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:25 smithi111 systemd[1]: Starting Ceph mds.b for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:37:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:25 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2567612447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:26.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:25 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2567612447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:26.361 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:26 smithi111 podman[61973]: 2024-04-21 08:37:25.954018607 +0000 UTC m=+0.018113479 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:37:26.361 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:26 smithi111 podman[61973]: 2024-04-21 08:37:26.059438792 +0000 UTC m=+0.123533637 container create e1ed5b9cf8df733d6c79e456121e8acd6e7f64c9c6da986e8e591be4c2efe962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-21T08:37:26.361 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:26 smithi111 podman[61973]: 2024-04-21 08:37:26.270311919 +0000 UTC m=+0.334406773 container init e1ed5b9cf8df733d6c79e456121e8acd6e7f64c9c6da986e8e591be4c2efe962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-21T08:37:26.361 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:26 smithi111 podman[61973]: 2024-04-21 08:37:26.272926454 +0000 UTC m=+0.337021310 container start e1ed5b9cf8df733d6c79e456121e8acd6e7f64c9c6da986e8e591be4c2efe962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-21T08:37:26.361 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:26 smithi111 ceph-mds[61992]: -- 172.21.15.111:0/1562655146 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ef008bfa40 con 0x55ef00892800 2024-04-21T08:37:26.361 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:26 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b[61988]: starting mds.b at 2024-04-21T08:37:26.361 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:26 smithi111 bash[61973]: e1ed5b9cf8df733d6c79e456121e8acd6e7f64c9c6da986e8e591be4c2efe962 2024-04-21T08:37:26.671 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:26 smithi111 systemd[1]: Started Ceph mds.b for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:37:26.786 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:26.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.887+0000 7f9e14985640 1 -- 172.21.15.94:0/13710930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e10101fc0 msgr2=0x7f9e10102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:26.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.887+0000 7f9e14985640 1 --2- 172.21.15.94:0/13710930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e10101fc0 0x7f9e10102440 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009900 tx=0x7f9e0002bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:26.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 -- 172.21.15.94:0/13710930 shutdown_connections 2024-04-21T08:37:26.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 --2- 172.21.15.94:0/13710930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e10102980 0x7f9e101092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:26.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 --2- 172.21.15.94:0/13710930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e10101fc0 0x7f9e10102440 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:26.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 --2- 172.21.15.94:0/13710930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e10100e70 0x7f9e10101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:26.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 -- 172.21.15.94:0/13710930 >> 172.21.15.94:0/13710930 conn(0x7f9e100fc3e0 msgr2=0x7f9e100fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:26.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 -- 172.21.15.94:0/13710930 shutdown_connections 2024-04-21T08:37:26.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 -- 172.21.15.94:0/13710930 wait complete. 2024-04-21T08:37:26.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 Processor -- start 2024-04-21T08:37:26.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 -- start start 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e10100e70 0x7f9e101970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.888+0000 7f9e14985640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e10101fc0 0x7f9e10197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e14985640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e10102980 0x7f9e1019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e14985640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1010c9c0 con 0x7f9e10102980 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e14985640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1010c840 con 0x7f9e10100e70 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e14985640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1010cb40 con 0x7f9e10101fc0 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e10102980 0x7f9e1019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e10102980 0x7f9e1019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54016/0 (socket says 172.21.15.94:54016) 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0ed9d640 1 -- 172.21.15.94:0/314846048 learned_addr learned my addr 172.21.15.94:0/314846048 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0dd9b640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e10101fc0 0x7f9e10197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0e59c640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e10100e70 0x7f9e101970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0ed9d640 1 -- 172.21.15.94:0/314846048 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e10101fc0 msgr2=0x7f9e10197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0ed9d640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e10101fc0 0x7f9e10197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:26.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0ed9d640 1 -- 172.21.15.94:0/314846048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e10100e70 msgr2=0x7f9e101970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:26.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0ed9d640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e10100e70 0x7f9e101970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:26.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0ed9d640 1 -- 172.21.15.94:0/314846048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1019ed10 con 0x7f9e10102980 2024-04-21T08:37:26.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e0ed9d640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e10102980 0x7f9e1019e6a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009fd0 tx=0x7f9e08007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:26.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9dff7fe640 1 -- 172.21.15.94:0/314846048 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0800de10 con 0x7f9e10102980 2024-04-21T08:37:26.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e14985640 1 -- 172.21.15.94:0/314846048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e10069140 con 0x7f9e10102980 2024-04-21T08:37:26.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9dff7fe640 1 -- 172.21.15.94:0/314846048 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e08003e20 con 0x7f9e10102980 2024-04-21T08:37:26.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9dff7fe640 1 -- 172.21.15.94:0/314846048 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08004810 con 0x7f9e10102980 2024-04-21T08:37:26.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.889+0000 7f9e14985640 1 -- 172.21.15.94:0/314846048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e10069660 con 0x7f9e10102980 2024-04-21T08:37:26.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.890+0000 7f9e14985640 1 -- 172.21.15.94:0/314846048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e10068de0 con 0x7f9e10102980 2024-04-21T08:37:26.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.892+0000 7f9dff7fe640 1 -- 172.21.15.94:0/314846048 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9e08003970 con 0x7f9e10102980 2024-04-21T08:37:26.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.893+0000 7f9dff7fe640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9df80a3d50 0x7f9df80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:26.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.893+0000 7f9dff7fe640 1 -- 172.21.15.94:0/314846048 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9e080d1a60 con 0x7f9e10102980 2024-04-21T08:37:26.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.893+0000 7f9e0e59c640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9df80a3d50 0x7f9df80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:26.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.893+0000 7f9e0e59c640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9df80a3d50 0x7f9df80a6210 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009bd0 tx=0x7f9e04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:26.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:26.896+0000 7f9dff7fe640 1 -- 172.21.15.94:0/314846048 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e0809e290 con 0x7f9e10102980 2024-04-21T08:37:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:26 smithi111 ceph-mon[31493]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:26 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:26 smithi094 ceph-mon[27988]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:26 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:27.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:26 smithi173 ceph-mon[31428]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:27.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:27.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:27.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:27.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:27.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:27.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:26 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:27.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.119+0000 7f9e14985640 1 -- 172.21.15.94:0/314846048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e10107de0 con 0x7f9e10102980 2024-04-21T08:37:27.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.120+0000 7f9dff7fe640 1 -- 172.21.15.94:0/314846048 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10494 (secure 0 0 0) 0x7f9e080cfe20 con 0x7f9e10102980 2024-04-21T08:37:27.120 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:27.120 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-21T08:37:27.120 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 13 2024-04-21T08:37:27.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 -- 172.21.15.94:0/314846048 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9df80a3d50 msgr2=0x7f9df80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9df80a3d50 0x7f9df80a6210 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009bd0 tx=0x7f9e04005f90 comp rx=0 tx=0).stop 2024-04-21T08:37:27.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 -- 172.21.15.94:0/314846048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e10102980 msgr2=0x7f9e1019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e10102980 0x7f9e1019e6a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009fd0 tx=0x7f9e08007590 comp rx=0 tx=0).stop 2024-04-21T08:37:27.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 -- 172.21.15.94:0/314846048 shutdown_connections 2024-04-21T08:37:27.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9df80a3d50 0x7f9df80a6210 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e10101fc0 0x7f9e10197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e10102980 0x7f9e1019e6a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 --2- 172.21.15.94:0/314846048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e10100e70 0x7f9e101970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 -- 172.21.15.94:0/314846048 >> 172.21.15.94:0/314846048 conn(0x7f9e100fc3e0 msgr2=0x7f9e101053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:27.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 -- 172.21.15.94:0/314846048 shutdown_connections 2024-04-21T08:37:27.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.122+0000 7f9e14985640 1 -- 172.21.15.94:0/314846048 wait complete. 2024-04-21T08:37:27.134 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:37:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.247+0000 7f88c27db640 1 -- 172.21.15.94:0/590935652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88bc102a50 msgr2=0x7f88bc104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.247+0000 7f88c27db640 1 --2- 172.21.15.94:0/590935652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88bc102a50 0x7f88bc104e40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009900 tx=0x7f88ac02bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.247+0000 7f88c27db640 1 -- 172.21.15.94:0/590935652 shutdown_connections 2024-04-21T08:37:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.247+0000 7f88c27db640 1 --2- 172.21.15.94:0/590935652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88bc107cb0 0x7f88bc10a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.247+0000 7f88c27db640 1 --2- 172.21.15.94:0/590935652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc105380 0x7f88bc107770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.247+0000 7f88c27db640 1 --2- 172.21.15.94:0/590935652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88bc102a50 0x7f88bc104e40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.247+0000 7f88c27db640 1 -- 172.21.15.94:0/590935652 >> 172.21.15.94:0/590935652 conn(0x7f88bc0fc3e0 msgr2=0x7f88bc0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.247+0000 7f88c27db640 1 -- 172.21.15.94:0/590935652 shutdown_connections 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.247+0000 7f88c27db640 1 -- 172.21.15.94:0/590935652 wait complete. 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c27db640 1 Processor -- start 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c27db640 1 -- start start 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c27db640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc102a50 0x7f88bc19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c27db640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88bc105380 0x7f88bc19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c27db640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88bc107cb0 0x7f88bc1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c27db640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc10dac0 con 0x7f88bc107cb0 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c27db640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc10d940 con 0x7f88bc102a50 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c27db640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc10dc40 con 0x7f88bc105380 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c0d78640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88bc107cb0 0x7f88bc1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c0d78640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88bc107cb0 0x7f88bc1a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54040/0 (socket says 172.21.15.94:54040) 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88c0d78640 1 -- 172.21.15.94:0/739200638 learned_addr learned my addr 172.21.15.94:0/739200638 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88bb7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88bc105380 0x7f88bc19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.248+0000 7f88bbfff640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc102a50 0x7f88bc19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88bbfff640 1 -- 172.21.15.94:0/739200638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88bc105380 msgr2=0x7f88bc19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88bbfff640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88bc105380 0x7f88bc19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88bbfff640 1 -- 172.21.15.94:0/739200638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88bc107cb0 msgr2=0x7f88bc1a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88bbfff640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88bc107cb0 0x7f88bc1a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88bbfff640 1 -- 172.21.15.94:0/739200638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88bc1a2020 con 0x7f88bc102a50 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88c0d78640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88bc107cb0 0x7f88bc1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88bbfff640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc102a50 0x7f88bc19a400 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f88ac000c00 tx=0x7f88ac005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88b97fa640 1 -- 172.21.15.94:0/739200638 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ac043020 con 0x7f88bc102a50 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88c27db640 1 -- 172.21.15.94:0/739200638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88bc1a22b0 con 0x7f88bc102a50 2024-04-21T08:37:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88b97fa640 1 -- 172.21.15.94:0/739200638 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88ac004350 con 0x7f88bc102a50 2024-04-21T08:37:27.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88b97fa640 1 -- 172.21.15.94:0/739200638 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ac003b50 con 0x7f88bc102a50 2024-04-21T08:37:27.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.249+0000 7f88c27db640 1 -- 172.21.15.94:0/739200638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88bc1a2850 con 0x7f88bc102a50 2024-04-21T08:37:27.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.250+0000 7f88c27db640 1 -- 172.21.15.94:0/739200638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88bc1097a0 con 0x7f88bc102a50 2024-04-21T08:37:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.251+0000 7f88b97fa640 1 -- 172.21.15.94:0/739200638 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f88ac046020 con 0x7f88bc102a50 2024-04-21T08:37:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.252+0000 7f88b97fa640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f88a40a3bb0 0x7f88a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.252+0000 7f88b97fa640 1 -- 172.21.15.94:0/739200638 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f88ac0f25a0 con 0x7f88bc102a50 2024-04-21T08:37:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.252+0000 7f88bb7fe640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f88a40a3bb0 0x7f88a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.252+0000 7f88bb7fe640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f88a40a3bb0 0x7f88a40a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f88bc19b910 tx=0x7f88b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:27.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.255+0000 7f88b97fa640 1 -- 172.21.15.94:0/739200638 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88ac04d020 con 0x7f88bc102a50 2024-04-21T08:37:27.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.441+0000 7f88c27db640 1 -- 172.21.15.94:0/739200638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88bc02d790 con 0x7f88bc102a50 2024-04-21T08:37:27.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.442+0000 7f88b97fa640 1 -- 172.21.15.94:0/739200638 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18420 (secure 0 0 0) 0x7f88ac0bf520 con 0x7f88bc102a50 2024-04-21T08:37:27.442 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:27.442 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":81,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:36:43.263319+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 -- 172.21.15.94:0/739200638 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f88a40a3bb0 msgr2=0x7f88a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f88a40a3bb0 0x7f88a40a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f88bc19b910 tx=0x7f88b0005f90 comp rx=0 tx=0).stop 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 -- 172.21.15.94:0/739200638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc102a50 msgr2=0x7f88bc19a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc102a50 0x7f88bc19a400 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f88ac000c00 tx=0x7f88ac005dc0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 -- 172.21.15.94:0/739200638 shutdown_connections 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f88a40a3bb0 0x7f88a40a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88bc105380 0x7f88bc19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88bc107cb0 0x7f88bc1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 --2- 172.21.15.94:0/739200638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc102a50 0x7f88bc19a400 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 -- 172.21.15.94:0/739200638 >> 172.21.15.94:0/739200638 conn(0x7f88bc0fc3e0 msgr2=0x7f88bc1022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 -- 172.21.15.94:0/739200638 shutdown_connections 2024-04-21T08:37:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.444+0000 7f88c27db640 1 -- 172.21.15.94:0/739200638 wait complete. 2024-04-21T08:37:27.455 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:37:27.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:37:27.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.560+0000 7f59e6677640 1 -- 172.21.15.94:0/1162973522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59e0101fe0 msgr2=0x7f59e0102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.560+0000 7f59e6677640 1 --2- 172.21.15.94:0/1162973522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59e0101fe0 0x7f59e0102460 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009900 tx=0x7f59d402bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:27.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.560+0000 7f59e6677640 1 -- 172.21.15.94:0/1162973522 shutdown_connections 2024-04-21T08:37:27.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.560+0000 7f59e6677640 1 --2- 172.21.15.94:0/1162973522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e01029a0 0x7f59e01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.560+0000 7f59e6677640 1 --2- 172.21.15.94:0/1162973522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59e0101fe0 0x7f59e0102460 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.560+0000 7f59e6677640 1 --2- 172.21.15.94:0/1162973522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e0100e90 0x7f59e01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.560+0000 7f59e6677640 1 -- 172.21.15.94:0/1162973522 >> 172.21.15.94:0/1162973522 conn(0x7f59e00fc400 msgr2=0x7f59e00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.560+0000 7f59e6677640 1 -- 172.21.15.94:0/1162973522 shutdown_connections 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.560+0000 7f59e6677640 1 -- 172.21.15.94:0/1162973522 wait complete. 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e6677640 1 Processor -- start 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e6677640 1 -- start start 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e6677640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e0100e90 0x7f59e01981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e6677640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e0101fe0 0x7f59e0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e6677640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59e01029a0 0x7f59e019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e6677640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f59e010c9d0 con 0x7f59e01029a0 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e6677640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f59e010c850 con 0x7f59e0101fe0 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e6677640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59e010cb50 con 0x7f59e0100e90 2024-04-21T08:37:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e4c14640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59e01029a0 0x7f59e019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e4c14640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59e01029a0 0x7f59e019f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54064/0 (socket says 172.21.15.94:54064) 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59e4c14640 1 -- 172.21.15.94:0/609653255 learned_addr learned my addr 172.21.15.94:0/609653255 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59dffff640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e0100e90 0x7f59e01981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.561+0000 7f59df7fe640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e0101fe0 0x7f59e0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59e4c14640 1 -- 172.21.15.94:0/609653255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e0100e90 msgr2=0x7f59e01981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59e4c14640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e0100e90 0x7f59e01981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59e4c14640 1 -- 172.21.15.94:0/609653255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e0101fe0 msgr2=0x7f59e0198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59e4c14640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e0101fe0 0x7f59e0198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59e4c14640 1 -- 172.21.15.94:0/609653255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e019fe10 con 0x7f59e01029a0 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59e4c14640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59e01029a0 0x7f59e019f7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f59d800eb10 tx=0x7f59d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59dd7fa640 1 -- 172.21.15.94:0/609653255 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d8004b20 con 0x7f59e01029a0 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59e6677640 1 -- 172.21.15.94:0/609653255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e01a0100 con 0x7f59e01029a0 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59dd7fa640 1 -- 172.21.15.94:0/609653255 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d8004cc0 con 0x7f59e01029a0 2024-04-21T08:37:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59e6677640 1 -- 172.21.15.94:0/609653255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e01a0620 con 0x7f59e01029a0 2024-04-21T08:37:27.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.562+0000 7f59dd7fa640 1 -- 172.21.15.94:0/609653255 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d8018630 con 0x7f59e01029a0 2024-04-21T08:37:27.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.563+0000 7f59e6677640 1 -- 172.21.15.94:0/609653255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e01a08e0 con 0x7f59e01029a0 2024-04-21T08:37:27.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.564+0000 7f59dd7fa640 1 -- 172.21.15.94:0/609653255 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f59d8005000 con 0x7f59e01029a0 2024-04-21T08:37:27.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.565+0000 7f59dd7fa640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f59c80a3bb0 0x7f59c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.565+0000 7f59dffff640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f59c80a3bb0 0x7f59c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.565+0000 7f59dd7fa640 1 -- 172.21.15.94:0/609653255 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f59d80d0650 con 0x7f59e01029a0 2024-04-21T08:37:27.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.565+0000 7f59dffff640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f59c80a3bb0 0x7f59c80a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f59d000bbe0 tx=0x7f59d000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:27.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.569+0000 7f59dd7fa640 1 -- 172.21.15.94:0/609653255 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f59d809d4d0 con 0x7f59e01029a0 2024-04-21T08:37:27.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:27 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] up:boot 2024-04-21T08:37:27.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:27 smithi094 ceph-mon[27988]: fsmap 12 up:standby 2024-04-21T08:37:27.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:27 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:37:27.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:27 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/314846048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:27.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:27 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/739200638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:27.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:27 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] up:boot 2024-04-21T08:37:27.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:27 smithi173 ceph-mon[31428]: fsmap 12 up:standby 2024-04-21T08:37:27.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:27 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:37:27.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:27 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/314846048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:27.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:27 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/739200638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:27.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.762+0000 7f59e6677640 1 -- 172.21.15.94:0/609653255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59e01a0bd0 con 0x7f59e01029a0 2024-04-21T08:37:27.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.762+0000 7f59dd7fa640 1 -- 172.21.15.94:0/609653255 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f59d809cdd0 con 0x7f59e01029a0 2024-04-21T08:37:27.762 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:37:27.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.764+0000 7f59e6677640 1 -- 172.21.15.94:0/609653255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f59c80a3bb0 msgr2=0x7f59c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.764+0000 7f59e6677640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f59c80a3bb0 0x7f59c80a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f59d000bbe0 tx=0x7f59d000b390 comp rx=0 tx=0).stop 2024-04-21T08:37:27.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.764+0000 7f59e6677640 1 -- 172.21.15.94:0/609653255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59e01029a0 msgr2=0x7f59e019f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.764+0000 7f59e6677640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59e01029a0 0x7f59e019f7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f59d800eb10 tx=0x7f59d800c2d0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.764+0000 7f59e6677640 1 -- 172.21.15.94:0/609653255 shutdown_connections 2024-04-21T08:37:27.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.764+0000 7f59e6677640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f59c80a3bb0 0x7f59c80a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.764+0000 7f59e6677640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59e01029a0 0x7f59e019f7a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.764+0000 7f59e6677640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e0101fe0 0x7f59e0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.764+0000 7f59e6677640 1 --2- 172.21.15.94:0/609653255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e0100e90 0x7f59e01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.764+0000 7f59e6677640 1 -- 172.21.15.94:0/609653255 >> 172.21.15.94:0/609653255 conn(0x7f59e00fc400 msgr2=0x7f59e00690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:27.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.765+0000 7f59e6677640 1 -- 172.21.15.94:0/609653255 shutdown_connections 2024-04-21T08:37:27.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.765+0000 7f59e6677640 1 -- 172.21.15.94:0/609653255 wait complete. 2024-04-21T08:37:27.776 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-21T08:37:27.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.874+0000 7f0549cc0640 1 -- 172.21.15.94:0/2093473250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0544100630 msgr2=0x7f0544100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.874+0000 7f0549cc0640 1 --2- 172.21.15.94:0/2093473250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0544100630 0x7f0544100a50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0534009900 tx=0x7f053402bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.874+0000 7f0549cc0640 1 -- 172.21.15.94:0/2093473250 shutdown_connections 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.874+0000 7f0549cc0640 1 --2- 172.21.15.94:0/2093473250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0544101820 0x7f0544101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 --2- 172.21.15.94:0/2093473250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05441021e0 0x7f0544108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 --2- 172.21.15.94:0/2093473250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0544100630 0x7f0544100a50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 -- 172.21.15.94:0/2093473250 >> 172.21.15.94:0/2093473250 conn(0x7f05440fbbc0 msgr2=0x7f05440fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 -- 172.21.15.94:0/2093473250 shutdown_connections 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 -- 172.21.15.94:0/2093473250 wait complete. 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 Processor -- start 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 -- start start 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0544100630 0x7f05440fdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0544101820 0x7f054419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441021e0 0x7f054419f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f054410c160 con 0x7f0544101820 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f054410bfe0 con 0x7f0544100630 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.875+0000 7f0549cc0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f054410c2e0 con 0x7f05441021e0 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f053affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0544101820 0x7f054419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f053affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0544101820 0x7f054419cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54074/0 (socket says 172.21.15.94:54074) 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f053affd640 1 -- 172.21.15.94:0/84399150 learned_addr learned my addr 172.21.15.94:0/84399150 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f0543fff640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441021e0 0x7f054419f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f05437fe640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0544100630 0x7f05440fdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f053affd640 1 -- 172.21.15.94:0/84399150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441021e0 msgr2=0x7f054419f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f053affd640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441021e0 0x7f054419f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f053affd640 1 -- 172.21.15.94:0/84399150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0544100630 msgr2=0x7f05440fdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f053affd640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0544100630 0x7f05440fdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f053affd640 1 -- 172.21.15.94:0/84399150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054419fed0 con 0x7f0544101820 2024-04-21T08:37:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f053affd640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0544101820 0x7f054419cfa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f052c00d770 tx=0x7f052c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f05417fa640 1 -- 172.21.15.94:0/84399150 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c017070 con 0x7f0544101820 2024-04-21T08:37:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f05417fa640 1 -- 172.21.15.94:0/84399150 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f052c0054d0 con 0x7f0544101820 2024-04-21T08:37:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f0549cc0640 1 -- 172.21.15.94:0/84399150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05441a0160 con 0x7f0544101820 2024-04-21T08:37:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f05417fa640 1 -- 172.21.15.94:0/84399150 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c004c30 con 0x7f0544101820 2024-04-21T08:37:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.876+0000 7f0549cc0640 1 -- 172.21.15.94:0/84399150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05441a0660 con 0x7f0544101820 2024-04-21T08:37:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.877+0000 7f0549cc0640 1 -- 172.21.15.94:0/84399150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05441081f0 con 0x7f0544101820 2024-04-21T08:37:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.878+0000 7f05417fa640 1 -- 172.21.15.94:0/84399150 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f052c004e10 con 0x7f0544101820 2024-04-21T08:37:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.879+0000 7f05417fa640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f05300a3bb0 0x7f05300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.879+0000 7f05437fe640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f05300a3bb0 0x7f05300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.879+0000 7f05417fa640 1 -- 172.21.15.94:0/84399150 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f052c0cfa20 con 0x7f0544101820 2024-04-21T08:37:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.879+0000 7f05437fe640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f05300a3bb0 0x7f05300a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0534036040 tx=0x7f0534000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:27.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:27.883+0000 7f05417fa640 1 -- 172.21.15.94:0/84399150 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f052c09c8a0 con 0x7f0544101820 2024-04-21T08:37:27.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:27 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] up:boot 2024-04-21T08:37:27.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:27 smithi111 ceph-mon[31493]: fsmap 12 up:standby 2024-04-21T08:37:27.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:27 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:37:27.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:27 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/314846048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:27.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:27 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/739200638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.091+0000 7f0549cc0640 1 -- 172.21.15.94:0/84399150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f054402d810 con 0x7f0544101820 2024-04-21T08:37:28.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.091+0000 7f05417fa640 1 -- 172.21.15.94:0/84399150 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f052c09c1a0 con 0x7f0544101820 2024-04-21T08:37:28.091 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:28.091 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-04-21T08:37:28.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:37:28.091 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-04-21T08:37:28.092 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-04-21T08:37:28.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:37:28.092 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-04-21T08:37:28.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.093+0000 7f0549cc0640 1 -- 172.21.15.94:0/84399150 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f05300a3bb0 msgr2=0x7f05300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:28.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.093+0000 7f0549cc0640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f05300a3bb0 0x7f05300a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0534036040 tx=0x7f0534000bc0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.093+0000 7f0549cc0640 1 -- 172.21.15.94:0/84399150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0544101820 msgr2=0x7f054419cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:28.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.093+0000 7f0549cc0640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0544101820 0x7f054419cfa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f052c00d770 tx=0x7f052c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:37:28.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.093+0000 7f0549cc0640 1 -- 172.21.15.94:0/84399150 shutdown_connections 2024-04-21T08:37:28.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.093+0000 7f0549cc0640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f05300a3bb0 0x7f05300a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.093+0000 7f0549cc0640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05441021e0 0x7f054419f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.093+0000 7f0549cc0640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0544101820 0x7f054419cfa0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.094+0000 7f0549cc0640 1 --2- 172.21.15.94:0/84399150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0544100630 0x7f05440fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.094+0000 7f0549cc0640 1 -- 172.21.15.94:0/84399150 >> 172.21.15.94:0/84399150 conn(0x7f05440fbbc0 msgr2=0x7f0544102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:28.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.094+0000 7f0549cc0640 1 -- 172.21.15.94:0/84399150 shutdown_connections 2024-04-21T08:37:28.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.094+0000 7f0549cc0640 1 -- 172.21.15.94:0/84399150 wait complete. 2024-04-21T08:37:28.108 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-21T08:37:28.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-21T08:37:28.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.218+0000 7fb190ee9640 1 -- 172.21.15.94:0/3335332488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb18c1032e0 msgr2=0x7fb18c103760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:28.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.218+0000 7fb190ee9640 1 --2- 172.21.15.94:0/3335332488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb18c1032e0 0x7fb18c103760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009930 tx=0x7fb17c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:28.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.218+0000 7fb190ee9640 1 -- 172.21.15.94:0/3335332488 shutdown_connections 2024-04-21T08:37:28.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.218+0000 7fb190ee9640 1 --2- 172.21.15.94:0/3335332488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb18c103ca0 0x7fb18c10a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.218+0000 7fb190ee9640 1 --2- 172.21.15.94:0/3335332488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb18c1032e0 0x7fb18c103760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.218+0000 7fb190ee9640 1 --2- 172.21.15.94:0/3335332488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb18c100270 0x7fb18c100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.218+0000 7fb190ee9640 1 -- 172.21.15.94:0/3335332488 >> 172.21.15.94:0/3335332488 conn(0x7fb18c0fbb80 msgr2=0x7fb18c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.218+0000 7fb190ee9640 1 -- 172.21.15.94:0/3335332488 shutdown_connections 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.218+0000 7fb190ee9640 1 -- 172.21.15.94:0/3335332488 wait complete. 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb190ee9640 1 Processor -- start 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb190ee9640 1 -- start start 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb190ee9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb18c100270 0x7fb18c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb190ee9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb18c1032e0 0x7fb18c101290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb190ee9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb18c103ca0 0x7fb18c1017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb190ee9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c10dae0 con 0x7fb18c103ca0 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb190ee9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c10d960 con 0x7fb18c100270 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb190ee9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c10dc60 con 0x7fb18c1032e0 2024-04-21T08:37:28.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb18ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb18c103ca0 0x7fb18c1017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:28.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb189d9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb18c1032e0 0x7fb18c101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:28.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb18a59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb18c100270 0x7fb18c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:28.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb18ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb18c103ca0 0x7fb18c1017d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54094/0 (socket says 172.21.15.94:54094) 2024-04-21T08:37:28.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb189d9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb18c1032e0 0x7fb18c101290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:36216/0 (socket says 172.21.15.94:36216) 2024-04-21T08:37:28.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.219+0000 7fb18ad9d640 1 -- 172.21.15.94:0/2433349456 learned_addr learned my addr 172.21.15.94:0/2433349456 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:28.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb18ad9d640 1 -- 172.21.15.94:0/2433349456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb18c1032e0 msgr2=0x7fb18c101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:28.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb18ad9d640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb18c1032e0 0x7fb18c101290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb18ad9d640 1 -- 172.21.15.94:0/2433349456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb18c100270 msgr2=0x7fb18c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:28.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb18ad9d640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb18c100270 0x7fb18c102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb18ad9d640 1 -- 172.21.15.94:0/2433349456 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18c102060 con 0x7fb18c103ca0 2024-04-21T08:37:28.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb18a59c640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb18c100270 0x7fb18c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:37:28.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb189d9b640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb18c1032e0 0x7fb18c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:37:28.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb18ad9d640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb18c103ca0 0x7fb18c1017d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb184009fd0 tx=0x7fb18400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:28.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb17b7fe640 1 -- 172.21.15.94:0/2433349456 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb184017070 con 0x7fb18c103ca0 2024-04-21T08:37:28.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb17b7fe640 1 -- 172.21.15.94:0/2433349456 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb184003e20 con 0x7fb18c103ca0 2024-04-21T08:37:28.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb190ee9640 1 -- 172.21.15.94:0/2433349456 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18c1a2380 con 0x7fb18c103ca0 2024-04-21T08:37:28.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb17b7fe640 1 -- 172.21.15.94:0/2433349456 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1840106c0 con 0x7fb18c103ca0 2024-04-21T08:37:28.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.220+0000 7fb190ee9640 1 -- 172.21.15.94:0/2433349456 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18c1a2850 con 0x7fb18c103ca0 2024-04-21T08:37:28.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.221+0000 7fb190ee9640 1 -- 172.21.15.94:0/2433349456 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb18c109550 con 0x7fb18c103ca0 2024-04-21T08:37:28.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.222+0000 7fb17b7fe640 1 -- 172.21.15.94:0/2433349456 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb184004140 con 0x7fb18c103ca0 2024-04-21T08:37:28.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.223+0000 7fb17b7fe640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1740a3bb0 0x7fb1740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:28.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.223+0000 7fb18a59c640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1740a3bb0 0x7fb1740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:28.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.223+0000 7fb17b7fe640 1 -- 172.21.15.94:0/2433349456 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb1840cfb90 con 0x7fb18c103ca0 2024-04-21T08:37:28.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.223+0000 7fb18a59c640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1740a3bb0 0x7fb1740a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb18c068e20 tx=0x7fb180005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:28.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.227+0000 7fb17b7fe640 1 -- 172.21.15.94:0/2433349456 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb1840047e0 con 0x7fb18c103ca0 2024-04-21T08:37:28.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:28.422+0000 7fb190ee9640 1 -- 172.21.15.94:0/2433349456 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb18c02d5c0 con 0x7fb18c103ca0 2024-04-21T08:37:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/609653255' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:37:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/84399150' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2433349456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:37:28.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:28 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:29.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:29.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/609653255' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:37:29.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/84399150' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:37:29.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:29.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:29.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2433349456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:37:29.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:28 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:29.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:29.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/609653255' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:37:29.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/84399150' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2433349456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:37:29.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:28 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:29.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.398+0000 7fb17b7fe640 1 -- 172.21.15.94:0/2433349456 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fb18409c3c0 con 0x7fb18c103ca0 2024-04-21T08:37:29.398 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-04-21T08:37:29.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 -- 172.21.15.94:0/2433349456 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1740a3bb0 msgr2=0x7fb1740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1740a3bb0 0x7fb1740a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb18c068e20 tx=0x7fb180005f90 comp rx=0 tx=0).stop 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 -- 172.21.15.94:0/2433349456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb18c103ca0 msgr2=0x7fb18c1017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb18c103ca0 0x7fb18c1017d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb184009fd0 tx=0x7fb18400b260 comp rx=0 tx=0).stop 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 -- 172.21.15.94:0/2433349456 shutdown_connections 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1740a3bb0 0x7fb1740a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb18c1032e0 0x7fb18c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb18c103ca0 0x7fb18c1017d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 --2- 172.21.15.94:0/2433349456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb18c100270 0x7fb18c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 -- 172.21.15.94:0/2433349456 >> 172.21.15.94:0/2433349456 conn(0x7fb18c0fbb80 msgr2=0x7fb18c106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 -- 172.21.15.94:0/2433349456 shutdown_connections 2024-04-21T08:37:29.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.401+0000 7fb190ee9640 1 -- 172.21.15.94:0/2433349456 wait complete. 2024-04-21T08:37:29.416 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-21T08:37:29.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 -- 172.21.15.94:0/3941007203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4360102a90 msgr2=0x7f4360104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:29.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 --2- 172.21.15.94:0/3941007203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4360102a90 0x7f4360104e80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4350009900 tx=0x7f435002bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:29.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 -- 172.21.15.94:0/3941007203 shutdown_connections 2024-04-21T08:37:29.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 --2- 172.21.15.94:0/3941007203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43601053c0 0x7f43601077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:29.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 --2- 172.21.15.94:0/3941007203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4360107cf0 0x7f436010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:29.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 --2- 172.21.15.94:0/3941007203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4360102a90 0x7f4360104e80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 -- 172.21.15.94:0/3941007203 >> 172.21.15.94:0/3941007203 conn(0x7f43600fc420 msgr2=0x7f43600fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 -- 172.21.15.94:0/3941007203 shutdown_connections 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 -- 172.21.15.94:0/3941007203 wait complete. 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 Processor -- start 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.523+0000 7f43663e6640 1 -- start start 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f43663e6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4360102a90 0x7f43601981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f43663e6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43601053c0 0x7f4360198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f43663e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4360107cf0 0x7f436019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f43663e6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f436010daf0 con 0x7f4360107cf0 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f43663e6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f436010d970 con 0x7f43601053c0 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f43663e6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f436010dc70 con 0x7f4360102a90 2024-04-21T08:37:29.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f4364983640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4360107cf0 0x7f436019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f4364983640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4360107cf0 0x7f436019f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54124/0 (socket says 172.21.15.94:54124) 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f4364983640 1 -- 172.21.15.94:0/657294423 learned_addr learned my addr 172.21.15.94:0/657294423 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f435f7fe640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43601053c0 0x7f4360198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f435ffff640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4360102a90 0x7f43601981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f4364983640 1 -- 172.21.15.94:0/657294423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4360102a90 msgr2=0x7f43601981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f4364983640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4360102a90 0x7f43601981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f4364983640 1 -- 172.21.15.94:0/657294423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43601053c0 msgr2=0x7f4360198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f4364983640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43601053c0 0x7f4360198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f4364983640 1 -- 172.21.15.94:0/657294423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436019fdf0 con 0x7f4360107cf0 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f435ffff640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4360102a90 0x7f43601981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:29.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f4364983640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4360107cf0 0x7f436019f780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f435800cce0 tx=0x7f435800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:29.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f435d7fa640 1 -- 172.21.15.94:0/657294423 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435800f450 con 0x7f4360107cf0 2024-04-21T08:37:29.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.524+0000 7f435d7fa640 1 -- 172.21.15.94:0/657294423 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f435800faa0 con 0x7f4360107cf0 2024-04-21T08:37:29.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.525+0000 7f435d7fa640 1 -- 172.21.15.94:0/657294423 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4358018b70 con 0x7f4360107cf0 2024-04-21T08:37:29.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.525+0000 7f43663e6640 1 -- 172.21.15.94:0/657294423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43601a00e0 con 0x7f4360107cf0 2024-04-21T08:37:29.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.525+0000 7f43663e6640 1 -- 172.21.15.94:0/657294423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43601a05e0 con 0x7f4360107cf0 2024-04-21T08:37:29.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.526+0000 7f43663e6640 1 -- 172.21.15.94:0/657294423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43600fdf10 con 0x7f4360107cf0 2024-04-21T08:37:29.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.527+0000 7f435d7fa640 1 -- 172.21.15.94:0/657294423 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4358018d10 con 0x7f4360107cf0 2024-04-21T08:37:29.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.527+0000 7f435d7fa640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3bb0 0x7f43480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:29.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.527+0000 7f435ffff640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3bb0 0x7f43480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:29.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.527+0000 7f435d7fa640 1 -- 172.21.15.94:0/657294423 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f43580d05c0 con 0x7f4360107cf0 2024-04-21T08:37:29.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.528+0000 7f435ffff640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3bb0 0x7f43480a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4350000c00 tx=0x7f43500095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:29.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.531+0000 7f435d7fa640 1 -- 172.21.15.94:0/657294423 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4358014030 con 0x7f4360107cf0 2024-04-21T08:37:29.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:29.729+0000 7f43663e6640 1 -- 172.21.15.94:0/657294423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f436002d5c0 con 0x7f4360107cf0 2024-04-21T08:37:30.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.401+0000 7f435d7fa640 1 -- 172.21.15.94:0/657294423 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f435809d360 con 0x7f4360107cf0 2024-04-21T08:37:30.401 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.403+0000 7f43663e6640 1 -- 172.21.15.94:0/657294423 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3bb0 msgr2=0x7f43480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.403+0000 7f43663e6640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3bb0 0x7f43480a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4350000c00 tx=0x7f43500095f0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.403+0000 7f43663e6640 1 -- 172.21.15.94:0/657294423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4360107cf0 msgr2=0x7f436019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.403+0000 7f43663e6640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4360107cf0 0x7f436019f780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f435800cce0 tx=0x7f435800a620 comp rx=0 tx=0).stop 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.403+0000 7f43663e6640 1 -- 172.21.15.94:0/657294423 shutdown_connections 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.403+0000 7f43663e6640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3bb0 0x7f43480a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.403+0000 7f43663e6640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4360107cf0 0x7f436019f780 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.403+0000 7f43663e6640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43601053c0 0x7f4360198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.403+0000 7f43663e6640 1 --2- 172.21.15.94:0/657294423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4360102a90 0x7f43601981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.403+0000 7f43663e6640 1 -- 172.21.15.94:0/657294423 >> 172.21.15.94:0/657294423 conn(0x7f43600fc420 msgr2=0x7f4360103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.404+0000 7f43663e6640 1 -- 172.21.15.94:0/657294423 shutdown_connections 2024-04-21T08:37:30.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.404+0000 7f43663e6640 1 -- 172.21.15.94:0/657294423 wait complete. 2024-04-21T08:37:30.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.525+0000 7fc57d196640 1 -- 172.21.15.94:0/3694609522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc578103b50 msgr2=0x7fc57810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.525+0000 7fc57d196640 1 --2- 172.21.15.94:0/3694609522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc578103b50 0x7fc57810a3e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc570009ca0 tx=0x7fc57002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.525+0000 7fc57d196640 1 -- 172.21.15.94:0/3694609522 shutdown_connections 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.525+0000 7fc57d196640 1 --2- 172.21.15.94:0/3694609522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc578103b50 0x7fc57810a3e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.525+0000 7fc57d196640 1 --2- 172.21.15.94:0/3694609522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc578103190 0x7fc578103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.525+0000 7fc57d196640 1 --2- 172.21.15.94:0/3694609522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc578101fa0 0x7fc5781023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.525+0000 7fc57d196640 1 -- 172.21.15.94:0/3694609522 >> 172.21.15.94:0/3694609522 conn(0x7fc5780fd530 msgr2=0x7fc5780ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57d196640 1 -- 172.21.15.94:0/3694609522 shutdown_connections 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57d196640 1 -- 172.21.15.94:0/3694609522 wait complete. 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57d196640 1 Processor -- start 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57d196640 1 -- start start 2024-04-21T08:37:30.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57d196640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc578101fa0 0x7fc578198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57d196640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc578103190 0x7fc578198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57d196640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc578103b50 0x7fc57819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57d196640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc57810dae0 con 0x7fc578103b50 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57d196640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc57810d960 con 0x7fc578101fa0 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57d196640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc57810dc60 con 0x7fc578103190 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc578103b50 0x7fc57819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc578103b50 0x7fc57819f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54142/0 (socket says 172.21.15.94:54142) 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.526+0000 7fc57759e640 1 -- 172.21.15.94:0/4207840016 learned_addr learned my addr 172.21.15.94:0/4207840016 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc56ffff640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc578103190 0x7fc578198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc576d9d640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc578101fa0 0x7fc578198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc57759e640 1 -- 172.21.15.94:0/4207840016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc578103190 msgr2=0x7fc578198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc57759e640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc578103190 0x7fc578198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc57759e640 1 -- 172.21.15.94:0/4207840016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc578101fa0 msgr2=0x7fc578198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc57759e640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc578101fa0 0x7fc578198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:30.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc57759e640 1 -- 172.21.15.94:0/4207840016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc57819fd70 con 0x7fc578103b50 2024-04-21T08:37:30.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc576d9d640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc578101fa0 0x7fc578198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:30.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc57759e640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc578103b50 0x7fc57819f700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc57002c700 tx=0x7fc5700093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:30.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc574d99640 1 -- 172.21.15.94:0/4207840016 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570041020 con 0x7fc578103b50 2024-04-21T08:37:30.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc574d99640 1 -- 172.21.15.94:0/4207840016 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc570034d60 con 0x7fc578103b50 2024-04-21T08:37:30.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc57d196640 1 -- 172.21.15.94:0/4207840016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5781a0000 con 0x7fc578103b50 2024-04-21T08:37:30.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc574d99640 1 -- 172.21.15.94:0/4207840016 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570039040 con 0x7fc578103b50 2024-04-21T08:37:30.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.527+0000 7fc57d196640 1 -- 172.21.15.94:0/4207840016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5781a0540 con 0x7fc578103b50 2024-04-21T08:37:30.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.528+0000 7fc57d196640 1 -- 172.21.15.94:0/4207840016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc578109340 con 0x7fc578103b50 2024-04-21T08:37:30.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.530+0000 7fc574d99640 1 -- 172.21.15.94:0/4207840016 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc570044020 con 0x7fc578103b50 2024-04-21T08:37:30.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.530+0000 7fc574d99640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc5640a3c30 0x7fc5640a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:30.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.530+0000 7fc576d9d640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc5640a3c30 0x7fc5640a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:30.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.530+0000 7fc574d99640 1 -- 172.21.15.94:0/4207840016 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7fc5700f2660 con 0x7fc578103b50 2024-04-21T08:37:30.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.531+0000 7fc576d9d640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc5640a3c30 0x7fc5640a60f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc578102ff0 tx=0x7fc568005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:30.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.534+0000 7fc574d99640 1 -- 172.21.15.94:0/4207840016 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc57004b020 con 0x7fc578103b50 2024-04-21T08:37:30.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2433349456' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:37:30.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:37:30.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: pgmap v282: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:30.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/657294423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:37:30.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:30.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:30.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:30.752+0000 7fc57d196640 1 -- 172.21.15.94:0/4207840016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc57802d5c0 con 0x7fc578103b50 2024-04-21T08:37:30.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2433349456' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:37:30.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:37:30.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: pgmap v282: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:30.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/657294423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:37:30.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2433349456' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: pgmap v282: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:30.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/657294423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:37:30.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:30.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:30.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:30.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:37:31.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.567+0000 7fc574d99640 1 -- 172.21.15.94:0/4207840016 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fc5700bf1f0 con 0x7fc578103b50 2024-04-21T08:37:31.567 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 2 and data pool 3 2024-04-21T08:37:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 -- 172.21.15.94:0/4207840016 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc5640a3c30 msgr2=0x7fc5640a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc5640a3c30 0x7fc5640a60f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc578102ff0 tx=0x7fc568005f90 comp rx=0 tx=0).stop 2024-04-21T08:37:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 -- 172.21.15.94:0/4207840016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc578103b50 msgr2=0x7fc57819f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc578103b50 0x7fc57819f700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc57002c700 tx=0x7fc5700093f0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 -- 172.21.15.94:0/4207840016 shutdown_connections 2024-04-21T08:37:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc5640a3c30 0x7fc5640a60f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc578103190 0x7fc578198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc578103b50 0x7fc57819f700 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 --2- 172.21.15.94:0/4207840016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc578101fa0 0x7fc578198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 -- 172.21.15.94:0/4207840016 >> 172.21.15.94:0/4207840016 conn(0x7fc5780fd530 msgr2=0x7fc578106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 -- 172.21.15.94:0/4207840016 shutdown_connections 2024-04-21T08:37:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.570+0000 7fc57d196640 1 -- 172.21.15.94:0/4207840016 wait complete. 2024-04-21T08:37:31.585 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-21T08:37:31.586 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-21T08:37:31.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:31 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/657294423' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:37:31.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:31 smithi111 ceph-mon[31493]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:37:31.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:31 smithi111 ceph-mon[31493]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:37:31.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:31 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4207840016' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:37:31.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:31 smithi111 ceph-mon[31493]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:37:31.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.697+0000 7f805e392640 1 -- 172.21.15.94:0/433335947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8058107cd0 msgr2=0x7f805810a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.697+0000 7f805e392640 1 --2- 172.21.15.94:0/433335947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8058107cd0 0x7f805810a0c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8050009ca0 tx=0x7f805002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.697+0000 7f805e392640 1 -- 172.21.15.94:0/433335947 shutdown_connections 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.697+0000 7f805e392640 1 --2- 172.21.15.94:0/433335947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8058107cd0 0x7f805810a0c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.697+0000 7f805e392640 1 --2- 172.21.15.94:0/433335947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80581053a0 0x7f8058107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.697+0000 7f805e392640 1 --2- 172.21.15.94:0/433335947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8058102a70 0x7f8058104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.697+0000 7f805e392640 1 -- 172.21.15.94:0/433335947 >> 172.21.15.94:0/433335947 conn(0x7f80580fc400 msgr2=0x7f80580fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.698+0000 7f805e392640 1 -- 172.21.15.94:0/433335947 shutdown_connections 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.698+0000 7f805e392640 1 -- 172.21.15.94:0/433335947 wait complete. 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.698+0000 7f805e392640 1 Processor -- start 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.698+0000 7f805e392640 1 -- start start 2024-04-21T08:37:31.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.698+0000 7f805e392640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8058102a70 0x7f8058198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:31.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.698+0000 7f805e392640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80581053a0 0x7f80581987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:31.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.698+0000 7f805e392640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8058107cd0 0x7f805819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:31.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.698+0000 7f805e392640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f805810dae0 con 0x7f8058107cd0 2024-04-21T08:37:31.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.698+0000 7f805e392640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f805810d960 con 0x7f8058102a70 2024-04-21T08:37:31.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805e392640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f805810dc60 con 0x7f80581053a0 2024-04-21T08:37:31.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805c92f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8058107cd0 0x7f805819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805c92f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8058107cd0 0x7f805819f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54162/0 (socket says 172.21.15.94:54162) 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805c92f640 1 -- 172.21.15.94:0/1255867510 learned_addr learned my addr 172.21.15.94:0/1255867510 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f80577fe640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80581053a0 0x7f80581987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f8057fff640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8058102a70 0x7f8058198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805c92f640 1 -- 172.21.15.94:0/1255867510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80581053a0 msgr2=0x7f80581987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805c92f640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80581053a0 0x7f80581987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805c92f640 1 -- 172.21.15.94:0/1255867510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8058102a70 msgr2=0x7f8058198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805c92f640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8058102a70 0x7f8058198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805c92f640 1 -- 172.21.15.94:0/1255867510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805819fe30 con 0x7f8058107cd0 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805c92f640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8058107cd0 0x7f805819f7c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f805002c700 tx=0x7f80500093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f80557fa640 1 -- 172.21.15.94:0/1255867510 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050041020 con 0x7f8058107cd0 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805e392640 1 -- 172.21.15.94:0/1255867510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80581a00c0 con 0x7f8058107cd0 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f80557fa640 1 -- 172.21.15.94:0/1255867510 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8050034d60 con 0x7f8058107cd0 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f80557fa640 1 -- 172.21.15.94:0/1255867510 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050039040 con 0x7f8058107cd0 2024-04-21T08:37:31.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.699+0000 7f805e392640 1 -- 172.21.15.94:0/1255867510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80581a05b0 con 0x7f8058107cd0 2024-04-21T08:37:31.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.701+0000 7f805e392640 1 -- 172.21.15.94:0/1255867510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80580fdef0 con 0x7f8058107cd0 2024-04-21T08:37:31.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.702+0000 7f80557fa640 1 -- 172.21.15.94:0/1255867510 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8050044020 con 0x7f8058107cd0 2024-04-21T08:37:31.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.702+0000 7f80557fa640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f80400a3bb0 0x7f80400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:31.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.702+0000 7f8057fff640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f80400a3bb0 0x7f80400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:31.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.702+0000 7f80557fa640 1 -- 172.21.15.94:0/1255867510 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f80500f21b0 con 0x7f8058107cd0 2024-04-21T08:37:31.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.703+0000 7f8057fff640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f80400a3bb0 0x7f80400a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8048005fd0 tx=0x7f804800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:31.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.707+0000 7f80557fa640 1 -- 172.21.15.94:0/1255867510 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f805004b020 con 0x7f8058107cd0 2024-04-21T08:37:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:31 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/657294423' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:37:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:31 smithi094 ceph-mon[27988]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:37:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:31 smithi094 ceph-mon[27988]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:37:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:31 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4207840016' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:37:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:31 smithi094 ceph-mon[27988]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:37:31.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:31 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/657294423' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:37:31.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:31 smithi173 ceph-mon[31428]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:37:31.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:31 smithi173 ceph-mon[31428]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:37:31.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:31 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4207840016' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:37:31.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:31 smithi173 ceph-mon[31428]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:37:31.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:31.893+0000 7f805e392640 1 -- 172.21.15.94:0/1255867510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f805802d770 con 0x7f8058107cd0 2024-04-21T08:37:32.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.432+0000 7f80557fa640 1 -- 172.21.15.94:0/1255867510 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f80500bece0 con 0x7f8058107cd0 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 -- 172.21.15.94:0/1255867510 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f80400a3bb0 msgr2=0x7f80400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f80400a3bb0 0x7f80400a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8048005fd0 tx=0x7f804800b390 comp rx=0 tx=0).stop 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 -- 172.21.15.94:0/1255867510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8058107cd0 msgr2=0x7f805819f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8058107cd0 0x7f805819f7c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f805002c700 tx=0x7f80500093f0 comp rx=0 tx=0).stop 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 -- 172.21.15.94:0/1255867510 shutdown_connections 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f80400a3bb0 0x7f80400a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80581053a0 0x7f80581987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8058107cd0 0x7f805819f7c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 --2- 172.21.15.94:0/1255867510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8058102a70 0x7f8058198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 -- 172.21.15.94:0/1255867510 >> 172.21.15.94:0/1255867510 conn(0x7f80580fc400 msgr2=0x7f80581034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 -- 172.21.15.94:0/1255867510 shutdown_connections 2024-04-21T08:37:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.435+0000 7f805e392640 1 -- 172.21.15.94:0/1255867510 wait complete. 2024-04-21T08:37:32.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-21T08:37:32.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.555+0000 7feb01a74640 1 -- 172.21.15.94:0/1666593635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feafc101fa0 msgr2=0x7feafc1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:32.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.555+0000 7feb01a74640 1 --2- 172.21.15.94:0/1666593635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feafc101fa0 0x7feafc1023c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7feaec009900 tx=0x7feaec02bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:32.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.555+0000 7feb01a74640 1 -- 172.21.15.94:0/1666593635 shutdown_connections 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.555+0000 7feb01a74640 1 --2- 172.21.15.94:0/1666593635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feafc103190 0x7feafc103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.555+0000 7feb01a74640 1 --2- 172.21.15.94:0/1666593635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feafc103b50 0x7feafc10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.555+0000 7feb01a74640 1 --2- 172.21.15.94:0/1666593635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feafc101fa0 0x7feafc1023c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.555+0000 7feb01a74640 1 -- 172.21.15.94:0/1666593635 >> 172.21.15.94:0/1666593635 conn(0x7feafc0fd530 msgr2=0x7feafc0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.555+0000 7feb01a74640 1 -- 172.21.15.94:0/1666593635 shutdown_connections 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.555+0000 7feb01a74640 1 -- 172.21.15.94:0/1666593635 wait complete. 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feb01a74640 1 Processor -- start 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feb01a74640 1 -- start start 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feb01a74640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feafc101fa0 0x7feafc198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feb01a74640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feafc103190 0x7feafc1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feb01a74640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feafc103b50 0x7feafc19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feb01a74640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7feafc10db40 con 0x7feafc103190 2024-04-21T08:37:32.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feb01a74640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feafc10d9c0 con 0x7feafc101fa0 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feb01a74640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feafc10dcc0 con 0x7feafc103b50 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafaffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feafc103190 0x7feafc1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafaffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feafc103190 0x7feafc1987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54184/0 (socket says 172.21.15.94:54184) 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafaffd640 1 -- 172.21.15.94:0/543591152 learned_addr learned my addr 172.21.15.94:0/543591152 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafb7fe640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feafc101fa0 0x7feafc198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafbfff640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feafc103b50 0x7feafc19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafaffd640 1 -- 172.21.15.94:0/543591152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feafc103b50 msgr2=0x7feafc19f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafaffd640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feafc103b50 0x7feafc19f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafaffd640 1 -- 172.21.15.94:0/543591152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feafc101fa0 msgr2=0x7feafc198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafaffd640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feafc101fa0 0x7feafc198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafaffd640 1 -- 172.21.15.94:0/543591152 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feafc19fe80 con 0x7feafc103190 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafbfff640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feafc103b50 0x7feafc19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.556+0000 7feafb7fe640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feafc101fa0 0x7feafc198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:32.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.557+0000 7feafaffd640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feafc103190 0x7feafc1987a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7feaf000d770 tx=0x7feaf000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:32.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.557+0000 7feaf8ff9640 1 -- 172.21.15.94:0/543591152 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf0017070 con 0x7feafc103190 2024-04-21T08:37:32.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.557+0000 7feaf8ff9640 1 -- 172.21.15.94:0/543591152 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaf00054d0 con 0x7feafc103190 2024-04-21T08:37:32.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.557+0000 7feb01a74640 1 -- 172.21.15.94:0/543591152 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feafc1a0170 con 0x7feafc103190 2024-04-21T08:37:32.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.557+0000 7feaf8ff9640 1 -- 172.21.15.94:0/543591152 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf0004c30 con 0x7feafc103190 2024-04-21T08:37:32.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.557+0000 7feb01a74640 1 -- 172.21.15.94:0/543591152 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feafc1a06f0 con 0x7feafc103190 2024-04-21T08:37:32.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.558+0000 7feb01a74640 1 -- 172.21.15.94:0/543591152 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feafc109390 con 0x7feafc103190 2024-04-21T08:37:32.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.559+0000 7feaf8ff9640 1 -- 172.21.15.94:0/543591152 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7feaf0004e10 con 0x7feafc103190 2024-04-21T08:37:32.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.559+0000 7feaf8ff9640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7feae40a3bb0 0x7feae40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:32.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.559+0000 7feafb7fe640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7feae40a3bb0 0x7feae40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:32.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.559+0000 7feaf8ff9640 1 -- 172.21.15.94:0/543591152 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7feaf00cffa0 con 0x7feafc103190 2024-04-21T08:37:32.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.560+0000 7feafb7fe640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7feae40a3bb0 0x7feae40a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feaec0096c0 tx=0x7feaec000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:32.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.564+0000 7feaf8ff9640 1 -- 172.21.15.94:0/543591152 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feaf009c950 con 0x7feafc103190 2024-04-21T08:37:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:32 smithi111 ceph-mon[31493]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:37:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:32 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4207840016' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:37:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:32 smithi111 ceph-mon[31493]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:37:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:32 smithi111 ceph-mon[31493]: pgmap v285: 97 pgs: 32 creating+peering, 59 unknown, 6 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:32 smithi111 ceph-mon[31493]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:37:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:32 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] up:active 2024-04-21T08:37:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:32 smithi111 ceph-mon[31493]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:32 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1255867510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:37:32.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:32.743+0000 7feb01a74640 1 -- 172.21.15.94:0/543591152 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7feafc062b50 con 0x7feafc103190 2024-04-21T08:37:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:32 smithi094 ceph-mon[27988]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:37:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:32 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4207840016' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:37:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:32 smithi094 ceph-mon[27988]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:37:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:32 smithi094 ceph-mon[27988]: pgmap v285: 97 pgs: 32 creating+peering, 59 unknown, 6 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:32 smithi094 ceph-mon[27988]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:37:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:32 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] up:active 2024-04-21T08:37:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:32 smithi094 ceph-mon[27988]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:32.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:32 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1255867510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:37:32.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:32 smithi173 ceph-mon[31428]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:37:32.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:32 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4207840016' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:37:32.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:32 smithi173 ceph-mon[31428]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:37:32.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:32 smithi173 ceph-mon[31428]: pgmap v285: 97 pgs: 32 creating+peering, 59 unknown, 6 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:37:32.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:32 smithi173 ceph-mon[31428]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:37:32.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:32 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] up:active 2024-04-21T08:37:32.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:32 smithi173 ceph-mon[31428]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:32.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:32 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1255867510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:37:33.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:33 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1255867510' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:37:33.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:33 smithi111 ceph-mon[31493]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:37:33.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:33 smithi111 ceph-mon[31493]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:37:33.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:33 smithi111 ceph-mon[31493]: Cluster is now healthy 2024-04-21T08:37:33.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:33 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/543591152' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:37:33.759 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:33 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1255867510' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:37:33.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:33 smithi094 ceph-mon[27988]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:37:33.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:33 smithi094 ceph-mon[27988]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:37:33.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:33 smithi094 ceph-mon[27988]: Cluster is now healthy 2024-04-21T08:37:33.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:33 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/543591152' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:37:33.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:33 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1255867510' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:37:33.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:33 smithi173 ceph-mon[31428]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:37:33.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:33 smithi173 ceph-mon[31428]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:37:33.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:33 smithi173 ceph-mon[31428]: Cluster is now healthy 2024-04-21T08:37:33.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:33 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/543591152' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:37:34.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.576+0000 7feaf8ff9640 1 -- 172.21.15.94:0/543591152 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7feaf009c250 con 0x7feafc103190 2024-04-21T08:37:34.576 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data_ec' created 2024-04-21T08:37:34.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 -- 172.21.15.94:0/543591152 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7feae40a3bb0 msgr2=0x7feae40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7feae40a3bb0 0x7feae40a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feaec0096c0 tx=0x7feaec000bc0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 -- 172.21.15.94:0/543591152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feafc103190 msgr2=0x7feafc1987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feafc103190 0x7feafc1987a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7feaf000d770 tx=0x7feaf000dc30 comp rx=0 tx=0).stop 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 -- 172.21.15.94:0/543591152 shutdown_connections 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7feae40a3bb0 0x7feae40a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feafc103b50 0x7feafc19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feafc103190 0x7feafc1987a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 --2- 172.21.15.94:0/543591152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feafc101fa0 0x7feafc198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 -- 172.21.15.94:0/543591152 >> 172.21.15.94:0/543591152 conn(0x7feafc0fd530 msgr2=0x7feafc106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 -- 172.21.15.94:0/543591152 shutdown_connections 2024-04-21T08:37:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.579+0000 7feb01a74640 1 -- 172.21.15.94:0/543591152 wait complete. 2024-04-21T08:37:34.596 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-21T08:37:34.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.700+0000 7f1a0993d640 1 -- 172.21.15.94:0/3358499557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a04107d50 msgr2=0x7f1a0410a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:34.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.700+0000 7f1a0993d640 1 --2- 172.21.15.94:0/3358499557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a04107d50 0x7f1a0410a140 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009ca0 tx=0x7f19f402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.700+0000 7f1a0993d640 1 -- 172.21.15.94:0/3358499557 shutdown_connections 2024-04-21T08:37:34.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.700+0000 7f1a0993d640 1 --2- 172.21.15.94:0/3358499557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a04105420 0x7f1a04107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.700+0000 7f1a0993d640 1 --2- 172.21.15.94:0/3358499557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a04107d50 0x7f1a0410a140 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.700+0000 7f1a0993d640 1 --2- 172.21.15.94:0/3358499557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04102af0 0x7f1a04104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.700+0000 7f1a0993d640 1 -- 172.21.15.94:0/3358499557 >> 172.21.15.94:0/3358499557 conn(0x7f1a040fc420 msgr2=0x7f1a040fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.700+0000 7f1a0993d640 1 -- 172.21.15.94:0/3358499557 shutdown_connections 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.700+0000 7f1a0993d640 1 -- 172.21.15.94:0/3358499557 wait complete. 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a0993d640 1 Processor -- start 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a0993d640 1 -- start start 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a0993d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04102af0 0x7f1a04198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a0993d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a04105420 0x7f1a04198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a0993d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a04107d50 0x7f1a0419f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a0993d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0410dbe0 con 0x7f1a04105420 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a0993d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0410da60 con 0x7f1a04102af0 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a0993d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0410dd60 con 0x7f1a04107d50 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a027fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a04105420 0x7f1a04198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a027fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a04105420 0x7f1a04198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48660/0 (socket says 172.21.15.94:48660) 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a027fc640 1 -- 172.21.15.94:0/3847426326 learned_addr learned my addr 172.21.15.94:0/3847426326 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:34.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a037fe640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a04107d50 0x7f1a0419f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a02ffd640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04102af0 0x7f1a04198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a027fc640 1 -- 172.21.15.94:0/3847426326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a04107d50 msgr2=0x7f1a0419f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a027fc640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a04107d50 0x7f1a0419f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a027fc640 1 -- 172.21.15.94:0/3847426326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04102af0 msgr2=0x7f1a04198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a027fc640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04102af0 0x7f1a04198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a027fc640 1 -- 172.21.15.94:0/3847426326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a0419ff30 con 0x7f1a04105420 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.701+0000 7f1a02ffd640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04102af0 0x7f1a04198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.702+0000 7f1a027fc640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a04105420 0x7f1a04198850 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f19f800b450 tx=0x7f19f800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.702+0000 7f1a0893b640 1 -- 172.21.15.94:0/3847426326 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f8015020 con 0x7f1a04105420 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.702+0000 7f1a0993d640 1 -- 172.21.15.94:0/3847426326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a041a0220 con 0x7f1a04105420 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.702+0000 7f1a0893b640 1 -- 172.21.15.94:0/3847426326 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19f8011d40 con 0x7f1a04105420 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.702+0000 7f1a0893b640 1 -- 172.21.15.94:0/3847426326 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f800f340 con 0x7f1a04105420 2024-04-21T08:37:34.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.702+0000 7f1a0993d640 1 -- 172.21.15.94:0/3847426326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a041a07a0 con 0x7f1a04105420 2024-04-21T08:37:34.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.703+0000 7f1a0993d640 1 -- 172.21.15.94:0/3847426326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a040fdf10 con 0x7f1a04105420 2024-04-21T08:37:34.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.705+0000 7f1a0893b640 1 -- 172.21.15.94:0/3847426326 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f19f8018020 con 0x7f1a04105420 2024-04-21T08:37:34.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.705+0000 7f1a0893b640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f19ec0a3bb0 0x7f19ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:34.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.705+0000 7f1a02ffd640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f19ec0a3bb0 0x7f19ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:34.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.705+0000 7f1a0893b640 1 -- 172.21.15.94:0/3847426326 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f19f80d0780 con 0x7f1a04105420 2024-04-21T08:37:34.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.706+0000 7f1a02ffd640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f19ec0a3bb0 0x7f19ec0a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f19fc006fd0 tx=0x7f19fc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:34.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.709+0000 7f1a0893b640 1 -- 172.21.15.94:0/3847426326 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f19f801f020 con 0x7f1a04105420 2024-04-21T08:37:34.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:34.887+0000 7f1a0993d640 1 -- 172.21.15.94:0/3847426326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f1a0402d5c0 con 0x7f1a04105420 2024-04-21T08:37:34.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:34 smithi111 ceph-mon[31493]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:37:34.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:34 smithi111 ceph-mon[31493]: pgmap v288: 97 pgs: 32 creating+peering, 33 unknown, 32 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-04-21T08:37:35.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:34 smithi094 ceph-mon[27988]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:37:35.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:34 smithi094 ceph-mon[27988]: pgmap v288: 97 pgs: 32 creating+peering, 33 unknown, 32 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-04-21T08:37:35.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:34 smithi173 ceph-mon[31428]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:37:35.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:34 smithi173 ceph-mon[31428]: pgmap v288: 97 pgs: 32 creating+peering, 33 unknown, 32 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-04-21T08:37:35.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.582+0000 7f1a0893b640 1 -- 172.21.15.94:0/3847426326 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f19f800d040 con 0x7f1a04105420 2024-04-21T08:37:35.582 INFO:teuthology.orchestra.run.smithi094.stderr:set pool 4 allow_ec_overwrites to true 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 -- 172.21.15.94:0/3847426326 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f19ec0a3bb0 msgr2=0x7f19ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f19ec0a3bb0 0x7f19ec0a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f19fc006fd0 tx=0x7f19fc008040 comp rx=0 tx=0).stop 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 -- 172.21.15.94:0/3847426326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a04105420 msgr2=0x7f1a04198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a04105420 0x7f1a04198850 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f19f800b450 tx=0x7f19f800b910 comp rx=0 tx=0).stop 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 -- 172.21.15.94:0/3847426326 shutdown_connections 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f19ec0a3bb0 0x7f19ec0a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a04107d50 0x7f1a0419f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a04105420 0x7f1a04198850 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 --2- 172.21.15.94:0/3847426326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04102af0 0x7f1a04198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 -- 172.21.15.94:0/3847426326 >> 172.21.15.94:0/3847426326 conn(0x7f1a040fc420 msgr2=0x7f1a04103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.585+0000 7f1a0993d640 1 -- 172.21.15.94:0/3847426326 shutdown_connections 2024-04-21T08:37:35.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.586+0000 7f1a0993d640 1 -- 172.21.15.94:0/3847426326 wait complete. 2024-04-21T08:37:35.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-21T08:37:35.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.711+0000 7fedff36d640 1 -- 172.21.15.94:0/892879452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedf8105400 msgr2=0x7fedf81077f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:35.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.711+0000 7fedff36d640 1 --2- 172.21.15.94:0/892879452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedf8105400 0x7fedf81077f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fede8009900 tx=0x7fede802bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:35.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.711+0000 7fedff36d640 1 -- 172.21.15.94:0/892879452 shutdown_connections 2024-04-21T08:37:35.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.711+0000 7fedff36d640 1 --2- 172.21.15.94:0/892879452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedf8107d30 0x7fedf810a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.711+0000 7fedff36d640 1 --2- 172.21.15.94:0/892879452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedf8105400 0x7fedf81077f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.711+0000 7fedff36d640 1 --2- 172.21.15.94:0/892879452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedf8102ad0 0x7fedf8104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.711+0000 7fedff36d640 1 -- 172.21.15.94:0/892879452 >> 172.21.15.94:0/892879452 conn(0x7fedf80fc440 msgr2=0x7fedf80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.711+0000 7fedff36d640 1 -- 172.21.15.94:0/892879452 shutdown_connections 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.711+0000 7fedff36d640 1 -- 172.21.15.94:0/892879452 wait complete. 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedff36d640 1 Processor -- start 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedff36d640 1 -- start start 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedff36d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedf8102ad0 0x7fedf8198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedff36d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedf8105400 0x7fedf81987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedff36d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedf8107d30 0x7fedf819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedff36d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fedf810db10 con 0x7fedf8105400 2024-04-21T08:37:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedff36d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fedf810d990 con 0x7fedf8107d30 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedff36d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fedf810dc90 con 0x7fedf8102ad0 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfc908640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedf8105400 0x7fedf81987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfc908640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedf8105400 0x7fedf81987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48682/0 (socket says 172.21.15.94:48682) 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfc908640 1 -- 172.21.15.94:0/3218163304 learned_addr learned my addr 172.21.15.94:0/3218163304 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfd109640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedf8102ad0 0x7fedf8198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfd90a640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedf8107d30 0x7fedf819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfc908640 1 -- 172.21.15.94:0/3218163304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedf8102ad0 msgr2=0x7fedf8198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfc908640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedf8102ad0 0x7fedf8198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfc908640 1 -- 172.21.15.94:0/3218163304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedf8107d30 msgr2=0x7fedf819f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfc908640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedf8107d30 0x7fedf819f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfc908640 1 -- 172.21.15.94:0/3218163304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf819fe90 con 0x7fedf8105400 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfd90a640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedf8107d30 0x7fedf819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedfc908640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedf8105400 0x7fedf81987b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fede8000c00 tx=0x7fede8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.712+0000 7fedee7fc640 1 -- 172.21.15.94:0/3218163304 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8043020 con 0x7fedf8105400 2024-04-21T08:37:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.713+0000 7fedff36d640 1 -- 172.21.15.94:0/3218163304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf81a0120 con 0x7fedf8105400 2024-04-21T08:37:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.713+0000 7fedee7fc640 1 -- 172.21.15.94:0/3218163304 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fede8004aa0 con 0x7fedf8105400 2024-04-21T08:37:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.713+0000 7fedee7fc640 1 -- 172.21.15.94:0/3218163304 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8003d00 con 0x7fedf8105400 2024-04-21T08:37:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.713+0000 7fedff36d640 1 -- 172.21.15.94:0/3218163304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf81a06c0 con 0x7fedf8105400 2024-04-21T08:37:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.714+0000 7fedff36d640 1 -- 172.21.15.94:0/3218163304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf80fdf30 con 0x7fedf8105400 2024-04-21T08:37:35.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.715+0000 7fedee7fc640 1 -- 172.21.15.94:0/3218163304 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fede8046020 con 0x7fedf8105400 2024-04-21T08:37:35.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.715+0000 7fedee7fc640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fede40a3bb0 0x7fede40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:35.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.715+0000 7fedfd109640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fede40a3bb0 0x7fede40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:35.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.715+0000 7fedee7fc640 1 -- 172.21.15.94:0/3218163304 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7fede80f29e0 con 0x7fedf8105400 2024-04-21T08:37:35.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.716+0000 7fedfd109640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fede40a3bb0 0x7fede40a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fedf4005fd0 tx=0x7fedf400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.720+0000 7fedee7fc640 1 -- 172.21.15.94:0/3218163304 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fede804d020 con 0x7fedf8105400 2024-04-21T08:37:35.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:35 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/543591152' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:37:35.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:35 smithi111 ceph-mon[31493]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:37:35.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:35 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3847426326' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:37:35.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:35.938+0000 7fedff36d640 1 -- 172.21.15.94:0/3218163304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fedf802d5c0 con 0x7fedf8105400 2024-04-21T08:37:36.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:35 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/543591152' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:37:36.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:35 smithi094 ceph-mon[27988]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:37:36.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:35 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3847426326' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:37:36.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:35 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/543591152' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:37:36.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:35 smithi173 ceph-mon[31428]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:37:36.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:35 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3847426326' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:37:36.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.584+0000 7fedee7fc640 1 -- 172.21.15.94:0/3218163304 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7fede80bf1a0 con 0x7fedf8105400 2024-04-21T08:37:36.584 INFO:teuthology.orchestra.run.smithi094.stderr:added data pool 4 to fsmap 2024-04-21T08:37:36.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 -- 172.21.15.94:0/3218163304 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fede40a3bb0 msgr2=0x7fede40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fede40a3bb0 0x7fede40a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fedf4005fd0 tx=0x7fedf400b390 comp rx=0 tx=0).stop 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 -- 172.21.15.94:0/3218163304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedf8105400 msgr2=0x7fedf81987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedf8105400 0x7fedf81987b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fede8000c00 tx=0x7fede8004270 comp rx=0 tx=0).stop 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 -- 172.21.15.94:0/3218163304 shutdown_connections 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fede40a3bb0 0x7fede40a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedf8107d30 0x7fedf819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedf8105400 0x7fedf81987b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 --2- 172.21.15.94:0/3218163304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedf8102ad0 0x7fedf8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 -- 172.21.15.94:0/3218163304 >> 172.21.15.94:0/3218163304 conn(0x7fedf80fc440 msgr2=0x7fedf8104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 -- 172.21.15.94:0/3218163304 shutdown_connections 2024-04-21T08:37:36.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.587+0000 7fedff36d640 1 -- 172.21.15.94:0/3218163304 wait complete. 2024-04-21T08:37:36.599 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:36.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.699+0000 7f9df1c1f640 1 -- 172.21.15.94:0/2107272400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dec1053f0 msgr2=0x7f9dec1077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:36.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.699+0000 7f9df1c1f640 1 --2- 172.21.15.94:0/2107272400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dec1053f0 0x7f9dec1077e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009900 tx=0x7f9de002bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:36.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.699+0000 7f9df1c1f640 1 -- 172.21.15.94:0/2107272400 shutdown_connections 2024-04-21T08:37:36.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.699+0000 7f9df1c1f640 1 --2- 172.21.15.94:0/2107272400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dec107d20 0x7f9dec10a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.699+0000 7f9df1c1f640 1 --2- 172.21.15.94:0/2107272400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dec1053f0 0x7f9dec1077e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.699+0000 7f9df1c1f640 1 --2- 172.21.15.94:0/2107272400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dec102ac0 0x7f9dec104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.699+0000 7f9df1c1f640 1 -- 172.21.15.94:0/2107272400 >> 172.21.15.94:0/2107272400 conn(0x7f9dec0fc410 msgr2=0x7f9dec0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.699+0000 7f9df1c1f640 1 -- 172.21.15.94:0/2107272400 shutdown_connections 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.699+0000 7f9df1c1f640 1 -- 172.21.15.94:0/2107272400 wait complete. 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.699+0000 7f9df1c1f640 1 Processor -- start 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9df1c1f640 1 -- start start 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9df1c1f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dec102ac0 0x7f9dec198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9df1c1f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dec1053f0 0x7f9dec198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9df1c1f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dec107d20 0x7f9dec19f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9df1c1f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9dec10db90 con 0x7f9dec107d20 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9df1c1f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9dec10da10 con 0x7f9dec102ac0 2024-04-21T08:37:36.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9df1c1f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9dec10dd10 con 0x7f9dec1053f0 2024-04-21T08:37:36.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9debfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dec107d20 0x7f9dec19f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:36.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9debfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dec107d20 0x7f9dec19f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48708/0 (socket says 172.21.15.94:48708) 2024-04-21T08:37:36.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9debfff640 1 -- 172.21.15.94:0/378443369 learned_addr learned my addr 172.21.15.94:0/378443369 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:36.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9deaffd640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dec1053f0 0x7f9dec198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:36.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9deb7fe640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dec102ac0 0x7f9dec198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:36.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9debfff640 1 -- 172.21.15.94:0/378443369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dec1053f0 msgr2=0x7f9dec198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:36.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9debfff640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dec1053f0 0x7f9dec198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9debfff640 1 -- 172.21.15.94:0/378443369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dec102ac0 msgr2=0x7f9dec198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:36.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9debfff640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dec102ac0 0x7f9dec198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9debfff640 1 -- 172.21.15.94:0/378443369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dec19ff70 con 0x7f9dec107d20 2024-04-21T08:37:36.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9deaffd640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dec1053f0 0x7f9dec198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:36.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.700+0000 7f9debfff640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dec107d20 0x7f9dec19f900 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9de400b970 tx=0x7f9de400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:36.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.701+0000 7f9de8ff9640 1 -- 172.21.15.94:0/378443369 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4019070 con 0x7f9dec107d20 2024-04-21T08:37:36.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.701+0000 7f9df1c1f640 1 -- 172.21.15.94:0/378443369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dec1a0260 con 0x7f9dec107d20 2024-04-21T08:37:36.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.701+0000 7f9de8ff9640 1 -- 172.21.15.94:0/378443369 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9de4014440 con 0x7f9dec107d20 2024-04-21T08:37:36.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.701+0000 7f9de8ff9640 1 -- 172.21.15.94:0/378443369 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de40075d0 con 0x7f9dec107d20 2024-04-21T08:37:36.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.701+0000 7f9df1c1f640 1 -- 172.21.15.94:0/378443369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dec1a0880 con 0x7f9dec107d20 2024-04-21T08:37:36.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.702+0000 7f9df1c1f640 1 -- 172.21.15.94:0/378443369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dec0fdf00 con 0x7f9dec107d20 2024-04-21T08:37:36.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.703+0000 7f9de8ff9640 1 -- 172.21.15.94:0/378443369 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9de401c070 con 0x7f9dec107d20 2024-04-21T08:37:36.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.703+0000 7f9de8ff9640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9dd40a3bb0 0x7f9dd40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:36.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.703+0000 7f9deb7fe640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9dd40a3bb0 0x7f9dd40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:36.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.703+0000 7f9de8ff9640 1 -- 172.21.15.94:0/378443369 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9de40cfc70 con 0x7f9dec107d20 2024-04-21T08:37:36.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.704+0000 7f9deb7fe640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9dd40a3bb0 0x7f9dd40a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00a810 tx=0x7f9ddc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:36.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.708+0000 7f9de8ff9640 1 -- 172.21.15.94:0/378443369 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9de409c410 con 0x7f9dec107d20 2024-04-21T08:37:36.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.925+0000 7f9df1c1f640 1 -- 172.21.15.94:0/378443369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9dec062b50 con 0x7f9dec107d20 2024-04-21T08:37:36.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.926+0000 7f9de8ff9640 1 -- 172.21.15.94:0/378443369 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11812 (secure 0 0 0) 0x7f9de409bd10 con 0x7f9dec107d20 2024-04-21T08:37:36.926 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:36.926 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:37:35.940006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24488,"qdb_cluster":[24488]},"id":1}]} 2024-04-21T08:37:36.926 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 16 2024-04-21T08:37:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 -- 172.21.15.94:0/378443369 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9dd40a3bb0 msgr2=0x7f9dd40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9dd40a3bb0 0x7f9dd40a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00a810 tx=0x7f9ddc008040 comp rx=0 tx=0).stop 2024-04-21T08:37:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 -- 172.21.15.94:0/378443369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dec107d20 msgr2=0x7f9dec19f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dec107d20 0x7f9dec19f900 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9de400b970 tx=0x7f9de400be30 comp rx=0 tx=0).stop 2024-04-21T08:37:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 -- 172.21.15.94:0/378443369 shutdown_connections 2024-04-21T08:37:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9dd40a3bb0 0x7f9dd40a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dec1053f0 0x7f9dec198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dec107d20 0x7f9dec19f900 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 --2- 172.21.15.94:0/378443369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dec102ac0 0x7f9dec198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 -- 172.21.15.94:0/378443369 >> 172.21.15.94:0/378443369 conn(0x7f9dec0fc410 msgr2=0x7f9dec103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:36.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 -- 172.21.15.94:0/378443369 shutdown_connections 2024-04-21T08:37:36.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:36.928+0000 7f9df1c1f640 1 -- 172.21.15.94:0/378443369 wait complete. 2024-04-21T08:37:36.937 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:36 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b[61988]: 2024-04-21T08:37:36.683+0000 7f081fb22700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-21T08:37:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:36 smithi111 ceph-mon[31493]: pgmap v290: 129 pgs: 28 creating+peering, 16 unknown, 85 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 9 op/s 2024-04-21T08:37:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:36 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3847426326' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:37:36.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:36 smithi111 ceph-mon[31493]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:37:36.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:36 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3218163304' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:37:36.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:36 smithi111 ceph-mon[31493]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:37:36.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:36 smithi094 ceph-mon[27988]: pgmap v290: 129 pgs: 28 creating+peering, 16 unknown, 85 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 9 op/s 2024-04-21T08:37:36.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:36 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3847426326' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:37:36.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:36 smithi094 ceph-mon[27988]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:37:36.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:36 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3218163304' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:37:36.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:36 smithi094 ceph-mon[27988]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:37:36.940 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:37:37.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:36 smithi173 ceph-mon[31428]: pgmap v290: 129 pgs: 28 creating+peering, 16 unknown, 85 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 9 op/s 2024-04-21T08:37:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:36 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3847426326' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:37:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:36 smithi173 ceph-mon[31428]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:37:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:36 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3218163304' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:37:37.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:36 smithi173 ceph-mon[31428]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:37:37.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.038+0000 7f003046d640 1 -- 172.21.15.94:0/3948019359 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0028105320 msgr2=0x7f0028107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.038+0000 7f003046d640 1 --2- 172.21.15.94:0/3948019359 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0028105320 0x7f0028107710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0018009900 tx=0x7f001802bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:37.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 -- 172.21.15.94:0/3948019359 shutdown_connections 2024-04-21T08:37:37.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 --2- 172.21.15.94:0/3948019359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0028107c50 0x7f002810a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 --2- 172.21.15.94:0/3948019359 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0028105320 0x7f0028107710 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 --2- 172.21.15.94:0/3948019359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00281029f0 0x7f0028104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 -- 172.21.15.94:0/3948019359 >> 172.21.15.94:0/3948019359 conn(0x7f00280fc3e0 msgr2=0x7f00280fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:37.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 -- 172.21.15.94:0/3948019359 shutdown_connections 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 -- 172.21.15.94:0/3948019359 wait complete. 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 Processor -- start 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 -- start start 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00281029f0 0x7f0028102360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0028105320 0x7f00281009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0028107c50 0x7f0028100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f002810da80 con 0x7f0028107c50 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f002810d900 con 0x7f00281029f0 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f003046d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f002810dc00 con 0x7f0028105320 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f002ea0a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0028107c50 0x7f0028100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f002ea0a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0028107c50 0x7f0028100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48724/0 (socket says 172.21.15.94:48724) 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.039+0000 7f002ea0a640 1 -- 172.21.15.94:0/3269774912 learned_addr learned my addr 172.21.15.94:0/3269774912 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f002e209640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00281029f0 0x7f0028102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f002da08640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0028105320 0x7f00281009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f002ea0a640 1 -- 172.21.15.94:0/3269774912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0028105320 msgr2=0x7f00281009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f002ea0a640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0028105320 0x7f00281009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f002ea0a640 1 -- 172.21.15.94:0/3269774912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00281029f0 msgr2=0x7f0028102360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f002ea0a640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00281029f0 0x7f0028102360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f002ea0a640 1 -- 172.21.15.94:0/3269774912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0028101770 con 0x7f0028107c50 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f002da08640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0028105320 0x7f00281009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f002e209640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00281029f0 0x7f0028102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f002ea0a640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0028107c50 0x7f0028100ee0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f002000eaa0 tx=0x7f002000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f001f7fe640 1 -- 172.21.15.94:0/3269774912 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002000cdb0 con 0x7f0028107c50 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f001f7fe640 1 -- 172.21.15.94:0/3269774912 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0020005020 con 0x7f0028107c50 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f003046d640 1 -- 172.21.15.94:0/3269774912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00281a22a0 con 0x7f0028107c50 2024-04-21T08:37:37.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f001f7fe640 1 -- 172.21.15.94:0/3269774912 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00200116d0 con 0x7f0028107c50 2024-04-21T08:37:37.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.040+0000 7f003046d640 1 -- 172.21.15.94:0/3269774912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00281a27f0 con 0x7f0028107c50 2024-04-21T08:37:37.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.041+0000 7f003046d640 1 -- 172.21.15.94:0/3269774912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00280fded0 con 0x7f0028107c50 2024-04-21T08:37:37.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.042+0000 7f001f7fe640 1 -- 172.21.15.94:0/3269774912 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f00200051d0 con 0x7f0028107c50 2024-04-21T08:37:37.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.042+0000 7f001f7fe640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f00140a3c30 0x7f00140a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.043+0000 7f002e209640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f00140a3c30 0x7f00140a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.043+0000 7f001f7fe640 1 -- 172.21.15.94:0/3269774912 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f00200d0c20 con 0x7f0028107c50 2024-04-21T08:37:37.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.043+0000 7f002e209640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f00140a3c30 0x7f00140a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0024005d00 tx=0x7f0024005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:37.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.046+0000 7f001f7fe640 1 -- 172.21.15.94:0/3269774912 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0020016030 con 0x7f0028107c50 2024-04-21T08:37:37.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.222+0000 7f003046d640 1 -- 172.21.15.94:0/3269774912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0028062b50 con 0x7f0028107c50 2024-04-21T08:37:37.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.224+0000 7f001f7fe640 1 -- 172.21.15.94:0/3269774912 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24092 (secure 0 0 0) 0x7f002009d440 con 0x7f0028107c50 2024-04-21T08:37:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:37.224 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:37:35.939829+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 -- 172.21.15.94:0/3269774912 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f00140a3c30 msgr2=0x7f00140a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f00140a3c30 0x7f00140a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0024005d00 tx=0x7f0024005c70 comp rx=0 tx=0).stop 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 -- 172.21.15.94:0/3269774912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0028107c50 msgr2=0x7f0028100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0028107c50 0x7f0028100ee0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f002000eaa0 tx=0x7f002000ef60 comp rx=0 tx=0).stop 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 -- 172.21.15.94:0/3269774912 shutdown_connections 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f00140a3c30 0x7f00140a60f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0028105320 0x7f00281009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0028107c50 0x7f0028100ee0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 --2- 172.21.15.94:0/3269774912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00281029f0 0x7f0028102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 -- 172.21.15.94:0/3269774912 >> 172.21.15.94:0/3269774912 conn(0x7f00280fc3e0 msgr2=0x7f00281034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 -- 172.21.15.94:0/3269774912 shutdown_connections 2024-04-21T08:37:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.226+0000 7f003046d640 1 -- 172.21.15.94:0/3269774912 wait complete. 2024-04-21T08:37:37.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:37:37.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.336+0000 7fcbf0eba640 1 -- 172.21.15.94:0/2477162735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbec1029f0 msgr2=0x7fcbec104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.336+0000 7fcbf0eba640 1 --2- 172.21.15.94:0/2477162735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbec1029f0 0x7fcbec104de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009ca0 tx=0x7fcbdc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.336+0000 7fcbf0eba640 1 -- 172.21.15.94:0/2477162735 shutdown_connections 2024-04-21T08:37:37.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.336+0000 7fcbf0eba640 1 --2- 172.21.15.94:0/2477162735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbec107c50 0x7fcbec10a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.336+0000 7fcbf0eba640 1 --2- 172.21.15.94:0/2477162735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbec105320 0x7fcbec107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.336+0000 7fcbf0eba640 1 --2- 172.21.15.94:0/2477162735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbec1029f0 0x7fcbec104de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.336+0000 7fcbf0eba640 1 -- 172.21.15.94:0/2477162735 >> 172.21.15.94:0/2477162735 conn(0x7fcbec0fc3e0 msgr2=0x7fcbec0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbf0eba640 1 -- 172.21.15.94:0/2477162735 shutdown_connections 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbf0eba640 1 -- 172.21.15.94:0/2477162735 wait complete. 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbf0eba640 1 Processor -- start 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbf0eba640 1 -- start start 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbf0eba640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbec1029f0 0x7fcbec1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbf0eba640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbec105320 0x7fcbec198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbf0eba640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbec107c50 0x7fcbec19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbf0eba640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec10dac0 con 0x7fcbec107c50 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbf0eba640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec10d940 con 0x7fcbec105320 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbf0eba640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec10dc40 con 0x7fcbec1029f0 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbead9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbec107c50 0x7fcbec19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbead9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbec107c50 0x7fcbec19f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48732/0 (socket says 172.21.15.94:48732) 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.337+0000 7fcbead9d640 1 -- 172.21.15.94:0/747370948 learned_addr learned my addr 172.21.15.94:0/747370948 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbea59c640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbec1029f0 0x7fcbec1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbe9d9b640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbec105320 0x7fcbec198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbead9d640 1 -- 172.21.15.94:0/747370948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbec1029f0 msgr2=0x7fcbec1981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbead9d640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbec1029f0 0x7fcbec1981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbead9d640 1 -- 172.21.15.94:0/747370948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbec105320 msgr2=0x7fcbec198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbead9d640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbec105320 0x7fcbec198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbead9d640 1 -- 172.21.15.94:0/747370948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbec19fde0 con 0x7fcbec107c50 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbe9d9b640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbec105320 0x7fcbec198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbead9d640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbec107c50 0x7fcbec19f770 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400d750 tx=0x7fcbe400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbdb7fe640 1 -- 172.21.15.94:0/747370948 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4017070 con 0x7fcbec107c50 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbdb7fe640 1 -- 172.21.15.94:0/747370948 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbe40054d0 con 0x7fcbec107c50 2024-04-21T08:37:37.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbf0eba640 1 -- 172.21.15.94:0/747370948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbec1a00d0 con 0x7fcbec107c50 2024-04-21T08:37:37.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbdb7fe640 1 -- 172.21.15.94:0/747370948 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4004c60 con 0x7fcbec107c50 2024-04-21T08:37:37.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.338+0000 7fcbf0eba640 1 -- 172.21.15.94:0/747370948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbec1a0620 con 0x7fcbec107c50 2024-04-21T08:37:37.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.339+0000 7fcbf0eba640 1 -- 172.21.15.94:0/747370948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbec0fded0 con 0x7fcbec107c50 2024-04-21T08:37:37.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.340+0000 7fcbdb7fe640 1 -- 172.21.15.94:0/747370948 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcbe4009bd0 con 0x7fcbec107c50 2024-04-21T08:37:37.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.341+0000 7fcbdb7fe640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcbd40a3bb0 0x7fcbd40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.341+0000 7fcbea59c640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcbd40a3bb0 0x7fcbd40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.341+0000 7fcbdb7fe640 1 -- 172.21.15.94:0/747370948 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcbe40d0430 con 0x7fcbec107c50 2024-04-21T08:37:37.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.341+0000 7fcbea59c640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcbd40a3bb0 0x7fcbd40a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc006010 tx=0x7fcbdc032340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:37.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.344+0000 7fcbdb7fe640 1 -- 172.21.15.94:0/747370948 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcbe409cbd0 con 0x7fcbec107c50 2024-04-21T08:37:37.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.525+0000 7fcbf0eba640 1 -- 172.21.15.94:0/747370948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcbec062b50 con 0x7fcbec107c50 2024-04-21T08:37:37.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.527+0000 7fcbdb7fe640 1 -- 172.21.15.94:0/747370948 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24092 (secure 0 0 0) 0x7fcbe40cf020 con 0x7fcbec107c50 2024-04-21T08:37:37.527 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:37.527 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:37:35.939829+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 -- 172.21.15.94:0/747370948 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcbd40a3bb0 msgr2=0x7fcbd40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcbd40a3bb0 0x7fcbd40a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc006010 tx=0x7fcbdc032340 comp rx=0 tx=0).stop 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 -- 172.21.15.94:0/747370948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbec107c50 msgr2=0x7fcbec19f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbec107c50 0x7fcbec19f770 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400d750 tx=0x7fcbe400dc10 comp rx=0 tx=0).stop 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 -- 172.21.15.94:0/747370948 shutdown_connections 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcbd40a3bb0 0x7fcbd40a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbec107c50 0x7fcbec19f770 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbec105320 0x7fcbec198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 --2- 172.21.15.94:0/747370948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbec1029f0 0x7fcbec1981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 -- 172.21.15.94:0/747370948 >> 172.21.15.94:0/747370948 conn(0x7fcbec0fc3e0 msgr2=0x7fcbec1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 -- 172.21.15.94:0/747370948 shutdown_connections 2024-04-21T08:37:37.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.529+0000 7fcbf0eba640 1 -- 172.21.15.94:0/747370948 wait complete. 2024-04-21T08:37:37.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 -- 172.21.15.94:0/1178711031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9834100e70 msgr2=0x7f9834101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 --2- 172.21.15.94:0/1178711031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9834100e70 0x7f9834101290 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9824009ca0 tx=0x7f982402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 -- 172.21.15.94:0/1178711031 shutdown_connections 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 --2- 172.21.15.94:0/1178711031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9834101fc0 0x7f9834102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 --2- 172.21.15.94:0/1178711031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9834102980 0x7f98341092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 --2- 172.21.15.94:0/1178711031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9834100e70 0x7f9834101290 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 -- 172.21.15.94:0/1178711031 >> 172.21.15.94:0/1178711031 conn(0x7f98340fc3e0 msgr2=0x7f98340fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 -- 172.21.15.94:0/1178711031 shutdown_connections 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 -- 172.21.15.94:0/1178711031 wait complete. 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 Processor -- start 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 -- start start 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9834100e70 0x7f9834197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.656+0000 7f9839855640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9834101fc0 0x7f98341975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9839855640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9834102980 0x7f983419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9839855640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f983410c9c0 con 0x7f9834101fc0 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9839855640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f983410c840 con 0x7f9834100e70 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9839855640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f983410cb40 con 0x7f9834102980 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f98327fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9834101fc0 0x7f98341975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f98327fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9834101fc0 0x7f98341975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48750/0 (socket says 172.21.15.94:48750) 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f98327fc640 1 -- 172.21.15.94:0/1809006200 learned_addr learned my addr 172.21.15.94:0/1809006200 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f98327fc640 1 -- 172.21.15.94:0/1809006200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9834102980 msgr2=0x7f983419e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f98327fc640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9834102980 0x7f983419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9832ffd640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9834100e70 0x7f9834197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f98327fc640 1 -- 172.21.15.94:0/1809006200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9834100e70 msgr2=0x7f9834197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f98327fc640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9834100e70 0x7f9834197080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f98327fc640 1 -- 172.21.15.94:0/1809006200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983419eca0 con 0x7f9834101fc0 2024-04-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9832ffd640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9834100e70 0x7f9834197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:37.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f98327fc640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9834101fc0 0x7f98341975c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f982800d950 tx=0x7f982800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:37.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9838853640 1 -- 172.21.15.94:0/1809006200 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828017070 con 0x7f9834101fc0 2024-04-21T08:37:37.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9838853640 1 -- 172.21.15.94:0/1809006200 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9828009c30 con 0x7f9834101fc0 2024-04-21T08:37:37.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9839855640 1 -- 172.21.15.94:0/1809006200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f983419ef90 con 0x7f9834101fc0 2024-04-21T08:37:37.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9838853640 1 -- 172.21.15.94:0/1809006200 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982800f620 con 0x7f9834101fc0 2024-04-21T08:37:37.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.657+0000 7f9839855640 1 -- 172.21.15.94:0/1809006200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f983419f400 con 0x7f9834101fc0 2024-04-21T08:37:37.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.658+0000 7f9839855640 1 -- 172.21.15.94:0/1809006200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9834108250 con 0x7f9834101fc0 2024-04-21T08:37:37.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.659+0000 7f9838853640 1 -- 172.21.15.94:0/1809006200 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f982801a070 con 0x7f9834101fc0 2024-04-21T08:37:37.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.660+0000 7f9838853640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f981c0a3bb0 0x7f981c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.660+0000 7f9832ffd640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f981c0a3bb0 0x7f981c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.660+0000 7f9838853640 1 -- 172.21.15.94:0/1809006200 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f98280d0170 con 0x7f9834101fc0 2024-04-21T08:37:37.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.661+0000 7f9832ffd640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f981c0a3bb0 0x7f981c0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9824037040 tx=0x7f9824005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:37.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.665+0000 7f9838853640 1 -- 172.21.15.94:0/1809006200 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9828009de0 con 0x7f9834101fc0 2024-04-21T08:37:37.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.881+0000 7f9839855640 1 -- 172.21.15.94:0/1809006200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9834062bf0 con 0x7f9834101fc0 2024-04-21T08:37:37.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.881+0000 7f9838853640 1 -- 172.21.15.94:0/1809006200 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11812 (secure 0 0 0) 0x7f982809c910 con 0x7f9834101fc0 2024-04-21T08:37:37.881 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:37.882 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24416,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:37:35.940006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24488,"qdb_cluster":[24488]},"id":1}]} 2024-04-21T08:37:37.882 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 16 2024-04-21T08:37:37.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.883+0000 7f9839855640 1 -- 172.21.15.94:0/1809006200 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f981c0a3bb0 msgr2=0x7f981c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.883+0000 7f9839855640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f981c0a3bb0 0x7f981c0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9824037040 tx=0x7f9824005ef0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.883+0000 7f9839855640 1 -- 172.21.15.94:0/1809006200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9834101fc0 msgr2=0x7f98341975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.883+0000 7f9839855640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9834101fc0 0x7f98341975c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f982800d950 tx=0x7f982800de10 comp rx=0 tx=0).stop 2024-04-21T08:37:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.883+0000 7f9839855640 1 -- 172.21.15.94:0/1809006200 shutdown_connections 2024-04-21T08:37:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.883+0000 7f9839855640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f981c0a3bb0 0x7f981c0a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.883+0000 7f9839855640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9834102980 0x7f983419e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.883+0000 7f9839855640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9834101fc0 0x7f98341975c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.883+0000 7f9839855640 1 --2- 172.21.15.94:0/1809006200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9834100e70 0x7f9834197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.883+0000 7f9839855640 1 -- 172.21.15.94:0/1809006200 >> 172.21.15.94:0/1809006200 conn(0x7f98340fc3e0 msgr2=0x7f9834105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.884+0000 7f9839855640 1 -- 172.21.15.94:0/1809006200 shutdown_connections 2024-04-21T08:37:37.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.884+0000 7f9839855640 1 -- 172.21.15.94:0/1809006200 wait complete. 2024-04-21T08:37:37.893 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:37 smithi094 ceph-mon[27988]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:37:37.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:37 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3218163304' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:37:37.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:37 smithi094 ceph-mon[27988]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:37.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:37 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/378443369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:37.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:37 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3269774912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:37.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:37 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/747370948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:37.895 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:37:37.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:37 smithi111 ceph-mon[31493]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:37:37.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:37 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3218163304' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:37:37.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:37 smithi111 ceph-mon[31493]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:37.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:37 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/378443369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:37.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:37 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3269774912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:37.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:37 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/747370948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:37.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.993+0000 7f934ff2b640 1 -- 172.21.15.94:0/113970402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9348101fd0 msgr2=0x7f9348102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.993+0000 7f934ff2b640 1 --2- 172.21.15.94:0/113970402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9348101fd0 0x7f9348102450 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9338009900 tx=0x7f933802bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:37.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.993+0000 7f934ff2b640 1 -- 172.21.15.94:0/113970402 shutdown_connections 2024-04-21T08:37:37.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.993+0000 7f934ff2b640 1 --2- 172.21.15.94:0/113970402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9348102990 0x7f93481092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.993+0000 7f934ff2b640 1 --2- 172.21.15.94:0/113970402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9348101fd0 0x7f9348102450 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.993+0000 7f934ff2b640 1 --2- 172.21.15.94:0/113970402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9348100e80 0x7f93481012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.993+0000 7f934ff2b640 1 -- 172.21.15.94:0/113970402 >> 172.21.15.94:0/113970402 conn(0x7f93480fc410 msgr2=0x7f93480fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.993+0000 7f934ff2b640 1 -- 172.21.15.94:0/113970402 shutdown_connections 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.993+0000 7f934ff2b640 1 -- 172.21.15.94:0/113970402 wait complete. 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.993+0000 7f934ff2b640 1 Processor -- start 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934ff2b640 1 -- start start 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934ff2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9348100e80 0x7f934806b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934ff2b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9348101fd0 0x7f934819cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934ff2b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9348102990 0x7f934819f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934ff2b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f934810c9c0 con 0x7f9348100e80 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934ff2b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f934810c840 con 0x7f9348101fd0 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934ff2b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f934810cb40 con 0x7f9348102990 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934dcc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9348100e80 0x7f934806b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934dcc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9348100e80 0x7f934806b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48782/0 (socket says 172.21.15.94:48782) 2024-04-21T08:37:37.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934dcc7640 1 -- 172.21.15.94:0/1124120859 learned_addr learned my addr 172.21.15.94:0/1124120859 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934d4c6640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9348101fd0 0x7f934819cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934dcc7640 1 -- 172.21.15.94:0/1124120859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9348102990 msgr2=0x7f934819f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934e4c8640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9348102990 0x7f934819f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934dcc7640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9348102990 0x7f934819f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934dcc7640 1 -- 172.21.15.94:0/1124120859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9348101fd0 msgr2=0x7f934819cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934dcc7640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9348101fd0 0x7f934819cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934dcc7640 1 -- 172.21.15.94:0/1124120859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934819fd70 con 0x7f9348100e80 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934d4c6640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9348101fd0 0x7f934819cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934dcc7640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9348100e80 0x7f934806b130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f934400c920 tx=0x7f934400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f933effd640 1 -- 172.21.15.94:0/1124120859 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344007c70 con 0x7f9348100e80 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f933effd640 1 -- 172.21.15.94:0/1124120859 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9344007e10 con 0x7f9348100e80 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f933effd640 1 -- 172.21.15.94:0/1124120859 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344018820 con 0x7f9348100e80 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934ff2b640 1 -- 172.21.15.94:0/1124120859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93481a0060 con 0x7f9348100e80 2024-04-21T08:37:37.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.994+0000 7f934ff2b640 1 -- 172.21.15.94:0/1124120859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93481a05d0 con 0x7f9348100e80 2024-04-21T08:37:37.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.995+0000 7f934ff2b640 1 -- 172.21.15.94:0/1124120859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9348108130 con 0x7f9348100e80 2024-04-21T08:37:37.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.996+0000 7f933effd640 1 -- 172.21.15.94:0/1124120859 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f934400f450 con 0x7f9348100e80 2024-04-21T08:37:37.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.997+0000 7f933effd640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f93340a3c30 0x7f93340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:37.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.997+0000 7f934d4c6640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f93340a3c30 0x7f93340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:37.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.997+0000 7f933effd640 1 -- 172.21.15.94:0/1124120859 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f93440d07c0 con 0x7f9348100e80 2024-04-21T08:37:37.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:37.997+0000 7f934d4c6640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f93340a3c30 0x7f93340a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9338006010 tx=0x7f9338000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:38.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.001+0000 7f933effd640 1 -- 172.21.15.94:0/1124120859 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9344014030 con 0x7f9348100e80 2024-04-21T08:37:38.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:37 smithi173 ceph-mon[31428]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:37:38.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:37 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3218163304' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:37:38.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:37 smithi173 ceph-mon[31428]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:38.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:37 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/378443369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:38.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:37 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3269774912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:38.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:37 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/747370948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:38.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.181+0000 7f934ff2b640 1 -- 172.21.15.94:0/1124120859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9348062b80 con 0x7f9348100e80 2024-04-21T08:37:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.182+0000 7f933effd640 1 -- 172.21.15.94:0/1124120859 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24092 (secure 0 0 0) 0x7f934401f300 con 0x7f9348100e80 2024-04-21T08:37:38.182 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:38.182 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:37:35.939829+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:37:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 -- 172.21.15.94:0/1124120859 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f93340a3c30 msgr2=0x7f93340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f93340a3c30 0x7f93340a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9338006010 tx=0x7f9338000bc0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 -- 172.21.15.94:0/1124120859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9348100e80 msgr2=0x7f934806b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9348100e80 0x7f934806b130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f934400c920 tx=0x7f934400cde0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 -- 172.21.15.94:0/1124120859 shutdown_connections 2024-04-21T08:37:38.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f93340a3c30 0x7f93340a60f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9348102990 0x7f934819f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9348101fd0 0x7f934819cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 --2- 172.21.15.94:0/1124120859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9348100e80 0x7f934806b130 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 -- 172.21.15.94:0/1124120859 >> 172.21.15.94:0/1124120859 conn(0x7f93480fc410 msgr2=0x7f9348105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:38.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 -- 172.21.15.94:0/1124120859 shutdown_connections 2024-04-21T08:37:38.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.184+0000 7f934ff2b640 1 -- 172.21.15.94:0/1124120859 wait complete. 2024-04-21T08:37:38.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-21T08:37:30.753966+0000', 'modified': '2024-04-21T08:37:35.940006+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24488}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24488': {'gid': 24488, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3878872423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3878872423}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3878872423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24488, 'qdb_cluster': [24488]} 2024-04-21T08:37:38.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T08:37:38.196 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:37:38.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 -- 172.21.15.94:0/650764171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ae8103230 msgr2=0x7f0ae81036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 --2- 172.21.15.94:0/650764171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ae8103230 0x7f0ae81036b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009930 tx=0x7f0ad802bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:38.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 -- 172.21.15.94:0/650764171 shutdown_connections 2024-04-21T08:37:38.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 --2- 172.21.15.94:0/650764171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ae8103bf0 0x7f0ae810a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 --2- 172.21.15.94:0/650764171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ae8103230 0x7f0ae81036b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 --2- 172.21.15.94:0/650764171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ae8100260 0x7f0ae8100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 -- 172.21.15.94:0/650764171 >> 172.21.15.94:0/650764171 conn(0x7f0ae80fbb90 msgr2=0x7f0ae80fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:38.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 -- 172.21.15.94:0/650764171 shutdown_connections 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 -- 172.21.15.94:0/650764171 wait complete. 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 Processor -- start 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.297+0000 7f0aef406640 1 -- start start 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aef406640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ae8100260 0x7f0ae8198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aef406640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ae8103230 0x7f0ae81987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aef406640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ae8103bf0 0x7f0ae819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aef406640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae810db00 con 0x7f0ae8103bf0 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aef406640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae810d980 con 0x7f0ae8100260 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aef406640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae810dc80 con 0x7f0ae8103230 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed9a3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ae8103bf0 0x7f0ae819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed9a3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ae8103bf0 0x7f0ae819f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48798/0 (socket says 172.21.15.94:48798) 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed9a3640 1 -- 172.21.15.94:0/2890155170 learned_addr learned my addr 172.21.15.94:0/2890155170 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:38.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ae8100260 0x7f0ae8198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ae8103230 0x7f0ae81987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed9a3640 1 -- 172.21.15.94:0/2890155170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ae8103230 msgr2=0x7f0ae81987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed9a3640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ae8103230 0x7f0ae81987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed9a3640 1 -- 172.21.15.94:0/2890155170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ae8100260 msgr2=0x7f0ae8198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed9a3640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ae8100260 0x7f0ae8198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed9a3640 1 -- 172.21.15.94:0/2890155170 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae819fe30 con 0x7f0ae8103bf0 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ae8100260 0x7f0ae8198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ae8103230 0x7f0ae81987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aed9a3640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ae8103bf0 0x7f0ae819f7c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009fd0 tx=0x7f0ae000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0ade7fc640 1 -- 172.21.15.94:0/2890155170 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0017070 con 0x7f0ae8103bf0 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aef406640 1 -- 172.21.15.94:0/2890155170 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae81a0120 con 0x7f0ae8103bf0 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0ade7fc640 1 -- 172.21.15.94:0/2890155170 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ae0003e20 con 0x7f0ae8103bf0 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0ade7fc640 1 -- 172.21.15.94:0/2890155170 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0010460 con 0x7f0ae8103bf0 2024-04-21T08:37:38.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.298+0000 7f0aef406640 1 -- 172.21.15.94:0/2890155170 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae81a0620 con 0x7f0ae8103bf0 2024-04-21T08:37:38.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0ade7fc640 1 -- 172.21.15.94:0/2890155170 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0ae0003970 con 0x7f0ae8103bf0 2024-04-21T08:37:38.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0af1ffc740 1 Processor -- start 2024-04-21T08:37:38.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0af1ffc740 1 -- start start 2024-04-21T08:37:38.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2898e40 0x5635e2899260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2899830 0x5635e2899cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5635e27fdb10 con 0x5635e2899830 2024-04-21T08:37:38.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5635e279b240 con 0x5635e289a2d0 2024-04-21T08:37:38.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5635e27f9ad0 con 0x5635e2898e40 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0aed1a2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2899830 0x5635e2899cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0aed1a2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2899830 0x5635e2899cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48812/0 (socket says 172.21.15.94:48812) 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0aed1a2640 1 -- 172.21.15.94:0/533603731 learned_addr learned my addr 172.21.15.94:0/533603731 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.301+0000 7f0ade7fc640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ad40a3bb0 0x7f0ad40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/533603731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/533603731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2898e40 0x5635e2899260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aed9a3640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ad40a3bb0 0x7f0ad40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aed1a2640 1 -- 172.21.15.94:0/533603731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2898e40 msgr2=0x5635e2899260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/533603731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2898e40 0x5635e2899260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aed1a2640 1 -- 172.21.15.94:0/533603731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 msgr2=0x5635e28a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/533603731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aed1a2640 1 -- 172.21.15.94:0/533603731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635e25102d0 con 0x5635e2899830 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/533603731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:38.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0ade7fc640 1 -- 172.21.15.94:0/2890155170 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0ae00d04b0 con 0x7f0ae8103bf0 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/533603731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2899830 0x5635e2899cb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400bb70 tx=0x7f0ae400d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff24964df9813465 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0ad37fe640 1 -- 172.21.15.94:0/533603731 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4005220 con 0x5635e2899830 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0aed9a3640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ad40a3bb0 0x7f0ad40a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00cddc0 tx=0x7f0ae0066ba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0ad37fe640 1 -- 172.21.15.94:0/533603731 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ae4004c50 con 0x5635e2899830 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.302+0000 7f0ad37fe640 1 -- 172.21.15.94:0/533603731 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae400f7a0 con 0x5635e2899830 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 -- 172.21.15.94:0/533603731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2899830 msgr2=0x5635e2899cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/533603731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2899830 0x5635e2899cb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400bb70 tx=0x7f0ae400d710 comp rx=0 tx=0).stop 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 -- 172.21.15.94:0/533603731 shutdown_connections 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/533603731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/533603731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2899830 0x5635e2899cb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/533603731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2898e40 0x5635e2899260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 -- 172.21.15.94:0/533603731 >> 172.21.15.94:0/533603731 conn(0x5635e2894f40 msgr2=0x5635e2893490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 -- 172.21.15.94:0/533603731 shutdown_connections 2024-04-21T08:37:38.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 -- 172.21.15.94:0/533603731 wait complete. 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 -- start start 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 Processor -- start 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28a33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.303+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5635e28a14b0 con 0x5635e2898e40 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5635e2786190 con 0x5635e289a2d0 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5635e28a1630 con 0x5635e2899830 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0af1ffc740 10 client.0 _ll_register_callbacks cb 0x5635e27818b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0aec9a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0aec9a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48826/0 (socket says 172.21.15.94:48826) 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0aec9a1640 1 -- 172.21.15.94:0/1091463568 learned_addr learned my addr 172.21.15.94:0/1091463568 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28a33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0aec9a1640 1 -- 172.21.15.94:0/1091463568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 msgr2=0x5635e28a33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28a33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0aec9a1640 1 -- 172.21.15.94:0/1091463568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 msgr2=0x5635e28aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0aec9a1640 1 -- 172.21.15.94:0/1091463568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635e25da6b0 con 0x5635e2898e40 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a2eb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009900 tx=0x7f0ad8005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf436418c5d9ef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0ad0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/1091463568 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8045020 con 0x5635e2898e40 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/1091463568 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0ad8004410 con 0x5635e2898e40 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1091463568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ae4004c50 con 0x5635e2898e40 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/1091463568 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f0ad8050050 con 0x5635e2898e40 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0af1ffc740 10 client.14745 fetch_fsmap learned FSMap version 16 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.304+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1091463568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x5635e28ad860 con 0x5635e2898e40 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.305+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/1091463568 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ae4004c50 con 0x5635e2898e40 2024-04-21T08:37:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.305+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/1091463568 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7f0ad8041680 con 0x5635e2898e40 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.305+0000 7f0af1ffc740 10 client.14745 fetch_fsmap finished waiting for FSMap version 16 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.305+0000 7f0af1ffc740 10 client.14745 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.305+0000 7f0af1ffc740 20 client.14745 populate_metadata read hostname 'smithi094' 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.305+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635e28aff10 0x5635e28b1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.305+0000 7f0af1ffc740 4 client.14745 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.305+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1091463568 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635e280b530 con 0x5635e28aff10 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.305+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635e28aff10 0x5635e28b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0ad0ff9640 10 client.14745 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635e28aff10 0x5635e28b1380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcd3784d8bb3e1e7 server_cookie=2495b863b0a2e016 in_seq=0 out_seq=0 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/1091463568 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635e280b530 con 0x5635e28aff10 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0ad0ff9640 10 client.14745 handle_command_reply: tid=0 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0af1ffc740 1 client.14745 shutdown 2024-04-21T08:37:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1091463568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 msgr2=0x5635e28a2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a2eb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009900 tx=0x7f0ad8005e60 comp rx=0 tx=0).stop 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1091463568 shutdown_connections 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635e28aff10 0x5635e28b1380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28a33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0ad0ff9640 0 client.14745 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/1091463568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a2eb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.306+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1091463568 >> 172.21.15.94:0/1091463568 conn(0x5635e2894f40 msgr2=0x5635e2892a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.307+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1091463568 shutdown_connections 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.307+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1091463568 wait complete. 2024-04-21T08:37:38.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.307+0000 7f0af1ffc740 20 client.14745 trim_cache size 0 max 16384 2024-04-21T08:37:38.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.324+0000 7f0af1ffc740 1 Processor -- start 2024-04-21T08:37:38.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0af1ffc740 1 -- start start 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28aa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28a1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5635e28af6c0 con 0x5635e2898e40 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5635e28afd90 con 0x5635e289a2d0 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5635e28b0460 con 0x5635e2899830 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0aec9a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0aec9a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48842/0 (socket says 172.21.15.94:48842) 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0aec9a1640 1 -- 172.21.15.94:0/1494572581 learned_addr learned my addr 172.21.15.94:0/1494572581 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0aec9a1640 1 -- 172.21.15.94:0/1494572581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 msgr2=0x5635e28aa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/1494572581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28aa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0aec9a1640 1 -- 172.21.15.94:0/1494572581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 msgr2=0x5635e28a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/1494572581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0aec9a1640 1 -- 172.21.15.94:0/1494572581 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635e2894630 con 0x5635e2898e40 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/1494572581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a0e20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0ad802c170 tx=0x7f0ad8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c4edc6fe9ad98f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0ad2ffd640 1 -- 172.21.15.94:0/1494572581 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8059070 con 0x5635e2898e40 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0ad2ffd640 1 -- 172.21.15.94:0/1494572581 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ad802c8e0 con 0x5635e2898e40 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.325+0000 7f0ad2ffd640 1 -- 172.21.15.94:0/1494572581 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8045030 con 0x5635e2898e40 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1494572581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 msgr2=0x5635e28a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/1494572581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a0e20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0ad802c170 tx=0x7f0ad8005b50 comp rx=0 tx=0).stop 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1494572581 shutdown_connections 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/1494572581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28aa3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/1494572581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e289a2d0 0x5635e28a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/1494572581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e2898e40 0x5635e28a0e20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1494572581 >> 172.21.15.94:0/1494572581 conn(0x5635e2894f40 msgr2=0x5635e2893400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:38.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1494572581 shutdown_connections 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 -- 172.21.15.94:0/1494572581 wait complete. 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 -- start start 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.326+0000 7f0af1ffc740 1 Processor -- start 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e2898e40 0x5635e28a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28a6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0af1ffc740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e289a2d0 0x5635e28a92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5635e28b1200 con 0x5635e289a2d0 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5635e28b0b30 con 0x5635e2898e40 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0af1ffc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5635e28b18d0 con 0x5635e2899830 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0af1ffc740 10 client.0 _ll_register_callbacks cb 0x7f0ad8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aec9a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e289a2d0 0x5635e28a92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aec9a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e289a2d0 0x5635e28a92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48846/0 (socket says 172.21.15.94:48846) 2024-04-21T08:37:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aec9a1640 1 -- 172.21.15.94:0/212937504 learned_addr learned my addr 172.21.15.94:0/212937504 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e2898e40 0x5635e28a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aec9a1640 1 -- 172.21.15.94:0/212937504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 msgr2=0x5635e28a6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28a6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aec9a1640 1 -- 172.21.15.94:0/212937504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e2898e40 msgr2=0x5635e28a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e2898e40 0x5635e28a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aec9a1640 1 -- 172.21.15.94:0/212937504 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635e28a9890 con 0x5635e289a2d0 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28a6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0aec9a1640 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e289a2d0 0x5635e28a92a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8005c40 tx=0x7f0ad8009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168533902be43206 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0ad0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/212937504 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8045070 con 0x5635e289a2d0 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/212937504 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0ad8041650 con 0x5635e289a2d0 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0af1ffc740 1 -- 172.21.15.94:0/212937504 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5635e28053a0 con 0x5635e289a2d0 2024-04-21T08:37:38.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/212937504 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f0ad8059070 con 0x5635e289a2d0 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0af1ffc740 10 client.14751 fetch_fsmap learned FSMap version 16 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.327+0000 7f0af1ffc740 1 -- 172.21.15.94:0/212937504 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x5635e2916140 con 0x5635e289a2d0 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/212937504 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5635e28053a0 con 0x5635e289a2d0 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/212937504 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7f0ad80039b0 con 0x5635e289a2d0 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0af1ffc740 10 client.14751 fetch_fsmap finished waiting for FSMap version 16 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0af1ffc740 10 client.14751 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0af1ffc740 20 client.14751 populate_metadata read hostname 'smithi094' 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635e291a940 0x5635e2919650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0af1ffc740 4 client.14751 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0af1ffc740 1 -- 172.21.15.94:0/212937504 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "status"}) -- 0x5635e2781960 con 0x5635e291a940 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635e291a940 0x5635e2919650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0ad0ff9640 10 client.14751 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.328+0000 7f0aed1a2640 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635e291a940 0x5635e2919650 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ded66b1222ef159 server_cookie=76491ced9af1a6ba in_seq=0 out_seq=0 2024-04-21T08:37:38.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0ad0ff9640 1 -- 172.21.15.94:0/212937504 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5635e2781960 con 0x5635e291a940 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0ad0ff9640 10 client.14751 handle_command_reply: tid=0 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 client.14751 shutdown 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 -- 172.21.15.94:0/212937504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e289a2d0 msgr2=0x5635e28a92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e289a2d0 0x5635e28a92a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8005c40 tx=0x7f0ad8009680 comp rx=0 tx=0).stop 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 -- 172.21.15.94:0/212937504 shutdown_connections 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635e291a940 0x5635e2919650 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635e2899830 0x5635e28a6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635e289a2d0 0x5635e28a92a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 --2- 172.21.15.94:0/212937504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635e2898e40 0x5635e28a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0ad0ff9640 0 client.14751 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 -- 172.21.15.94:0/212937504 >> 172.21.15.94:0/212937504 conn(0x5635e2894f40 msgr2=0x5635e28a0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 -- 172.21.15.94:0/212937504 shutdown_connections 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 1 -- 172.21.15.94:0/212937504 wait complete. 2024-04-21T08:37:38.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0af1ffc740 20 client.14751 trim_cache size 0 max 16384 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24488, 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 6.7599812210000003, 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 16, 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 89, 2024-04-21T08:37:38.336 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:37:38.337 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 12.031800826 2024-04-21T08:37:38.337 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:37:38.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0aef406640 1 -- 172.21.15.94:0/2890155170 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ad40a3bb0 msgr2=0x7f0ad40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0aef406640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ad40a3bb0 0x7f0ad40a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00cddc0 tx=0x7f0ae0066ba0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0aef406640 1 -- 172.21.15.94:0/2890155170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ae8103bf0 msgr2=0x7f0ae819f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:38.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0aef406640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ae8103bf0 0x7f0ae819f7c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009fd0 tx=0x7f0ae000b260 comp rx=0 tx=0).stop 2024-04-21T08:37:38.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0aef406640 1 -- 172.21.15.94:0/2890155170 shutdown_connections 2024-04-21T08:37:38.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0aef406640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ad40a3bb0 0x7f0ad40a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0aef406640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ae8103230 0x7f0ae81987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0aef406640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ae8103bf0 0x7f0ae819f7c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0aef406640 1 --2- 172.21.15.94:0/2890155170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ae8100260 0x7f0ae8198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:38.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.329+0000 7f0aef406640 1 -- 172.21.15.94:0/2890155170 >> 172.21.15.94:0/2890155170 conn(0x7f0ae80fbb90 msgr2=0x7f0ae81066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:38.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.330+0000 7f0aef406640 1 -- 172.21.15.94:0/2890155170 shutdown_connections 2024-04-21T08:37:38.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:38.330+0000 7f0aef406640 1 -- 172.21.15.94:0/2890155170 wait complete. 2024-04-21T08:37:38.340 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T08:37:38.340 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:37:38.340 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph fs ls 2024-04-21T08:37:38.576 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:37:38.841 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:38 smithi094 ceph-mon[27988]: pgmap v293: 129 pgs: 1 creating+activating, 16 creating+peering, 5 unknown, 107 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-04-21T08:37:38.842 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:38 smithi094 ceph-mon[27988]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:37:38.842 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:38 smithi094 ceph-mon[27988]: Cluster is now healthy 2024-04-21T08:37:38.842 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:38 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1809006200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:38.842 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:38 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1124120859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:38.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:38 smithi111 ceph-mon[31493]: pgmap v293: 129 pgs: 1 creating+activating, 16 creating+peering, 5 unknown, 107 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-04-21T08:37:38.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:38 smithi111 ceph-mon[31493]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:37:38.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:38 smithi111 ceph-mon[31493]: Cluster is now healthy 2024-04-21T08:37:38.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:38 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1809006200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:38.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:38 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1124120859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:39.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:38 smithi173 ceph-mon[31428]: pgmap v293: 129 pgs: 1 creating+activating, 16 creating+peering, 5 unknown, 107 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-04-21T08:37:39.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:38 smithi173 ceph-mon[31428]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:37:39.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:38 smithi173 ceph-mon[31428]: Cluster is now healthy 2024-04-21T08:37:39.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:38 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1809006200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:37:39.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:38 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1124120859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:40.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.352+0000 7fc23ac57700 1 -- 172.21.15.94:0/2788495587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc234107980 msgr2=0x7fc234107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.352+0000 7fc23ac57700 1 --2- 172.21.15.94:0/2788495587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc234107980 0x7fc234107d60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc228009990 tx=0x7fc228009ca0 comp rx=0 tx=0).stop 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 -- 172.21.15.94:0/2788495587 shutdown_connections 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 --2- 172.21.15.94:0/2788495587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2341022f0 0x7fc23410e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 --2- 172.21.15.94:0/2788495587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc234107980 0x7fc234107d60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 --2- 172.21.15.94:0/2788495587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234101930 0x7fc234101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 -- 172.21.15.94:0/2788495587 >> 172.21.15.94:0/2788495587 conn(0x7fc2340788b0 msgr2=0x7fc234078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 -- 172.21.15.94:0/2788495587 shutdown_connections 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 -- 172.21.15.94:0/2788495587 wait complete. 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 Processor -- start 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 -- start start 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc234101930 0x7fc234199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2341022f0 0x7fc2341996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234107980 0x7fc23419da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc234111b60 con 0x7fc2341022f0 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341119e0 con 0x7fc234107980 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.353+0000 7fc23ac57700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc234111ce0 con 0x7fc234101930 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc233fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2341022f0 0x7fc2341996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc233fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2341022f0 0x7fc2341996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48870/0 (socket says 172.21.15.94:48870) 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc233fff700 1 -- 172.21.15.94:0/3075501567 learned_addr learned my addr 172.21.15.94:0/3075501567 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc2389f3700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc234101930 0x7fc234199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc2391f4700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234107980 0x7fc23419da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc233fff700 1 -- 172.21.15.94:0/3075501567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc234101930 msgr2=0x7fc234199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc233fff700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc234101930 0x7fc234199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc233fff700 1 -- 172.21.15.94:0/3075501567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234107980 msgr2=0x7fc23419da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:40.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc233fff700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234107980 0x7fc23419da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:40.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc233fff700 1 -- 172.21.15.94:0/3075501567 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23419e180 con 0x7fc2341022f0 2024-04-21T08:37:40.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc233fff700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2341022f0 0x7fc2341996b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc22000b530 tx=0x7fc22000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:40.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc231ffb700 1 -- 172.21.15.94:0/3075501567 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220015020 con 0x7fc2341022f0 2024-04-21T08:37:40.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc23ac57700 1 -- 172.21.15.94:0/3075501567 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc23419e470 con 0x7fc2341022f0 2024-04-21T08:37:40.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc231ffb700 1 -- 172.21.15.94:0/3075501567 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc220011ce0 con 0x7fc2341022f0 2024-04-21T08:37:40.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc231ffb700 1 -- 172.21.15.94:0/3075501567 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220010dd0 con 0x7fc2341022f0 2024-04-21T08:37:40.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.354+0000 7fc23ac57700 1 -- 172.21.15.94:0/3075501567 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2341a5cf0 con 0x7fc2341022f0 2024-04-21T08:37:40.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.355+0000 7fc23ac57700 1 -- 172.21.15.94:0/3075501567 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc234069730 con 0x7fc2341022f0 2024-04-21T08:37:40.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.356+0000 7fc231ffb700 1 -- 172.21.15.94:0/3075501567 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc220018020 con 0x7fc2341022f0 2024-04-21T08:37:40.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.357+0000 7fc231ffb700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc2240a3c40 0x7fc2240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:40.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.357+0000 7fc2389f3700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc2240a3c40 0x7fc2240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:40.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.357+0000 7fc231ffb700 1 -- 172.21.15.94:0/3075501567 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc2200cfd00 con 0x7fc2341022f0 2024-04-21T08:37:40.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.357+0000 7fc2389f3700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc2240a3c40 0x7fc2240a6100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc22800b5c0 tx=0x7fc228005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:40.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.361+0000 7fc231ffb700 1 -- 172.21.15.94:0/3075501567 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc22001f020 con 0x7fc2341022f0 2024-04-21T08:37:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.611+0000 7fc23ac57700 1 -- 172.21.15.94:0/3075501567 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc2341a5f20 con 0x7fc2341022f0 2024-04-21T08:37:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.611+0000 7fc231ffb700 1 -- 172.21.15.94:0/3075501567 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7fc22009be50 con 0x7fc2341022f0 2024-04-21T08:37:40.611 INFO:teuthology.orchestra.run.smithi094.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T08:37:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 -- 172.21.15.94:0/3075501567 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc2240a3c40 msgr2=0x7fc2240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc2240a3c40 0x7fc2240a6100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc22800b5c0 tx=0x7fc228005960 comp rx=0 tx=0).stop 2024-04-21T08:37:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 -- 172.21.15.94:0/3075501567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2341022f0 msgr2=0x7fc2341996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2341022f0 0x7fc2341996b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc22000b530 tx=0x7fc22000b840 comp rx=0 tx=0).stop 2024-04-21T08:37:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 -- 172.21.15.94:0/3075501567 shutdown_connections 2024-04-21T08:37:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc2240a3c40 0x7fc2240a6100 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234107980 0x7fc23419da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc234101930 0x7fc234199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 --2- 172.21.15.94:0/3075501567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2341022f0 0x7fc2341996b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 -- 172.21.15.94:0/3075501567 >> 172.21.15.94:0/3075501567 conn(0x7fc2340788b0 msgr2=0x7fc234105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 -- 172.21.15.94:0/3075501567 shutdown_connections 2024-04-21T08:37:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:40.614+0000 7fc23ac57700 1 -- 172.21.15.94:0/3075501567 wait complete. 2024-04-21T08:37:40.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:40 smithi111 ceph-mon[31493]: pgmap v294: 129 pgs: 1 creating+activating, 16 creating+peering, 112 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-21T08:37:40.995 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:40 smithi094 ceph-mon[27988]: pgmap v294: 129 pgs: 1 creating+activating, 16 creating+peering, 112 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-21T08:37:41.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:40 smithi173 ceph-mon[31428]: pgmap v294: 129 pgs: 1 creating+activating, 16 creating+peering, 112 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-21T08:37:41.352 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-21T08:37:41.353 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T08:37:41.353 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi094.front.sepia.ceph.com 2024-04-21T08:37:41.353 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-21T08:37:41.353 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:37:41.353 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-21T08:37:41.353 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T08:37:41.353 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T08:37:41.353 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T08:37:41.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:41.353 DEBUG:teuthology.orchestra.run.smithi094:> ip addr 2024-04-21T08:37:41.370 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T08:37:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T08:37:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T08:37:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4c brd ff:ff:ff:ff:ff:ff 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f0 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4d brd ff:ff:ff:ff:ff:ff 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f1 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:62 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:63 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: inet 172.21.15.94/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft 42194sec preferred_lft 42194sec 2024-04-21T08:37:41.371 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::cabc:3417:31e5:23d5/64 scope link noprefixroute 2024-04-21T08:37:41.372 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:37:41.372 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T08:37:41.372 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:37:41.372 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-21T08:37:41.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add name ceph-brx type bridge 2024-04-21T08:37:41.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr flush dev ceph-brx 2024-04-21T08:37:41.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set ceph-brx up 2024-04-21T08:37:41.373 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T08:37:41.373 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-21T08:37:41.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:37:41.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:37:41.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:41.552 DEBUG:teuthology.orchestra.run.smithi094:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T08:37:41.575 INFO:teuthology.orchestra.run.smithi094.stdout:1 2024-04-21T08:37:41.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:41.576 DEBUG:teuthology.orchestra.run.smithi094:> route 2024-04-21T08:37:41.647 INFO:teuthology.orchestra.run.smithi094.stdout:Kernel IP routing table 2024-04-21T08:37:41.647 INFO:teuthology.orchestra.run.smithi094.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T08:37:41.647 INFO:teuthology.orchestra.run.smithi094.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-21T08:37:41.647 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-21T08:37:41.647 INFO:teuthology.orchestra.run.smithi094.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T08:37:41.647 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:37:41.648 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-21T08:37:41.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-21T08:37:41.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-21T08:37:41.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-21T08:37:41.648 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-21T08:37:41.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:37:41.831 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:41 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3075501567' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:37:41.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:41 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3075501567' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:37:41.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:37:41.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:41.938 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-04-21T08:37:41.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:41.954 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list-id 2024-04-21T08:37:42.008 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:37:42.008 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-21T08:37:42.008 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:37:42.008 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-21T08:37:42.008 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-21T08:37:42.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:41 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3075501567' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:37:42.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:37:42.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:37:42.123 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-21T08:37:42.124 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:37:42.124 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-21T08:37:42.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-21T08:37:42.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T08:37:42.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-21T08:37:42.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-21T08:37:42.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-21T08:37:42.124 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-21T08:37:42.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:37:42.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:37:42.247 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:37:42.247 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-21T08:37:42.247 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set brx.0 up 2024-04-21T08:37:42.247 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T08:37:42.247 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-21T08:37:42.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:37:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:37:42.321 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-21T08:37:42.321 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:37:42.321 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:37:42.336 INFO:teuthology.orchestra.run.smithi094.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T08:37:42.336 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:37:42.336 DEBUG:teuthology.orchestra.run.smithi094:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:37:42.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo modprobe fuse 2024-04-21T08:37:42.456 DEBUG:teuthology.orchestra.run.smithi094:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T08:37:42.512 INFO:teuthology.orchestra.run.smithi094.stdout:/proc 2024-04-21T08:37:42.512 INFO:teuthology.orchestra.run.smithi094.stdout:/sys 2024-04-21T08:37:42.512 INFO:teuthology.orchestra.run.smithi094.stdout:/dev 2024-04-21T08:37:42.512 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/security 2024-04-21T08:37:42.512 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/shm 2024-04-21T08:37:42.512 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/pts 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/run 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/cgroup 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/pstore 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/bpf 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/ 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/selinux 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/mqueue 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/debug 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/hugepages 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/tracing 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/config 2024-04-21T08:37:42.513 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/fuse/connections 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/run/user/1000 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/b99249c2025ca746201f8c02ad2c26fbe574483f84cd34a04c3896cdb1ded182/merged 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/b70cc6eb0a815af975db75765bc60a678949b9deab84dc0e45c77fca41413945/merged 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/run/user/0 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/13f2f7fa62b402c1c2c2781f3f1dc51a47d88a05e3c3018ea8a2c8d3cf68bf32/merged 2024-04-21T08:37:42.514 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/cbb788570ad7dc629f5b1466bcb47bbc17ddc00bc35d5765ab9e2892e3a2c657/merged 2024-04-21T08:37:42.515 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/f3444ba9f1e92ced913d22b295637daf7ab546c7ca35e9c405ced206e35203f4/merged 2024-04-21T08:37:42.515 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/3e296cfa4e759f81fb4fc4a4714b8bd70bb4d218df1811ae3d874ca6f1fa50f4/merged 2024-04-21T08:37:42.515 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/01ef8418978330e6307f645403cb38bc723931ab480fc2ff19034f10e5b324d1/merged 2024-04-21T08:37:42.515 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/2ebe427004a60ed3c6cca82362a798d17b35838a8ff3e330445f71d219bbdea2/merged 2024-04-21T08:37:42.515 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/4842c6ca69add00dd13f36c8b2b9743d6677fd96109c9e70efb429e2a2d1e379/merged 2024-04-21T08:37:42.515 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/34b10c40db371f4ebd02a2e4ea5f0933296724621c50435e8ae7f1a97db8a6db/merged 2024-04-21T08:37:42.515 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns 2024-04-21T08:37:42.515 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:37:42.515 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:37:42.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:42.515 DEBUG:teuthology.orchestra.run.smithi094:> ls /sys/fs/fuse/connections 2024-04-21T08:37:42.567 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-21T08:37:42.567 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-21T08:37:42.610 DEBUG:teuthology.orchestra.run.smithi094:> sudo modprobe fuse 2024-04-21T08:37:42.633 DEBUG:teuthology.orchestra.run.smithi094:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/proc 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/sys 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/dev 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/security 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/shm 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/pts 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/run 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/cgroup 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/pstore 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/bpf 2024-04-21T08:37:42.689 INFO:teuthology.orchestra.run.smithi094.stdout:/ 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/selinux 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/mqueue 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/debug 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/hugepages 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/tracing 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/config 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/fuse/connections 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T08:37:42.690 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/run/user/1000 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/b99249c2025ca746201f8c02ad2c26fbe574483f84cd34a04c3896cdb1ded182/merged 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/b70cc6eb0a815af975db75765bc60a678949b9deab84dc0e45c77fca41413945/merged 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/run/user/0 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/13f2f7fa62b402c1c2c2781f3f1dc51a47d88a05e3c3018ea8a2c8d3cf68bf32/merged 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/cbb788570ad7dc629f5b1466bcb47bbc17ddc00bc35d5765ab9e2892e3a2c657/merged 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/f3444ba9f1e92ced913d22b295637daf7ab546c7ca35e9c405ced206e35203f4/merged 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/3e296cfa4e759f81fb4fc4a4714b8bd70bb4d218df1811ae3d874ca6f1fa50f4/merged 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/01ef8418978330e6307f645403cb38bc723931ab480fc2ff19034f10e5b324d1/merged 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/2ebe427004a60ed3c6cca82362a798d17b35838a8ff3e330445f71d219bbdea2/merged 2024-04-21T08:37:42.691 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/4842c6ca69add00dd13f36c8b2b9743d6677fd96109c9e70efb429e2a2d1e379/merged 2024-04-21T08:37:42.692 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/34b10c40db371f4ebd02a2e4ea5f0933296724621c50435e8ae7f1a97db8a6db/merged 2024-04-21T08:37:42.692 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns 2024-04-21T08:37:42.692 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:37:42.692 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:37:42.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:42.692 DEBUG:teuthology.orchestra.run.smithi094:> ls /sys/fs/fuse/connections 2024-04-21T08:37:42.711 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:2024-04-21T08:37:42.712+0000 7f8bb2243480 -1 init, newargv = 0x563b578c3ef0 newargc=15 2024-04-21T08:37:42.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:ceph-fuse[71596]: starting ceph client 2024-04-21T08:37:42.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:ceph-fuse[71596]: starting fuse 2024-04-21T08:37:42.726 INFO:teuthology.orchestra.run.smithi094.stdout:115 2024-04-21T08:37:42.726 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> sudo stdin-killer -- python3 -c ' 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> import glob 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> import re 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> import os 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> import subprocess 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> def _find_admin_socket(client_name): 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> files = glob.glob(asok_path) 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> # Given a non-glob path, it better be there 2024-04-21T08:37:42.727 DEBUG:teuthology.orchestra.run.smithi094:> if "*" not in asok_path: 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> assert(len(files) == 1) 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> return files[0] 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> for f in files: 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> contents = proc_f.read() 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> if mountpoint in contents: 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> return f 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> print(_find_admin_socket("client.admin")) 2024-04-21T08:37:42.728 DEBUG:teuthology.orchestra.run.smithi094:> ' 2024-04-21T08:37:42.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:42 smithi094 ceph-mon[27988]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-21T08:37:42.836 INFO:teuthology.orchestra.run.smithi094.stdout:/var/run/ceph/ceph-client.admin.71596.asok 2024-04-21T08:37:42.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:37:42.846 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.71596.asok 2024-04-21T08:37:42.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:42.846 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.71596.asok status 2024-04-21T08:37:42.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:42 smithi111 ceph-mon[31493]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-21T08:37:42.978 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "metadata": { 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_sha1": "a630765c1f93c6e61d9b40c4b76437611d5ef29e", 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "entity_id": "admin", 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "hostname": "smithi094", 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "pid": "71596", 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "root": "/" 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "dentry_count": 0, 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "dentry_pinned_count": 0, 2024-04-21T08:37:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 14763, 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "inst": { 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "name": { 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client", 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 14763 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": { 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "192.168.0.1:0", 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 2883348666 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": { 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "192.168.0.1:0", 2024-04-21T08:37:42.980 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 2883348666 2024-04-21T08:37:42.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:37:42.981 INFO:teuthology.orchestra.run.smithi094.stdout: "inst_str": "client.14763 192.168.0.1:0/2883348666", 2024-04-21T08:37:42.981 INFO:teuthology.orchestra.run.smithi094.stdout: "addr_str": "192.168.0.1:0/2883348666", 2024-04-21T08:37:42.981 INFO:teuthology.orchestra.run.smithi094.stdout: "inode_count": 1, 2024-04-21T08:37:42.981 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_epoch": 16, 2024-04-21T08:37:42.981 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_epoch": 89, 2024-04-21T08:37:42.981 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_epoch_barrier": 0, 2024-04-21T08:37:42.981 INFO:teuthology.orchestra.run.smithi094.stdout: "blocklisted": false, 2024-04-21T08:37:42.981 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs" 2024-04-21T08:37:42.981 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:37:42.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:42.987 DEBUG:teuthology.orchestra.run.smithi094:> stat --file-system '--printf=%T 2024-04-21T08:37:42.987 DEBUG:teuthology.orchestra.run.smithi094:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:37:43.003 INFO:teuthology.orchestra.run.smithi094.stdout:fuseblk 2024-04-21T08:37:43.003 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:37:43.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:43.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:37:43.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:42 smithi173 ceph-mon[31428]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-21T08:37:43.068 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-21T08:37:43.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:37:43.176 INFO:teuthology.orchestra.run.smithi094.stdout:# file: . 2024-04-21T08:37:43.177 INFO:teuthology.orchestra.run.smithi094.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-21T08:37:43.177 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:37:43.182 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:37:43.182 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:37:43.199 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:37:43.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.336+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/3990337512 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a44102980 msgr2=0x7f4a441092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.336+0000 7f4a4a7b2640 1 --2- 172.21.15.94:0/3990337512 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a44102980 0x7f4a441092a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009ca0 tx=0x7f4a3c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.336+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/3990337512 shutdown_connections 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.336+0000 7f4a4a7b2640 1 --2- 172.21.15.94:0/3990337512 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a44102980 0x7f4a441092a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.336+0000 7f4a4a7b2640 1 --2- 172.21.15.94:0/3990337512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a44101fc0 0x7f4a44102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.336+0000 7f4a4a7b2640 1 --2- 172.21.15.94:0/3990337512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44100e70 0x7f4a44101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.336+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/3990337512 >> 172.21.15.94:0/3990337512 conn(0x7f4a440fc420 msgr2=0x7f4a440fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.336+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/3990337512 shutdown_connections 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.336+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/3990337512 wait complete. 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a4a7b2640 1 Processor -- start 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a4a7b2640 1 -- start start 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a4a7b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a44100e70 0x7f4a4406b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a4a7b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44101fc0 0x7f4a4419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a4a7b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a44102980 0x7f4a4419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a4a7b2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4410c9c0 con 0x7f4a44100e70 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a4a7b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4410c840 con 0x7f4a44102980 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a4a7b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4410cb40 con 0x7f4a44101fc0 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a43fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a44100e70 0x7f4a4406b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a43fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a44100e70 0x7f4a4406b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48880/0 (socket says 172.21.15.94:48880) 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a43fff640 1 -- 172.21.15.94:0/747862799 learned_addr learned my addr 172.21.15.94:0/747862799 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a437fe640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44101fc0 0x7f4a4419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.337+0000 7f4a48d4f640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a44102980 0x7f4a4419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a43fff640 1 -- 172.21.15.94:0/747862799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44101fc0 msgr2=0x7f4a4419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a43fff640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44101fc0 0x7f4a4419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a43fff640 1 -- 172.21.15.94:0/747862799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a44102980 msgr2=0x7f4a4419f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a43fff640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a44102980 0x7f4a4419f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a43fff640 1 -- 172.21.15.94:0/747862799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4419fdc0 con 0x7f4a44100e70 2024-04-21T08:37:43.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a437fe640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44101fc0 0x7f4a4419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:43.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a43fff640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a44100e70 0x7f4a4406b1b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400d770 tx=0x7f4a3400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:43.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a417fa640 1 -- 172.21.15.94:0/747862799 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34017070 con 0x7f4a44100e70 2024-04-21T08:37:43.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a417fa640 1 -- 172.21.15.94:0/747862799 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a340054d0 con 0x7f4a44100e70 2024-04-21T08:37:43.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/747862799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4419fff0 con 0x7f4a44100e70 2024-04-21T08:37:43.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a417fa640 1 -- 172.21.15.94:0/747862799 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34004c30 con 0x7f4a44100e70 2024-04-21T08:37:43.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.338+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/747862799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a441a0460 con 0x7f4a44100e70 2024-04-21T08:37:43.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.339+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/747862799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a44108180 con 0x7f4a44100e70 2024-04-21T08:37:43.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.340+0000 7f4a417fa640 1 -- 172.21.15.94:0/747862799 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4a34004dd0 con 0x7f4a44100e70 2024-04-21T08:37:43.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.341+0000 7f4a417fa640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.341+0000 7f4a437fe640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.341+0000 7f4a417fa640 1 -- 172.21.15.94:0/747862799 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4a340d0550 con 0x7f4a44100e70 2024-04-21T08:37:43.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.341+0000 7f4a437fe640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4a4406af20 tx=0x7f4a3800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:43.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.345+0000 7f4a417fa640 1 -- 172.21.15.94:0/747862799 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a3409cc90 con 0x7f4a44100e70 2024-04-21T08:37:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.525+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/747862799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a4402d830 con 0x7f4a44100e70 2024-04-21T08:37:43.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.526+0000 7f4a417fa640 1 -- 172.21.15.94:0/747862799 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24092 (secure 0 0 0) 0x7f4a3409c590 con 0x7f4a44100e70 2024-04-21T08:37:43.527 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:43.527 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:37:35.939829+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:37:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.528+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/747862799 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a2c0a3bb0 msgr2=0x7f4a2c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.528+0000 7f4a4a7b2640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4a4406af20 tx=0x7f4a3800a560 comp rx=0 tx=0).stop 2024-04-21T08:37:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.528+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/747862799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a44100e70 msgr2=0x7f4a4406b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.528+0000 7f4a4a7b2640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a44100e70 0x7f4a4406b1b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400d770 tx=0x7f4a3400dc30 comp rx=0 tx=0).stop 2024-04-21T08:37:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.529+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/747862799 shutdown_connections 2024-04-21T08:37:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.529+0000 7f4a4a7b2640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.529+0000 7f4a4a7b2640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a44101fc0 0x7f4a4419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.529+0000 7f4a4a7b2640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a44102980 0x7f4a4419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.529+0000 7f4a4a7b2640 1 --2- 172.21.15.94:0/747862799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a44100e70 0x7f4a4406b1b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.529+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/747862799 >> 172.21.15.94:0/747862799 conn(0x7f4a440fc420 msgr2=0x7f4a44105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:43.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.529+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/747862799 shutdown_connections 2024-04-21T08:37:43.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.529+0000 7f4a4a7b2640 1 -- 172.21.15.94:0/747862799 wait complete. 2024-04-21T08:37:43.540 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:37:43.541 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:37:43.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.646+0000 7f36aae3a640 1 -- 172.21.15.94:0/1748282154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36a4101fe0 msgr2=0x7f36a4102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.646+0000 7f36aae3a640 1 --2- 172.21.15.94:0/1748282154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36a4101fe0 0x7f36a4102460 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3694009ca0 tx=0x7f369402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.646+0000 7f36aae3a640 1 -- 172.21.15.94:0/1748282154 shutdown_connections 2024-04-21T08:37:43.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.646+0000 7f36aae3a640 1 --2- 172.21.15.94:0/1748282154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a41029a0 0x7f36a41092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.646+0000 7f36aae3a640 1 --2- 172.21.15.94:0/1748282154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36a4101fe0 0x7f36a4102460 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.646+0000 7f36aae3a640 1 --2- 172.21.15.94:0/1748282154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a4100e90 0x7f36a41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.646+0000 7f36aae3a640 1 -- 172.21.15.94:0/1748282154 >> 172.21.15.94:0/1748282154 conn(0x7f36a40fc400 msgr2=0x7f36a40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:43.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.646+0000 7f36aae3a640 1 -- 172.21.15.94:0/1748282154 shutdown_connections 2024-04-21T08:37:43.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.646+0000 7f36aae3a640 1 -- 172.21.15.94:0/1748282154 wait complete. 2024-04-21T08:37:43.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36aae3a640 1 Processor -- start 2024-04-21T08:37:43.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36aae3a640 1 -- start start 2024-04-21T08:37:43.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36aae3a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a4100e90 0x7f36a410b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36aae3a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a4101fe0 0x7f36a4109e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36aae3a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36a41029a0 0x7f36a410a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36aae3a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f36a410db70 con 0x7f36a41029a0 2024-04-21T08:37:43.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36aae3a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f36a410d9f0 con 0x7f36a4100e90 2024-04-21T08:37:43.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36aae3a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f36a410dcf0 con 0x7f36a4101fe0 2024-04-21T08:37:43.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a93d7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36a41029a0 0x7f36a410a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a93d7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36a41029a0 0x7f36a410a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47376/0 (socket says 172.21.15.94:47376) 2024-04-21T08:37:43.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a93d7640 1 -- 172.21.15.94:0/3613188896 learned_addr learned my addr 172.21.15.94:0/3613188896 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:43.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a3fff640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a4101fe0 0x7f36a4109e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a8bd6640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a4100e90 0x7f36a410b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a93d7640 1 -- 172.21.15.94:0/3613188896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a4101fe0 msgr2=0x7f36a4109e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a93d7640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a4101fe0 0x7f36a4109e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a93d7640 1 -- 172.21.15.94:0/3613188896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a4100e90 msgr2=0x7f36a410b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a93d7640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a4100e90 0x7f36a410b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a93d7640 1 -- 172.21.15.94:0/3613188896 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a410ac40 con 0x7f36a41029a0 2024-04-21T08:37:43.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.647+0000 7f36a8bd6640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a4100e90 0x7f36a410b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:43.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.648+0000 7f36a93d7640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36a41029a0 0x7f36a410a3b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f368c00d750 tx=0x7f368c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:43.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.648+0000 7f36a1ffb640 1 -- 172.21.15.94:0/3613188896 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368c017070 con 0x7f36a41029a0 2024-04-21T08:37:43.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.648+0000 7f36a1ffb640 1 -- 172.21.15.94:0/3613188896 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f368c0054d0 con 0x7f36a41029a0 2024-04-21T08:37:43.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.648+0000 7f36aae3a640 1 -- 172.21.15.94:0/3613188896 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a41a23a0 con 0x7f36a41029a0 2024-04-21T08:37:43.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.648+0000 7f36a1ffb640 1 -- 172.21.15.94:0/3613188896 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368c004c60 con 0x7f36a41029a0 2024-04-21T08:37:43.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.648+0000 7f36aae3a640 1 -- 172.21.15.94:0/3613188896 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a41a28c0 con 0x7f36a41029a0 2024-04-21T08:37:43.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.649+0000 7f36aae3a640 1 -- 172.21.15.94:0/3613188896 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a4108290 con 0x7f36a41029a0 2024-04-21T08:37:43.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.650+0000 7f36a1ffb640 1 -- 172.21.15.94:0/3613188896 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f368c009bd0 con 0x7f36a41029a0 2024-04-21T08:37:43.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.650+0000 7f36a1ffb640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f36900a3bb0 0x7f36900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.651+0000 7f36a1ffb640 1 -- 172.21.15.94:0/3613188896 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f368c0cffe0 con 0x7f36a41029a0 2024-04-21T08:37:43.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.651+0000 7f36a8bd6640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f36900a3bb0 0x7f36900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.651+0000 7f36a8bd6640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f36900a3bb0 0x7f36900a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f369c009930 tx=0x7f369c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:43.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.655+0000 7f36a1ffb640 1 -- 172.21.15.94:0/3613188896 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f368c09c780 con 0x7f36a41029a0 2024-04-21T08:37:43.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.833+0000 7f36aae3a640 1 -- 172.21.15.94:0/3613188896 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f36a4062b50 con 0x7f36a41029a0 2024-04-21T08:37:43.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.834+0000 7f36a1ffb640 1 -- 172.21.15.94:0/3613188896 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f368c09c080 con 0x7f36a41029a0 2024-04-21T08:37:43.835 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:37:43.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 -- 172.21.15.94:0/3613188896 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f36900a3bb0 msgr2=0x7f36900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f36900a3bb0 0x7f36900a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f369c009930 tx=0x7f369c008040 comp rx=0 tx=0).stop 2024-04-21T08:37:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 -- 172.21.15.94:0/3613188896 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36a41029a0 msgr2=0x7f36a410a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36a41029a0 0x7f36a410a3b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f368c00d750 tx=0x7f368c00dc10 comp rx=0 tx=0).stop 2024-04-21T08:37:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 -- 172.21.15.94:0/3613188896 shutdown_connections 2024-04-21T08:37:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f36900a3bb0 0x7f36900a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a4101fe0 0x7f36a4109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36a41029a0 0x7f36a410a3b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 --2- 172.21.15.94:0/3613188896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a4100e90 0x7f36a410b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 -- 172.21.15.94:0/3613188896 >> 172.21.15.94:0/3613188896 conn(0x7f36a40fc400 msgr2=0x7f36a4105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 -- 172.21.15.94:0/3613188896 shutdown_connections 2024-04-21T08:37:43.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.836+0000 7f36aae3a640 1 -- 172.21.15.94:0/3613188896 wait complete. 2024-04-21T08:37:43.845 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:43 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/747862799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:43.847 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:37:43.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:43 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/747862799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:43.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.948+0000 7f657298f640 1 -- 172.21.15.94:0/1696141655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f656c103c00 msgr2=0x7f656c10a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.948+0000 7f657298f640 1 --2- 172.21.15.94:0/1696141655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f656c103c00 0x7f656c10a490 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6564009cd0 tx=0x7f656402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.948+0000 7f657298f640 1 -- 172.21.15.94:0/1696141655 shutdown_connections 2024-04-21T08:37:43.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.948+0000 7f657298f640 1 --2- 172.21.15.94:0/1696141655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f656c103240 0x7f656c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.948+0000 7f657298f640 1 --2- 172.21.15.94:0/1696141655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f656c103c00 0x7f656c10a490 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.948+0000 7f657298f640 1 --2- 172.21.15.94:0/1696141655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c100270 0x7f656c100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.948+0000 7f657298f640 1 -- 172.21.15.94:0/1696141655 >> 172.21.15.94:0/1696141655 conn(0x7f656c0fbb80 msgr2=0x7f656c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:43.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.948+0000 7f657298f640 1 -- 172.21.15.94:0/1696141655 shutdown_connections 2024-04-21T08:37:43.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.948+0000 7f657298f640 1 -- 172.21.15.94:0/1696141655 wait complete. 2024-04-21T08:37:43.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f657298f640 1 Processor -- start 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f657298f640 1 -- start start 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f657298f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c100270 0x7f656c198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f657298f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f656c103240 0x7f656c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f657298f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f656c103c00 0x7f656c19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f657298f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f656c10db10 con 0x7f656c103240 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f657298f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f656c10d990 con 0x7f656c100270 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f657298f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f656c10dc90 con 0x7f656c103c00 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f656b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f656c103240 0x7f656c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f656b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f656c103240 0x7f656c1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47394/0 (socket says 172.21.15.94:47394) 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f656b7fe640 1 -- 172.21.15.94:0/436972651 learned_addr learned my addr 172.21.15.94:0/436972651 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f656bfff640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c100270 0x7f656c198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f6570f2c640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f656c103c00 0x7f656c19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f656b7fe640 1 -- 172.21.15.94:0/436972651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f656c103c00 msgr2=0x7f656c19f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f656b7fe640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f656c103c00 0x7f656c19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f656b7fe640 1 -- 172.21.15.94:0/436972651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c100270 msgr2=0x7f656c198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f656b7fe640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c100270 0x7f656c198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.949+0000 7f656b7fe640 1 -- 172.21.15.94:0/436972651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c19fdb0 con 0x7f656c103240 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.950+0000 7f656b7fe640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f656c103240 0x7f656c1986d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f656000d770 tx=0x7f656000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.950+0000 7f65697fa640 1 -- 172.21.15.94:0/436972651 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560017070 con 0x7f656c103240 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.950+0000 7f65697fa640 1 -- 172.21.15.94:0/436972651 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65600054d0 con 0x7f656c103240 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.950+0000 7f657298f640 1 -- 172.21.15.94:0/436972651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656c1a00a0 con 0x7f656c103240 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.950+0000 7f65697fa640 1 -- 172.21.15.94:0/436972651 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560004c30 con 0x7f656c103240 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.950+0000 7f657298f640 1 -- 172.21.15.94:0/436972651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f656c1a05a0 con 0x7f656c103240 2024-04-21T08:37:43.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.951+0000 7f657298f640 1 -- 172.21.15.94:0/436972651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f656c109370 con 0x7f656c103240 2024-04-21T08:37:43.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.952+0000 7f65697fa640 1 -- 172.21.15.94:0/436972651 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6560004e10 con 0x7f656c103240 2024-04-21T08:37:43.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.953+0000 7f65697fa640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f65540a3bb0 0x7f65540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:43.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.953+0000 7f656bfff640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f65540a3bb0 0x7f65540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:43.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.953+0000 7f65697fa640 1 -- 172.21.15.94:0/436972651 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f65600d01b0 con 0x7f656c103240 2024-04-21T08:37:43.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.953+0000 7f656bfff640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f65540a3bb0 0x7f65540a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f655c005fd0 tx=0x7f655c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:43.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:43.957+0000 7f65697fa640 1 -- 172.21.15.94:0/436972651 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f656009c950 con 0x7f656c103240 2024-04-21T08:37:44.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:43 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/747862799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:44.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.134+0000 7f657298f640 1 -- 172.21.15.94:0/436972651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f656c062b80 con 0x7f656c103240 2024-04-21T08:37:44.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.135+0000 7f65697fa640 1 -- 172.21.15.94:0/436972651 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f656009c250 con 0x7f656c103240 2024-04-21T08:37:44.135 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:37:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 -- 172.21.15.94:0/436972651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f65540a3bb0 msgr2=0x7f65540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f65540a3bb0 0x7f65540a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f655c005fd0 tx=0x7f655c005850 comp rx=0 tx=0).stop 2024-04-21T08:37:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 -- 172.21.15.94:0/436972651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f656c103240 msgr2=0x7f656c1986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f656c103240 0x7f656c1986d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f656000d770 tx=0x7f656000dc30 comp rx=0 tx=0).stop 2024-04-21T08:37:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 -- 172.21.15.94:0/436972651 shutdown_connections 2024-04-21T08:37:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f65540a3bb0 0x7f65540a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f656c103c00 0x7f656c19f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f656c103240 0x7f656c1986d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 --2- 172.21.15.94:0/436972651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c100270 0x7f656c198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 -- 172.21.15.94:0/436972651 >> 172.21.15.94:0/436972651 conn(0x7f656c0fbb80 msgr2=0x7f656c106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 -- 172.21.15.94:0/436972651 shutdown_connections 2024-04-21T08:37:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.137+0000 7f657298f640 1 -- 172.21.15.94:0/436972651 wait complete. 2024-04-21T08:37:44.148 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:37:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.248+0000 7f9ecace0640 1 -- 172.21.15.94:0/2751655349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec4105340 msgr2=0x7f9ec4107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.248+0000 7f9ecace0640 1 --2- 172.21.15.94:0/2751655349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec4105340 0x7f9ec4107730 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8009900 tx=0x7f9eb802bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.248+0000 7f9ecace0640 1 -- 172.21.15.94:0/2751655349 shutdown_connections 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.248+0000 7f9ecace0640 1 --2- 172.21.15.94:0/2751655349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec4107c70 0x7f9ec410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.248+0000 7f9ecace0640 1 --2- 172.21.15.94:0/2751655349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec4105340 0x7f9ec4107730 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.248+0000 7f9ecace0640 1 --2- 172.21.15.94:0/2751655349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec4102a10 0x7f9ec4104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.248+0000 7f9ecace0640 1 -- 172.21.15.94:0/2751655349 >> 172.21.15.94:0/2751655349 conn(0x7f9ec40fc400 msgr2=0x7f9ec40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ecace0640 1 -- 172.21.15.94:0/2751655349 shutdown_connections 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ecace0640 1 -- 172.21.15.94:0/2751655349 wait complete. 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ecace0640 1 Processor -- start 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ecace0640 1 -- start start 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ecace0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec4102a10 0x7f9ec4198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ecace0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec4105340 0x7f9ec4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ecace0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec4107c70 0x7f9ec419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ecace0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec410daa0 con 0x7f9ec4105340 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ecace0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec410d920 con 0x7f9ec4107c70 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ecace0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec410dc20 con 0x7f9ec4102a10 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ec3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec4105340 0x7f9ec4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ec3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec4105340 0x7f9ec4198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47402/0 (socket says 172.21.15.94:47402) 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ec3fff640 1 -- 172.21.15.94:0/2303976274 learned_addr learned my addr 172.21.15.94:0/2303976274 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.249+0000 7f9ec8a7c640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec4102a10 0x7f9ec4198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ec3fff640 1 -- 172.21.15.94:0/2303976274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec4102a10 msgr2=0x7f9ec4198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ec3fff640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec4102a10 0x7f9ec4198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ec927d640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec4107c70 0x7f9ec419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ec3fff640 1 -- 172.21.15.94:0/2303976274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec4107c70 msgr2=0x7f9ec419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ec3fff640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec4107c70 0x7f9ec419f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ec3fff640 1 -- 172.21.15.94:0/2303976274 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec419fe20 con 0x7f9ec4105340 2024-04-21T08:37:44.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ec3fff640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec4105340 0x7f9ec4198740 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8000c00 tx=0x7f9eb8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:44.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ec1ffb640 1 -- 172.21.15.94:0/2303976274 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8043020 con 0x7f9ec4105340 2024-04-21T08:37:44.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ec1ffb640 1 -- 172.21.15.94:0/2303976274 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eb8004440 con 0x7f9ec4105340 2024-04-21T08:37:44.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ecace0640 1 -- 172.21.15.94:0/2303976274 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ec41a00b0 con 0x7f9ec4105340 2024-04-21T08:37:44.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ec1ffb640 1 -- 172.21.15.94:0/2303976274 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8003c90 con 0x7f9ec4105340 2024-04-21T08:37:44.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.250+0000 7f9ecace0640 1 -- 172.21.15.94:0/2303976274 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ec41a0650 con 0x7f9ec4105340 2024-04-21T08:37:44.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.251+0000 7f9ecace0640 1 -- 172.21.15.94:0/2303976274 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ec40fdef0 con 0x7f9ec4105340 2024-04-21T08:37:44.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.252+0000 7f9ec1ffb640 1 -- 172.21.15.94:0/2303976274 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9eb8046020 con 0x7f9ec4105340 2024-04-21T08:37:44.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.253+0000 7f9ec1ffb640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9eac0a3c30 0x7f9eac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.253+0000 7f9ec8a7c640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9eac0a3c30 0x7f9eac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.253+0000 7f9ec1ffb640 1 -- 172.21.15.94:0/2303976274 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9eb80f2c80 con 0x7f9ec4105340 2024-04-21T08:37:44.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.254+0000 7f9ec8a7c640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9eac0a3c30 0x7f9eac0a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009bd0 tx=0x7f9eb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:44.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.257+0000 7f9ec1ffb640 1 -- 172.21.15.94:0/2303976274 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9eb804d020 con 0x7f9ec4105340 2024-04-21T08:37:44.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.434+0000 7f9ecace0640 1 -- 172.21.15.94:0/2303976274 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9ec402d810 con 0x7f9ec4105340 2024-04-21T08:37:44.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.435+0000 7f9ec1ffb640 1 -- 172.21.15.94:0/2303976274 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f9eb80bf420 con 0x7f9ec4105340 2024-04-21T08:37:44.435 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:37:44.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.436+0000 7f9ecace0640 1 -- 172.21.15.94:0/2303976274 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9eac0a3c30 msgr2=0x7f9eac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.436+0000 7f9ecace0640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9eac0a3c30 0x7f9eac0a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009bd0 tx=0x7f9eb4005f90 comp rx=0 tx=0).stop 2024-04-21T08:37:44.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.437+0000 7f9ecace0640 1 -- 172.21.15.94:0/2303976274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec4105340 msgr2=0x7f9ec4198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.437+0000 7f9ecace0640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec4105340 0x7f9ec4198740 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8000c00 tx=0x7f9eb8004040 comp rx=0 tx=0).stop 2024-04-21T08:37:44.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.437+0000 7f9ecace0640 1 -- 172.21.15.94:0/2303976274 shutdown_connections 2024-04-21T08:37:44.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.437+0000 7f9ecace0640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9eac0a3c30 0x7f9eac0a60f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.437+0000 7f9ecace0640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec4107c70 0x7f9ec419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.437+0000 7f9ecace0640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec4105340 0x7f9ec4198740 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.437+0000 7f9ecace0640 1 --2- 172.21.15.94:0/2303976274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec4102a10 0x7f9ec4198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.437+0000 7f9ecace0640 1 -- 172.21.15.94:0/2303976274 >> 172.21.15.94:0/2303976274 conn(0x7f9ec40fc400 msgr2=0x7f9ec41048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:44.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.437+0000 7f9ecace0640 1 -- 172.21.15.94:0/2303976274 shutdown_connections 2024-04-21T08:37:44.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.437+0000 7f9ecace0640 1 -- 172.21.15.94:0/2303976274 wait complete. 2024-04-21T08:37:44.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:37:44.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.549+0000 7f6c1832c640 1 -- 172.21.15.94:0/1853451210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c10102a10 msgr2=0x7f6c10104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.549+0000 7f6c1832c640 1 --2- 172.21.15.94:0/1853451210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c10102a10 0x7f6c10104e00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009900 tx=0x7f6c0c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.549+0000 7f6c1832c640 1 -- 172.21.15.94:0/1853451210 shutdown_connections 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.549+0000 7f6c1832c640 1 --2- 172.21.15.94:0/1853451210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c10105340 0x7f6c10107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.549+0000 7f6c1832c640 1 --2- 172.21.15.94:0/1853451210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c10107c70 0x7f6c1010a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.549+0000 7f6c1832c640 1 --2- 172.21.15.94:0/1853451210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c10102a10 0x7f6c10104e00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 -- 172.21.15.94:0/1853451210 >> 172.21.15.94:0/1853451210 conn(0x7f6c100fc400 msgr2=0x7f6c100fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 -- 172.21.15.94:0/1853451210 shutdown_connections 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 -- 172.21.15.94:0/1853451210 wait complete. 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 Processor -- start 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 -- start start 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c10102a10 0x7f6c10198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c10105340 0x7f6c10198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c10107c70 0x7f6c1019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1010dad0 con 0x7f6c10102a10 2024-04-21T08:37:44.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1010d950 con 0x7f6c10107c70 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c1832c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1010dc50 con 0x7f6c10105340 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c160c8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c10102a10 0x7f6c10198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c160c8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c10102a10 0x7f6c10198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47412/0 (socket says 172.21.15.94:47412) 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c160c8640 1 -- 172.21.15.94:0/2943651099 learned_addr learned my addr 172.21.15.94:0/2943651099 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.550+0000 7f6c158c7640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c10105340 0x7f6c10198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c168c9640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c10107c70 0x7f6c1019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c160c8640 1 -- 172.21.15.94:0/2943651099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c10105340 msgr2=0x7f6c10198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c160c8640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c10105340 0x7f6c10198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c160c8640 1 -- 172.21.15.94:0/2943651099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c10107c70 msgr2=0x7f6c1019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c160c8640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c10107c70 0x7f6c1019f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c160c8640 1 -- 172.21.15.94:0/2943651099 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1019fe70 con 0x7f6c10102a10 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c160c8640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c10102a10 0x7f6c10198250 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c006010 tx=0x7f6c0c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:44.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c077fe640 1 -- 172.21.15.94:0/2943651099 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c043020 con 0x7f6c10102a10 2024-04-21T08:37:44.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c1832c640 1 -- 172.21.15.94:0/2943651099 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c101a0100 con 0x7f6c10102a10 2024-04-21T08:37:44.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c077fe640 1 -- 172.21.15.94:0/2943651099 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c0c004aa0 con 0x7f6c10102a10 2024-04-21T08:37:44.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c077fe640 1 -- 172.21.15.94:0/2943651099 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c003c70 con 0x7f6c10102a10 2024-04-21T08:37:44.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.551+0000 7f6c1832c640 1 -- 172.21.15.94:0/2943651099 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c101a0650 con 0x7f6c10102a10 2024-04-21T08:37:44.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.552+0000 7f6c1832c640 1 -- 172.21.15.94:0/2943651099 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c100fdef0 con 0x7f6c10102a10 2024-04-21T08:37:44.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.553+0000 7f6c077fe640 1 -- 172.21.15.94:0/2943651099 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6c0c046020 con 0x7f6c10102a10 2024-04-21T08:37:44.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.554+0000 7f6c077fe640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6bfc0a3bb0 0x7f6bfc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.554+0000 7f6c158c7640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6bfc0a3bb0 0x7f6bfc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.554+0000 7f6c077fe640 1 -- 172.21.15.94:0/2943651099 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6c0c0f30b0 con 0x7f6c10102a10 2024-04-21T08:37:44.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.554+0000 7f6c158c7640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6bfc0a3bb0 0x7f6bfc0a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6c10199760 tx=0x7f6c0000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:44.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.557+0000 7f6c077fe640 1 -- 172.21.15.94:0/2943651099 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c0c04d020 con 0x7f6c10102a10 2024-04-21T08:37:44.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.734+0000 7f6c1832c640 1 -- 172.21.15.94:0/2943651099 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6c10062b50 con 0x7f6c10102a10 2024-04-21T08:37:44.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.735+0000 7f6c077fe640 1 -- 172.21.15.94:0/2943651099 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f6c0c0bf850 con 0x7f6c10102a10 2024-04-21T08:37:44.735 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:37:44.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 -- 172.21.15.94:0/2943651099 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6bfc0a3bb0 msgr2=0x7f6bfc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6bfc0a3bb0 0x7f6bfc0a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6c10199760 tx=0x7f6c0000b390 comp rx=0 tx=0).stop 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 -- 172.21.15.94:0/2943651099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c10102a10 msgr2=0x7f6c10198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c10102a10 0x7f6c10198250 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c006010 tx=0x7f6c0c0042f0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 -- 172.21.15.94:0/2943651099 shutdown_connections 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6bfc0a3bb0 0x7f6bfc0a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c10105340 0x7f6c10198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c10107c70 0x7f6c1019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 --2- 172.21.15.94:0/2943651099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c10102a10 0x7f6c10198250 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 -- 172.21.15.94:0/2943651099 >> 172.21.15.94:0/2943651099 conn(0x7f6c100fc400 msgr2=0x7f6c101048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 -- 172.21.15.94:0/2943651099 shutdown_connections 2024-04-21T08:37:44.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.737+0000 7f6c1832c640 1 -- 172.21.15.94:0/2943651099 wait complete. 2024-04-21T08:37:44.746 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:44 smithi094 ceph-mon[27988]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2024-04-21T08:37:44.746 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:44 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3613188896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:37:44.746 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:44 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/436972651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:37:44.746 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:44 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2303976274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:37:44.747 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 -- 172.21.15.94:0/2372997139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a70105420 msgr2=0x7f9a70107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 --2- 172.21.15.94:0/2372997139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a70105420 0x7f9a70107810 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009900 tx=0x7f9a6402bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 -- 172.21.15.94:0/2372997139 shutdown_connections 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 --2- 172.21.15.94:0/2372997139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a70107d50 0x7f9a7010a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 --2- 172.21.15.94:0/2372997139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a70105420 0x7f9a70107810 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 --2- 172.21.15.94:0/2372997139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a70102af0 0x7f9a70104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 -- 172.21.15.94:0/2372997139 >> 172.21.15.94:0/2372997139 conn(0x7f9a700fc440 msgr2=0x7f9a700fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 -- 172.21.15.94:0/2372997139 shutdown_connections 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 -- 172.21.15.94:0/2372997139 wait complete. 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 Processor -- start 2024-04-21T08:37:44.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.847+0000 7f9a76281640 1 -- start start 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a76281640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a70102af0 0x7f9a70198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a76281640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a70105420 0x7f9a70198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a76281640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a70107d50 0x7f9a7019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a76281640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7010db00 con 0x7f9a70102af0 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a76281640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7010d980 con 0x7f9a70107d50 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a76281640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7010dc80 con 0x7f9a70105420 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a70102af0 0x7f9a70198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a70102af0 0x7f9a70198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47424/0 (socket says 172.21.15.94:47424) 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6ffff640 1 -- 172.21.15.94:0/3109357077 learned_addr learned my addr 172.21.15.94:0/3109357077 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6f7fe640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a70105420 0x7f9a70198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6ffff640 1 -- 172.21.15.94:0/3109357077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a70105420 msgr2=0x7f9a70198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6ffff640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a70105420 0x7f9a70198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6ffff640 1 -- 172.21.15.94:0/3109357077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a70107d50 msgr2=0x7f9a7019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:44.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6ffff640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a70107d50 0x7f9a7019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:44.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6ffff640 1 -- 172.21.15.94:0/3109357077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7019fe70 con 0x7f9a70102af0 2024-04-21T08:37:44.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6f7fe640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a70105420 0x7f9a70198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:44.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6ffff640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a70102af0 0x7f9a70198250 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000c970 tx=0x7f9a6000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:44.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6d7fa640 1 -- 172.21.15.94:0/3109357077 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60007bb0 con 0x7f9a70102af0 2024-04-21T08:37:44.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a76281640 1 -- 172.21.15.94:0/3109357077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a701a0160 con 0x7f9a70102af0 2024-04-21T08:37:44.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6d7fa640 1 -- 172.21.15.94:0/3109357077 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a60007d50 con 0x7f9a70102af0 2024-04-21T08:37:44.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a6d7fa640 1 -- 172.21.15.94:0/3109357077 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6000f450 con 0x7f9a70102af0 2024-04-21T08:37:44.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.848+0000 7f9a76281640 1 -- 172.21.15.94:0/3109357077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a701a06b0 con 0x7f9a70102af0 2024-04-21T08:37:44.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.850+0000 7f9a76281640 1 -- 172.21.15.94:0/3109357077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a700fdf30 con 0x7f9a70102af0 2024-04-21T08:37:44.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.851+0000 7f9a6d7fa640 1 -- 172.21.15.94:0/3109357077 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9a60004f20 con 0x7f9a70102af0 2024-04-21T08:37:44.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.851+0000 7f9a6d7fa640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9a580a3bb0 0x7f9a580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:44.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.851+0000 7f9a6f7fe640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9a580a3bb0 0x7f9a580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:44.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.851+0000 7f9a6d7fa640 1 -- 172.21.15.94:0/3109357077 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9a600d0460 con 0x7f9a70102af0 2024-04-21T08:37:44.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.852+0000 7f9a6f7fe640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9a580a3bb0 0x7f9a580a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9a64036040 tx=0x7f9a64000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:44.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:44.856+0000 7f9a6d7fa640 1 -- 172.21.15.94:0/3109357077 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a60014030 con 0x7f9a70102af0 2024-04-21T08:37:44.912 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:44 smithi173 ceph-mon[31428]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2024-04-21T08:37:44.912 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:44 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3613188896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:37:44.912 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:44 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/436972651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:37:44.912 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:44 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2303976274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:37:44.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:44 smithi111 ceph-mon[31493]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2024-04-21T08:37:44.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:44 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3613188896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:37:44.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:44 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/436972651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:37:44.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:44 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2303976274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:37:45.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.044+0000 7f9a76281640 1 -- 172.21.15.94:0/3109357077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9a7002d5c0 con 0x7f9a70102af0 2024-04-21T08:37:45.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.045+0000 7f9a6d7fa640 1 -- 172.21.15.94:0/3109357077 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7f9a6009cc00 con 0x7f9a70102af0 2024-04-21T08:37:45.047 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/441167573 2024-04-22T08:30:29.483746+0000 2024-04-21T08:37:45.047 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/2502723264 2024-04-22T08:30:29.483746+0000 2024-04-21T08:37:45.047 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/613544885 2024-04-22T08:30:29.483746+0000 2024-04-21T08:37:45.047 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/3877440809 2024-04-22T08:30:29.483746+0000 2024-04-21T08:37:45.047 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3977681577 2024-04-22T08:30:03.511107+0000 2024-04-21T08:37:45.047 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3965359185 2024-04-22T08:30:03.511107+0000 2024-04-21T08:37:45.048 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/1761971189 2024-04-22T08:30:03.511107+0000 2024-04-21T08:37:45.048 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/2629131251 2024-04-22T08:30:03.511107+0000 2024-04-21T08:37:45.048 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/3877440809 2024-04-22T08:30:29.483746+0000 2024-04-21T08:37:45.048 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/2629131251 2024-04-22T08:30:03.511107+0000 2024-04-21T08:37:45.049 INFO:teuthology.orchestra.run.smithi094.stderr:listed 10 entries 2024-04-21T08:37:45.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 -- 172.21.15.94:0/3109357077 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9a580a3bb0 msgr2=0x7f9a580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9a580a3bb0 0x7f9a580a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9a64036040 tx=0x7f9a64000bc0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 -- 172.21.15.94:0/3109357077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a70102af0 msgr2=0x7f9a70198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a70102af0 0x7f9a70198250 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000c970 tx=0x7f9a6000ce30 comp rx=0 tx=0).stop 2024-04-21T08:37:45.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 -- 172.21.15.94:0/3109357077 shutdown_connections 2024-04-21T08:37:45.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9a580a3bb0 0x7f9a580a6070 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a70105420 0x7f9a70198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a70107d50 0x7f9a7019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 --2- 172.21.15.94:0/3109357077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a70102af0 0x7f9a70198250 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 -- 172.21.15.94:0/3109357077 >> 172.21.15.94:0/3109357077 conn(0x7f9a700fc440 msgr2=0x7f9a70103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:45.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 -- 172.21.15.94:0/3109357077 shutdown_connections 2024-04-21T08:37:45.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.047+0000 7f9a76281640 1 -- 172.21.15.94:0/3109357077 wait complete. 2024-04-21T08:37:45.058 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:37:45.058 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:37:45.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:37:45.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.212+0000 7f718f8c1640 1 -- 172.21.15.94:0/2823811971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7188107cc0 msgr2=0x7f718810a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.212+0000 7f718f8c1640 1 --2- 172.21.15.94:0/2823811971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7188107cc0 0x7f718810a0b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7180009ca0 tx=0x7f718002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.212+0000 7f718f8c1640 1 -- 172.21.15.94:0/2823811971 shutdown_connections 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.212+0000 7f718f8c1640 1 --2- 172.21.15.94:0/2823811971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7188105390 0x7f7188107780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.212+0000 7f718f8c1640 1 --2- 172.21.15.94:0/2823811971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7188107cc0 0x7f718810a0b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.212+0000 7f718f8c1640 1 --2- 172.21.15.94:0/2823811971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7188102a60 0x7f7188104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.212+0000 7f718f8c1640 1 -- 172.21.15.94:0/2823811971 >> 172.21.15.94:0/2823811971 conn(0x7f71880fc410 msgr2=0x7f71880fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.212+0000 7f718f8c1640 1 -- 172.21.15.94:0/2823811971 shutdown_connections 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.212+0000 7f718f8c1640 1 -- 172.21.15.94:0/2823811971 wait complete. 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718f8c1640 1 Processor -- start 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718f8c1640 1 -- start start 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718f8c1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7188102a60 0x7f71881980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718f8c1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7188105390 0x7f7188198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718f8c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7188107cc0 0x7f718819f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718f8c1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f718810dac0 con 0x7f7188107cc0 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718f8c1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f718810d940 con 0x7f7188102a60 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718f8c1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f718810dc40 con 0x7f7188105390 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718de5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7188107cc0 0x7f718819f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718de5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7188107cc0 0x7f718819f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47442/0 (socket says 172.21.15.94:47442) 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718de5e640 1 -- 172.21.15.94:0/3641161678 learned_addr learned my addr 172.21.15.94:0/3641161678 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718ce5c640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7188105390 0x7f7188198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718d65d640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7188102a60 0x7f71881980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718de5e640 1 -- 172.21.15.94:0/3641161678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7188105390 msgr2=0x7f7188198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718de5e640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7188105390 0x7f7188198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718de5e640 1 -- 172.21.15.94:0/3641161678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7188102a60 msgr2=0x7f71881980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718de5e640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7188102a60 0x7f71881980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718de5e640 1 -- 172.21.15.94:0/3641161678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718819fd00 con 0x7f7188107cc0 2024-04-21T08:37:45.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.213+0000 7f718d65d640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7188102a60 0x7f71881980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:45.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.214+0000 7f718de5e640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7188107cc0 0x7f718819f690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f718002c700 tx=0x7f7180034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:45.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.214+0000 7f717e7fc640 1 -- 172.21.15.94:0/3641161678 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7180039040 con 0x7f7188107cc0 2024-04-21T08:37:45.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.214+0000 7f718f8c1640 1 -- 172.21.15.94:0/3641161678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718819ff90 con 0x7f7188107cc0 2024-04-21T08:37:45.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.214+0000 7f717e7fc640 1 -- 172.21.15.94:0/3641161678 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7180044070 con 0x7f7188107cc0 2024-04-21T08:37:45.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.214+0000 7f717e7fc640 1 -- 172.21.15.94:0/3641161678 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71800042a0 con 0x7f7188107cc0 2024-04-21T08:37:45.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.214+0000 7f718f8c1640 1 -- 172.21.15.94:0/3641161678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71881a03f0 con 0x7f7188107cc0 2024-04-21T08:37:45.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.215+0000 7f718f8c1640 1 -- 172.21.15.94:0/3641161678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71880fdf00 con 0x7f7188107cc0 2024-04-21T08:37:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.216+0000 7f717e7fc640 1 -- 172.21.15.94:0/3641161678 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7180003970 con 0x7f7188107cc0 2024-04-21T08:37:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.216+0000 7f717e7fc640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f71740a3bb0 0x7f71740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.216+0000 7f718d65d640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f71740a3bb0 0x7f71740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.216+0000 7f717e7fc640 1 -- 172.21.15.94:0/3641161678 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f71800f2a00 con 0x7f7188107cc0 2024-04-21T08:37:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.217+0000 7f718d65d640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f71740a3bb0 0x7f71740a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f718400baa0 tx=0x7f7184005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.221+0000 7f717e7fc640 1 -- 172.21.15.94:0/3641161678 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f71800bf1a0 con 0x7f7188107cc0 2024-04-21T08:37:45.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.399+0000 7f718f8c1640 1 -- 172.21.15.94:0/3641161678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7188062c50 con 0x7f7188107cc0 2024-04-21T08:37:45.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.401+0000 7f717e7fc640 1 -- 172.21.15.94:0/3641161678 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24092 (secure 0 0 0) 0x7f71800df0e0 con 0x7f7188107cc0 2024-04-21T08:37:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:45.401 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:37:35.939829+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:37:45.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.402+0000 7f718f8c1640 1 -- 172.21.15.94:0/3641161678 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f71740a3bb0 msgr2=0x7f71740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.402+0000 7f718f8c1640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f71740a3bb0 0x7f71740a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f718400baa0 tx=0x7f7184005d90 comp rx=0 tx=0).stop 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.403+0000 7f718f8c1640 1 -- 172.21.15.94:0/3641161678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7188107cc0 msgr2=0x7f718819f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.403+0000 7f718f8c1640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7188107cc0 0x7f718819f690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f718002c700 tx=0x7f7180034760 comp rx=0 tx=0).stop 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.403+0000 7f718f8c1640 1 -- 172.21.15.94:0/3641161678 shutdown_connections 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.403+0000 7f718f8c1640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f71740a3bb0 0x7f71740a6070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.403+0000 7f718f8c1640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7188105390 0x7f7188198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.403+0000 7f718f8c1640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7188107cc0 0x7f718819f690 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.403+0000 7f718f8c1640 1 --2- 172.21.15.94:0/3641161678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7188102a60 0x7f71881980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.403+0000 7f718f8c1640 1 -- 172.21.15.94:0/3641161678 >> 172.21.15.94:0/3641161678 conn(0x7f71880fc410 msgr2=0x7f71881033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.403+0000 7f718f8c1640 1 -- 172.21.15.94:0/3641161678 shutdown_connections 2024-04-21T08:37:45.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.403+0000 7f718f8c1640 1 -- 172.21.15.94:0/3641161678 wait complete. 2024-04-21T08:37:45.413 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:37:45.414 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:37:45.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.516+0000 7f6557dd5640 1 -- 172.21.15.94:0/460253643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6550107c90 msgr2=0x7f655010a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.516+0000 7f6557dd5640 1 --2- 172.21.15.94:0/460253643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6550107c90 0x7f655010a0e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6548009ca0 tx=0x7f654802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.516+0000 7f6557dd5640 1 -- 172.21.15.94:0/460253643 shutdown_connections 2024-04-21T08:37:45.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.516+0000 7f6557dd5640 1 --2- 172.21.15.94:0/460253643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6550107c90 0x7f655010a0e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.516+0000 7f6557dd5640 1 --2- 172.21.15.94:0/460253643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6550105360 0x7f6550107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.516+0000 7f6557dd5640 1 --2- 172.21.15.94:0/460253643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6550102a30 0x7f6550104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.516+0000 7f6557dd5640 1 -- 172.21.15.94:0/460253643 >> 172.21.15.94:0/460253643 conn(0x7f65500fc420 msgr2=0x7f65500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:45.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.516+0000 7f6557dd5640 1 -- 172.21.15.94:0/460253643 shutdown_connections 2024-04-21T08:37:45.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.516+0000 7f6557dd5640 1 -- 172.21.15.94:0/460253643 wait complete. 2024-04-21T08:37:45.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6557dd5640 1 Processor -- start 2024-04-21T08:37:45.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6557dd5640 1 -- start start 2024-04-21T08:37:45.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6557dd5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6550102a30 0x7f6550198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6557dd5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6550105360 0x7f6550198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6557dd5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6550107c90 0x7f655019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6557dd5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f655010db50 con 0x7f6550102a30 2024-04-21T08:37:45.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6557dd5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f655010d9d0 con 0x7f6550105360 2024-04-21T08:37:45.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6557dd5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f655010dcd0 con 0x7f6550107c90 2024-04-21T08:37:45.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6555b71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6550102a30 0x7f6550198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6555b71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6550102a30 0x7f6550198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47450/0 (socket says 172.21.15.94:47450) 2024-04-21T08:37:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6555b71640 1 -- 172.21.15.94:0/3600945397 learned_addr learned my addr 172.21.15.94:0/3600945397 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6556372640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6550107c90 0x7f655019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6555370640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6550105360 0x7f6550198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6555b71640 1 -- 172.21.15.94:0/3600945397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6550107c90 msgr2=0x7f655019f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6555b71640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6550107c90 0x7f655019f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6555b71640 1 -- 172.21.15.94:0/3600945397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6550105360 msgr2=0x7f6550198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6555b71640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6550105360 0x7f6550198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6555b71640 1 -- 172.21.15.94:0/3600945397 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655019fe40 con 0x7f6550102a30 2024-04-21T08:37:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6555b71640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6550102a30 0x7f6550198220 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f654c00d950 tx=0x7f654c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.517+0000 7f6546ffd640 1 -- 172.21.15.94:0/3600945397 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c017030 con 0x7f6550102a30 2024-04-21T08:37:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.518+0000 7f6557dd5640 1 -- 172.21.15.94:0/3600945397 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65501a0130 con 0x7f6550102a30 2024-04-21T08:37:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.518+0000 7f6546ffd640 1 -- 172.21.15.94:0/3600945397 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f654c004020 con 0x7f6550102a30 2024-04-21T08:37:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.518+0000 7f6546ffd640 1 -- 172.21.15.94:0/3600945397 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c0047a0 con 0x7f6550102a30 2024-04-21T08:37:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.518+0000 7f6557dd5640 1 -- 172.21.15.94:0/3600945397 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65501a0680 con 0x7f6550102a30 2024-04-21T08:37:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.519+0000 7f6557dd5640 1 -- 172.21.15.94:0/3600945397 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65500fdf10 con 0x7f6550102a30 2024-04-21T08:37:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.520+0000 7f6546ffd640 1 -- 172.21.15.94:0/3600945397 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f654c009a30 con 0x7f6550102a30 2024-04-21T08:37:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.520+0000 7f6546ffd640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f653c0a3bb0 0x7f653c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.521+0000 7f6555370640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f653c0a3bb0 0x7f653c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.521+0000 7f6546ffd640 1 -- 172.21.15.94:0/3600945397 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f654c0d0100 con 0x7f6550102a30 2024-04-21T08:37:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.521+0000 7f6555370640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f653c0a3bb0 0x7f653c0a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6550199730 tx=0x7f6540008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:45.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.525+0000 7f6546ffd640 1 -- 172.21.15.94:0/3600945397 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f654c09c8a0 con 0x7f6550102a30 2024-04-21T08:37:45.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.711+0000 7f6557dd5640 1 -- 172.21.15.94:0/3600945397 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6550062b50 con 0x7f6550102a30 2024-04-21T08:37:45.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.711+0000 7f6546ffd640 1 -- 172.21.15.94:0/3600945397 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f654c09c1a0 con 0x7f6550102a30 2024-04-21T08:37:45.712 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:37:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 -- 172.21.15.94:0/3600945397 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f653c0a3bb0 msgr2=0x7f653c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f653c0a3bb0 0x7f653c0a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6550199730 tx=0x7f6540008040 comp rx=0 tx=0).stop 2024-04-21T08:37:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 -- 172.21.15.94:0/3600945397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6550102a30 msgr2=0x7f6550198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6550102a30 0x7f6550198220 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f654c00d950 tx=0x7f654c00de10 comp rx=0 tx=0).stop 2024-04-21T08:37:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 -- 172.21.15.94:0/3600945397 shutdown_connections 2024-04-21T08:37:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f653c0a3bb0 0x7f653c0a6070 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6550107c90 0x7f655019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6550105360 0x7f6550198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 --2- 172.21.15.94:0/3600945397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6550102a30 0x7f6550198220 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 -- 172.21.15.94:0/3600945397 >> 172.21.15.94:0/3600945397 conn(0x7f65500fc420 msgr2=0x7f6550103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:45.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 -- 172.21.15.94:0/3600945397 shutdown_connections 2024-04-21T08:37:45.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.713+0000 7f6557dd5640 1 -- 172.21.15.94:0/3600945397 wait complete. 2024-04-21T08:37:45.724 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:45 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2943651099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:37:45.724 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:45 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3109357077' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:37:45.724 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:45 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3641161678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:45.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:37:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.826+0000 7fd7a50f2640 1 -- 172.21.15.94:0/3649226121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a0102a00 msgr2=0x7fd7a0109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.826+0000 7fd7a50f2640 1 --2- 172.21.15.94:0/3649226121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a0102a00 0x7fd7a0109320 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd798009ca0 tx=0x7fd79802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.826+0000 7fd7a50f2640 1 -- 172.21.15.94:0/3649226121 shutdown_connections 2024-04-21T08:37:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.826+0000 7fd7a50f2640 1 --2- 172.21.15.94:0/3649226121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a0102040 0x7fd7a01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.826+0000 7fd7a50f2640 1 --2- 172.21.15.94:0/3649226121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a0102a00 0x7fd7a0109320 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.826+0000 7fd7a50f2640 1 --2- 172.21.15.94:0/3649226121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7a0100e60 0x7fd7a0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.826+0000 7fd7a50f2640 1 -- 172.21.15.94:0/3649226121 >> 172.21.15.94:0/3649226121 conn(0x7fd7a00fc440 msgr2=0x7fd7a00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.826+0000 7fd7a50f2640 1 -- 172.21.15.94:0/3649226121 shutdown_connections 2024-04-21T08:37:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.826+0000 7fd7a50f2640 1 -- 172.21.15.94:0/3649226121 wait complete. 2024-04-21T08:37:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd7a50f2640 1 Processor -- start 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd7a50f2640 1 -- start start 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd7a50f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a0100e60 0x7fd7a01981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd7a50f2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7a0102040 0x7fd7a0198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd7a50f2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a0102a00 0x7fd7a019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd7a50f2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a010ca00 con 0x7fd7a0100e60 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd7a50f2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a010c880 con 0x7fd7a0102040 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd7a50f2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a010cb80 con 0x7fd7a0102a00 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd79ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a0100e60 0x7fd7a01981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd79ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a0100e60 0x7fd7a01981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47482/0 (socket says 172.21.15.94:47482) 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd79ed9d640 1 -- 172.21.15.94:0/1307984551 learned_addr learned my addr 172.21.15.94:0/1307984551 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd79f59e640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a0102a00 0x7fd7a019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd79e59c640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7a0102040 0x7fd7a0198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd79ed9d640 1 -- 172.21.15.94:0/1307984551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a0102a00 msgr2=0x7fd7a019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd79ed9d640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a0102a00 0x7fd7a019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd79ed9d640 1 -- 172.21.15.94:0/1307984551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7a0102040 msgr2=0x7fd7a0198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd79ed9d640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7a0102040 0x7fd7a0198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.827+0000 7fd79ed9d640 1 -- 172.21.15.94:0/1307984551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a019fde0 con 0x7fd7a0100e60 2024-04-21T08:37:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.828+0000 7fd79ed9d640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a0100e60 0x7fd7a01981c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd79000c970 tx=0x7fd79000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.828+0000 7fd787fff640 1 -- 172.21.15.94:0/1307984551 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd790004220 con 0x7fd7a0100e60 2024-04-21T08:37:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.828+0000 7fd7a50f2640 1 -- 172.21.15.94:0/1307984551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a01a00d0 con 0x7fd7a0100e60 2024-04-21T08:37:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.828+0000 7fd787fff640 1 -- 172.21.15.94:0/1307984551 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7900043c0 con 0x7fd7a0100e60 2024-04-21T08:37:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.828+0000 7fd787fff640 1 -- 172.21.15.94:0/1307984551 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79000f450 con 0x7fd7a0100e60 2024-04-21T08:37:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.828+0000 7fd7a50f2640 1 -- 172.21.15.94:0/1307984551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a01a0670 con 0x7fd7a0100e60 2024-04-21T08:37:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.829+0000 7fd7a50f2640 1 -- 172.21.15.94:0/1307984551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a01a0930 con 0x7fd7a0100e60 2024-04-21T08:37:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.830+0000 7fd787fff640 1 -- 172.21.15.94:0/1307984551 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd790003610 con 0x7fd7a0100e60 2024-04-21T08:37:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.830+0000 7fd787fff640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd7880a3bb0 0x7fd7880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.831+0000 7fd79e59c640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd7880a3bb0 0x7fd7880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.831+0000 7fd787fff640 1 -- 172.21.15.94:0/1307984551 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd7900d0750 con 0x7fd7a0100e60 2024-04-21T08:37:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.831+0000 7fd79e59c640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd7880a3bb0 0x7fd7880a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7a01996d0 tx=0x7fd794005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:45.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:45.834+0000 7fd787fff640 1 -- 172.21.15.94:0/1307984551 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd790014030 con 0x7fd7a0100e60 2024-04-21T08:37:45.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:45 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2943651099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:37:45.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:45 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3109357077' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:37:45.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:45 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3641161678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:46.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:45 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2943651099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:37:46.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:45 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3109357077' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:37:46.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:45 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3641161678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:46.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.011+0000 7fd7a50f2640 1 -- 172.21.15.94:0/1307984551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd7a01a0b00 con 0x7fd7a0100e60 2024-04-21T08:37:46.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.012+0000 7fd787fff640 1 -- 172.21.15.94:0/1307984551 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fd79009c8a0 con 0x7fd7a0100e60 2024-04-21T08:37:46.012 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 -- 172.21.15.94:0/1307984551 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd7880a3bb0 msgr2=0x7fd7880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd7880a3bb0 0x7fd7880a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7a01996d0 tx=0x7fd794005f90 comp rx=0 tx=0).stop 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 -- 172.21.15.94:0/1307984551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a0100e60 msgr2=0x7fd7a01981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a0100e60 0x7fd7a01981c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd79000c970 tx=0x7fd79000ce30 comp rx=0 tx=0).stop 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 -- 172.21.15.94:0/1307984551 shutdown_connections 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd7880a3bb0 0x7fd7880a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7a0102a00 0x7fd7a019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7a0102040 0x7fd7a0198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 --2- 172.21.15.94:0/1307984551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a0100e60 0x7fd7a01981c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 -- 172.21.15.94:0/1307984551 >> 172.21.15.94:0/1307984551 conn(0x7fd7a00fc440 msgr2=0x7fd7a006a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 -- 172.21.15.94:0/1307984551 shutdown_connections 2024-04-21T08:37:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.014+0000 7fd7a50f2640 1 -- 172.21.15.94:0/1307984551 wait complete. 2024-04-21T08:37:46.024 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:37:46.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.125+0000 7f764a848640 1 -- 172.21.15.94:0/4023585685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7644105360 msgr2=0x7f7644107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.125+0000 7f764a848640 1 --2- 172.21.15.94:0/4023585685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7644105360 0x7f7644107750 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7634009900 tx=0x7f763402bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.125+0000 7f764a848640 1 -- 172.21.15.94:0/4023585685 shutdown_connections 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.125+0000 7f764a848640 1 --2- 172.21.15.94:0/4023585685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7644107c90 0x7f764410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.125+0000 7f764a848640 1 --2- 172.21.15.94:0/4023585685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7644105360 0x7f7644107750 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.125+0000 7f764a848640 1 --2- 172.21.15.94:0/4023585685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7644102a30 0x7f7644104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.125+0000 7f764a848640 1 -- 172.21.15.94:0/4023585685 >> 172.21.15.94:0/4023585685 conn(0x7f76440fc420 msgr2=0x7f76440fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.125+0000 7f764a848640 1 -- 172.21.15.94:0/4023585685 shutdown_connections 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.125+0000 7f764a848640 1 -- 172.21.15.94:0/4023585685 wait complete. 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.125+0000 7f764a848640 1 Processor -- start 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f764a848640 1 -- start start 2024-04-21T08:37:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f764a848640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7644102a30 0x7f7644198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f764a848640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7644105360 0x7f7644198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f764a848640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7644107c90 0x7f764419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f764a848640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f764410dab0 con 0x7f7644107c90 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f764a848640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f764410d930 con 0x7f7644105360 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f764a848640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f764410dc30 con 0x7f7644102a30 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f7648de5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7644107c90 0x7f764419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f7648de5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7644107c90 0x7f764419f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47496/0 (socket says 172.21.15.94:47496) 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f7643fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7644102a30 0x7f7644198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f7648de5640 1 -- 172.21.15.94:0/3397512261 learned_addr learned my addr 172.21.15.94:0/3397512261 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f76437fe640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7644105360 0x7f7644198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f7648de5640 1 -- 172.21.15.94:0/3397512261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7644102a30 msgr2=0x7f7644198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f7648de5640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7644102a30 0x7f7644198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f7648de5640 1 -- 172.21.15.94:0/3397512261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7644105360 msgr2=0x7f7644198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f7648de5640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7644105360 0x7f7644198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f7648de5640 1 -- 172.21.15.94:0/3397512261 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f764419fe50 con 0x7f7644107c90 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f76437fe640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7644105360 0x7f7644198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f7648de5640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7644107c90 0x7f764419f7e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f763c00eaa0 tx=0x7f763c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.126+0000 7f76417fa640 1 -- 172.21.15.94:0/3397512261 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763c00cdb0 con 0x7f7644107c90 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.127+0000 7f764a848640 1 -- 172.21.15.94:0/3397512261 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76441a0140 con 0x7f7644107c90 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.127+0000 7f76417fa640 1 -- 172.21.15.94:0/3397512261 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f763c005020 con 0x7f7644107c90 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.127+0000 7f76417fa640 1 -- 172.21.15.94:0/3397512261 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763c0116d0 con 0x7f7644107c90 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.127+0000 7f764a848640 1 -- 172.21.15.94:0/3397512261 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76441a0690 con 0x7f7644107c90 2024-04-21T08:37:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.128+0000 7f764a848640 1 -- 172.21.15.94:0/3397512261 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76440fdf10 con 0x7f7644107c90 2024-04-21T08:37:46.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.129+0000 7f76417fa640 1 -- 172.21.15.94:0/3397512261 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f763c0051d0 con 0x7f7644107c90 2024-04-21T08:37:46.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.129+0000 7f76417fa640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f762c0a3bb0 0x7f762c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.129+0000 7f7643fff640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f762c0a3bb0 0x7f762c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.129+0000 7f76417fa640 1 -- 172.21.15.94:0/3397512261 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f763c0d07d0 con 0x7f7644107c90 2024-04-21T08:37:46.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.130+0000 7f7643fff640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f762c0a3bb0 0x7f762c0a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7638005d00 tx=0x7f763800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:46.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.134+0000 7f76417fa640 1 -- 172.21.15.94:0/3397512261 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f763c016030 con 0x7f7644107c90 2024-04-21T08:37:46.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.312+0000 7f764a848640 1 -- 172.21.15.94:0/3397512261 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7644062b50 con 0x7f7644107c90 2024-04-21T08:37:46.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.313+0000 7f76417fa640 1 -- 172.21.15.94:0/3397512261 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f763c09cf70 con 0x7f7644107c90 2024-04-21T08:37:46.312 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.314+0000 7f764a848640 1 -- 172.21.15.94:0/3397512261 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f762c0a3bb0 msgr2=0x7f762c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f762c0a3bb0 0x7f762c0a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7638005d00 tx=0x7f763800b410 comp rx=0 tx=0).stop 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 -- 172.21.15.94:0/3397512261 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7644107c90 msgr2=0x7f764419f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7644107c90 0x7f764419f7e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f763c00eaa0 tx=0x7f763c00ef60 comp rx=0 tx=0).stop 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 -- 172.21.15.94:0/3397512261 shutdown_connections 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f762c0a3bb0 0x7f762c0a6070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7644107c90 0x7f764419f7e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7644105360 0x7f7644198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 --2- 172.21.15.94:0/3397512261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7644102a30 0x7f7644198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 -- 172.21.15.94:0/3397512261 >> 172.21.15.94:0/3397512261 conn(0x7f76440fc420 msgr2=0x7f76441048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 -- 172.21.15.94:0/3397512261 shutdown_connections 2024-04-21T08:37:46.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.315+0000 7f764a848640 1 -- 172.21.15.94:0/3397512261 wait complete. 2024-04-21T08:37:46.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:37:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 -- 172.21.15.94:0/3853813179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde4c101fa0 msgr2=0x7fde4c1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 --2- 172.21.15.94:0/3853813179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde4c101fa0 0x7fde4c1023c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fde48009ca0 tx=0x7fde4802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 -- 172.21.15.94:0/3853813179 shutdown_connections 2024-04-21T08:37:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 --2- 172.21.15.94:0/3853813179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde4c103b50 0x7fde4c10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 --2- 172.21.15.94:0/3853813179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde4c103190 0x7fde4c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 --2- 172.21.15.94:0/3853813179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde4c101fa0 0x7fde4c1023c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 -- 172.21.15.94:0/3853813179 >> 172.21.15.94:0/3853813179 conn(0x7fde4c0fd530 msgr2=0x7fde4c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 -- 172.21.15.94:0/3853813179 shutdown_connections 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 -- 172.21.15.94:0/3853813179 wait complete. 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 Processor -- start 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 -- start start 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde4c101fa0 0x7fde4c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.429+0000 7fde5425c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde4c103190 0x7fde4c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde5425c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde4c103b50 0x7fde4c19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde5425c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fde4c10db20 con 0x7fde4c103190 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde5425c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fde4c10d9a0 con 0x7fde4c103b50 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde5425c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fde4c10dca0 con 0x7fde4c101fa0 2024-04-21T08:37:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde517f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde4c103190 0x7fde4c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde517f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde4c103190 0x7fde4c1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47518/0 (socket says 172.21.15.94:47518) 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde517f7640 1 -- 172.21.15.94:0/1043895493 learned_addr learned my addr 172.21.15.94:0/1043895493 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde51ff8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde4c101fa0 0x7fde4c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde527f9640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde4c103b50 0x7fde4c19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde517f7640 1 -- 172.21.15.94:0/1043895493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde4c101fa0 msgr2=0x7fde4c1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde517f7640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde4c101fa0 0x7fde4c1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde517f7640 1 -- 172.21.15.94:0/1043895493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde4c103b50 msgr2=0x7fde4c19f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde517f7640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde4c103b50 0x7fde4c19f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde517f7640 1 -- 172.21.15.94:0/1043895493 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4c19fe80 con 0x7fde4c103190 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde527f9640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde4c103b50 0x7fde4c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde517f7640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde4c103190 0x7fde4c1987f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00b4a0 tx=0x7fde3c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde42ffd640 1 -- 172.21.15.94:0/1043895493 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c014070 con 0x7fde4c103190 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde42ffd640 1 -- 172.21.15.94:0/1043895493 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde3c010ac0 con 0x7fde4c103190 2024-04-21T08:37:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde5425c640 1 -- 172.21.15.94:0/1043895493 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde4c1a0170 con 0x7fde4c103190 2024-04-21T08:37:46.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde42ffd640 1 -- 172.21.15.94:0/1043895493 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c017c30 con 0x7fde4c103190 2024-04-21T08:37:46.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.430+0000 7fde5425c640 1 -- 172.21.15.94:0/1043895493 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde4c1a06c0 con 0x7fde4c103190 2024-04-21T08:37:46.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.431+0000 7fde5425c640 1 -- 172.21.15.94:0/1043895493 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde4c1093e0 con 0x7fde4c103190 2024-04-21T08:37:46.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.432+0000 7fde42ffd640 1 -- 172.21.15.94:0/1043895493 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fde3c017070 con 0x7fde4c103190 2024-04-21T08:37:46.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.433+0000 7fde42ffd640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde380a3bb0 0x7fde380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.433+0000 7fde51ff8640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde380a3bb0 0x7fde380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.433+0000 7fde42ffd640 1 -- 172.21.15.94:0/1043895493 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fde3c0cfd70 con 0x7fde4c103190 2024-04-21T08:37:46.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.433+0000 7fde51ff8640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde380a3bb0 0x7fde380a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fde48032380 tx=0x7fde48000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:46.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.436+0000 7fde42ffd640 1 -- 172.21.15.94:0/1043895493 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde3c09c510 con 0x7fde4c103190 2024-04-21T08:37:46.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.616+0000 7fde5425c640 1 -- 172.21.15.94:0/1043895493 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fde4c02d810 con 0x7fde4c103190 2024-04-21T08:37:46.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.616+0000 7fde42ffd640 1 -- 172.21.15.94:0/1043895493 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fde3c09be10 con 0x7fde4c103190 2024-04-21T08:37:46.616 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:37:46.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 -- 172.21.15.94:0/1043895493 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde380a3bb0 msgr2=0x7fde380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde380a3bb0 0x7fde380a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fde48032380 tx=0x7fde48000f40 comp rx=0 tx=0).stop 2024-04-21T08:37:46.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 -- 172.21.15.94:0/1043895493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde4c103190 msgr2=0x7fde4c1987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde4c103190 0x7fde4c1987f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00b4a0 tx=0x7fde3c00b960 comp rx=0 tx=0).stop 2024-04-21T08:37:46.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 -- 172.21.15.94:0/1043895493 shutdown_connections 2024-04-21T08:37:46.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fde380a3bb0 0x7fde380a6070 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde4c103b50 0x7fde4c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde4c103190 0x7fde4c1987f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 --2- 172.21.15.94:0/1043895493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde4c101fa0 0x7fde4c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 -- 172.21.15.94:0/1043895493 >> 172.21.15.94:0/1043895493 conn(0x7fde4c0fd530 msgr2=0x7fde4c1065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:46.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 -- 172.21.15.94:0/1043895493 shutdown_connections 2024-04-21T08:37:46.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.618+0000 7fde5425c640 1 -- 172.21.15.94:0/1043895493 wait complete. 2024-04-21T08:37:46.629 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:37:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.729+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2656016580 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a8102a10 msgr2=0x7f78a8104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.729+0000 7f78ad0c7640 1 --2- 172.21.15.94:0/2656016580 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a8102a10 0x7f78a8104e00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7898009ca0 tx=0x7f789802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.729+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2656016580 shutdown_connections 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.729+0000 7f78ad0c7640 1 --2- 172.21.15.94:0/2656016580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78a8105340 0x7f78a8107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.729+0000 7f78ad0c7640 1 --2- 172.21.15.94:0/2656016580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a8107c70 0x7f78a810a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.729+0000 7f78ad0c7640 1 --2- 172.21.15.94:0/2656016580 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a8102a10 0x7f78a8104e00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.729+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2656016580 >> 172.21.15.94:0/2656016580 conn(0x7f78a80fc400 msgr2=0x7f78a80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.729+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2656016580 shutdown_connections 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.729+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2656016580 wait complete. 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78ad0c7640 1 Processor -- start 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78ad0c7640 1 -- start start 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78ad0c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a8102a10 0x7f78a8198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78ad0c7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78a8105340 0x7f78a81987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78ad0c7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a8107c70 0x7f78a819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78ad0c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f78a810dad0 con 0x7f78a8102a10 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78ad0c7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f78a810d950 con 0x7f78a8107c70 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78ad0c7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f78a810dc50 con 0x7f78a8105340 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78a6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a8102a10 0x7f78a8198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78a6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a8102a10 0x7f78a8198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47530/0 (socket says 172.21.15.94:47530) 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78a6d9d640 1 -- 172.21.15.94:0/2427079636 learned_addr learned my addr 172.21.15.94:0/2427079636 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.730+0000 7f78a659c640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78a8105340 0x7f78a81987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f78a759e640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a8107c70 0x7f78a819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f78a6d9d640 1 -- 172.21.15.94:0/2427079636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78a8105340 msgr2=0x7f78a81987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f78a6d9d640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78a8105340 0x7f78a81987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f78a6d9d640 1 -- 172.21.15.94:0/2427079636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a8107c70 msgr2=0x7f78a819f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f78a6d9d640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a8107c70 0x7f78a819f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f78a6d9d640 1 -- 172.21.15.94:0/2427079636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a819feb0 con 0x7f78a8102a10 2024-04-21T08:37:46.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f78a6d9d640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a8102a10 0x7f78a8198290 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7898005e30 tx=0x7f7898034830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:46.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f788ffff640 1 -- 172.21.15.94:0/2427079636 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7898039040 con 0x7f78a8102a10 2024-04-21T08:37:46.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f788ffff640 1 -- 172.21.15.94:0/2427079636 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7898035c30 con 0x7f78a8102a10 2024-04-21T08:37:46.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2427079636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a81a0140 con 0x7f78a8102a10 2024-04-21T08:37:46.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f788ffff640 1 -- 172.21.15.94:0/2427079636 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78980042a0 con 0x7f78a8102a10 2024-04-21T08:37:46.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.731+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2427079636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a81a0630 con 0x7f78a8102a10 2024-04-21T08:37:46.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.732+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2427079636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a80fdef0 con 0x7f78a8102a10 2024-04-21T08:37:46.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.733+0000 7f788ffff640 1 -- 172.21.15.94:0/2427079636 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7898046030 con 0x7f78a8102a10 2024-04-21T08:37:46.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.733+0000 7f788ffff640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f78900a3bb0 0x7f78900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.734+0000 7f78a659c640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f78900a3bb0 0x7f78900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.734+0000 7f788ffff640 1 -- 172.21.15.94:0/2427079636 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f78980f2080 con 0x7f78a8102a10 2024-04-21T08:37:46.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.734+0000 7f78a659c640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f78900a3bb0 0x7f78900a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f78a81997a0 tx=0x7f789c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:46.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.738+0000 7f788ffff640 1 -- 172.21.15.94:0/2427079636 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f789804d060 con 0x7f78a8102a10 2024-04-21T08:37:46.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.927+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2427079636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f78a802d7e0 con 0x7f78a8102a10 2024-04-21T08:37:46.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.928+0000 7f788ffff640 1 -- 172.21.15.94:0/2427079636 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7f78980be8b0 con 0x7f78a8102a10 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/441167573 2024-04-22T08:30:29.483746+0000 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/2502723264 2024-04-22T08:30:29.483746+0000 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/613544885 2024-04-22T08:30:29.483746+0000 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/3877440809 2024-04-22T08:30:29.483746+0000 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3977681577 2024-04-22T08:30:03.511107+0000 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3965359185 2024-04-22T08:30:03.511107+0000 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/1761971189 2024-04-22T08:30:03.511107+0000 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/2629131251 2024-04-22T08:30:03.511107+0000 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/3877440809 2024-04-22T08:30:29.483746+0000 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/2629131251 2024-04-22T08:30:03.511107+0000 2024-04-21T08:37:46.928 INFO:teuthology.orchestra.run.smithi094.stderr:listed 10 entries 2024-04-21T08:37:46.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2427079636 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f78900a3bb0 msgr2=0x7f78900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f78900a3bb0 0x7f78900a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f78a81997a0 tx=0x7f789c00b390 comp rx=0 tx=0).stop 2024-04-21T08:37:46.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2427079636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a8102a10 msgr2=0x7f78a8198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a8102a10 0x7f78a8198290 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7898005e30 tx=0x7f7898034830 comp rx=0 tx=0).stop 2024-04-21T08:37:46.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2427079636 shutdown_connections 2024-04-21T08:37:46.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f78900a3bb0 0x7f78900a6070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78a8105340 0x7f78a81987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a8107c70 0x7f78a819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 --2- 172.21.15.94:0/2427079636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a8102a10 0x7f78a8198290 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2427079636 >> 172.21.15.94:0/2427079636 conn(0x7f78a80fc400 msgr2=0x7f78a8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:46.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2427079636 shutdown_connections 2024-04-21T08:37:46.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:46.930+0000 7f78ad0c7640 1 -- 172.21.15.94:0/2427079636 wait complete. 2024-04-21T08:37:46.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:46 smithi111 ceph-mon[31493]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 716 B/s wr, 0 op/s 2024-04-21T08:37:46.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:46 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3600945397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:37:46.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:46 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1307984551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:37:46.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:46 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3397512261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:37:46.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:46 smithi094 ceph-mon[27988]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 716 B/s wr, 0 op/s 2024-04-21T08:37:46.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:46 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3600945397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:37:46.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:46 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1307984551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:37:46.940 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:46 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3397512261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:37:46.941 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi094.front.sepia.ceph.com... 2024-04-21T08:37:46.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:46.941 DEBUG:teuthology.orchestra.run.smithi094:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:37:46.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:ceph-fuse[71596]: fuse finished with error 0 and tester_r 0 2024-04-21T08:37:46.964 INFO:teuthology.orchestra.run:waiting for 300 2024-04-21T08:37:47.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:46 smithi173 ceph-mon[31428]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 716 B/s wr, 0 op/s 2024-04-21T08:37:47.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:46 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3600945397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:37:47.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:46 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1307984551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:37:47.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:46 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3397512261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:37:47.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:47 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1043895493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:37:47.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:47 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2427079636' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:37:48.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:47 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1043895493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:37:48.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:47 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2427079636' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:37:48.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:47 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1043895493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:37:48.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:47 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2427079636' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:37:48.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:48 smithi111 ceph-mon[31493]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T08:37:49.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:48 smithi094 ceph-mon[27988]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T08:37:49.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:48 smithi173 ceph-mon[31428]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T08:37:50.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:50 smithi111 ceph-mon[31493]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:50 smithi094 ceph-mon[27988]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:51.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:50 smithi173 ceph-mon[31428]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:52.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:52 smithi111 ceph-mon[31493]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:53.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:52 smithi094 ceph-mon[27988]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:53.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:52 smithi173 ceph-mon[31428]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:53.066 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:37:53.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:37:53.066 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-21T08:37:53.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:37:53.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 -- 172.21.15.94:0/2678970463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8af0105370 msgr2=0x7f8af0107760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 --2- 172.21.15.94:0/2678970463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8af0105370 0x7f8af0107760 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0009900 tx=0x7f8ae002bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:53.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 -- 172.21.15.94:0/2678970463 shutdown_connections 2024-04-21T08:37:53.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 --2- 172.21.15.94:0/2678970463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af0107ca0 0x7f8af010a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 --2- 172.21.15.94:0/2678970463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8af0105370 0x7f8af0107760 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 --2- 172.21.15.94:0/2678970463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8af0102a40 0x7f8af0104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 -- 172.21.15.94:0/2678970463 >> 172.21.15.94:0/2678970463 conn(0x7f8af00fc470 msgr2=0x7f8af00fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:53.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 -- 172.21.15.94:0/2678970463 shutdown_connections 2024-04-21T08:37:53.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 -- 172.21.15.94:0/2678970463 wait complete. 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 Processor -- start 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.237+0000 7f8af7499640 1 -- start start 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af7499640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8af0102a40 0x7f8af0198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af7499640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8af0105370 0x7f8af01987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af7499640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af0107ca0 0x7f8af019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af7499640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8af010db40 con 0x7f8af0102a40 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af7499640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8af010d9c0 con 0x7f8af0107ca0 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af7499640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8af010dcc0 con 0x7f8af0105370 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5235640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8af0102a40 0x7f8af0198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5235640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8af0102a40 0x7f8af0198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47542/0 (socket says 172.21.15.94:47542) 2024-04-21T08:37:53.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5235640 1 -- 172.21.15.94:0/2044273958 learned_addr learned my addr 172.21.15.94:0/2044273958 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af4a34640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8af0105370 0x7f8af01987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5a36640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af0107ca0 0x7f8af019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5235640 1 -- 172.21.15.94:0/2044273958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8af0105370 msgr2=0x7f8af01987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5235640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8af0105370 0x7f8af01987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5235640 1 -- 172.21.15.94:0/2044273958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af0107ca0 msgr2=0x7f8af019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5235640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af0107ca0 0x7f8af019f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5235640 1 -- 172.21.15.94:0/2044273958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af019fe30 con 0x7f8af0102a40 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5a36640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af0107ca0 0x7f8af019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af5235640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8af0102a40 0x7f8af0198260 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800ba50 tx=0x7f8ae800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8ae67fc640 1 -- 172.21.15.94:0/2044273958 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8019070 con 0x7f8af0102a40 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8ae67fc640 1 -- 172.21.15.94:0/2044273958 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ae800cbe0 con 0x7f8af0102a40 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8ae67fc640 1 -- 172.21.15.94:0/2044273958 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae80075d0 con 0x7f8af0102a40 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af7499640 1 -- 172.21.15.94:0/2044273958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8af01a0120 con 0x7f8af0102a40 2024-04-21T08:37:53.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.238+0000 7f8af7499640 1 -- 172.21.15.94:0/2044273958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8af01a06a0 con 0x7f8af0102a40 2024-04-21T08:37:53.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.239+0000 7f8af7499640 1 -- 172.21.15.94:0/2044273958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8af00fdf60 con 0x7f8af0102a40 2024-04-21T08:37:53.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.240+0000 7f8ae67fc640 1 -- 172.21.15.94:0/2044273958 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8ae801c070 con 0x7f8af0102a40 2024-04-21T08:37:53.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.241+0000 7f8ae67fc640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8adc0a3bb0 0x7f8adc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.241+0000 7f8af4a34640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8adc0a3bb0 0x7f8adc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.241+0000 7f8ae67fc640 1 -- 172.21.15.94:0/2044273958 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8ae80d0220 con 0x7f8af0102a40 2024-04-21T08:37:53.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.241+0000 7f8af4a34640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8adc0a3bb0 0x7f8adc0a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ae002c840 tx=0x7f8ae0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:53.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.245+0000 7f8ae67fc640 1 -- 172.21.15.94:0/2044273958 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ae809c9c0 con 0x7f8af0102a40 2024-04-21T08:37:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.421+0000 7f8af7499640 1 -- 172.21.15.94:0/2044273958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8af0062b50 con 0x7f8af0102a40 2024-04-21T08:37:53.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.423+0000 7f8ae67fc640 1 -- 172.21.15.94:0/2044273958 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24092 (secure 0 0 0) 0x7f8ae809c2c0 con 0x7f8af0102a40 2024-04-21T08:37:53.422 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:53.422 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:37:35.939829+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 -- 172.21.15.94:0/2044273958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8adc0a3bb0 msgr2=0x7f8adc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8adc0a3bb0 0x7f8adc0a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ae002c840 tx=0x7f8ae0000bc0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 -- 172.21.15.94:0/2044273958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8af0102a40 msgr2=0x7f8af0198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8af0102a40 0x7f8af0198260 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800ba50 tx=0x7f8ae800bf10 comp rx=0 tx=0).stop 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 -- 172.21.15.94:0/2044273958 shutdown_connections 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8adc0a3bb0 0x7f8adc0a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8af0105370 0x7f8af01987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af0107ca0 0x7f8af019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 --2- 172.21.15.94:0/2044273958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8af0102a40 0x7f8af0198260 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 -- 172.21.15.94:0/2044273958 >> 172.21.15.94:0/2044273958 conn(0x7f8af00fc470 msgr2=0x7f8af0103380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:53.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 -- 172.21.15.94:0/2044273958 shutdown_connections 2024-04-21T08:37:53.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.425+0000 7f8af7499640 1 -- 172.21.15.94:0/2044273958 wait complete. 2024-04-21T08:37:53.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:37:53.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.534+0000 7fed9469a640 1 -- 172.21.15.94:0/1238407833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed8c107c70 msgr2=0x7fed8c10a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.534+0000 7fed9469a640 1 --2- 172.21.15.94:0/1238407833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed8c107c70 0x7fed8c10a0c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fed84009c90 tx=0x7fed8402c0a0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.534+0000 7fed9469a640 1 -- 172.21.15.94:0/1238407833 shutdown_connections 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.534+0000 7fed9469a640 1 --2- 172.21.15.94:0/1238407833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c105340 0x7fed8c107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.534+0000 7fed9469a640 1 --2- 172.21.15.94:0/1238407833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed8c107c70 0x7fed8c10a0c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.534+0000 7fed9469a640 1 --2- 172.21.15.94:0/1238407833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8c102a10 0x7fed8c104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.534+0000 7fed9469a640 1 -- 172.21.15.94:0/1238407833 >> 172.21.15.94:0/1238407833 conn(0x7fed8c0fc420 msgr2=0x7fed8c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.534+0000 7fed9469a640 1 -- 172.21.15.94:0/1238407833 shutdown_connections 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.534+0000 7fed9469a640 1 -- 172.21.15.94:0/1238407833 wait complete. 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed9469a640 1 Processor -- start 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed9469a640 1 -- start start 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed9469a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8c102a10 0x7fed8c1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed9469a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed8c105340 0x7fed8c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed9469a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c107c70 0x7fed8c19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed9469a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c10dad0 con 0x7fed8c105340 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed9469a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c10d950 con 0x7fed8c102a10 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed9469a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c10dc50 con 0x7fed8c107c70 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed91c35640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed8c105340 0x7fed8c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed91c35640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed8c105340 0x7fed8c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47560/0 (socket says 172.21.15.94:47560) 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed91c35640 1 -- 172.21.15.94:0/2538531564 learned_addr learned my addr 172.21.15.94:0/2538531564 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed92436640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8c102a10 0x7fed8c1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed92c37640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c107c70 0x7fed8c19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed91c35640 1 -- 172.21.15.94:0/2538531564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c107c70 msgr2=0x7fed8c19f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed91c35640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c107c70 0x7fed8c19f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed91c35640 1 -- 172.21.15.94:0/2538531564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8c102a10 msgr2=0x7fed8c1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed91c35640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8c102a10 0x7fed8c1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed91c35640 1 -- 172.21.15.94:0/2538531564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8c19fdf0 con 0x7fed8c105340 2024-04-21T08:37:53.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed92436640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8c102a10 0x7fed8c1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:53.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.535+0000 7fed91c35640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed8c105340 0x7fed8c198710 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00d740 tx=0x7fed7c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:53.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.536+0000 7fed837fe640 1 -- 172.21.15.94:0/2538531564 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c017070 con 0x7fed8c105340 2024-04-21T08:37:53.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.536+0000 7fed9469a640 1 -- 172.21.15.94:0/2538531564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed8c1a00e0 con 0x7fed8c105340 2024-04-21T08:37:53.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.536+0000 7fed837fe640 1 -- 172.21.15.94:0/2538531564 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed7c00de50 con 0x7fed8c105340 2024-04-21T08:37:53.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.536+0000 7fed837fe640 1 -- 172.21.15.94:0/2538531564 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c004ce0 con 0x7fed8c105340 2024-04-21T08:37:53.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.536+0000 7fed9469a640 1 -- 172.21.15.94:0/2538531564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed8c1a0630 con 0x7fed8c105340 2024-04-21T08:37:53.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.536+0000 7fed9469a640 1 -- 172.21.15.94:0/2538531564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed8c0fdf10 con 0x7fed8c105340 2024-04-21T08:37:53.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.538+0000 7fed837fe640 1 -- 172.21.15.94:0/2538531564 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fed7c004e80 con 0x7fed8c105340 2024-04-21T08:37:53.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.538+0000 7fed837fe640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed780a3bb0 0x7fed780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.538+0000 7fed92436640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed780a3bb0 0x7fed780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.538+0000 7fed837fe640 1 -- 172.21.15.94:0/2538531564 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fed7c0d03a0 con 0x7fed8c105340 2024-04-21T08:37:53.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.539+0000 7fed92436640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed780a3bb0 0x7fed780a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed88005d00 tx=0x7fed88005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:53.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.542+0000 7fed837fe640 1 -- 172.21.15.94:0/2538531564 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed7c09cb40 con 0x7fed8c105340 2024-04-21T08:37:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.721+0000 7fed9469a640 1 -- 172.21.15.94:0/2538531564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed8c02d7c0 con 0x7fed8c105340 2024-04-21T08:37:53.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.723+0000 7fed837fe640 1 -- 172.21.15.94:0/2538531564 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24092 (secure 0 0 0) 0x7fed7c09c440 con 0x7fed8c105340 2024-04-21T08:37:53.722 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:53.722 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:37:35.939829+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:37:53.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.724+0000 7fed9469a640 1 -- 172.21.15.94:0/2538531564 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed780a3bb0 msgr2=0x7fed780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.724+0000 7fed9469a640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed780a3bb0 0x7fed780a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed88005d00 tx=0x7fed88005c70 comp rx=0 tx=0).stop 2024-04-21T08:37:53.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.725+0000 7fed9469a640 1 -- 172.21.15.94:0/2538531564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed8c105340 msgr2=0x7fed8c198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.725+0000 7fed9469a640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed8c105340 0x7fed8c198710 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00d740 tx=0x7fed7c00dc00 comp rx=0 tx=0).stop 2024-04-21T08:37:53.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.725+0000 7fed9469a640 1 -- 172.21.15.94:0/2538531564 shutdown_connections 2024-04-21T08:37:53.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.725+0000 7fed9469a640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed780a3bb0 0x7fed780a6070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.725+0000 7fed9469a640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c107c70 0x7fed8c19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.725+0000 7fed9469a640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed8c105340 0x7fed8c198710 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.725+0000 7fed9469a640 1 --2- 172.21.15.94:0/2538531564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8c102a10 0x7fed8c1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.725+0000 7fed9469a640 1 -- 172.21.15.94:0/2538531564 >> 172.21.15.94:0/2538531564 conn(0x7fed8c0fc420 msgr2=0x7fed8c103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:53.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.725+0000 7fed9469a640 1 -- 172.21.15.94:0/2538531564 shutdown_connections 2024-04-21T08:37:53.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.725+0000 7fed9469a640 1 -- 172.21.15.94:0/2538531564 wait complete. 2024-04-21T08:37:53.733 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:53 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2044273958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:53.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-21T08:37:53.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 -- 172.21.15.94:0/2647455799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd180102ad0 msgr2=0x7fd180104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 --2- 172.21.15.94:0/2647455799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd180102ad0 0x7fd180104ec0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd170009900 tx=0x7fd17002bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:53.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 -- 172.21.15.94:0/2647455799 shutdown_connections 2024-04-21T08:37:53.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 --2- 172.21.15.94:0/2647455799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd180107d30 0x7fd18010a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 --2- 172.21.15.94:0/2647455799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd180105400 0x7fd1801077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 --2- 172.21.15.94:0/2647455799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd180102ad0 0x7fd180104ec0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 -- 172.21.15.94:0/2647455799 >> 172.21.15.94:0/2647455799 conn(0x7fd1800fc400 msgr2=0x7fd1800fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 -- 172.21.15.94:0/2647455799 shutdown_connections 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 -- 172.21.15.94:0/2647455799 wait complete. 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 Processor -- start 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.838+0000 7fd184c14640 1 -- start start 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd184c14640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd180102ad0 0x7fd180198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd184c14640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd180105400 0x7fd180198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd184c14640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd180107d30 0x7fd18019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd184c14640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd18010dae0 con 0x7fd180107d30 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd184c14640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd18010d960 con 0x7fd180105400 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd184c14640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd18010dc60 con 0x7fd180102ad0 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd180107d30 0x7fd18019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd180107d30 0x7fd18019f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51548/0 (socket says 172.21.15.94:51548) 2024-04-21T08:37:53.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17ed9d640 1 -- 172.21.15.94:0/954451062 learned_addr learned my addr 172.21.15.94:0/954451062 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17dd9b640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd180105400 0x7fd180198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17e59c640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd180102ad0 0x7fd180198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17ed9d640 1 -- 172.21.15.94:0/954451062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd180102ad0 msgr2=0x7fd180198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17ed9d640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd180102ad0 0x7fd180198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17ed9d640 1 -- 172.21.15.94:0/954451062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd180105400 msgr2=0x7fd180198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17ed9d640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd180105400 0x7fd180198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17ed9d640 1 -- 172.21.15.94:0/954451062 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18019fe40 con 0x7fd180107d30 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17dd9b640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd180105400 0x7fd180198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd17ed9d640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd180107d30 0x7fd18019f7d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd17800eaa0 tx=0x7fd17800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd16f7fe640 1 -- 172.21.15.94:0/954451062 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17800cdb0 con 0x7fd180107d30 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd184c14640 1 -- 172.21.15.94:0/954451062 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1801a0130 con 0x7fd180107d30 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd16f7fe640 1 -- 172.21.15.94:0/954451062 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd178005020 con 0x7fd180107d30 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd16f7fe640 1 -- 172.21.15.94:0/954451062 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1780116d0 con 0x7fd180107d30 2024-04-21T08:37:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.839+0000 7fd184c14640 1 -- 172.21.15.94:0/954451062 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1801a06d0 con 0x7fd180107d30 2024-04-21T08:37:53.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.840+0000 7fd184c14640 1 -- 172.21.15.94:0/954451062 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1800fdef0 con 0x7fd180107d30 2024-04-21T08:37:53.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.842+0000 7fd16f7fe640 1 -- 172.21.15.94:0/954451062 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd1780054a0 con 0x7fd180107d30 2024-04-21T08:37:53.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.842+0000 7fd16f7fe640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd1680a3bb0 0x7fd1680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:53.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.842+0000 7fd17e59c640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd1680a3bb0 0x7fd1680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:53.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.842+0000 7fd16f7fe640 1 -- 172.21.15.94:0/954451062 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd1780d0c10 con 0x7fd180107d30 2024-04-21T08:37:53.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.843+0000 7fd17e59c640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd1680a3bb0 0x7fd1680a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd1700096c0 tx=0x7fd170000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:53.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:53.846+0000 7fd16f7fe640 1 -- 172.21.15.94:0/954451062 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd178016030 con 0x7fd180107d30 2024-04-21T08:37:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:53 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2044273958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:54.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:53 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2044273958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:54.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.062+0000 7fd184c14640 1 -- 172.21.15.94:0/954451062 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd180062b50 con 0x7fd180107d30 2024-04-21T08:37:54.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.640+0000 7fd16f7fe640 1 -- 172.21.15.94:0/954451062 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7fd17809d3b0 con 0x7fd180107d30 2024-04-21T08:37:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 -- 172.21.15.94:0/954451062 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd1680a3bb0 msgr2=0x7fd1680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd1680a3bb0 0x7fd1680a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd1700096c0 tx=0x7fd170000bc0 comp rx=0 tx=0).stop 2024-04-21T08:37:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 -- 172.21.15.94:0/954451062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd180107d30 msgr2=0x7fd18019f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd180107d30 0x7fd18019f7d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd17800eaa0 tx=0x7fd17800ef60 comp rx=0 tx=0).stop 2024-04-21T08:37:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 -- 172.21.15.94:0/954451062 shutdown_connections 2024-04-21T08:37:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd1680a3bb0 0x7fd1680a6070 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd180107d30 0x7fd18019f7d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd180105400 0x7fd180198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 --2- 172.21.15.94:0/954451062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd180102ad0 0x7fd180198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 -- 172.21.15.94:0/954451062 >> 172.21.15.94:0/954451062 conn(0x7fd1800fc400 msgr2=0x7fd180103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:54.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 -- 172.21.15.94:0/954451062 shutdown_connections 2024-04-21T08:37:54.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.643+0000 7fd184c14640 1 -- 172.21.15.94:0/954451062 wait complete. 2024-04-21T08:37:54.654 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-21T08:37:54.655 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-21T08:37:54.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.763+0000 7f2399001640 1 -- 172.21.15.94:0/2731732523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2394101f90 msgr2=0x7f2394102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.763+0000 7f2399001640 1 --2- 172.21.15.94:0/2731732523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2394101f90 0x7f2394102410 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2384009900 tx=0x7f238402bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.763+0000 7f2399001640 1 -- 172.21.15.94:0/2731732523 shutdown_connections 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.763+0000 7f2399001640 1 --2- 172.21.15.94:0/2731732523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2394102950 0x7f23941091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.763+0000 7f2399001640 1 --2- 172.21.15.94:0/2731732523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2394101f90 0x7f2394102410 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.763+0000 7f2399001640 1 --2- 172.21.15.94:0/2731732523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394100da0 0x7f23941011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.763+0000 7f2399001640 1 -- 172.21.15.94:0/2731732523 >> 172.21.15.94:0/2731732523 conn(0x7f23940fc310 msgr2=0x7f23940fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.763+0000 7f2399001640 1 -- 172.21.15.94:0/2731732523 shutdown_connections 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.763+0000 7f2399001640 1 -- 172.21.15.94:0/2731732523 wait complete. 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.763+0000 7f2399001640 1 Processor -- start 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f2399001640 1 -- start start 2024-04-21T08:37:54.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f2399001640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394100da0 0x7f2394198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f2399001640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2394101f90 0x7f2394198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f2399001640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2394102950 0x7f239419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f2399001640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f239410d9f0 con 0x7f2394101f90 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f2399001640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f239410d870 con 0x7f2394102950 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f2399001640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f239410db70 con 0x7f2394100da0 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2394101f90 0x7f2394198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2394101f90 0x7f2394198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51574/0 (socket says 172.21.15.94:51574) 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239259c640 1 -- 172.21.15.94:0/1842511306 learned_addr learned my addr 172.21.15.94:0/1842511306 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f2392d9d640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394100da0 0x7f2394198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239359e640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2394102950 0x7f239419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239259c640 1 -- 172.21.15.94:0/1842511306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394100da0 msgr2=0x7f2394198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239259c640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394100da0 0x7f2394198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239259c640 1 -- 172.21.15.94:0/1842511306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2394102950 msgr2=0x7f239419f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239259c640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2394102950 0x7f239419f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239259c640 1 -- 172.21.15.94:0/1842511306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239419fc40 con 0x7f2394101f90 2024-04-21T08:37:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239359e640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2394102950 0x7f239419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f239259c640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2394101f90 0x7f2394198560 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2384036040 tx=0x7f23840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.764+0000 7f237bfff640 1 -- 172.21.15.94:0/1842511306 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2384043020 con 0x7f2394101f90 2024-04-21T08:37:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.765+0000 7f237bfff640 1 -- 172.21.15.94:0/1842511306 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23840054c0 con 0x7f2394101f90 2024-04-21T08:37:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.765+0000 7f2399001640 1 -- 172.21.15.94:0/1842511306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f239419fed0 con 0x7f2394101f90 2024-04-21T08:37:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.765+0000 7f237bfff640 1 -- 172.21.15.94:0/1842511306 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2384003db0 con 0x7f2394101f90 2024-04-21T08:37:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.765+0000 7f2399001640 1 -- 172.21.15.94:0/1842511306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23941a0390 con 0x7f2394101f90 2024-04-21T08:37:54.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.766+0000 7f2399001640 1 -- 172.21.15.94:0/1842511306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23941080c0 con 0x7f2394101f90 2024-04-21T08:37:54.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.766+0000 7f237bfff640 1 -- 172.21.15.94:0/1842511306 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2384046020 con 0x7f2394101f90 2024-04-21T08:37:54.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.767+0000 7f237bfff640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f237c0a3bb0 0x7f237c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:54.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.767+0000 7f2392d9d640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f237c0a3bb0 0x7f237c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:54.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.767+0000 7f237bfff640 1 -- 172.21.15.94:0/1842511306 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f23840f2af0 con 0x7f2394101f90 2024-04-21T08:37:54.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.767+0000 7f2392d9d640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f237c0a3bb0 0x7f237c0a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f23880058c0 tx=0x7f238800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:54.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.771+0000 7f237bfff640 1 -- 172.21.15.94:0/1842511306 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f238404d020 con 0x7f2394101f90 2024-04-21T08:37:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:54 smithi111 ceph-mon[31493]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:54 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2538531564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:54 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/954451062' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:37:54.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:54.988+0000 7f2399001640 1 -- 172.21.15.94:0/1842511306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f2394062b50 con 0x7f2394101f90 2024-04-21T08:37:55.009 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:54 smithi094 ceph-mon[27988]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:54 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2538531564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:54 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/954451062' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:37:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:54 smithi173 ceph-mon[31428]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:54 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2538531564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:37:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:54 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/954451062' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:37:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.646+0000 7f237bfff640 1 -- 172.21.15.94:0/1842511306 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f23840bf290 con 0x7f2394101f90 2024-04-21T08:37:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 -- 172.21.15.94:0/1842511306 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f237c0a3bb0 msgr2=0x7f237c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f237c0a3bb0 0x7f237c0a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f23880058c0 tx=0x7f238800a2d0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 -- 172.21.15.94:0/1842511306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2394101f90 msgr2=0x7f2394198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2394101f90 0x7f2394198560 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2384036040 tx=0x7f23840042f0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 -- 172.21.15.94:0/1842511306 shutdown_connections 2024-04-21T08:37:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f237c0a3bb0 0x7f237c0a6070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2394102950 0x7f239419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2394101f90 0x7f2394198560 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 --2- 172.21.15.94:0/1842511306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394100da0 0x7f2394198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 -- 172.21.15.94:0/1842511306 >> 172.21.15.94:0/1842511306 conn(0x7f23940fc310 msgr2=0x7f2394105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.648+0000 7f2399001640 1 -- 172.21.15.94:0/1842511306 shutdown_connections 2024-04-21T08:37:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.649+0000 7f2399001640 1 -- 172.21.15.94:0/1842511306 wait complete. 2024-04-21T08:37:55.661 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-21T08:37:55.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 -- 172.21.15.94:0/4037122713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f1c102a90 msgr2=0x7f0f1c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 --2- 172.21.15.94:0/4037122713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f1c102a90 0x7f0f1c104e80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009900 tx=0x7f0f1002bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 -- 172.21.15.94:0/4037122713 shutdown_connections 2024-04-21T08:37:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 --2- 172.21.15.94:0/4037122713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c107cf0 0x7f0f1c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 --2- 172.21.15.94:0/4037122713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f1c1053c0 0x7f0f1c1077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 --2- 172.21.15.94:0/4037122713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f1c102a90 0x7f0f1c104e80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 -- 172.21.15.94:0/4037122713 >> 172.21.15.94:0/4037122713 conn(0x7f0f1c0fc440 msgr2=0x7f0f1c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 -- 172.21.15.94:0/4037122713 shutdown_connections 2024-04-21T08:37:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 -- 172.21.15.94:0/4037122713 wait complete. 2024-04-21T08:37:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 Processor -- start 2024-04-21T08:37:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.765+0000 7f0f2296d640 1 -- start start 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f2296d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c102a90 0x7f0f1c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f2296d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f1c1053c0 0x7f0f1c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f2296d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f1c107cf0 0x7f0f1c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f2296d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c10daf0 con 0x7f0f1c107cf0 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f2296d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c10d970 con 0x7f0f1c1053c0 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f2296d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c10dc70 con 0x7f0f1c102a90 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f20f0a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f1c107cf0 0x7f0f1c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f20f0a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f1c107cf0 0x7f0f1c19f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51586/0 (socket says 172.21.15.94:51586) 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f20f0a640 1 -- 172.21.15.94:0/65360151 learned_addr learned my addr 172.21.15.94:0/65360151 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f1bfff640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c102a90 0x7f0f1c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f1b7fe640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f1c1053c0 0x7f0f1c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f20f0a640 1 -- 172.21.15.94:0/65360151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c102a90 msgr2=0x7f0f1c198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f20f0a640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c102a90 0x7f0f1c198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f20f0a640 1 -- 172.21.15.94:0/65360151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f1c1053c0 msgr2=0x7f0f1c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f20f0a640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f1c1053c0 0x7f0f1c198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f20f0a640 1 -- 172.21.15.94:0/65360151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1c19fe50 con 0x7f0f1c107cf0 2024-04-21T08:37:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f20f0a640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f1c107cf0 0x7f0f1c19f7e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400eaa0 tx=0x7f0f1400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f197fa640 1 -- 172.21.15.94:0/65360151 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1400cdb0 con 0x7f0f1c107cf0 2024-04-21T08:37:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f197fa640 1 -- 172.21.15.94:0/65360151 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f14005020 con 0x7f0f1c107cf0 2024-04-21T08:37:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f2296d640 1 -- 172.21.15.94:0/65360151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1c1a0140 con 0x7f0f1c107cf0 2024-04-21T08:37:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f197fa640 1 -- 172.21.15.94:0/65360151 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f140116d0 con 0x7f0f1c107cf0 2024-04-21T08:37:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.766+0000 7f0f2296d640 1 -- 172.21.15.94:0/65360151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f1c1a0690 con 0x7f0f1c107cf0 2024-04-21T08:37:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.767+0000 7f0f2296d640 1 -- 172.21.15.94:0/65360151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f1c0fdf30 con 0x7f0f1c107cf0 2024-04-21T08:37:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.768+0000 7f0f197fa640 1 -- 172.21.15.94:0/65360151 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0f140051d0 con 0x7f0f1c107cf0 2024-04-21T08:37:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.769+0000 7f0f197fa640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0f040a3bb0 0x7f0f040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.769+0000 7f0f1bfff640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0f040a3bb0 0x7f0f040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.769+0000 7f0f197fa640 1 -- 172.21.15.94:0/65360151 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0f140d0750 con 0x7f0f1c107cf0 2024-04-21T08:37:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.770+0000 7f0f1bfff640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0f040a3bb0 0x7f0f040a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009230 tx=0x7f0f10005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.774+0000 7f0f197fa640 1 -- 172.21.15.94:0/65360151 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f14016030 con 0x7f0f1c107cf0 2024-04-21T08:37:55.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:55 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/954451062' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:37:55.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:55 smithi111 ceph-mon[31493]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:55.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:55 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1842511306' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-21T08:37:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:55.986+0000 7f0f2296d640 1 -- 172.21.15.94:0/65360151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f0f1c062b50 con 0x7f0f1c107cf0 2024-04-21T08:37:56.009 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:55 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/954451062' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:37:56.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:55 smithi094 ceph-mon[27988]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:56.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:55 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1842511306' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-21T08:37:56.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:55 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/954451062' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:37:56.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:55 smithi173 ceph-mon[31428]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:56.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:55 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1842511306' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-21T08:37:56.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.651+0000 7f0f197fa640 1 -- 172.21.15.94:0/65360151 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f0f1409c8a0 con 0x7f0f1c107cf0 2024-04-21T08:37:56.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.653+0000 7f0f2296d640 1 -- 172.21.15.94:0/65360151 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0f040a3bb0 msgr2=0x7f0f040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0f040a3bb0 0x7f0f040a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009230 tx=0x7f0f10005fd0 comp rx=0 tx=0).stop 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 -- 172.21.15.94:0/65360151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f1c107cf0 msgr2=0x7f0f1c19f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f1c107cf0 0x7f0f1c19f7e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400eaa0 tx=0x7f0f1400ef60 comp rx=0 tx=0).stop 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 -- 172.21.15.94:0/65360151 shutdown_connections 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0f040a3bb0 0x7f0f040a6070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f1c107cf0 0x7f0f1c19f7e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f1c1053c0 0x7f0f1c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 --2- 172.21.15.94:0/65360151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c102a90 0x7f0f1c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 -- 172.21.15.94:0/65360151 >> 172.21.15.94:0/65360151 conn(0x7f0f1c0fc440 msgr2=0x7f0f1c103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 -- 172.21.15.94:0/65360151 shutdown_connections 2024-04-21T08:37:56.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.654+0000 7f0f2296d640 1 -- 172.21.15.94:0/65360151 wait complete. 2024-04-21T08:37:56.668 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-21T08:37:56.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.773+0000 7fed872c1640 1 -- 172.21.15.94:0/751555987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed80101fa0 msgr2=0x7fed801023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:56.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.773+0000 7fed872c1640 1 --2- 172.21.15.94:0/751555987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed80101fa0 0x7fed801023c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fed7c009900 tx=0x7fed7c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:56.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.773+0000 7fed872c1640 1 -- 172.21.15.94:0/751555987 shutdown_connections 2024-04-21T08:37:56.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.773+0000 7fed872c1640 1 --2- 172.21.15.94:0/751555987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80103190 0x7fed80103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.773+0000 7fed872c1640 1 --2- 172.21.15.94:0/751555987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed80103b50 0x7fed8010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.773+0000 7fed872c1640 1 --2- 172.21.15.94:0/751555987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed80101fa0 0x7fed801023c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.773+0000 7fed872c1640 1 -- 172.21.15.94:0/751555987 >> 172.21.15.94:0/751555987 conn(0x7fed800fd530 msgr2=0x7fed800ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.773+0000 7fed872c1640 1 -- 172.21.15.94:0/751555987 shutdown_connections 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.773+0000 7fed872c1640 1 -- 172.21.15.94:0/751555987 wait complete. 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed872c1640 1 Processor -- start 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed872c1640 1 -- start start 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed872c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed80101fa0 0x7fed80075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed872c1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed80103190 0x7fed80074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed872c1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80103b50 0x7fed80074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed872c1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fed8010dae0 con 0x7fed80101fa0 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed872c1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fed8010d960 con 0x7fed80103190 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed872c1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed8010dc60 con 0x7fed80103b50 2024-04-21T08:37:56.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8505d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed80101fa0 0x7fed80075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8505d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed80101fa0 0x7fed80075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51610/0 (socket says 172.21.15.94:51610) 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8505d640 1 -- 172.21.15.94:0/4185737442 learned_addr learned my addr 172.21.15.94:0/4185737442 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8485c640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed80103190 0x7fed80074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8585e640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80103b50 0x7fed80074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8505d640 1 -- 172.21.15.94:0/4185737442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80103b50 msgr2=0x7fed80074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8505d640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80103b50 0x7fed80074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8505d640 1 -- 172.21.15.94:0/4185737442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed80103190 msgr2=0x7fed80074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8505d640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed80103190 0x7fed80074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8505d640 1 -- 172.21.15.94:0/4185737442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed80075110 con 0x7fed80101fa0 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed8505d640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed80101fa0 0x7fed80075d00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fed7c000c00 tx=0x7fed7c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.774+0000 7fed767fc640 1 -- 172.21.15.94:0/4185737442 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c043020 con 0x7fed80101fa0 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.775+0000 7fed872c1640 1 -- 172.21.15.94:0/4185737442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed801a2350 con 0x7fed80101fa0 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.775+0000 7fed767fc640 1 -- 172.21.15.94:0/4185737442 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed7c004aa0 con 0x7fed80101fa0 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.775+0000 7fed767fc640 1 -- 172.21.15.94:0/4185737442 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c003d00 con 0x7fed80101fa0 2024-04-21T08:37:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.775+0000 7fed872c1640 1 -- 172.21.15.94:0/4185737442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed801a28f0 con 0x7fed80101fa0 2024-04-21T08:37:56.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.776+0000 7fed872c1640 1 -- 172.21.15.94:0/4185737442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed80109480 con 0x7fed80101fa0 2024-04-21T08:37:56.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.777+0000 7fed767fc640 1 -- 172.21.15.94:0/4185737442 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fed7c046020 con 0x7fed80101fa0 2024-04-21T08:37:56.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.777+0000 7fed767fc640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed6c0a3bb0 0x7fed6c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:56.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.777+0000 7fed8485c640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed6c0a3bb0 0x7fed6c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:56.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.777+0000 7fed767fc640 1 -- 172.21.15.94:0/4185737442 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fed7c0f3060 con 0x7fed80101fa0 2024-04-21T08:37:56.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.778+0000 7fed8485c640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed6c0a3bb0 0x7fed6c0a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fed80075a70 tx=0x7fed7000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.781+0000 7fed767fc640 1 -- 172.21.15.94:0/4185737442 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed7c04d020 con 0x7fed80101fa0 2024-04-21T08:37:56.937 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:37:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b[61988]: 2024-04-21T08:37:56.684+0000 7f081fb22700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-21T08:37:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:56 smithi111 ceph-mon[31493]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:56 smithi111 ceph-mon[31493]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-21T08:37:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:56 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1842511306' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-21T08:37:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:56 smithi111 ceph-mon[31493]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-21T08:37:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:56 smithi111 ceph-mon[31493]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:56.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:56 smithi111 ceph-mon[31493]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-21T08:37:56.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:56 smithi111 ceph-mon[31493]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-21T08:37:56.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:56 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/65360151' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:37:56.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:56.995+0000 7fed872c1640 1 -- 172.21.15.94:0/4185737442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fed8002d5c0 con 0x7fed80101fa0 2024-04-21T08:37:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:56 smithi094 ceph-mon[27988]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:56 smithi094 ceph-mon[27988]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-21T08:37:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:56 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1842511306' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-21T08:37:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:56 smithi094 ceph-mon[27988]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-21T08:37:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:56 smithi094 ceph-mon[27988]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:56 smithi094 ceph-mon[27988]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-21T08:37:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:56 smithi094 ceph-mon[27988]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-21T08:37:57.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:56 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/65360151' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:37:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:56 smithi173 ceph-mon[31428]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:37:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:56 smithi173 ceph-mon[31428]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-21T08:37:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:56 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1842511306' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-21T08:37:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:56 smithi173 ceph-mon[31428]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-21T08:37:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:56 smithi173 ceph-mon[31428]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:37:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:56 smithi173 ceph-mon[31428]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-21T08:37:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:56 smithi173 ceph-mon[31428]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-21T08:37:57.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:56 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/65360151' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:37:57.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.703+0000 7fed767fc640 1 -- 172.21.15.94:0/4185737442 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fed7c0bf800 con 0x7fed80101fa0 2024-04-21T08:37:57.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 -- 172.21.15.94:0/4185737442 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed6c0a3bb0 msgr2=0x7fed6c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:57.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed6c0a3bb0 0x7fed6c0a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fed80075a70 tx=0x7fed7000b390 comp rx=0 tx=0).stop 2024-04-21T08:37:57.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 -- 172.21.15.94:0/4185737442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed80101fa0 msgr2=0x7fed80075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:57.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed80101fa0 0x7fed80075d00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fed7c000c00 tx=0x7fed7c004270 comp rx=0 tx=0).stop 2024-04-21T08:37:57.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 -- 172.21.15.94:0/4185737442 shutdown_connections 2024-04-21T08:37:57.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fed6c0a3bb0 0x7fed6c0a6070 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:57.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80103b50 0x7fed80074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:57.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed80103190 0x7fed80074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:57.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 --2- 172.21.15.94:0/4185737442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed80101fa0 0x7fed80075d00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:57.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 -- 172.21.15.94:0/4185737442 >> 172.21.15.94:0/4185737442 conn(0x7fed800fd530 msgr2=0x7fed80106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:57.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 -- 172.21.15.94:0/4185737442 shutdown_connections 2024-04-21T08:37:57.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.706+0000 7fed872c1640 1 -- 172.21.15.94:0/4185737442 wait complete. 2024-04-21T08:37:57.716 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-21T08:37:57.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.819+0000 7f3b71a3a640 1 -- 172.21.15.94:0/4228117883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b6c1053c0 msgr2=0x7f3b6c1077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.819+0000 7f3b71a3a640 1 --2- 172.21.15.94:0/4228117883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b6c1053c0 0x7f3b6c1077b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009900 tx=0x7f3b6002bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.819+0000 7f3b71a3a640 1 -- 172.21.15.94:0/4228117883 shutdown_connections 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.819+0000 7f3b71a3a640 1 --2- 172.21.15.94:0/4228117883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b6c107cf0 0x7f3b6c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.819+0000 7f3b71a3a640 1 --2- 172.21.15.94:0/4228117883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b6c1053c0 0x7f3b6c1077b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 --2- 172.21.15.94:0/4228117883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6c102a90 0x7f3b6c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 -- 172.21.15.94:0/4228117883 >> 172.21.15.94:0/4228117883 conn(0x7f3b6c0fc400 msgr2=0x7f3b6c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 -- 172.21.15.94:0/4228117883 shutdown_connections 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 -- 172.21.15.94:0/4228117883 wait complete. 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 Processor -- start 2024-04-21T08:37:57.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 -- start start 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b6c102a90 0x7f3b6c198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6c1053c0 0x7f3b6c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b6c107cf0 0x7f3b6c19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c10dac0 con 0x7f3b6c102a90 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c10d940 con 0x7f3b6c107cf0 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.820+0000 7f3b71a3a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c10dc40 con 0x7f3b6c1053c0 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b6c102a90 0x7f3b6c198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b6c102a90 0x7f3b6c198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51624/0 (socket says 172.21.15.94:51624) 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6affd640 1 -- 172.21.15.94:0/2090703211 learned_addr learned my addr 172.21.15.94:0/2090703211 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6b7fe640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b6c107cf0 0x7f3b6c19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6a7fc640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6c1053c0 0x7f3b6c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6affd640 1 -- 172.21.15.94:0/2090703211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6c1053c0 msgr2=0x7f3b6c1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6affd640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6c1053c0 0x7f3b6c1987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:57.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6affd640 1 -- 172.21.15.94:0/2090703211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b6c107cf0 msgr2=0x7f3b6c19f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6affd640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b6c107cf0 0x7f3b6c19f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6affd640 1 -- 172.21.15.94:0/2090703211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b6c19fe80 con 0x7f3b6c102a90 2024-04-21T08:37:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6b7fe640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b6c107cf0 0x7f3b6c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:37:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b6affd640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b6c102a90 0x7f3b6c198260 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00ea80 tx=0x7f3b5c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b70a38640 1 -- 172.21.15.94:0/2090703211 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c00ce40 con 0x7f3b6c102a90 2024-04-21T08:37:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b70a38640 1 -- 172.21.15.94:0/2090703211 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b5c005020 con 0x7f3b6c102a90 2024-04-21T08:37:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b71a3a640 1 -- 172.21.15.94:0/2090703211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b6c1a0170 con 0x7f3b6c102a90 2024-04-21T08:37:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b70a38640 1 -- 172.21.15.94:0/2090703211 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0116d0 con 0x7f3b6c102a90 2024-04-21T08:37:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.821+0000 7f3b71a3a640 1 -- 172.21.15.94:0/2090703211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b6c1a0640 con 0x7f3b6c102a90 2024-04-21T08:37:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.822+0000 7f3b71a3a640 1 -- 172.21.15.94:0/2090703211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b6c0fdef0 con 0x7f3b6c102a90 2024-04-21T08:37:57.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.823+0000 7f3b70a38640 1 -- 172.21.15.94:0/2090703211 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3b5c0054d0 con 0x7f3b6c102a90 2024-04-21T08:37:57.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.824+0000 7f3b70a38640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b540a3bb0 0x7f3b540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:57.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.824+0000 7f3b6a7fc640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b540a3bb0 0x7f3b540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:57.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.824+0000 7f3b70a38640 1 -- 172.21.15.94:0/2090703211 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3b5c0d0860 con 0x7f3b6c102a90 2024-04-21T08:37:57.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.825+0000 7f3b6a7fc640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b540a3bb0 0x7f3b540a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3b60005ab0 tx=0x7f3b60005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:57.829+0000 7f3b70a38640 1 -- 172.21.15.94:0/2090703211 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b5c016030 con 0x7f3b6c102a90 2024-04-21T08:37:57.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/65360151' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:37:57.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] up:active 2024-04-21T08:37:57.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-21T08:37:57.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-21T08:37:57.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: Cluster is now healthy 2024-04-21T08:37:57.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-21T08:37:57.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-21T08:37:57.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-21T08:37:57.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] up:active 2024-04-21T08:37:57.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:37:57.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:57 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4185737442' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:37:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/65360151' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:37:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] up:active 2024-04-21T08:37:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-21T08:37:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-21T08:37:58.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: Cluster is now healthy 2024-04-21T08:37:58.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-21T08:37:58.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-21T08:37:58.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-21T08:37:58.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] up:active 2024-04-21T08:37:58.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:37:58.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:57 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4185737442' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:37:58.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/65360151' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:37:58.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] up:active 2024-04-21T08:37:58.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-21T08:37:58.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-21T08:37:58.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: Cluster is now healthy 2024-04-21T08:37:58.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-21T08:37:58.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-21T08:37:58.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-21T08:37:58.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] up:active 2024-04-21T08:37:58.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:37:58.012 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:57 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4185737442' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:37:58.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.034+0000 7f3b71a3a640 1 -- 172.21.15.94:0/2090703211 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3b6c062390 con 0x7f3b540a3bb0 2024-04-21T08:37:58.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.058+0000 7f3b70a38640 1 -- 172.21.15.94:0/2090703211 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3b6c062390 con 0x7f3b540a3bb0 2024-04-21T08:37:58.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 -- 172.21.15.94:0/2090703211 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b540a3bb0 msgr2=0x7f3b540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b540a3bb0 0x7f3b540a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3b60005ab0 tx=0x7f3b60005fd0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 -- 172.21.15.94:0/2090703211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b6c102a90 msgr2=0x7f3b6c198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b6c102a90 0x7f3b6c198260 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00ea80 tx=0x7f3b5c00ef40 comp rx=0 tx=0).stop 2024-04-21T08:37:58.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 -- 172.21.15.94:0/2090703211 shutdown_connections 2024-04-21T08:37:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3b540a3bb0 0x7f3b540a6070 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6c1053c0 0x7f3b6c1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b6c107cf0 0x7f3b6c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 --2- 172.21.15.94:0/2090703211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b6c102a90 0x7f3b6c198260 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 -- 172.21.15.94:0/2090703211 >> 172.21.15.94:0/2090703211 conn(0x7f3b6c0fc400 msgr2=0x7f3b6c104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.060+0000 7f3b71a3a640 1 -- 172.21.15.94:0/2090703211 shutdown_connections 2024-04-21T08:37:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.061+0000 7f3b71a3a640 1 -- 172.21.15.94:0/2090703211 wait complete. 2024-04-21T08:37:58.070 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-21T08:37:58.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.174+0000 7f87fcbb2640 1 -- 172.21.15.94:0/724520473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f8103580 msgr2=0x7f87f81039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.174+0000 7f87fcbb2640 1 --2- 172.21.15.94:0/724520473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f8103580 0x7f87f81039a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009ca0 tx=0x7f87f002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.174+0000 7f87fcbb2640 1 -- 172.21.15.94:0/724520473 shutdown_connections 2024-04-21T08:37:58.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.174+0000 7f87fcbb2640 1 --2- 172.21.15.94:0/724520473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87f81009b0 0x7f87f810a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.174+0000 7f87fcbb2640 1 --2- 172.21.15.94:0/724520473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f80ffff0 0x7f87f8100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.174+0000 7f87fcbb2640 1 --2- 172.21.15.94:0/724520473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f8103580 0x7f87f81039a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.174+0000 7f87fcbb2640 1 -- 172.21.15.94:0/724520473 >> 172.21.15.94:0/724520473 conn(0x7f87f80fbb90 msgr2=0x7f87f80fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.174+0000 7f87fcbb2640 1 -- 172.21.15.94:0/724520473 shutdown_connections 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.174+0000 7f87fcbb2640 1 -- 172.21.15.94:0/724520473 wait complete. 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87fcbb2640 1 Processor -- start 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87fcbb2640 1 -- start start 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87fcbb2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f80ffff0 0x7f87f81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87fcbb2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f81009b0 0x7f87f81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87fcbb2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87f8103580 0x7f87f819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87fcbb2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f87f810dad0 con 0x7f87f81009b0 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87fcbb2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f87f810d950 con 0x7f87f80ffff0 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87fcbb2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f87f810dc50 con 0x7f87f8103580 2024-04-21T08:37:58.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f81009b0 0x7f87f81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f80ffff0 0x7f87f81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f81009b0 0x7f87f81986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51638/0 (socket says 172.21.15.94:51638) 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f5d9b640 1 -- 172.21.15.94:0/1990400012 learned_addr learned my addr 172.21.15.94:0/1990400012 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f6d9d640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87f8103580 0x7f87f819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f5d9b640 1 -- 172.21.15.94:0/1990400012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87f8103580 msgr2=0x7f87f819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f5d9b640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87f8103580 0x7f87f819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f5d9b640 1 -- 172.21.15.94:0/1990400012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f80ffff0 msgr2=0x7f87f81981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f5d9b640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f80ffff0 0x7f87f81981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f5d9b640 1 -- 172.21.15.94:0/1990400012 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f819fdd0 con 0x7f87f81009b0 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.175+0000 7f87f6d9d640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87f8103580 0x7f87f819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:58.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.176+0000 7f87f5d9b640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f81009b0 0x7f87f81986f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00e9f0 tx=0x7f87ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.176+0000 7f87e77fe640 1 -- 172.21.15.94:0/1990400012 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00cd80 con 0x7f87f81009b0 2024-04-21T08:37:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.176+0000 7f87fcbb2640 1 -- 172.21.15.94:0/1990400012 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f81a00c0 con 0x7f87f81009b0 2024-04-21T08:37:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.176+0000 7f87e77fe640 1 -- 172.21.15.94:0/1990400012 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87ec0108e0 con 0x7f87f81009b0 2024-04-21T08:37:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.176+0000 7f87e77fe640 1 -- 172.21.15.94:0/1990400012 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec018980 con 0x7f87f81009b0 2024-04-21T08:37:58.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.176+0000 7f87fcbb2640 1 -- 172.21.15.94:0/1990400012 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f81a0610 con 0x7f87f81009b0 2024-04-21T08:37:58.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.177+0000 7f87fcbb2640 1 -- 172.21.15.94:0/1990400012 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87f8109510 con 0x7f87f81009b0 2024-04-21T08:37:58.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.178+0000 7f87e77fe640 1 -- 172.21.15.94:0/1990400012 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f87ec010430 con 0x7f87f81009b0 2024-04-21T08:37:58.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.179+0000 7f87e77fe640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f87e00a3bb0 0x7f87e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.179+0000 7f87f659c640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f87e00a3bb0 0x7f87e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.179+0000 7f87e77fe640 1 -- 172.21.15.94:0/1990400012 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f87ec0d09c0 con 0x7f87f81009b0 2024-04-21T08:37:58.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.179+0000 7f87f659c640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f87e00a3bb0 0x7f87e00a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87f0032380 tx=0x7f87f0000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:58.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.183+0000 7f87e77fe640 1 -- 172.21.15.94:0/1990400012 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f87ec0d2e20 con 0x7f87f81009b0 2024-04-21T08:37:58.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.387+0000 7f87fcbb2640 1 -- 172.21.15.94:0/1990400012 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f87f8062420 con 0x7f87e00a3bb0 2024-04-21T08:37:58.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.417+0000 7f87e77fe640 1 -- 172.21.15.94:0/1990400012 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f87f8062420 con 0x7f87e00a3bb0 2024-04-21T08:37:58.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 -- 172.21.15.94:0/1990400012 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f87e00a3bb0 msgr2=0x7f87e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f87e00a3bb0 0x7f87e00a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87f0032380 tx=0x7f87f0000f40 comp rx=0 tx=0).stop 2024-04-21T08:37:58.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 -- 172.21.15.94:0/1990400012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f81009b0 msgr2=0x7f87f81986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f81009b0 0x7f87f81986f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00e9f0 tx=0x7f87ec00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 -- 172.21.15.94:0/1990400012 shutdown_connections 2024-04-21T08:37:58.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f87e00a3bb0 0x7f87e00a6070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87f8103580 0x7f87f819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f81009b0 0x7f87f81986f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 --2- 172.21.15.94:0/1990400012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f80ffff0 0x7f87f81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 -- 172.21.15.94:0/1990400012 >> 172.21.15.94:0/1990400012 conn(0x7f87f80fbb90 msgr2=0x7f87f81066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:58.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 -- 172.21.15.94:0/1990400012 shutdown_connections 2024-04-21T08:37:58.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.419+0000 7f87fcbb2640 1 -- 172.21.15.94:0/1990400012 wait complete. 2024-04-21T08:37:58.427 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:37:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:37:58.039+0000 7fbced630700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:37:58.427 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:37:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:37:58.039+0000 7fbced630700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:37:58.427 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:37:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:37:58.039+0000 7fbced630700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:37:58.427 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:37:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:37:58.039+0000 7fbced630700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:37:58.428 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:37:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:37:58.039+0000 7fbced630700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:37:58.428 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-21T08:37:58.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.533+0000 7f61b8e99640 1 -- 172.21.15.94:0/3181411201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b41031a0 msgr2=0x7f61b4103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.533+0000 7f61b8e99640 1 --2- 172.21.15.94:0/3181411201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b41031a0 0x7f61b4103620 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f61a8009900 tx=0x7f61a802bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:58.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.533+0000 7f61b8e99640 1 -- 172.21.15.94:0/3181411201 shutdown_connections 2024-04-21T08:37:58.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.533+0000 7f61b8e99640 1 --2- 172.21.15.94:0/3181411201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b4103b60 0x7f61b410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.533+0000 7f61b8e99640 1 --2- 172.21.15.94:0/3181411201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b41031a0 0x7f61b4103620 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.533+0000 7f61b8e99640 1 --2- 172.21.15.94:0/3181411201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b4101fb0 0x7f61b41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.533+0000 7f61b8e99640 1 -- 172.21.15.94:0/3181411201 >> 172.21.15.94:0/3181411201 conn(0x7f61b40fd540 msgr2=0x7f61b40ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:58.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.533+0000 7f61b8e99640 1 -- 172.21.15.94:0/3181411201 shutdown_connections 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.533+0000 7f61b8e99640 1 -- 172.21.15.94:0/3181411201 wait complete. 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b8e99640 1 Processor -- start 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b8e99640 1 -- start start 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b8e99640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b4101fb0 0x7f61b4075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b8e99640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41031a0 0x7f61b4074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b8e99640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b4103b60 0x7f61b4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b8e99640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f61b410daf0 con 0x7f61b4103b60 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b8e99640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f61b410d970 con 0x7f61b41031a0 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b8e99640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f61b410dc70 con 0x7f61b4101fb0 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b4103b60 0x7f61b4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b4103b60 0x7f61b4074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51654/0 (socket says 172.21.15.94:51654) 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b2d9d640 1 -- 172.21.15.94:0/118717130 learned_addr learned my addr 172.21.15.94:0/118717130 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:58.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b1d9b640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41031a0 0x7f61b4074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b259c640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b4101fb0 0x7f61b4075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b2d9d640 1 -- 172.21.15.94:0/118717130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b4101fb0 msgr2=0x7f61b4075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b2d9d640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b4101fb0 0x7f61b4075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b2d9d640 1 -- 172.21.15.94:0/118717130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41031a0 msgr2=0x7f61b4074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b2d9d640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41031a0 0x7f61b4074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b2d9d640 1 -- 172.21.15.94:0/118717130 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b4075110 con 0x7f61b4103b60 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.534+0000 7f61b2d9d640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b4103b60 0x7f61b4074880 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00eaa0 tx=0x7f61ac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.535+0000 7f61a37fe640 1 -- 172.21.15.94:0/118717130 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac00cdb0 con 0x7f61b4103b60 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.535+0000 7f61a37fe640 1 -- 172.21.15.94:0/118717130 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61ac005020 con 0x7f61b4103b60 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.535+0000 7f61b8e99640 1 -- 172.21.15.94:0/118717130 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b41a22a0 con 0x7f61b4103b60 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.535+0000 7f61a37fe640 1 -- 172.21.15.94:0/118717130 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0116d0 con 0x7f61b4103b60 2024-04-21T08:37:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.535+0000 7f61b8e99640 1 -- 172.21.15.94:0/118717130 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b41a27c0 con 0x7f61b4103b60 2024-04-21T08:37:58.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.536+0000 7f61b8e99640 1 -- 172.21.15.94:0/118717130 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b40691c0 con 0x7f61b4103b60 2024-04-21T08:37:58.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.537+0000 7f61a37fe640 1 -- 172.21.15.94:0/118717130 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f61ac0051d0 con 0x7f61b4103b60 2024-04-21T08:37:58.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.537+0000 7f61a37fe640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f619c0a3bb0 0x7f619c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.537+0000 7f61b259c640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f619c0a3bb0 0x7f619c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.538+0000 7f61a37fe640 1 -- 172.21.15.94:0/118717130 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f61ac0d0c10 con 0x7f61b4103b60 2024-04-21T08:37:58.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.538+0000 7f61b259c640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f619c0a3bb0 0x7f619c0a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f61b4103000 tx=0x7f61a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:58.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.541+0000 7f61a37fe640 1 -- 172.21.15.94:0/118717130 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61ac016030 con 0x7f61b4103b60 2024-04-21T08:37:58.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.749+0000 7f61b8e99640 1 -- 172.21.15.94:0/118717130 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f61b40624e0 con 0x7f619c0a3bb0 2024-04-21T08:37:58.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.754+0000 7f61a37fe640 1 -- 172.21.15.94:0/118717130 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f61b40624e0 con 0x7f619c0a3bb0 2024-04-21T08:37:58.753 INFO:teuthology.orchestra.run.smithi094.stdout:/volumes/qa/sv_0/9a097733-feee-4e62-b1ed-30903c3b1783 2024-04-21T08:37:58.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 -- 172.21.15.94:0/118717130 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f619c0a3bb0 msgr2=0x7f619c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f619c0a3bb0 0x7f619c0a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f61b4103000 tx=0x7f61a4005f90 comp rx=0 tx=0).stop 2024-04-21T08:37:58.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 -- 172.21.15.94:0/118717130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b4103b60 msgr2=0x7f61b4074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b4103b60 0x7f61b4074880 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00eaa0 tx=0x7f61ac00ef60 comp rx=0 tx=0).stop 2024-04-21T08:37:58.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 -- 172.21.15.94:0/118717130 shutdown_connections 2024-04-21T08:37:58.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f619c0a3bb0 0x7f619c0a6070 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b4103b60 0x7f61b4074880 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41031a0 0x7f61b4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 --2- 172.21.15.94:0/118717130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b4101fb0 0x7f61b4075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 -- 172.21.15.94:0/118717130 >> 172.21.15.94:0/118717130 conn(0x7f61b40fd540 msgr2=0x7f61b41064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:58.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.756+0000 7f61b8e99640 1 -- 172.21.15.94:0/118717130 shutdown_connections 2024-04-21T08:37:58.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.757+0000 7f61b8e99640 1 -- 172.21.15.94:0/118717130 wait complete. 2024-04-21T08:37:58.759 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:58 smithi094 ceph-mon[27988]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 426 B/s wr, 1 op/s 2024-04-21T08:37:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:58 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4185737442' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:37:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:58 smithi094 ceph-mon[27988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:37:58.766 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-21T08:37:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.870+0000 7f5d83681640 1 -- 172.21.15.94:0/381151651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d7c107c70 msgr2=0x7f5d7c10a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.870+0000 7f5d83681640 1 --2- 172.21.15.94:0/381151651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d7c107c70 0x7f5d7c10a0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009ca0 tx=0x7f5d7402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.870+0000 7f5d83681640 1 -- 172.21.15.94:0/381151651 shutdown_connections 2024-04-21T08:37:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.870+0000 7f5d83681640 1 --2- 172.21.15.94:0/381151651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d7c107c70 0x7f5d7c10a0c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.870+0000 7f5d83681640 1 --2- 172.21.15.94:0/381151651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c105340 0x7f5d7c107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.870+0000 7f5d83681640 1 --2- 172.21.15.94:0/381151651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d7c102a10 0x7f5d7c104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 -- 172.21.15.94:0/381151651 >> 172.21.15.94:0/381151651 conn(0x7f5d7c0fc400 msgr2=0x7f5d7c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 -- 172.21.15.94:0/381151651 shutdown_connections 2024-04-21T08:37:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 -- 172.21.15.94:0/381151651 wait complete. 2024-04-21T08:37:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 Processor -- start 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 -- start start 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102a10 0x7f5d7c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d7c105340 0x7f5d7c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d7c107c70 0x7f5d7c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c10dae0 con 0x7f5d7c107c70 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c10d960 con 0x7f5d7c102a10 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d83681640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c10dc60 con 0x7f5d7c105340 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d81c1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d7c107c70 0x7f5d7c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d81c1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d7c107c70 0x7f5d7c19f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51672/0 (socket says 172.21.15.94:51672) 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.871+0000 7f5d81c1e640 1 -- 172.21.15.94:0/555287649 learned_addr learned my addr 172.21.15.94:0/555287649 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:58.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d80c1c640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d7c105340 0x7f5d7c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d81c1e640 1 -- 172.21.15.94:0/555287649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d7c105340 msgr2=0x7f5d7c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d81c1e640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d7c105340 0x7f5d7c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d8141d640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102a10 0x7f5d7c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d81c1e640 1 -- 172.21.15.94:0/555287649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102a10 msgr2=0x7f5d7c1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d81c1e640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102a10 0x7f5d7c1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d81c1e640 1 -- 172.21.15.94:0/555287649 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7c19fdd0 con 0x7f5d7c107c70 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d80c1c640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d7c105340 0x7f5d7c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d8141d640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102a10 0x7f5d7c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d81c1e640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d7c107c70 0x7f5d7c19f760 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5d7402c700 tx=0x7f5d74034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d727fc640 1 -- 172.21.15.94:0/555287649 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74039040 con 0x7f5d7c107c70 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d727fc640 1 -- 172.21.15.94:0/555287649 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d74044070 con 0x7f5d7c107c70 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d83681640 1 -- 172.21.15.94:0/555287649 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7c1a0060 con 0x7f5d7c107c70 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d727fc640 1 -- 172.21.15.94:0/555287649 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d740042a0 con 0x7f5d7c107c70 2024-04-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.872+0000 7f5d83681640 1 -- 172.21.15.94:0/555287649 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d7c1a0550 con 0x7f5d7c107c70 2024-04-21T08:37:58.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.873+0000 7f5d83681640 1 -- 172.21.15.94:0/555287649 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d7c0fdef0 con 0x7f5d7c107c70 2024-04-21T08:37:58.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.874+0000 7f5d727fc640 1 -- 172.21.15.94:0/555287649 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5d74003970 con 0x7f5d7c107c70 2024-04-21T08:37:58.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.875+0000 7f5d727fc640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5d680a3bb0 0x7f5d680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.875+0000 7f5d8141d640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5d680a3bb0 0x7f5d680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.875+0000 7f5d727fc640 1 -- 172.21.15.94:0/555287649 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5d740f2ae0 con 0x7f5d7c107c70 2024-04-21T08:37:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.875+0000 7f5d8141d640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5d680a3bb0 0x7f5d680a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009b40 tx=0x7f5d78005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:58.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:58.879+0000 7f5d727fc640 1 -- 172.21.15.94:0/555287649 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d740bf280 con 0x7f5d7c107c70 2024-04-21T08:37:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:58 smithi111 ceph-mon[31493]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 426 B/s wr, 1 op/s 2024-04-21T08:37:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:58 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4185737442' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:37:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:58 smithi111 ceph-mon[31493]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:37:59.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:58 smithi173 ceph-mon[31428]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 426 B/s wr, 1 op/s 2024-04-21T08:37:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:58 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4185737442' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:37:59.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:58 smithi173 ceph-mon[31428]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:37:59.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.084+0000 7f5d83681640 1 -- 172.21.15.94:0/555287649 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f5d7c062420 con 0x7f5d680a3bb0 2024-04-21T08:37:59.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.108+0000 7f5d727fc640 1 -- 172.21.15.94:0/555287649 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5d7c062420 con 0x7f5d680a3bb0 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.110+0000 7f5d83681640 1 -- 172.21.15.94:0/555287649 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5d680a3bb0 msgr2=0x7f5d680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.110+0000 7f5d83681640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5d680a3bb0 0x7f5d680a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009b40 tx=0x7f5d78005e50 comp rx=0 tx=0).stop 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.110+0000 7f5d83681640 1 -- 172.21.15.94:0/555287649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d7c107c70 msgr2=0x7f5d7c19f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.110+0000 7f5d83681640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d7c107c70 0x7f5d7c19f760 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5d7402c700 tx=0x7f5d74034760 comp rx=0 tx=0).stop 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.110+0000 7f5d83681640 1 -- 172.21.15.94:0/555287649 shutdown_connections 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.110+0000 7f5d83681640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5d680a3bb0 0x7f5d680a6070 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.110+0000 7f5d83681640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d7c105340 0x7f5d7c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.110+0000 7f5d83681640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d7c107c70 0x7f5d7c19f760 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.110+0000 7f5d83681640 1 --2- 172.21.15.94:0/555287649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102a10 0x7f5d7c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.110+0000 7f5d83681640 1 -- 172.21.15.94:0/555287649 >> 172.21.15.94:0/555287649 conn(0x7f5d7c0fc400 msgr2=0x7f5d7c1032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.111+0000 7f5d83681640 1 -- 172.21.15.94:0/555287649 shutdown_connections 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.111+0000 7f5d83681640 1 -- 172.21.15.94:0/555287649 wait complete. 2024-04-21T08:37:59.120 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-21T08:37:59.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.223+0000 7fac5a7e6640 1 -- 172.21.15.94:0/1696331611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac54101f80 msgr2=0x7fac541023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.223+0000 7fac5a7e6640 1 --2- 172.21.15.94:0/1696331611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac54101f80 0x7fac541023a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fac44009900 tx=0x7fac4402bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:59.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.223+0000 7fac5a7e6640 1 -- 172.21.15.94:0/1696331611 shutdown_connections 2024-04-21T08:37:59.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.223+0000 7fac5a7e6640 1 --2- 172.21.15.94:0/1696331611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac54103b30 0x7fac5410a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.223+0000 7fac5a7e6640 1 --2- 172.21.15.94:0/1696331611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac54103170 0x7fac541035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.223+0000 7fac5a7e6640 1 --2- 172.21.15.94:0/1696331611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac54101f80 0x7fac541023a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.223+0000 7fac5a7e6640 1 -- 172.21.15.94:0/1696331611 >> 172.21.15.94:0/1696331611 conn(0x7fac540fd510 msgr2=0x7fac540ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.223+0000 7fac5a7e6640 1 -- 172.21.15.94:0/1696331611 shutdown_connections 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.223+0000 7fac5a7e6640 1 -- 172.21.15.94:0/1696331611 wait complete. 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac5a7e6640 1 Processor -- start 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac5a7e6640 1 -- start start 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac5a7e6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac54101f80 0x7fac541981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac5a7e6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac54103170 0x7fac54198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac5a7e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac54103b30 0x7fac5419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac5a7e6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fac5410db10 con 0x7fac54103b30 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac5a7e6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fac5410d990 con 0x7fac54101f80 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac5a7e6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fac5410dc90 con 0x7fac54103170 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac58d83640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac54103b30 0x7fac5419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac53fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac54101f80 0x7fac541981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac53fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac54101f80 0x7fac541981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:33870/0 (socket says 172.21.15.94:33870) 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac58d83640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac54103b30 0x7fac5419f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51700/0 (socket says 172.21.15.94:51700) 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac53fff640 1 -- 172.21.15.94:0/359953110 learned_addr learned my addr 172.21.15.94:0/359953110 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.224+0000 7fac537fe640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac54103170 0x7fac54198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac53fff640 1 -- 172.21.15.94:0/359953110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac54103170 msgr2=0x7fac54198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac53fff640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac54103170 0x7fac54198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac53fff640 1 -- 172.21.15.94:0/359953110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac54103b30 msgr2=0x7fac5419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac53fff640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac54103b30 0x7fac5419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac53fff640 1 -- 172.21.15.94:0/359953110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5419fdf0 con 0x7fac54101f80 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac58d83640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac54103b30 0x7fac5419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac537fe640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac54103170 0x7fac54198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac53fff640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac54101f80 0x7fac541981d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac44005ab0 tx=0x7fac440042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac517fa640 1 -- 172.21.15.94:0/359953110 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44043020 con 0x7fac54101f80 2024-04-21T08:37:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac5a7e6640 1 -- 172.21.15.94:0/359953110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac541a0080 con 0x7fac54101f80 2024-04-21T08:37:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac517fa640 1 -- 172.21.15.94:0/359953110 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac440054c0 con 0x7fac54101f80 2024-04-21T08:37:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac517fa640 1 -- 172.21.15.94:0/359953110 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44003d10 con 0x7fac54101f80 2024-04-21T08:37:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.225+0000 7fac5a7e6640 1 -- 172.21.15.94:0/359953110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac541a0620 con 0x7fac54101f80 2024-04-21T08:37:59.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.226+0000 7fac5a7e6640 1 -- 172.21.15.94:0/359953110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac54109310 con 0x7fac54101f80 2024-04-21T08:37:59.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.227+0000 7fac517fa640 1 -- 172.21.15.94:0/359953110 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fac44046020 con 0x7fac54101f80 2024-04-21T08:37:59.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.228+0000 7fac517fa640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fac3c0a3bb0 0x7fac3c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.228+0000 7fac537fe640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fac3c0a3bb0 0x7fac3c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.228+0000 7fac517fa640 1 -- 172.21.15.94:0/359953110 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fac440f3080 con 0x7fac54101f80 2024-04-21T08:37:59.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.228+0000 7fac537fe640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fac3c0a3bb0 0x7fac3c0a6070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fac541996e0 tx=0x7fac48005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:59.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.232+0000 7fac517fa640 1 -- 172.21.15.94:0/359953110 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac4404d020 con 0x7fac54101f80 2024-04-21T08:37:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.438+0000 7fac5a7e6640 1 -- 172.21.15.94:0/359953110 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fac54062420 con 0x7fac3c0a3bb0 2024-04-21T08:37:59.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.443+0000 7fac517fa640 1 -- 172.21.15.94:0/359953110 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fac54062420 con 0x7fac3c0a3bb0 2024-04-21T08:37:59.442 INFO:teuthology.orchestra.run.smithi094.stdout:/volumes/qa/sv_1/2736019d-b614-4eb2-b627-65349b1b5fba 2024-04-21T08:37:59.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 -- 172.21.15.94:0/359953110 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fac3c0a3bb0 msgr2=0x7fac3c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fac3c0a3bb0 0x7fac3c0a6070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fac541996e0 tx=0x7fac48005ec0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 -- 172.21.15.94:0/359953110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac54101f80 msgr2=0x7fac541981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac54101f80 0x7fac541981d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac44005ab0 tx=0x7fac440042f0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 -- 172.21.15.94:0/359953110 shutdown_connections 2024-04-21T08:37:59.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fac3c0a3bb0 0x7fac3c0a6070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac54103170 0x7fac54198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac54103b30 0x7fac5419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 --2- 172.21.15.94:0/359953110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac54101f80 0x7fac541981d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 -- 172.21.15.94:0/359953110 >> 172.21.15.94:0/359953110 conn(0x7fac540fd510 msgr2=0x7fac541064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:59.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 -- 172.21.15.94:0/359953110 shutdown_connections 2024-04-21T08:37:59.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.445+0000 7fac5a7e6640 1 -- 172.21.15.94:0/359953110 wait complete. 2024-04-21T08:37:59.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:37:59.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.554+0000 7f29b65ce640 1 -- 172.21.15.94:0/3885731009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0100e70 msgr2=0x7f29b0101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.554+0000 7f29b65ce640 1 --2- 172.21.15.94:0/3885731009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0100e70 0x7f29b0101290 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009900 tx=0x7f29a402bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:59.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.554+0000 7f29b65ce640 1 -- 172.21.15.94:0/3885731009 shutdown_connections 2024-04-21T08:37:59.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.554+0000 7f29b65ce640 1 --2- 172.21.15.94:0/3885731009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29b0102980 0x7f29b01092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 --2- 172.21.15.94:0/3885731009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29b0101fc0 0x7f29b0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 --2- 172.21.15.94:0/3885731009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0100e70 0x7f29b0101290 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 -- 172.21.15.94:0/3885731009 >> 172.21.15.94:0/3885731009 conn(0x7f29b00fc3e0 msgr2=0x7f29b00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:59.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 -- 172.21.15.94:0/3885731009 shutdown_connections 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 -- 172.21.15.94:0/3885731009 wait complete. 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 Processor -- start 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 -- start start 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29b0100e70 0x7f29b0198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29b0101fc0 0x7f29b0198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0102980 0x7f29b019f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f29b010dc10 con 0x7f29b0102980 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f29b010da90 con 0x7f29b0101fc0 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.555+0000 7f29b65ce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f29b010dd90 con 0x7f29b0100e70 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b4b6b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0102980 0x7f29b019f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b4b6b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0102980 0x7f29b019f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51728/0 (socket says 172.21.15.94:51728) 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b4b6b640 1 -- 172.21.15.94:0/1566458981 learned_addr learned my addr 172.21.15.94:0/1566458981 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:59.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29affff640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29b0100e70 0x7f29b0198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29af7fe640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29b0101fc0 0x7f29b0198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b4b6b640 1 -- 172.21.15.94:0/1566458981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29b0100e70 msgr2=0x7f29b0198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b4b6b640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29b0100e70 0x7f29b0198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b4b6b640 1 -- 172.21.15.94:0/1566458981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29b0101fc0 msgr2=0x7f29b0198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b4b6b640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29b0101fc0 0x7f29b0198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b4b6b640 1 -- 172.21.15.94:0/1566458981 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b019ff60 con 0x7f29b0102980 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29af7fe640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29b0101fc0 0x7f29b0198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b4b6b640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0102980 0x7f29b019f8f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f29a800b970 tx=0x7f29a800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29ad7fa640 1 -- 172.21.15.94:0/1566458981 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a8019070 con 0x7f29b0102980 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29ad7fa640 1 -- 172.21.15.94:0/1566458981 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29a8014440 con 0x7f29b0102980 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b65ce640 1 -- 172.21.15.94:0/1566458981 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b01a0250 con 0x7f29b0102980 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29ad7fa640 1 -- 172.21.15.94:0/1566458981 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a80075d0 con 0x7f29b0102980 2024-04-21T08:37:59.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.556+0000 7f29b65ce640 1 -- 172.21.15.94:0/1566458981 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b01a07d0 con 0x7f29b0102980 2024-04-21T08:37:59.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.557+0000 7f29b65ce640 1 -- 172.21.15.94:0/1566458981 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b0108200 con 0x7f29b0102980 2024-04-21T08:37:59.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.558+0000 7f29ad7fa640 1 -- 172.21.15.94:0/1566458981 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f29a801c070 con 0x7f29b0102980 2024-04-21T08:37:59.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.559+0000 7f29ad7fa640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f29980a3bb0 0x7f29980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.559+0000 7f29affff640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f29980a3bb0 0x7f29980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.559+0000 7f29ad7fa640 1 -- 172.21.15.94:0/1566458981 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f29a80cfbc0 con 0x7f29b0102980 2024-04-21T08:37:59.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.559+0000 7f29affff640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f29980a3bb0 0x7f29980a6070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f29a4006010 tx=0x7f29a4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:59.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.564+0000 7f29ad7fa640 1 -- 172.21.15.94:0/1566458981 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29a809bd10 con 0x7f29b0102980 2024-04-21T08:37:59.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.673+0000 7f29ad7fa640 1 -- 172.21.15.94:0/1566458981 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f29a8098380 con 0x7f29b0102980 2024-04-21T08:37:59.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:59 smithi094 ceph-mon[27988]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:59.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:37:59 smithi094 ceph-mon[27988]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:59.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.796+0000 7f29b65ce640 1 -- 172.21.15.94:0/1566458981 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29b002d5c0 con 0x7f29b0102980 2024-04-21T08:37:59.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.797+0000 7f29ad7fa640 1 -- 172.21.15.94:0/1566458981 <== mon.0 v2:172.21.15.94:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11885 (secure 0 0 0) 0x7f29a80bae20 con 0x7f29b0102980 2024-04-21T08:37:59.796 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:37:59.796 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:37:56.995828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24446,24488]},"id":1}]} 2024-04-21T08:37:59.796 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-04-21T08:37:59.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 -- 172.21.15.94:0/1566458981 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f29980a3bb0 msgr2=0x7f29980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f29980a3bb0 0x7f29980a6070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f29a4006010 tx=0x7f29a4000bc0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 -- 172.21.15.94:0/1566458981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0102980 msgr2=0x7f29b019f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0102980 0x7f29b019f8f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f29a800b970 tx=0x7f29a800be30 comp rx=0 tx=0).stop 2024-04-21T08:37:59.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 -- 172.21.15.94:0/1566458981 shutdown_connections 2024-04-21T08:37:59.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f29980a3bb0 0x7f29980a6070 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0102980 0x7f29b019f8f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29b0101fc0 0x7f29b0198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 --2- 172.21.15.94:0/1566458981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29b0100e70 0x7f29b0198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 -- 172.21.15.94:0/1566458981 >> 172.21.15.94:0/1566458981 conn(0x7f29b00fc3e0 msgr2=0x7f29b01053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:59.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 -- 172.21.15.94:0/1566458981 shutdown_connections 2024-04-21T08:37:59.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.799+0000 7f29b65ce640 1 -- 172.21.15.94:0/1566458981 wait complete. 2024-04-21T08:37:59.809 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:37:59.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.908+0000 7fd17b069640 1 -- 172.21.15.94:0/248166157 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174103180 msgr2=0x7fd174103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.908+0000 7fd17b069640 1 --2- 172.21.15.94:0/248166157 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174103180 0x7fd174103600 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd168009900 tx=0x7fd16802bc30 comp rx=0 tx=0).stop 2024-04-21T08:37:59.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.908+0000 7fd17b069640 1 -- 172.21.15.94:0/248166157 shutdown_connections 2024-04-21T08:37:59.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.908+0000 7fd17b069640 1 --2- 172.21.15.94:0/248166157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd174103b40 0x7fd17410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.908+0000 7fd17b069640 1 --2- 172.21.15.94:0/248166157 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174103180 0x7fd174103600 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.908+0000 7fd17b069640 1 --2- 172.21.15.94:0/248166157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd174101f90 0x7fd1741023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.908+0000 7fd17b069640 1 -- 172.21.15.94:0/248166157 >> 172.21.15.94:0/248166157 conn(0x7fd1740fd500 msgr2=0x7fd1740ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:59.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.908+0000 7fd17b069640 1 -- 172.21.15.94:0/248166157 shutdown_connections 2024-04-21T08:37:59.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.908+0000 7fd17b069640 1 -- 172.21.15.94:0/248166157 wait complete. 2024-04-21T08:37:59.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd17b069640 1 Processor -- start 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd17b069640 1 -- start start 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd17b069640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd174101f90 0x7fd174198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd17b069640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd174103180 0x7fd174198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd17b069640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174103b40 0x7fd17419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd17b069640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd17410db10 con 0x7fd174103b40 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd17b069640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd17410d990 con 0x7fd174103180 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd17b069640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd17410dc90 con 0x7fd174101f90 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd179606640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174103b40 0x7fd17419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd179606640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174103b40 0x7fd17419f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51744/0 (socket says 172.21.15.94:51744) 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd179606640 1 -- 172.21.15.94:0/2710731849 learned_addr learned my addr 172.21.15.94:0/2710731849 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd173fff640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd174103180 0x7fd174198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd178e05640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd174101f90 0x7fd174198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd179606640 1 -- 172.21.15.94:0/2710731849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd174101f90 msgr2=0x7fd174198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd179606640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd174101f90 0x7fd174198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd179606640 1 -- 172.21.15.94:0/2710731849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd174103180 msgr2=0x7fd174198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd179606640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd174103180 0x7fd174198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.909+0000 7fd179606640 1 -- 172.21.15.94:0/2710731849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17419fe20 con 0x7fd174103b40 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.910+0000 7fd179606640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174103b40 0x7fd17419f7b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00b970 tx=0x7fd16c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.910+0000 7fd171ffb640 1 -- 172.21.15.94:0/2710731849 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c019070 con 0x7fd174103b40 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.910+0000 7fd171ffb640 1 -- 172.21.15.94:0/2710731849 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd16c014440 con 0x7fd174103b40 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.910+0000 7fd17b069640 1 -- 172.21.15.94:0/2710731849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1741a0110 con 0x7fd174103b40 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.910+0000 7fd171ffb640 1 -- 172.21.15.94:0/2710731849 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0075d0 con 0x7fd174103b40 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.910+0000 7fd17b069640 1 -- 172.21.15.94:0/2710731849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1741a0660 con 0x7fd174103b40 2024-04-21T08:37:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.911+0000 7fd17b069640 1 -- 172.21.15.94:0/2710731849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1741092b0 con 0x7fd174103b40 2024-04-21T08:37:59.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.912+0000 7fd171ffb640 1 -- 172.21.15.94:0/2710731849 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd16c01c070 con 0x7fd174103b40 2024-04-21T08:37:59.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.912+0000 7fd171ffb640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd15c0a3c40 0x7fd15c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.912+0000 7fd178e05640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd15c0a3c40 0x7fd15c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.912+0000 7fd171ffb640 1 -- 172.21.15.94:0/2710731849 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd16c0cfc40 con 0x7fd174103b40 2024-04-21T08:37:59.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.913+0000 7fd178e05640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd15c0a3c40 0x7fd15c0a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd164005fd0 tx=0x7fd164005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:59.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:37:59.917+0000 7fd171ffb640 1 -- 172.21.15.94:0/2710731849 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd16c0d6080 con 0x7fd174103b40 2024-04-21T08:37:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:59 smithi111 ceph-mon[31493]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:37:59 smithi111 ceph-mon[31493]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:00.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:59 smithi173 ceph-mon[31428]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:00.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:37:59 smithi173 ceph-mon[31428]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:00.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.094+0000 7fd17b069640 1 -- 172.21.15.94:0/2710731849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd174062b50 con 0x7fd174103b40 2024-04-21T08:38:00.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.096+0000 7fd171ffb640 1 -- 172.21.15.94:0/2710731849 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24092 (secure 0 0 0) 0x7fd16c09bd40 con 0x7fd174103b40 2024-04-21T08:38:00.095 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:38:00.095 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:37:35.939829+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:38:00.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 -- 172.21.15.94:0/2710731849 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd15c0a3c40 msgr2=0x7fd15c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd15c0a3c40 0x7fd15c0a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd164005fd0 tx=0x7fd164005850 comp rx=0 tx=0).stop 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 -- 172.21.15.94:0/2710731849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174103b40 msgr2=0x7fd17419f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174103b40 0x7fd17419f7b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00b970 tx=0x7fd16c00be30 comp rx=0 tx=0).stop 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 -- 172.21.15.94:0/2710731849 shutdown_connections 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd15c0a3c40 0x7fd15c0a6100 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174103b40 0x7fd17419f7b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd174103180 0x7fd174198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 --2- 172.21.15.94:0/2710731849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd174101f90 0x7fd174198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 -- 172.21.15.94:0/2710731849 >> 172.21.15.94:0/2710731849 conn(0x7fd1740fd500 msgr2=0x7fd174106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 -- 172.21.15.94:0/2710731849 shutdown_connections 2024-04-21T08:38:00.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.098+0000 7fd17b069640 1 -- 172.21.15.94:0/2710731849 wait complete. 2024-04-21T08:38:00.108 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-21T08:38:00.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:38:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.207+0000 7fbc89fa4640 1 -- 172.21.15.94:0/4254694231 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84103b40 msgr2=0x7fbc8410a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.207+0000 7fbc89fa4640 1 --2- 172.21.15.94:0/4254694231 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84103b40 0x7fbc8410a3d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009ca0 tx=0x7fbc7c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.207+0000 7fbc89fa4640 1 -- 172.21.15.94:0/4254694231 shutdown_connections 2024-04-21T08:38:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.207+0000 7fbc89fa4640 1 --2- 172.21.15.94:0/4254694231 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84103b40 0x7fbc8410a3d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.207+0000 7fbc89fa4640 1 --2- 172.21.15.94:0/4254694231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc84103180 0x7fbc84103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.207+0000 7fbc89fa4640 1 --2- 172.21.15.94:0/4254694231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84101f90 0x7fbc841023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.207+0000 7fbc89fa4640 1 -- 172.21.15.94:0/4254694231 >> 172.21.15.94:0/4254694231 conn(0x7fbc840fd500 msgr2=0x7fbc840ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.207+0000 7fbc89fa4640 1 -- 172.21.15.94:0/4254694231 shutdown_connections 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.207+0000 7fbc89fa4640 1 -- 172.21.15.94:0/4254694231 wait complete. 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc89fa4640 1 Processor -- start 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc89fa4640 1 -- start start 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc89fa4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84101f90 0x7fbc84198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc89fa4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc84103180 0x7fbc84198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc89fa4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84103b40 0x7fbc8419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc89fa4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8410dad0 con 0x7fbc84103b40 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc89fa4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8410d950 con 0x7fbc84103180 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc89fa4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8410dc50 con 0x7fbc84101f90 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc83fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84103b40 0x7fbc8419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc83fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84103b40 0x7fbc8419f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51758/0 (socket says 172.21.15.94:51758) 2024-04-21T08:38:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc83fff640 1 -- 172.21.15.94:0/2341094136 learned_addr learned my addr 172.21.15.94:0/2341094136 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.208+0000 7fbc837fe640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84101f90 0x7fbc84198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc83fff640 1 -- 172.21.15.94:0/2341094136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84101f90 msgr2=0x7fbc84198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc83fff640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84101f90 0x7fbc84198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc7bfff640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc84103180 0x7fbc84198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc83fff640 1 -- 172.21.15.94:0/2341094136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc84103180 msgr2=0x7fbc84198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc83fff640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc84103180 0x7fbc84198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc83fff640 1 -- 172.21.15.94:0/2341094136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc8419fd70 con 0x7fbc84103b40 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc7bfff640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc84103180 0x7fbc84198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc83fff640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84103b40 0x7fbc8419f700 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c02c700 tx=0x7fbc7c0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc817fa640 1 -- 172.21.15.94:0/2341094136 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c041020 con 0x7fbc84103b40 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc817fa640 1 -- 172.21.15.94:0/2341094136 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc7c034d60 con 0x7fbc84103b40 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc89fa4640 1 -- 172.21.15.94:0/2341094136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc841a0000 con 0x7fbc84103b40 2024-04-21T08:38:00.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc817fa640 1 -- 172.21.15.94:0/2341094136 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c039040 con 0x7fbc84103b40 2024-04-21T08:38:00.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.209+0000 7fbc89fa4640 1 -- 172.21.15.94:0/2341094136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc841a0540 con 0x7fbc84103b40 2024-04-21T08:38:00.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.210+0000 7fbc89fa4640 1 -- 172.21.15.94:0/2341094136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc841092b0 con 0x7fbc84103b40 2024-04-21T08:38:00.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.211+0000 7fbc817fa640 1 -- 172.21.15.94:0/2341094136 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbc7c044020 con 0x7fbc84103b40 2024-04-21T08:38:00.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.211+0000 7fbc817fa640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fbc700a3cc0 0x7fbc700a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:00.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.212+0000 7fbc837fe640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fbc700a3cc0 0x7fbc700a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:00.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.212+0000 7fbc817fa640 1 -- 172.21.15.94:0/2341094136 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbc7c0f2a40 con 0x7fbc84103b40 2024-04-21T08:38:00.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.212+0000 7fbc837fe640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fbc700a3cc0 0x7fbc700a6180 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbc84102fe0 tx=0x7fbc74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:00.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.215+0000 7fbc817fa640 1 -- 172.21.15.94:0/2341094136 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc7c04b020 con 0x7fbc84103b40 2024-04-21T08:38:00.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.389+0000 7fbc89fa4640 1 -- 172.21.15.94:0/2341094136 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbc84062390 con 0x7fbc700a3cc0 2024-04-21T08:38:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.398+0000 7fbc817fa640 1 -- 172.21.15.94:0/2341094136 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340506 (secure 0 0 0) 0x7fbc84062390 con 0x7fbc700a3cc0 2024-04-21T08:38:00.402 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:38:00.404 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-21T08:38:00.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 -- 172.21.15.94:0/2341094136 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fbc700a3cc0 msgr2=0x7fbc700a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fbc700a3cc0 0x7fbc700a6180 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbc84102fe0 tx=0x7fbc74005f90 comp rx=0 tx=0).stop 2024-04-21T08:38:00.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 -- 172.21.15.94:0/2341094136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84103b40 msgr2=0x7fbc8419f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84103b40 0x7fbc8419f700 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c02c700 tx=0x7fbc7c0093f0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 -- 172.21.15.94:0/2341094136 shutdown_connections 2024-04-21T08:38:00.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fbc700a3cc0 0x7fbc700a6180 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84103b40 0x7fbc8419f700 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc84103180 0x7fbc84198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 --2- 172.21.15.94:0/2341094136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84101f90 0x7fbc84198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 -- 172.21.15.94:0/2341094136 >> 172.21.15.94:0/2341094136 conn(0x7fbc840fd500 msgr2=0x7fbc84106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:00.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 -- 172.21.15.94:0/2341094136 shutdown_connections 2024-04-21T08:38:00.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.402+0000 7fbc89fa4640 1 -- 172.21.15.94:0/2341094136 wait complete. 2024-04-21T08:38:00.412 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-04-21T08:37:59.586075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595098,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":142,"num_read_kb":127,"num_write":255,"num_write_kb":2958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":198,"ondisk_log_size":198,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373252,"kb_used_data":11588,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568572,"statfs":{"total":1151940427776,"available":1151558217728,"internally_reserved":0,"allocated":11866112,"data_stored":7515782,"data_compressed":284622,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2320,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":29,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327492+0000","last_change":"2024-04-21T08:37:31.448345+0000","last_active":"2024-04-21T08:37:37.327492+0000","last_peered":"2024-04-21T08:37:37.327492+0000","last_clean":"2024-04-21T08:37:37.327492+0000","last_became_active":"2024-04-21T08:37:31.447878+0000","last_became_peered":"2024-04-21T08:37:31.447878+0000","last_unstale":"2024-04-21T08:37:37.327492+0000","last_undegraded":"2024-04-21T08:37:37.327492+0000","last_fullsized":"2024-04-21T08:37:37.327492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:45.589165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644842+0000","last_change":"2024-04-21T08:37:31.452895+0000","last_active":"2024-04-21T08:37:36.644842+0000","last_peered":"2024-04-21T08:37:36.644842+0000","last_clean":"2024-04-21T08:37:36.644842+0000","last_became_active":"2024-04-21T08:37:31.452550+0000","last_became_peered":"2024-04-21T08:37:31.452550+0000","last_unstale":"2024-04-21T08:37:36.644842+0000","last_undegraded":"2024-04-21T08:37:36.644842+0000","last_fullsized":"2024-04-21T08:37:36.644842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:02.497802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644670+0000","last_change":"2024-04-21T08:37:31.447552+0000","last_active":"2024-04-21T08:37:36.644670+0000","last_peered":"2024-04-21T08:37:36.644670+0000","last_clean":"2024-04-21T08:37:36.644670+0000","last_became_active":"2024-04-21T08:37:31.446044+0000","last_became_peered":"2024-04-21T08:37:31.446044+0000","last_unstale":"2024-04-21T08:37:36.644670+0000","last_undegraded":"2024-04-21T08:37:36.644670+0000","last_fullsized":"2024-04-21T08:37:36.644670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:22.801977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.794319+0000","last_change":"2024-04-21T08:37:31.452400+0000","last_active":"2024-04-21T08:37:37.794319+0000","last_peered":"2024-04-21T08:37:37.794319+0000","last_clean":"2024-04-21T08:37:37.794319+0000","last_became_active":"2024-04-21T08:37:31.451462+0000","last_became_peered":"2024-04-21T08:37:31.451462+0000","last_unstale":"2024-04-21T08:37:37.794319+0000","last_undegraded":"2024-04-21T08:37:37.794319+0000","last_fullsized":"2024-04-21T08:37:37.794319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:22.463481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463505+0000","last_change":"2024-04-21T08:37:31.448942+0000","last_active":"2024-04-21T08:37:37.463505+0000","last_peered":"2024-04-21T08:37:37.463505+0000","last_clean":"2024-04-21T08:37:37.463505+0000","last_became_active":"2024-04-21T08:37:31.448526+0000","last_became_peered":"2024-04-21T08:37:31.448526+0000","last_unstale":"2024-04-21T08:37:37.463505+0000","last_undegraded":"2024-04-21T08:37:37.463505+0000","last_fullsized":"2024-04-21T08:37:37.463505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:29.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583194+0000","last_change":"2024-04-21T08:37:31.452165+0000","last_active":"2024-04-21T08:37:36.583194+0000","last_peered":"2024-04-21T08:37:36.583194+0000","last_clean":"2024-04-21T08:37:36.583194+0000","last_became_active":"2024-04-21T08:37:31.451802+0000","last_became_peered":"2024-04-21T08:37:31.451802+0000","last_unstale":"2024-04-21T08:37:36.583194+0000","last_undegraded":"2024-04-21T08:37:36.583194+0000","last_fullsized":"2024-04-21T08:37:36.583194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:42.210544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793621+0000","last_change":"2024-04-21T08:37:31.444798+0000","last_active":"2024-04-21T08:37:37.793621+0000","last_peered":"2024-04-21T08:37:37.793621+0000","last_clean":"2024-04-21T08:37:37.793621+0000","last_became_active":"2024-04-21T08:37:31.444313+0000","last_became_peered":"2024-04-21T08:37:31.444313+0000","last_unstale":"2024-04-21T08:37:37.793621+0000","last_undegraded":"2024-04-21T08:37:37.793621+0000","last_fullsized":"2024-04-21T08:37:37.793621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:22:03.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.009783+0000","last_change":"2024-04-21T08:37:31.448257+0000","last_active":"2024-04-21T08:37:37.009783+0000","last_peered":"2024-04-21T08:37:37.009783+0000","last_clean":"2024-04-21T08:37:37.009783+0000","last_became_active":"2024-04-21T08:37:31.447845+0000","last_became_peered":"2024-04-21T08:37:31.447845+0000","last_unstale":"2024-04-21T08:37:37.009783+0000","last_undegraded":"2024-04-21T08:37:37.009783+0000","last_fullsized":"2024-04-21T08:37:37.009783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:19.956699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.470394+0000","last_change":"2024-04-21T08:37:31.443022+0000","last_active":"2024-04-21T08:37:37.470394+0000","last_peered":"2024-04-21T08:37:37.470394+0000","last_clean":"2024-04-21T08:37:37.470394+0000","last_became_active":"2024-04-21T08:37:31.442645+0000","last_became_peered":"2024-04-21T08:37:31.442645+0000","last_unstale":"2024-04-21T08:37:37.470394+0000","last_undegraded":"2024-04-21T08:37:37.470394+0000","last_fullsized":"2024-04-21T08:37:37.470394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:18:10.861281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583635+0000","last_change":"2024-04-21T08:37:31.453427+0000","last_active":"2024-04-21T08:37:36.583635+0000","last_peered":"2024-04-21T08:37:36.583635+0000","last_clean":"2024-04-21T08:37:36.583635+0000","last_became_active":"2024-04-21T08:37:31.453096+0000","last_became_peered":"2024-04-21T08:37:31.453096+0000","last_unstale":"2024-04-21T08:37:36.583635+0000","last_undegraded":"2024-04-21T08:37:36.583635+0000","last_fullsized":"2024-04-21T08:37:36.583635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:49.450351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793834+0000","last_change":"2024-04-21T08:37:31.449908+0000","last_active":"2024-04-21T08:37:37.793834+0000","last_peered":"2024-04-21T08:37:37.793834+0000","last_clean":"2024-04-21T08:37:37.793834+0000","last_became_active":"2024-04-21T08:37:31.449558+0000","last_became_peered":"2024-04-21T08:37:31.449558+0000","last_unstale":"2024-04-21T08:37:37.793834+0000","last_undegraded":"2024-04-21T08:37:37.793834+0000","last_fullsized":"2024-04-21T08:37:37.793834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:21.538689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.794213+0000","last_change":"2024-04-21T08:37:31.452973+0000","last_active":"2024-04-21T08:37:37.794213+0000","last_peered":"2024-04-21T08:37:37.794213+0000","last_clean":"2024-04-21T08:37:37.794213+0000","last_became_active":"2024-04-21T08:37:31.452586+0000","last_became_peered":"2024-04-21T08:37:31.452586+0000","last_unstale":"2024-04-21T08:37:37.794213+0000","last_undegraded":"2024-04-21T08:37:37.794213+0000","last_fullsized":"2024-04-21T08:37:37.794213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:09:11.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792575+0000","last_change":"2024-04-21T08:37:31.445801+0000","last_active":"2024-04-21T08:37:37.792575+0000","last_peered":"2024-04-21T08:37:37.792575+0000","last_clean":"2024-04-21T08:37:37.792575+0000","last_became_active":"2024-04-21T08:37:31.445478+0000","last_became_peered":"2024-04-21T08:37:31.445478+0000","last_unstale":"2024-04-21T08:37:37.792575+0000","last_undegraded":"2024-04-21T08:37:37.792575+0000","last_fullsized":"2024-04-21T08:37:37.792575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:33.352739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644406+0000","last_change":"2024-04-21T08:37:31.450735+0000","last_active":"2024-04-21T08:37:36.644406+0000","last_peered":"2024-04-21T08:37:36.644406+0000","last_clean":"2024-04-21T08:37:36.644406+0000","last_became_active":"2024-04-21T08:37:31.450384+0000","last_became_peered":"2024-04-21T08:37:31.450384+0000","last_unstale":"2024-04-21T08:37:36.644406+0000","last_undegraded":"2024-04-21T08:37:36.644406+0000","last_fullsized":"2024-04-21T08:37:36.644406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:45.950489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644416+0000","last_change":"2024-04-21T08:37:31.454690+0000","last_active":"2024-04-21T08:37:36.644416+0000","last_peered":"2024-04-21T08:37:36.644416+0000","last_clean":"2024-04-21T08:37:36.644416+0000","last_became_active":"2024-04-21T08:37:31.454330+0000","last_became_peered":"2024-04-21T08:37:31.454330+0000","last_unstale":"2024-04-21T08:37:36.644416+0000","last_undegraded":"2024-04-21T08:37:36.644416+0000","last_fullsized":"2024-04-21T08:37:36.644416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:52.644317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.469900+0000","last_change":"2024-04-21T08:37:31.441389+0000","last_active":"2024-04-21T08:37:37.469900+0000","last_peered":"2024-04-21T08:37:37.469900+0000","last_clean":"2024-04-21T08:37:37.469900+0000","last_became_active":"2024-04-21T08:37:31.441065+0000","last_became_peered":"2024-04-21T08:37:31.441065+0000","last_unstale":"2024-04-21T08:37:37.469900+0000","last_undegraded":"2024-04-21T08:37:37.469900+0000","last_fullsized":"2024-04-21T08:37:37.469900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:56.528154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327357+0000","last_change":"2024-04-21T08:37:31.446722+0000","last_active":"2024-04-21T08:37:37.327357+0000","last_peered":"2024-04-21T08:37:37.327357+0000","last_clean":"2024-04-21T08:37:37.327357+0000","last_became_active":"2024-04-21T08:37:31.446396+0000","last_became_peered":"2024-04-21T08:37:31.446396+0000","last_unstale":"2024-04-21T08:37:37.327357+0000","last_undegraded":"2024-04-21T08:37:37.327357+0000","last_fullsized":"2024-04-21T08:37:37.327357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:17.488633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582923+0000","last_change":"2024-04-21T08:37:31.444388+0000","last_active":"2024-04-21T08:37:36.582923+0000","last_peered":"2024-04-21T08:37:36.582923+0000","last_clean":"2024-04-21T08:37:36.582923+0000","last_became_active":"2024-04-21T08:37:31.444023+0000","last_became_peered":"2024-04-21T08:37:31.444023+0000","last_unstale":"2024-04-21T08:37:36.582923+0000","last_undegraded":"2024-04-21T08:37:36.582923+0000","last_fullsized":"2024-04-21T08:37:36.582923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:07:46.238941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583622+0000","last_change":"2024-04-21T08:37:31.447850+0000","last_active":"2024-04-21T08:37:36.583622+0000","last_peered":"2024-04-21T08:37:36.583622+0000","last_clean":"2024-04-21T08:37:36.583622+0000","last_became_active":"2024-04-21T08:37:31.447505+0000","last_became_peered":"2024-04-21T08:37:31.447505+0000","last_unstale":"2024-04-21T08:37:36.583622+0000","last_undegraded":"2024-04-21T08:37:36.583622+0000","last_fullsized":"2024-04-21T08:37:36.583622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:55.399410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583175+0000","last_change":"2024-04-21T08:37:35.608747+0000","last_active":"2024-04-21T08:37:36.583175+0000","last_peered":"2024-04-21T08:37:36.583175+0000","last_clean":"2024-04-21T08:37:36.583175+0000","last_became_active":"2024-04-21T08:37:35.608157+0000","last_became_peered":"2024-04-21T08:37:35.608157+0000","last_unstale":"2024-04-21T08:37:36.583175+0000","last_undegraded":"2024-04-21T08:37:36.583175+0000","last_fullsized":"2024-04-21T08:37:36.583175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:53.384190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:56.668232+0000","last_change":"2024-04-21T08:37:30.414311+0000","last_active":"2024-04-21T08:37:56.668232+0000","last_peered":"2024-04-21T08:37:56.668232+0000","last_clean":"2024-04-21T08:37:56.668232+0000","last_became_active":"2024-04-21T08:37:30.413776+0000","last_became_peered":"2024-04-21T08:37:30.413776+0000","last_unstale":"2024-04-21T08:37:56.668232+0000","last_undegraded":"2024-04-21T08:37:56.668232+0000","last_fullsized":"2024-04-21T08:37:56.668232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:48.628006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010284+0000","last_change":"2024-04-21T08:37:31.451813+0000","last_active":"2024-04-21T08:37:37.010284+0000","last_peered":"2024-04-21T08:37:37.010284+0000","last_clean":"2024-04-21T08:37:37.010284+0000","last_became_active":"2024-04-21T08:37:31.451447+0000","last_became_peered":"2024-04-21T08:37:31.451447+0000","last_unstale":"2024-04-21T08:37:37.010284+0000","last_undegraded":"2024-04-21T08:37:37.010284+0000","last_fullsized":"2024-04-21T08:37:37.010284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:46:45.235326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793183+0000","last_change":"2024-04-21T08:37:35.603518+0000","last_active":"2024-04-21T08:37:37.793183+0000","last_peered":"2024-04-21T08:37:37.793183+0000","last_clean":"2024-04-21T08:37:37.793183+0000","last_became_active":"2024-04-21T08:37:35.603038+0000","last_became_peered":"2024-04-21T08:37:35.603038+0000","last_unstale":"2024-04-21T08:37:37.793183+0000","last_undegraded":"2024-04-21T08:37:37.793183+0000","last_fullsized":"2024-04-21T08:37:37.793183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:06:29.224025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010426+0000","last_change":"2024-04-21T08:37:30.424608+0000","last_active":"2024-04-21T08:37:37.010426+0000","last_peered":"2024-04-21T08:37:37.010426+0000","last_clean":"2024-04-21T08:37:37.010426+0000","last_became_active":"2024-04-21T08:37:30.424259+0000","last_became_peered":"2024-04-21T08:37:30.424259+0000","last_unstale":"2024-04-21T08:37:37.010426+0000","last_undegraded":"2024-04-21T08:37:37.010426+0000","last_fullsized":"2024-04-21T08:37:37.010426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:38.007095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644687+0000","last_change":"2024-04-21T08:37:31.444656+0000","last_active":"2024-04-21T08:37:36.644687+0000","last_peered":"2024-04-21T08:37:36.644687+0000","last_clean":"2024-04-21T08:37:36.644687+0000","last_became_active":"2024-04-21T08:37:31.444300+0000","last_became_peered":"2024-04-21T08:37:31.444300+0000","last_unstale":"2024-04-21T08:37:36.644687+0000","last_undegraded":"2024-04-21T08:37:36.644687+0000","last_fullsized":"2024-04-21T08:37:36.644687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:40.901674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793547+0000","last_change":"2024-04-21T08:37:35.611732+0000","last_active":"2024-04-21T08:37:37.793547+0000","last_peered":"2024-04-21T08:37:37.793547+0000","last_clean":"2024-04-21T08:37:37.793547+0000","last_became_active":"2024-04-21T08:37:35.611060+0000","last_became_peered":"2024-04-21T08:37:35.611060+0000","last_unstale":"2024-04-21T08:37:37.793547+0000","last_undegraded":"2024-04-21T08:37:37.793547+0000","last_fullsized":"2024-04-21T08:37:37.793547+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:36:01.491949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.464007+0000","last_change":"2024-04-21T08:37:30.422350+0000","last_active":"2024-04-21T08:37:37.464007+0000","last_peered":"2024-04-21T08:37:37.464007+0000","last_clean":"2024-04-21T08:37:37.464007+0000","last_became_active":"2024-04-21T08:37:30.421659+0000","last_became_peered":"2024-04-21T08:37:30.421659+0000","last_unstale":"2024-04-21T08:37:37.464007+0000","last_undegraded":"2024-04-21T08:37:37.464007+0000","last_fullsized":"2024-04-21T08:37:37.464007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:09:29.894205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.470215+0000","last_change":"2024-04-21T08:37:31.443896+0000","last_active":"2024-04-21T08:37:37.470215+0000","last_peered":"2024-04-21T08:37:37.470215+0000","last_clean":"2024-04-21T08:37:37.470215+0000","last_became_active":"2024-04-21T08:37:31.442684+0000","last_became_peered":"2024-04-21T08:37:31.442684+0000","last_unstale":"2024-04-21T08:37:37.470215+0000","last_undegraded":"2024-04-21T08:37:37.470215+0000","last_fullsized":"2024-04-21T08:37:37.470215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:14:18.649565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793303+0000","last_change":"2024-04-21T08:37:35.600468+0000","last_active":"2024-04-21T08:37:37.793303+0000","last_peered":"2024-04-21T08:37:37.793303+0000","last_clean":"2024-04-21T08:37:37.793303+0000","last_became_active":"2024-04-21T08:37:35.599940+0000","last_became_peered":"2024-04-21T08:37:35.599940+0000","last_unstale":"2024-04-21T08:37:37.793303+0000","last_undegraded":"2024-04-21T08:37:37.793303+0000","last_fullsized":"2024-04-21T08:37:37.793303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:48.125434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463767+0000","last_change":"2024-04-21T08:37:30.416480+0000","last_active":"2024-04-21T08:37:37.463767+0000","last_peered":"2024-04-21T08:37:37.463767+0000","last_clean":"2024-04-21T08:37:37.463767+0000","last_became_active":"2024-04-21T08:37:30.416072+0000","last_became_peered":"2024-04-21T08:37:30.416072+0000","last_unstale":"2024-04-21T08:37:37.463767+0000","last_undegraded":"2024-04-21T08:37:37.463767+0000","last_fullsized":"2024-04-21T08:37:37.463767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:28:22.400926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793203+0000","last_change":"2024-04-21T08:37:31.457216+0000","last_active":"2024-04-21T08:37:37.793203+0000","last_peered":"2024-04-21T08:37:37.793203+0000","last_clean":"2024-04-21T08:37:37.793203+0000","last_became_active":"2024-04-21T08:37:31.456856+0000","last_became_peered":"2024-04-21T08:37:31.456856+0000","last_unstale":"2024-04-21T08:37:37.793203+0000","last_undegraded":"2024-04-21T08:37:37.793203+0000","last_fullsized":"2024-04-21T08:37:37.793203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:10.272886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583515+0000","last_change":"2024-04-21T08:37:35.601715+0000","last_active":"2024-04-21T08:37:36.583515+0000","last_peered":"2024-04-21T08:37:36.583515+0000","last_clean":"2024-04-21T08:37:36.583515+0000","last_became_active":"2024-04-21T08:37:35.601269+0000","last_became_peered":"2024-04-21T08:37:35.601269+0000","last_unstale":"2024-04-21T08:37:36.583515+0000","last_undegraded":"2024-04-21T08:37:36.583515+0000","last_fullsized":"2024-04-21T08:37:36.583515+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:57.192918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793165+0000","last_change":"2024-04-21T08:37:30.415510+0000","last_active":"2024-04-21T08:37:37.793165+0000","last_peered":"2024-04-21T08:37:37.793165+0000","last_clean":"2024-04-21T08:37:37.793165+0000","last_became_active":"2024-04-21T08:37:30.415199+0000","last_became_peered":"2024-04-21T08:37:30.415199+0000","last_unstale":"2024-04-21T08:37:37.793165+0000","last_undegraded":"2024-04-21T08:37:37.793165+0000","last_fullsized":"2024-04-21T08:37:37.793165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:32.336247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463505+0000","last_change":"2024-04-21T08:37:31.444022+0000","last_active":"2024-04-21T08:37:37.463505+0000","last_peered":"2024-04-21T08:37:37.463505+0000","last_clean":"2024-04-21T08:37:37.463505+0000","last_became_active":"2024-04-21T08:37:31.443609+0000","last_became_peered":"2024-04-21T08:37:31.443609+0000","last_unstale":"2024-04-21T08:37:37.463505+0000","last_undegraded":"2024-04-21T08:37:37.463505+0000","last_fullsized":"2024-04-21T08:37:37.463505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:39.978817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583458+0000","last_change":"2024-04-21T08:37:35.608954+0000","last_active":"2024-04-21T08:37:36.583458+0000","last_peered":"2024-04-21T08:37:36.583458+0000","last_clean":"2024-04-21T08:37:36.583458+0000","last_became_active":"2024-04-21T08:37:35.608364+0000","last_became_peered":"2024-04-21T08:37:35.608364+0000","last_unstale":"2024-04-21T08:37:36.583458+0000","last_undegraded":"2024-04-21T08:37:36.583458+0000","last_fullsized":"2024-04-21T08:37:36.583458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:04.574382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:56.665863+0000","last_change":"2024-04-21T08:37:30.419877+0000","last_active":"2024-04-21T08:37:56.665863+0000","last_peered":"2024-04-21T08:37:56.665863+0000","last_clean":"2024-04-21T08:37:56.665863+0000","last_became_active":"2024-04-21T08:37:30.419572+0000","last_became_peered":"2024-04-21T08:37:30.419572+0000","last_unstale":"2024-04-21T08:37:56.665863+0000","last_undegraded":"2024-04-21T08:37:56.665863+0000","last_fullsized":"2024-04-21T08:37:56.665863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:32:35.524089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793298+0000","last_change":"2024-04-21T08:37:31.458590+0000","last_active":"2024-04-21T08:37:37.793298+0000","last_peered":"2024-04-21T08:37:37.793298+0000","last_clean":"2024-04-21T08:37:37.793298+0000","last_became_active":"2024-04-21T08:37:31.458196+0000","last_became_peered":"2024-04-21T08:37:31.458196+0000","last_unstale":"2024-04-21T08:37:37.793298+0000","last_undegraded":"2024-04-21T08:37:37.793298+0000","last_fullsized":"2024-04-21T08:37:37.793298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:00.890859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327672+0000","last_change":"2024-04-21T08:37:35.604555+0000","last_active":"2024-04-21T08:37:37.327672+0000","last_peered":"2024-04-21T08:37:37.327672+0000","last_clean":"2024-04-21T08:37:37.327672+0000","last_became_active":"2024-04-21T08:37:35.603847+0000","last_became_peered":"2024-04-21T08:37:35.603847+0000","last_unstale":"2024-04-21T08:37:37.327672+0000","last_undegraded":"2024-04-21T08:37:37.327672+0000","last_fullsized":"2024-04-21T08:37:37.327672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:41:59.194494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792381+0000","last_change":"2024-04-21T08:37:30.415786+0000","last_active":"2024-04-21T08:37:37.792381+0000","last_peered":"2024-04-21T08:37:37.792381+0000","last_clean":"2024-04-21T08:37:37.792381+0000","last_became_active":"2024-04-21T08:37:30.415472+0000","last_became_peered":"2024-04-21T08:37:30.415472+0000","last_unstale":"2024-04-21T08:37:37.792381+0000","last_undegraded":"2024-04-21T08:37:37.792381+0000","last_fullsized":"2024-04-21T08:37:37.792381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.417132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.009927+0000","last_change":"2024-04-21T08:37:31.455720+0000","last_active":"2024-04-21T08:37:37.009927+0000","last_peered":"2024-04-21T08:37:37.009927+0000","last_clean":"2024-04-21T08:37:37.009927+0000","last_became_active":"2024-04-21T08:37:31.455334+0000","last_became_peered":"2024-04-21T08:37:31.455334+0000","last_unstale":"2024-04-21T08:37:37.009927+0000","last_undegraded":"2024-04-21T08:37:37.009927+0000","last_fullsized":"2024-04-21T08:37:37.009927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:06:55.220656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644802+0000","last_change":"2024-04-21T08:37:35.603305+0000","last_active":"2024-04-21T08:37:36.644802+0000","last_peered":"2024-04-21T08:37:36.644802+0000","last_clean":"2024-04-21T08:37:36.644802+0000","last_became_active":"2024-04-21T08:37:35.602854+0000","last_became_peered":"2024-04-21T08:37:35.602854+0000","last_unstale":"2024-04-21T08:37:36.644802+0000","last_undegraded":"2024-04-21T08:37:36.644802+0000","last_fullsized":"2024-04-21T08:37:36.644802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:10.993864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792869+0000","last_change":"2024-04-21T08:37:30.420797+0000","last_active":"2024-04-21T08:37:37.792869+0000","last_peered":"2024-04-21T08:37:37.792869+0000","last_clean":"2024-04-21T08:37:37.792869+0000","last_became_active":"2024-04-21T08:37:30.420485+0000","last_became_peered":"2024-04-21T08:37:30.420485+0000","last_unstale":"2024-04-21T08:37:37.792869+0000","last_undegraded":"2024-04-21T08:37:37.792869+0000","last_fullsized":"2024-04-21T08:37:37.792869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:03.275677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582283+0000","last_change":"2024-04-21T08:37:31.448406+0000","last_active":"2024-04-21T08:37:36.582283+0000","last_peered":"2024-04-21T08:37:36.582283+0000","last_clean":"2024-04-21T08:37:36.582283+0000","last_became_active":"2024-04-21T08:37:31.448085+0000","last_became_peered":"2024-04-21T08:37:31.448085+0000","last_unstale":"2024-04-21T08:37:36.582283+0000","last_undegraded":"2024-04-21T08:37:36.582283+0000","last_fullsized":"2024-04-21T08:37:36.582283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:49.574064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010048+0000","last_change":"2024-04-21T08:37:35.601338+0000","last_active":"2024-04-21T08:37:37.010048+0000","last_peered":"2024-04-21T08:37:37.010048+0000","last_clean":"2024-04-21T08:37:37.010048+0000","last_became_active":"2024-04-21T08:37:35.600890+0000","last_became_peered":"2024-04-21T08:37:35.600890+0000","last_unstale":"2024-04-21T08:37:37.010048+0000","last_undegraded":"2024-04-21T08:37:37.010048+0000","last_fullsized":"2024-04-21T08:37:37.010048+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:30:21.755850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:55.661291+0000","last_change":"2024-04-21T08:37:30.423320+0000","last_active":"2024-04-21T08:37:55.661291+0000","last_peered":"2024-04-21T08:37:55.661291+0000","last_clean":"2024-04-21T08:37:55.661291+0000","last_became_active":"2024-04-21T08:37:30.423020+0000","last_became_peered":"2024-04-21T08:37:30.423020+0000","last_unstale":"2024-04-21T08:37:55.661291+0000","last_undegraded":"2024-04-21T08:37:55.661291+0000","last_fullsized":"2024-04-21T08:37:55.661291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:08:27.313196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.470485+0000","last_change":"2024-04-21T08:37:31.442425+0000","last_active":"2024-04-21T08:37:37.470485+0000","last_peered":"2024-04-21T08:37:37.470485+0000","last_clean":"2024-04-21T08:37:37.470485+0000","last_became_active":"2024-04-21T08:37:31.442028+0000","last_became_peered":"2024-04-21T08:37:31.442028+0000","last_unstale":"2024-04-21T08:37:37.470485+0000","last_undegraded":"2024-04-21T08:37:37.470485+0000","last_fullsized":"2024-04-21T08:37:37.470485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:45.498656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582551+0000","last_change":"2024-04-21T08:37:35.607829+0000","last_active":"2024-04-21T08:37:36.582551+0000","last_peered":"2024-04-21T08:37:36.582551+0000","last_clean":"2024-04-21T08:37:36.582551+0000","last_became_active":"2024-04-21T08:37:35.607150+0000","last_became_peered":"2024-04-21T08:37:35.607150+0000","last_unstale":"2024-04-21T08:37:36.582551+0000","last_undegraded":"2024-04-21T08:37:36.582551+0000","last_fullsized":"2024-04-21T08:37:36.582551+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:01:34.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793870+0000","last_change":"2024-04-21T08:37:30.418613+0000","last_active":"2024-04-21T08:37:37.793870+0000","last_peered":"2024-04-21T08:37:37.793870+0000","last_clean":"2024-04-21T08:37:37.793870+0000","last_became_active":"2024-04-21T08:37:30.418300+0000","last_became_peered":"2024-04-21T08:37:30.418300+0000","last_unstale":"2024-04-21T08:37:37.793870+0000","last_undegraded":"2024-04-21T08:37:37.793870+0000","last_fullsized":"2024-04-21T08:37:37.793870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:17:35.370753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644684+0000","last_change":"2024-04-21T08:37:31.456902+0000","last_active":"2024-04-21T08:37:36.644684+0000","last_peered":"2024-04-21T08:37:36.644684+0000","last_clean":"2024-04-21T08:37:36.644684+0000","last_became_active":"2024-04-21T08:37:31.453637+0000","last_became_peered":"2024-04-21T08:37:31.453637+0000","last_unstale":"2024-04-21T08:37:36.644684+0000","last_undegraded":"2024-04-21T08:37:36.644684+0000","last_fullsized":"2024-04-21T08:37:36.644684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:48:59.387349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793938+0000","last_change":"2024-04-21T08:37:35.608280+0000","last_active":"2024-04-21T08:37:37.793938+0000","last_peered":"2024-04-21T08:37:37.793938+0000","last_clean":"2024-04-21T08:37:37.793938+0000","last_became_active":"2024-04-21T08:37:35.607898+0000","last_became_peered":"2024-04-21T08:37:35.607898+0000","last_unstale":"2024-04-21T08:37:37.793938+0000","last_undegraded":"2024-04-21T08:37:37.793938+0000","last_fullsized":"2024-04-21T08:37:37.793938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:55.689519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793121+0000","last_change":"2024-04-21T08:37:30.417352+0000","last_active":"2024-04-21T08:37:37.793121+0000","last_peered":"2024-04-21T08:37:37.793121+0000","last_clean":"2024-04-21T08:37:37.793121+0000","last_became_active":"2024-04-21T08:37:30.417000+0000","last_became_peered":"2024-04-21T08:37:30.417000+0000","last_unstale":"2024-04-21T08:37:37.793121+0000","last_undegraded":"2024-04-21T08:37:37.793121+0000","last_fullsized":"2024-04-21T08:37:37.793121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:56.087334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.470354+0000","last_change":"2024-04-21T08:37:31.458229+0000","last_active":"2024-04-21T08:37:37.470354+0000","last_peered":"2024-04-21T08:37:37.470354+0000","last_clean":"2024-04-21T08:37:37.470354+0000","last_became_active":"2024-04-21T08:37:31.456897+0000","last_became_peered":"2024-04-21T08:37:31.456897+0000","last_unstale":"2024-04-21T08:37:37.470354+0000","last_undegraded":"2024-04-21T08:37:37.470354+0000","last_fullsized":"2024-04-21T08:37:37.470354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:25:58.253081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582556+0000","last_change":"2024-04-21T08:37:35.601077+0000","last_active":"2024-04-21T08:37:36.582556+0000","last_peered":"2024-04-21T08:37:36.582556+0000","last_clean":"2024-04-21T08:37:36.582556+0000","last_became_active":"2024-04-21T08:37:35.600673+0000","last_became_peered":"2024-04-21T08:37:35.600673+0000","last_unstale":"2024-04-21T08:37:36.582556+0000","last_undegraded":"2024-04-21T08:37:36.582556+0000","last_fullsized":"2024-04-21T08:37:36.582556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:04.554208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:56.704816+0000","last_change":"2024-04-21T08:37:30.421440+0000","last_active":"2024-04-21T08:37:56.704816+0000","last_peered":"2024-04-21T08:37:56.704816+0000","last_clean":"2024-04-21T08:37:56.704816+0000","last_became_active":"2024-04-21T08:37:30.420988+0000","last_became_peered":"2024-04-21T08:37:30.420988+0000","last_unstale":"2024-04-21T08:37:56.704816+0000","last_undegraded":"2024-04-21T08:37:56.704816+0000","last_fullsized":"2024-04-21T08:37:56.704816+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:38:03.850705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463776+0000","last_change":"2024-04-21T08:37:31.447208+0000","last_active":"2024-04-21T08:37:37.463776+0000","last_peered":"2024-04-21T08:37:37.463776+0000","last_clean":"2024-04-21T08:37:37.463776+0000","last_became_active":"2024-04-21T08:37:31.444464+0000","last_became_peered":"2024-04-21T08:37:31.444464+0000","last_unstale":"2024-04-21T08:37:37.463776+0000","last_undegraded":"2024-04-21T08:37:37.463776+0000","last_fullsized":"2024-04-21T08:37:37.463776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:33.805676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793579+0000","last_change":"2024-04-21T08:37:35.603937+0000","last_active":"2024-04-21T08:37:37.793579+0000","last_peered":"2024-04-21T08:37:37.793579+0000","last_clean":"2024-04-21T08:37:37.793579+0000","last_became_active":"2024-04-21T08:37:35.603477+0000","last_became_peered":"2024-04-21T08:37:35.603477+0000","last_unstale":"2024-04-21T08:37:37.793579+0000","last_undegraded":"2024-04-21T08:37:37.793579+0000","last_fullsized":"2024-04-21T08:37:37.793579+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:40:33.901570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:55.661103+0000","last_change":"2024-04-21T08:37:30.416941+0000","last_active":"2024-04-21T08:37:55.661103+0000","last_peered":"2024-04-21T08:37:55.661103+0000","last_clean":"2024-04-21T08:37:55.661103+0000","last_became_active":"2024-04-21T08:37:30.416625+0000","last_became_peered":"2024-04-21T08:37:30.416625+0000","last_unstale":"2024-04-21T08:37:55.661103+0000","last_undegraded":"2024-04-21T08:37:55.661103+0000","last_fullsized":"2024-04-21T08:37:55.661103+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:07.782779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792496+0000","last_change":"2024-04-21T08:37:31.441707+0000","last_active":"2024-04-21T08:37:37.792496+0000","last_peered":"2024-04-21T08:37:37.792496+0000","last_clean":"2024-04-21T08:37:37.792496+0000","last_became_active":"2024-04-21T08:37:31.441366+0000","last_became_peered":"2024-04-21T08:37:31.441366+0000","last_unstale":"2024-04-21T08:37:37.792496+0000","last_undegraded":"2024-04-21T08:37:37.792496+0000","last_fullsized":"2024-04-21T08:37:37.792496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:16.573804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327512+0000","last_change":"2024-04-21T08:37:35.596214+0000","last_active":"2024-04-21T08:37:37.327512+0000","last_peered":"2024-04-21T08:37:37.327512+0000","last_clean":"2024-04-21T08:37:37.327512+0000","last_became_active":"2024-04-21T08:37:35.595773+0000","last_became_peered":"2024-04-21T08:37:35.595773+0000","last_unstale":"2024-04-21T08:37:37.327512+0000","last_undegraded":"2024-04-21T08:37:37.327512+0000","last_fullsized":"2024-04-21T08:37:37.327512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:07.538190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793056+0000","last_change":"2024-04-21T08:37:30.416324+0000","last_active":"2024-04-21T08:37:37.793056+0000","last_peered":"2024-04-21T08:37:37.793056+0000","last_clean":"2024-04-21T08:37:37.793056+0000","last_became_active":"2024-04-21T08:37:30.416024+0000","last_became_peered":"2024-04-21T08:37:30.416024+0000","last_unstale":"2024-04-21T08:37:37.793056+0000","last_undegraded":"2024-04-21T08:37:37.793056+0000","last_fullsized":"2024-04-21T08:37:37.793056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:54.111986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.794218+0000","last_change":"2024-04-21T08:37:31.453680+0000","last_active":"2024-04-21T08:37:37.794218+0000","last_peered":"2024-04-21T08:37:37.794218+0000","last_clean":"2024-04-21T08:37:37.794218+0000","last_became_active":"2024-04-21T08:37:31.453319+0000","last_became_peered":"2024-04-21T08:37:31.453319+0000","last_unstale":"2024-04-21T08:37:37.794218+0000","last_undegraded":"2024-04-21T08:37:37.794218+0000","last_fullsized":"2024-04-21T08:37:37.794218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:43.139789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582873+0000","last_change":"2024-04-21T08:37:35.608906+0000","last_active":"2024-04-21T08:37:36.582873+0000","last_peered":"2024-04-21T08:37:36.582873+0000","last_clean":"2024-04-21T08:37:36.582873+0000","last_became_active":"2024-04-21T08:37:35.608535+0000","last_became_peered":"2024-04-21T08:37:35.608535+0000","last_unstale":"2024-04-21T08:37:36.582873+0000","last_undegraded":"2024-04-21T08:37:36.582873+0000","last_fullsized":"2024-04-21T08:37:36.582873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:58:01.422308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793158+0000","last_change":"2024-04-21T08:37:30.414865+0000","last_active":"2024-04-21T08:37:37.793158+0000","last_peered":"2024-04-21T08:37:37.793158+0000","last_clean":"2024-04-21T08:37:37.793158+0000","last_became_active":"2024-04-21T08:37:30.414471+0000","last_became_peered":"2024-04-21T08:37:30.414471+0000","last_unstale":"2024-04-21T08:37:37.793158+0000","last_undegraded":"2024-04-21T08:37:37.793158+0000","last_fullsized":"2024-04-21T08:37:37.793158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:46.943574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793278+0000","last_change":"2024-04-21T08:37:31.456634+0000","last_active":"2024-04-21T08:37:37.793278+0000","last_peered":"2024-04-21T08:37:37.793278+0000","last_clean":"2024-04-21T08:37:37.793278+0000","last_became_active":"2024-04-21T08:37:31.456278+0000","last_became_peered":"2024-04-21T08:37:31.456278+0000","last_unstale":"2024-04-21T08:37:37.793278+0000","last_undegraded":"2024-04-21T08:37:37.793278+0000","last_fullsized":"2024-04-21T08:37:37.793278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:04.870024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792829+0000","last_change":"2024-04-21T08:37:35.601760+0000","last_active":"2024-04-21T08:37:37.792829+0000","last_peered":"2024-04-21T08:37:37.792829+0000","last_clean":"2024-04-21T08:37:37.792829+0000","last_became_active":"2024-04-21T08:37:35.601332+0000","last_became_peered":"2024-04-21T08:37:35.601332+0000","last_unstale":"2024-04-21T08:37:37.792829+0000","last_undegraded":"2024-04-21T08:37:37.792829+0000","last_fullsized":"2024-04-21T08:37:37.792829+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:04:03.694581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463377+0000","last_change":"2024-04-21T08:37:30.422419+0000","last_active":"2024-04-21T08:37:37.463377+0000","last_peered":"2024-04-21T08:37:37.463377+0000","last_clean":"2024-04-21T08:37:37.463377+0000","last_became_active":"2024-04-21T08:37:30.422123+0000","last_became_peered":"2024-04-21T08:37:30.422123+0000","last_unstale":"2024-04-21T08:37:37.463377+0000","last_undegraded":"2024-04-21T08:37:37.463377+0000","last_fullsized":"2024-04-21T08:37:37.463377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:32:54.029041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792868+0000","last_change":"2024-04-21T08:37:31.458992+0000","last_active":"2024-04-21T08:37:37.792868+0000","last_peered":"2024-04-21T08:37:37.792868+0000","last_clean":"2024-04-21T08:37:37.792868+0000","last_became_active":"2024-04-21T08:37:31.458635+0000","last_became_peered":"2024-04-21T08:37:31.458635+0000","last_unstale":"2024-04-21T08:37:37.792868+0000","last_undegraded":"2024-04-21T08:37:37.792868+0000","last_fullsized":"2024-04-21T08:37:37.792868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:48:14.325571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792914+0000","last_change":"2024-04-21T08:37:35.603739+0000","last_active":"2024-04-21T08:37:37.792914+0000","last_peered":"2024-04-21T08:37:37.792914+0000","last_clean":"2024-04-21T08:37:37.792914+0000","last_became_active":"2024-04-21T08:37:35.603310+0000","last_became_peered":"2024-04-21T08:37:35.603310+0000","last_unstale":"2024-04-21T08:37:37.792914+0000","last_undegraded":"2024-04-21T08:37:37.792914+0000","last_fullsized":"2024-04-21T08:37:37.792914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:19.162520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463925+0000","last_change":"2024-04-21T08:37:30.418104+0000","last_active":"2024-04-21T08:37:37.463925+0000","last_peered":"2024-04-21T08:37:37.463925+0000","last_clean":"2024-04-21T08:37:37.463925+0000","last_became_active":"2024-04-21T08:37:30.417814+0000","last_became_peered":"2024-04-21T08:37:30.417814+0000","last_unstale":"2024-04-21T08:37:37.463925+0000","last_undegraded":"2024-04-21T08:37:37.463925+0000","last_fullsized":"2024-04-21T08:37:37.463925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:50:56.696513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327371+0000","last_change":"2024-04-21T08:37:31.451952+0000","last_active":"2024-04-21T08:37:37.327371+0000","last_peered":"2024-04-21T08:37:37.327371+0000","last_clean":"2024-04-21T08:37:37.327371+0000","last_became_active":"2024-04-21T08:37:31.451588+0000","last_became_peered":"2024-04-21T08:37:31.451588+0000","last_unstale":"2024-04-21T08:37:37.327371+0000","last_undegraded":"2024-04-21T08:37:37.327371+0000","last_fullsized":"2024-04-21T08:37:37.327371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:06.506769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463621+0000","last_change":"2024-04-21T08:37:35.760661+0000","last_active":"2024-04-21T08:37:37.463621+0000","last_peered":"2024-04-21T08:37:37.463621+0000","last_clean":"2024-04-21T08:37:37.463621+0000","last_became_active":"2024-04-21T08:37:35.759925+0000","last_became_peered":"2024-04-21T08:37:35.759925+0000","last_unstale":"2024-04-21T08:37:37.463621+0000","last_undegraded":"2024-04-21T08:37:37.463621+0000","last_fullsized":"2024-04-21T08:37:37.463621+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:11:53.062251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582616+0000","last_change":"2024-04-21T08:37:30.416487+0000","last_active":"2024-04-21T08:37:36.582616+0000","last_peered":"2024-04-21T08:37:36.582616+0000","last_clean":"2024-04-21T08:37:36.582616+0000","last_became_active":"2024-04-21T08:37:30.416076+0000","last_became_peered":"2024-04-21T08:37:30.416076+0000","last_unstale":"2024-04-21T08:37:36.582616+0000","last_undegraded":"2024-04-21T08:37:36.582616+0000","last_fullsized":"2024-04-21T08:37:36.582616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:18:33.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583419+0000","last_change":"2024-04-21T08:37:31.448824+0000","last_active":"2024-04-21T08:37:36.583419+0000","last_peered":"2024-04-21T08:37:36.583419+0000","last_clean":"2024-04-21T08:37:36.583419+0000","last_became_active":"2024-04-21T08:37:31.448432+0000","last_became_peered":"2024-04-21T08:37:31.448432+0000","last_unstale":"2024-04-21T08:37:36.583419+0000","last_undegraded":"2024-04-21T08:37:36.583419+0000","last_fullsized":"2024-04-21T08:37:36.583419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:41.111272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792582+0000","last_change":"2024-04-21T08:37:35.604539+0000","last_active":"2024-04-21T08:37:37.792582+0000","last_peered":"2024-04-21T08:37:37.792582+0000","last_clean":"2024-04-21T08:37:37.792582+0000","last_became_active":"2024-04-21T08:37:35.604132+0000","last_became_peered":"2024-04-21T08:37:35.604132+0000","last_unstale":"2024-04-21T08:37:37.792582+0000","last_undegraded":"2024-04-21T08:37:37.792582+0000","last_fullsized":"2024-04-21T08:37:37.792582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:12.180730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793615+0000","last_change":"2024-04-21T08:37:30.418854+0000","last_active":"2024-04-21T08:37:37.793615+0000","last_peered":"2024-04-21T08:37:37.793615+0000","last_clean":"2024-04-21T08:37:37.793615+0000","last_became_active":"2024-04-21T08:37:30.418511+0000","last_became_peered":"2024-04-21T08:37:30.418511+0000","last_unstale":"2024-04-21T08:37:37.793615+0000","last_undegraded":"2024-04-21T08:37:37.793615+0000","last_fullsized":"2024-04-21T08:37:37.793615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:26.751924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583178+0000","last_change":"2024-04-21T08:37:31.453365+0000","last_active":"2024-04-21T08:37:36.583178+0000","last_peered":"2024-04-21T08:37:36.583178+0000","last_clean":"2024-04-21T08:37:36.583178+0000","last_became_active":"2024-04-21T08:37:31.453042+0000","last_became_peered":"2024-04-21T08:37:31.453042+0000","last_unstale":"2024-04-21T08:37:36.583178+0000","last_undegraded":"2024-04-21T08:37:36.583178+0000","last_fullsized":"2024-04-21T08:37:36.583178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:49:46.648754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792773+0000","last_change":"2024-04-21T08:37:35.760626+0000","last_active":"2024-04-21T08:37:37.792773+0000","last_peered":"2024-04-21T08:37:37.792773+0000","last_clean":"2024-04-21T08:37:37.792773+0000","last_became_active":"2024-04-21T08:37:35.760125+0000","last_became_peered":"2024-04-21T08:37:35.760125+0000","last_unstale":"2024-04-21T08:37:37.792773+0000","last_undegraded":"2024-04-21T08:37:37.792773+0000","last_fullsized":"2024-04-21T08:37:37.792773+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:58.366557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793012+0000","last_change":"2024-04-21T08:37:31.457654+0000","last_active":"2024-04-21T08:37:37.793012+0000","last_peered":"2024-04-21T08:37:37.793012+0000","last_clean":"2024-04-21T08:37:37.793012+0000","last_became_active":"2024-04-21T08:37:31.456950+0000","last_became_peered":"2024-04-21T08:37:31.456950+0000","last_unstale":"2024-04-21T08:37:37.793012+0000","last_undegraded":"2024-04-21T08:37:37.793012+0000","last_fullsized":"2024-04-21T08:37:37.793012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:02.863000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792278+0000","last_change":"2024-04-21T08:37:30.422021+0000","last_active":"2024-04-21T08:37:37.792278+0000","last_peered":"2024-04-21T08:37:37.792278+0000","last_clean":"2024-04-21T08:37:37.792278+0000","last_became_active":"2024-04-21T08:37:30.421711+0000","last_became_peered":"2024-04-21T08:37:30.421711+0000","last_unstale":"2024-04-21T08:37:37.792278+0000","last_undegraded":"2024-04-21T08:37:37.792278+0000","last_fullsized":"2024-04-21T08:37:37.792278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:01.612205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793684+0000","last_change":"2024-04-21T08:37:35.598041+0000","last_active":"2024-04-21T08:37:37.793684+0000","last_peered":"2024-04-21T08:37:37.793684+0000","last_clean":"2024-04-21T08:37:37.793684+0000","last_became_active":"2024-04-21T08:37:35.597635+0000","last_became_peered":"2024-04-21T08:37:35.597635+0000","last_unstale":"2024-04-21T08:37:37.793684+0000","last_undegraded":"2024-04-21T08:37:37.793684+0000","last_fullsized":"2024-04-21T08:37:37.793684+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:35:27.452340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010021+0000","last_change":"2024-04-21T08:37:31.450423+0000","last_active":"2024-04-21T08:37:37.010021+0000","last_peered":"2024-04-21T08:37:37.010021+0000","last_clean":"2024-04-21T08:37:37.010021+0000","last_became_active":"2024-04-21T08:37:31.450112+0000","last_became_peered":"2024-04-21T08:37:31.450112+0000","last_unstale":"2024-04-21T08:37:37.010021+0000","last_undegraded":"2024-04-21T08:37:37.010021+0000","last_fullsized":"2024-04-21T08:37:37.010021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:10:47.006769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793736+0000","last_change":"2024-04-21T08:37:30.416519+0000","last_active":"2024-04-21T08:37:37.793736+0000","last_peered":"2024-04-21T08:37:37.793736+0000","last_clean":"2024-04-21T08:37:37.793736+0000","last_became_active":"2024-04-21T08:37:30.416131+0000","last_became_peered":"2024-04-21T08:37:30.416131+0000","last_unstale":"2024-04-21T08:37:37.793736+0000","last_undegraded":"2024-04-21T08:37:37.793736+0000","last_fullsized":"2024-04-21T08:37:37.793736+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:12.280847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793112+0000","last_change":"2024-04-21T08:36:20.257759+0000","last_active":"2024-04-21T08:37:37.793112+0000","last_peered":"2024-04-21T08:37:37.793112+0000","last_clean":"2024-04-21T08:37:37.793112+0000","last_became_active":"2024-04-21T08:36:20.257313+0000","last_became_peered":"2024-04-21T08:36:20.257313+0000","last_unstale":"2024-04-21T08:37:37.793112+0000","last_undegraded":"2024-04-21T08:37:37.793112+0000","last_fullsized":"2024-04-21T08:37:37.793112+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_clean_scrub_stamp":"2024-04-21T08:32:56.546295+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:42:44.393447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793548+0000","last_change":"2024-04-21T08:37:35.761562+0000","last_active":"2024-04-21T08:37:37.793548+0000","last_peered":"2024-04-21T08:37:37.793548+0000","last_clean":"2024-04-21T08:37:37.793548+0000","last_became_active":"2024-04-21T08:37:35.760703+0000","last_became_peered":"2024-04-21T08:37:35.760703+0000","last_unstale":"2024-04-21T08:37:37.793548+0000","last_undegraded":"2024-04-21T08:37:37.793548+0000","last_fullsized":"2024-04-21T08:37:37.793548+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:12:47.511117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792272+0000","last_change":"2024-04-21T08:37:31.453645+0000","last_active":"2024-04-21T08:37:37.792272+0000","last_peered":"2024-04-21T08:37:37.792272+0000","last_clean":"2024-04-21T08:37:37.792272+0000","last_became_active":"2024-04-21T08:37:31.453297+0000","last_became_peered":"2024-04-21T08:37:31.453297+0000","last_unstale":"2024-04-21T08:37:37.792272+0000","last_undegraded":"2024-04-21T08:37:37.792272+0000","last_fullsized":"2024-04-21T08:37:37.792272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:58.976327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793168+0000","last_change":"2024-04-21T08:37:30.420560+0000","last_active":"2024-04-21T08:37:37.793168+0000","last_peered":"2024-04-21T08:37:37.793168+0000","last_clean":"2024-04-21T08:37:37.793168+0000","last_became_active":"2024-04-21T08:37:30.420223+0000","last_became_peered":"2024-04-21T08:37:30.420223+0000","last_unstale":"2024-04-21T08:37:37.793168+0000","last_undegraded":"2024-04-21T08:37:37.793168+0000","last_fullsized":"2024-04-21T08:37:37.793168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:43:55.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327440+0000","last_change":"2024-04-21T08:37:35.598294+0000","last_active":"2024-04-21T08:37:37.327440+0000","last_peered":"2024-04-21T08:37:37.327440+0000","last_clean":"2024-04-21T08:37:37.327440+0000","last_became_active":"2024-04-21T08:37:35.597883+0000","last_became_peered":"2024-04-21T08:37:35.597883+0000","last_unstale":"2024-04-21T08:37:37.327440+0000","last_undegraded":"2024-04-21T08:37:37.327440+0000","last_fullsized":"2024-04-21T08:37:37.327440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:24:05.144763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463885+0000","last_change":"2024-04-21T08:37:30.423182+0000","last_active":"2024-04-21T08:37:37.463885+0000","last_peered":"2024-04-21T08:37:37.463885+0000","last_clean":"2024-04-21T08:37:37.463885+0000","last_became_active":"2024-04-21T08:37:30.422792+0000","last_became_peered":"2024-04-21T08:37:30.422792+0000","last_unstale":"2024-04-21T08:37:37.463885+0000","last_undegraded":"2024-04-21T08:37:37.463885+0000","last_fullsized":"2024-04-21T08:37:37.463885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:44:00.727269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792681+0000","last_change":"2024-04-21T08:37:31.444013+0000","last_active":"2024-04-21T08:37:37.792681+0000","last_peered":"2024-04-21T08:37:37.792681+0000","last_clean":"2024-04-21T08:37:37.792681+0000","last_became_active":"2024-04-21T08:37:31.443684+0000","last_became_peered":"2024-04-21T08:37:31.443684+0000","last_unstale":"2024-04-21T08:37:37.792681+0000","last_undegraded":"2024-04-21T08:37:37.792681+0000","last_fullsized":"2024-04-21T08:37:37.792681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:00:35.400818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.009982+0000","last_change":"2024-04-21T08:37:35.759817+0000","last_active":"2024-04-21T08:37:37.009982+0000","last_peered":"2024-04-21T08:37:37.009982+0000","last_clean":"2024-04-21T08:37:37.009982+0000","last_became_active":"2024-04-21T08:37:35.759396+0000","last_became_peered":"2024-04-21T08:37:35.759396+0000","last_unstale":"2024-04-21T08:37:37.009982+0000","last_undegraded":"2024-04-21T08:37:37.009982+0000","last_fullsized":"2024-04-21T08:37:37.009982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:20:36.170880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:56.668254+0000","last_change":"2024-04-21T08:37:30.414102+0000","last_active":"2024-04-21T08:37:56.668254+0000","last_peered":"2024-04-21T08:37:56.668254+0000","last_clean":"2024-04-21T08:37:56.668254+0000","last_became_active":"2024-04-21T08:37:30.413670+0000","last_became_peered":"2024-04-21T08:37:30.413670+0000","last_unstale":"2024-04-21T08:37:56.668254+0000","last_undegraded":"2024-04-21T08:37:56.668254+0000","last_fullsized":"2024-04-21T08:37:56.668254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:12:49.696516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010371+0000","last_change":"2024-04-21T08:37:31.457451+0000","last_active":"2024-04-21T08:37:37.010371+0000","last_peered":"2024-04-21T08:37:37.010371+0000","last_clean":"2024-04-21T08:37:37.010371+0000","last_became_active":"2024-04-21T08:37:31.457120+0000","last_became_peered":"2024-04-21T08:37:31.457120+0000","last_unstale":"2024-04-21T08:37:37.010371+0000","last_undegraded":"2024-04-21T08:37:37.010371+0000","last_fullsized":"2024-04-21T08:37:37.010371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:07:48.025843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583148+0000","last_change":"2024-04-21T08:37:35.600815+0000","last_active":"2024-04-21T08:37:36.583148+0000","last_peered":"2024-04-21T08:37:36.583148+0000","last_clean":"2024-04-21T08:37:36.583148+0000","last_became_active":"2024-04-21T08:37:35.600380+0000","last_became_peered":"2024-04-21T08:37:35.600380+0000","last_unstale":"2024-04-21T08:37:36.583148+0000","last_undegraded":"2024-04-21T08:37:36.583148+0000","last_fullsized":"2024-04-21T08:37:36.583148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:19:33.710431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793767+0000","last_change":"2024-04-21T08:37:30.425592+0000","last_active":"2024-04-21T08:37:37.793767+0000","last_peered":"2024-04-21T08:37:37.793767+0000","last_clean":"2024-04-21T08:37:37.793767+0000","last_became_active":"2024-04-21T08:37:30.425274+0000","last_became_peered":"2024-04-21T08:37:30.425274+0000","last_unstale":"2024-04-21T08:37:37.793767+0000","last_undegraded":"2024-04-21T08:37:37.793767+0000","last_fullsized":"2024-04-21T08:37:37.793767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:27:29.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644745+0000","last_change":"2024-04-21T08:37:31.455393+0000","last_active":"2024-04-21T08:37:36.644745+0000","last_peered":"2024-04-21T08:37:36.644745+0000","last_clean":"2024-04-21T08:37:36.644745+0000","last_became_active":"2024-04-21T08:37:31.455065+0000","last_became_peered":"2024-04-21T08:37:31.455065+0000","last_unstale":"2024-04-21T08:37:36.644745+0000","last_undegraded":"2024-04-21T08:37:36.644745+0000","last_fullsized":"2024-04-21T08:37:36.644745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:05:23.386790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.794208+0000","last_change":"2024-04-21T08:37:35.760767+0000","last_active":"2024-04-21T08:37:37.794208+0000","last_peered":"2024-04-21T08:37:37.794208+0000","last_clean":"2024-04-21T08:37:37.794208+0000","last_became_active":"2024-04-21T08:37:35.760187+0000","last_became_peered":"2024-04-21T08:37:35.760187+0000","last_unstale":"2024-04-21T08:37:37.794208+0000","last_undegraded":"2024-04-21T08:37:37.794208+0000","last_fullsized":"2024-04-21T08:37:37.794208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:52:11.823765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:55.659724+0000","last_change":"2024-04-21T08:37:30.416696+0000","last_active":"2024-04-21T08:37:55.659724+0000","last_peered":"2024-04-21T08:37:55.659724+0000","last_clean":"2024-04-21T08:37:55.659724+0000","last_became_active":"2024-04-21T08:37:30.415868+0000","last_became_peered":"2024-04-21T08:37:30.415868+0000","last_unstale":"2024-04-21T08:37:55.659724+0000","last_undegraded":"2024-04-21T08:37:55.659724+0000","last_fullsized":"2024-04-21T08:37:55.659724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:48:08.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463414+0000","last_change":"2024-04-21T08:37:31.453861+0000","last_active":"2024-04-21T08:37:37.463414+0000","last_peered":"2024-04-21T08:37:37.463414+0000","last_clean":"2024-04-21T08:37:37.463414+0000","last_became_active":"2024-04-21T08:37:31.453562+0000","last_became_peered":"2024-04-21T08:37:31.453562+0000","last_unstale":"2024-04-21T08:37:37.463414+0000","last_undegraded":"2024-04-21T08:37:37.463414+0000","last_fullsized":"2024-04-21T08:37:37.463414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:17:50.513917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583018+0000","last_change":"2024-04-21T08:37:35.609217+0000","last_active":"2024-04-21T08:37:36.583018+0000","last_peered":"2024-04-21T08:37:36.583018+0000","last_clean":"2024-04-21T08:37:36.583018+0000","last_became_active":"2024-04-21T08:37:35.608786+0000","last_became_peered":"2024-04-21T08:37:35.608786+0000","last_unstale":"2024-04-21T08:37:36.583018+0000","last_undegraded":"2024-04-21T08:37:36.583018+0000","last_fullsized":"2024-04-21T08:37:36.583018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:45:50.945845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.009929+0000","last_change":"2024-04-21T08:37:30.417156+0000","last_active":"2024-04-21T08:37:37.009929+0000","last_peered":"2024-04-21T08:37:37.009929+0000","last_clean":"2024-04-21T08:37:37.009929+0000","last_became_active":"2024-04-21T08:37:30.415717+0000","last_became_peered":"2024-04-21T08:37:30.415717+0000","last_unstale":"2024-04-21T08:37:37.009929+0000","last_undegraded":"2024-04-21T08:37:37.009929+0000","last_fullsized":"2024-04-21T08:37:37.009929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:37.819082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327169+0000","last_change":"2024-04-21T08:37:31.444948+0000","last_active":"2024-04-21T08:37:37.327169+0000","last_peered":"2024-04-21T08:37:37.327169+0000","last_clean":"2024-04-21T08:37:37.327169+0000","last_became_active":"2024-04-21T08:37:31.444604+0000","last_became_peered":"2024-04-21T08:37:31.444604+0000","last_unstale":"2024-04-21T08:37:37.327169+0000","last_undegraded":"2024-04-21T08:37:37.327169+0000","last_fullsized":"2024-04-21T08:37:37.327169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.820503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793995+0000","last_change":"2024-04-21T08:37:35.611541+0000","last_active":"2024-04-21T08:37:37.793995+0000","last_peered":"2024-04-21T08:37:37.793995+0000","last_clean":"2024-04-21T08:37:37.793995+0000","last_became_active":"2024-04-21T08:37:35.611136+0000","last_became_peered":"2024-04-21T08:37:35.611136+0000","last_unstale":"2024-04-21T08:37:37.793995+0000","last_undegraded":"2024-04-21T08:37:37.793995+0000","last_fullsized":"2024-04-21T08:37:37.793995+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:28.254529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583035+0000","last_change":"2024-04-21T08:37:30.414216+0000","last_active":"2024-04-21T08:37:36.583035+0000","last_peered":"2024-04-21T08:37:36.583035+0000","last_clean":"2024-04-21T08:37:36.583035+0000","last_became_active":"2024-04-21T08:37:30.413890+0000","last_became_peered":"2024-04-21T08:37:30.413890+0000","last_unstale":"2024-04-21T08:37:36.583035+0000","last_undegraded":"2024-04-21T08:37:36.583035+0000","last_fullsized":"2024-04-21T08:37:36.583035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:16.972618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010035+0000","last_change":"2024-04-21T08:37:31.449307+0000","last_active":"2024-04-21T08:37:37.010035+0000","last_peered":"2024-04-21T08:37:37.010035+0000","last_clean":"2024-04-21T08:37:37.010035+0000","last_became_active":"2024-04-21T08:37:31.448980+0000","last_became_peered":"2024-04-21T08:37:31.448980+0000","last_unstale":"2024-04-21T08:37:37.010035+0000","last_undegraded":"2024-04-21T08:37:37.010035+0000","last_fullsized":"2024-04-21T08:37:37.010035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:30:31.939274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793935+0000","last_change":"2024-04-21T08:37:35.760616+0000","last_active":"2024-04-21T08:37:37.793935+0000","last_peered":"2024-04-21T08:37:37.793935+0000","last_clean":"2024-04-21T08:37:37.793935+0000","last_became_active":"2024-04-21T08:37:35.760131+0000","last_became_peered":"2024-04-21T08:37:35.760131+0000","last_unstale":"2024-04-21T08:37:37.793935+0000","last_undegraded":"2024-04-21T08:37:37.793935+0000","last_fullsized":"2024-04-21T08:37:37.793935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:51.510259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463508+0000","last_change":"2024-04-21T08:37:30.421935+0000","last_active":"2024-04-21T08:37:37.463508+0000","last_peered":"2024-04-21T08:37:37.463508+0000","last_clean":"2024-04-21T08:37:37.463508+0000","last_became_active":"2024-04-21T08:37:30.421600+0000","last_became_peered":"2024-04-21T08:37:30.421600+0000","last_unstale":"2024-04-21T08:37:37.463508+0000","last_undegraded":"2024-04-21T08:37:37.463508+0000","last_fullsized":"2024-04-21T08:37:37.463508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:28:03.160185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793806+0000","last_change":"2024-04-21T08:37:31.447234+0000","last_active":"2024-04-21T08:37:37.793806+0000","last_peered":"2024-04-21T08:37:37.793806+0000","last_clean":"2024-04-21T08:37:37.793806+0000","last_became_active":"2024-04-21T08:37:31.446876+0000","last_became_peered":"2024-04-21T08:37:31.446876+0000","last_unstale":"2024-04-21T08:37:37.793806+0000","last_undegraded":"2024-04-21T08:37:37.793806+0000","last_fullsized":"2024-04-21T08:37:37.793806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:51.784126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.469999+0000","last_change":"2024-04-21T08:37:35.598577+0000","last_active":"2024-04-21T08:37:37.469999+0000","last_peered":"2024-04-21T08:37:37.469999+0000","last_clean":"2024-04-21T08:37:37.469999+0000","last_became_active":"2024-04-21T08:37:35.598119+0000","last_became_peered":"2024-04-21T08:37:35.598119+0000","last_unstale":"2024-04-21T08:37:37.469999+0000","last_undegraded":"2024-04-21T08:37:37.469999+0000","last_fullsized":"2024-04-21T08:37:37.469999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:41.986674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583598+0000","last_change":"2024-04-21T08:37:30.425737+0000","last_active":"2024-04-21T08:37:36.583598+0000","last_peered":"2024-04-21T08:37:36.583598+0000","last_clean":"2024-04-21T08:37:36.583598+0000","last_became_active":"2024-04-21T08:37:30.425180+0000","last_became_peered":"2024-04-21T08:37:30.425180+0000","last_unstale":"2024-04-21T08:37:36.583598+0000","last_undegraded":"2024-04-21T08:37:36.583598+0000","last_fullsized":"2024-04-21T08:37:36.583598+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:08.646106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793988+0000","last_change":"2024-04-21T08:37:31.451505+0000","last_active":"2024-04-21T08:37:37.793988+0000","last_peered":"2024-04-21T08:37:37.793988+0000","last_clean":"2024-04-21T08:37:37.793988+0000","last_became_active":"2024-04-21T08:37:31.451146+0000","last_became_peered":"2024-04-21T08:37:31.451146+0000","last_unstale":"2024-04-21T08:37:37.793988+0000","last_undegraded":"2024-04-21T08:37:37.793988+0000","last_fullsized":"2024-04-21T08:37:37.793988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:27:01.793910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644629+0000","last_change":"2024-04-21T08:37:35.608995+0000","last_active":"2024-04-21T08:37:36.644629+0000","last_peered":"2024-04-21T08:37:36.644629+0000","last_clean":"2024-04-21T08:37:36.644629+0000","last_became_active":"2024-04-21T08:37:35.608587+0000","last_became_peered":"2024-04-21T08:37:35.608587+0000","last_unstale":"2024-04-21T08:37:36.644629+0000","last_undegraded":"2024-04-21T08:37:36.644629+0000","last_fullsized":"2024-04-21T08:37:36.644629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.829418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793277+0000","last_change":"2024-04-21T08:37:30.419026+0000","last_active":"2024-04-21T08:37:37.793277+0000","last_peered":"2024-04-21T08:37:37.793277+0000","last_clean":"2024-04-21T08:37:37.793277+0000","last_became_active":"2024-04-21T08:37:30.418706+0000","last_became_peered":"2024-04-21T08:37:30.418706+0000","last_unstale":"2024-04-21T08:37:37.793277+0000","last_undegraded":"2024-04-21T08:37:37.793277+0000","last_fullsized":"2024-04-21T08:37:37.793277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:24.441918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793705+0000","last_change":"2024-04-21T08:37:31.446561+0000","last_active":"2024-04-21T08:37:37.793705+0000","last_peered":"2024-04-21T08:37:37.793705+0000","last_clean":"2024-04-21T08:37:37.793705+0000","last_became_active":"2024-04-21T08:37:31.446162+0000","last_became_peered":"2024-04-21T08:37:31.446162+0000","last_unstale":"2024-04-21T08:37:37.793705+0000","last_undegraded":"2024-04-21T08:37:37.793705+0000","last_fullsized":"2024-04-21T08:37:37.793705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:38:24.684537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792653+0000","last_change":"2024-04-21T08:37:35.598748+0000","last_active":"2024-04-21T08:37:37.792653+0000","last_peered":"2024-04-21T08:37:37.792653+0000","last_clean":"2024-04-21T08:37:37.792653+0000","last_became_active":"2024-04-21T08:37:35.598353+0000","last_became_peered":"2024-04-21T08:37:35.598353+0000","last_unstale":"2024-04-21T08:37:37.792653+0000","last_undegraded":"2024-04-21T08:37:37.792653+0000","last_fullsized":"2024-04-21T08:37:37.792653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:47:36.519694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792754+0000","last_change":"2024-04-21T08:37:30.422298+0000","last_active":"2024-04-21T08:37:37.792754+0000","last_peered":"2024-04-21T08:37:37.792754+0000","last_clean":"2024-04-21T08:37:37.792754+0000","last_became_active":"2024-04-21T08:37:30.421297+0000","last_became_peered":"2024-04-21T08:37:30.421297+0000","last_unstale":"2024-04-21T08:37:37.792754+0000","last_undegraded":"2024-04-21T08:37:37.792754+0000","last_fullsized":"2024-04-21T08:37:37.792754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:16.249586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.794146+0000","last_change":"2024-04-21T08:37:31.452367+0000","last_active":"2024-04-21T08:37:37.794146+0000","last_peered":"2024-04-21T08:37:37.794146+0000","last_clean":"2024-04-21T08:37:37.794146+0000","last_became_active":"2024-04-21T08:37:31.452031+0000","last_became_peered":"2024-04-21T08:37:31.452031+0000","last_unstale":"2024-04-21T08:37:37.794146+0000","last_undegraded":"2024-04-21T08:37:37.794146+0000","last_fullsized":"2024-04-21T08:37:37.794146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:58.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327228+0000","last_change":"2024-04-21T08:37:31.451222+0000","last_active":"2024-04-21T08:37:37.327228+0000","last_peered":"2024-04-21T08:37:37.327228+0000","last_clean":"2024-04-21T08:37:37.327228+0000","last_became_active":"2024-04-21T08:37:31.450869+0000","last_became_peered":"2024-04-21T08:37:31.450869+0000","last_unstale":"2024-04-21T08:37:37.327228+0000","last_undegraded":"2024-04-21T08:37:37.327228+0000","last_fullsized":"2024-04-21T08:37:37.327228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:17.051330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583538+0000","last_change":"2024-04-21T08:37:31.453415+0000","last_active":"2024-04-21T08:37:36.583538+0000","last_peered":"2024-04-21T08:37:36.583538+0000","last_clean":"2024-04-21T08:37:36.583538+0000","last_became_active":"2024-04-21T08:37:31.453089+0000","last_became_peered":"2024-04-21T08:37:31.453089+0000","last_unstale":"2024-04-21T08:37:36.583538+0000","last_undegraded":"2024-04-21T08:37:36.583538+0000","last_fullsized":"2024-04-21T08:37:36.583538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:25:44.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582986+0000","last_change":"2024-04-21T08:37:31.446707+0000","last_active":"2024-04-21T08:37:36.582986+0000","last_peered":"2024-04-21T08:37:36.582986+0000","last_clean":"2024-04-21T08:37:36.582986+0000","last_became_active":"2024-04-21T08:37:31.446340+0000","last_became_peered":"2024-04-21T08:37:31.446340+0000","last_unstale":"2024-04-21T08:37:36.582986+0000","last_undegraded":"2024-04-21T08:37:36.582986+0000","last_fullsized":"2024-04-21T08:37:36.582986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:13.649607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644552+0000","last_change":"2024-04-21T08:37:31.455532+0000","last_active":"2024-04-21T08:37:36.644552+0000","last_peered":"2024-04-21T08:37:36.644552+0000","last_clean":"2024-04-21T08:37:36.644552+0000","last_became_active":"2024-04-21T08:37:31.453547+0000","last_became_peered":"2024-04-21T08:37:31.453547+0000","last_unstale":"2024-04-21T08:37:36.644552+0000","last_undegraded":"2024-04-21T08:37:36.644552+0000","last_fullsized":"2024-04-21T08:37:36.644552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:39:47.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792742+0000","last_change":"2024-04-21T08:37:31.452535+0000","last_active":"2024-04-21T08:37:37.792742+0000","last_peered":"2024-04-21T08:37:37.792742+0000","last_clean":"2024-04-21T08:37:37.792742+0000","last_became_active":"2024-04-21T08:37:31.451637+0000","last_became_peered":"2024-04-21T08:37:31.451637+0000","last_unstale":"2024-04-21T08:37:37.792742+0000","last_undegraded":"2024-04-21T08:37:37.792742+0000","last_fullsized":"2024-04-21T08:37:37.792742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:38:38.162780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327676+0000","last_change":"2024-04-21T08:37:31.451806+0000","last_active":"2024-04-21T08:37:37.327676+0000","last_peered":"2024-04-21T08:37:37.327676+0000","last_clean":"2024-04-21T08:37:37.327676+0000","last_became_active":"2024-04-21T08:37:31.451442+0000","last_became_peered":"2024-04-21T08:37:31.451442+0000","last_unstale":"2024-04-21T08:37:37.327676+0000","last_undegraded":"2024-04-21T08:37:37.327676+0000","last_fullsized":"2024-04-21T08:37:37.327676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:07.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583796+0000","last_change":"2024-04-21T08:37:31.444440+0000","last_active":"2024-04-21T08:37:36.583796+0000","last_peered":"2024-04-21T08:37:36.583796+0000","last_clean":"2024-04-21T08:37:36.583796+0000","last_became_active":"2024-04-21T08:37:31.444079+0000","last_became_peered":"2024-04-21T08:37:31.444079+0000","last_unstale":"2024-04-21T08:37:36.583796+0000","last_undegraded":"2024-04-21T08:37:36.583796+0000","last_fullsized":"2024-04-21T08:37:36.583796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:13.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010479+0000","last_change":"2024-04-21T08:37:31.453698+0000","last_active":"2024-04-21T08:37:37.010479+0000","last_peered":"2024-04-21T08:37:37.010479+0000","last_clean":"2024-04-21T08:37:37.010479+0000","last_became_active":"2024-04-21T08:37:31.453379+0000","last_became_peered":"2024-04-21T08:37:31.453379+0000","last_unstale":"2024-04-21T08:37:37.010479+0000","last_undegraded":"2024-04-21T08:37:37.010479+0000","last_fullsized":"2024-04-21T08:37:37.010479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:46:29.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792851+0000","last_change":"2024-04-21T08:37:31.444480+0000","last_active":"2024-04-21T08:37:37.792851+0000","last_peered":"2024-04-21T08:37:37.792851+0000","last_clean":"2024-04-21T08:37:37.792851+0000","last_became_active":"2024-04-21T08:37:31.444030+0000","last_became_peered":"2024-04-21T08:37:31.444030+0000","last_unstale":"2024-04-21T08:37:37.792851+0000","last_undegraded":"2024-04-21T08:37:37.792851+0000","last_fullsized":"2024-04-21T08:37:37.792851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:15:05.668885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010031+0000","last_change":"2024-04-21T08:37:31.456602+0000","last_active":"2024-04-21T08:37:37.010031+0000","last_peered":"2024-04-21T08:37:37.010031+0000","last_clean":"2024-04-21T08:37:37.010031+0000","last_became_active":"2024-04-21T08:37:31.456286+0000","last_became_peered":"2024-04-21T08:37:31.456286+0000","last_unstale":"2024-04-21T08:37:37.010031+0000","last_undegraded":"2024-04-21T08:37:37.010031+0000","last_fullsized":"2024-04-21T08:37:37.010031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:48:12.462037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.470257+0000","last_change":"2024-04-21T08:37:31.447008+0000","last_active":"2024-04-21T08:37:37.470257+0000","last_peered":"2024-04-21T08:37:37.470257+0000","last_clean":"2024-04-21T08:37:37.470257+0000","last_became_active":"2024-04-21T08:37:31.445192+0000","last_became_peered":"2024-04-21T08:37:31.445192+0000","last_unstale":"2024-04-21T08:37:37.470257+0000","last_undegraded":"2024-04-21T08:37:37.470257+0000","last_fullsized":"2024-04-21T08:37:37.470257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:08.025647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792578+0000","last_change":"2024-04-21T08:37:31.443288+0000","last_active":"2024-04-21T08:37:37.792578+0000","last_peered":"2024-04-21T08:37:37.792578+0000","last_clean":"2024-04-21T08:37:37.792578+0000","last_became_active":"2024-04-21T08:37:31.442586+0000","last_became_peered":"2024-04-21T08:37:31.442586+0000","last_unstale":"2024-04-21T08:37:37.792578+0000","last_undegraded":"2024-04-21T08:37:37.792578+0000","last_fullsized":"2024-04-21T08:37:37.792578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:01.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327502+0000","last_change":"2024-04-21T08:37:31.453899+0000","last_active":"2024-04-21T08:37:37.327502+0000","last_peered":"2024-04-21T08:37:37.327502+0000","last_clean":"2024-04-21T08:37:37.327502+0000","last_became_active":"2024-04-21T08:37:31.453527+0000","last_became_peered":"2024-04-21T08:37:31.453527+0000","last_unstale":"2024-04-21T08:37:37.327502+0000","last_undegraded":"2024-04-21T08:37:37.327502+0000","last_fullsized":"2024-04-21T08:37:37.327502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:23:18.429618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283745,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4730,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":74,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":18132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1073003,"data_compressed":50780,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":477999,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070497,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":477819,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":50780,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":478693,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":481713,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920820,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50780,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:38:00.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 -- 172.21.15.94:0/256574426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff61c102980 msgr2=0x7ff61c1092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 --2- 172.21.15.94:0/256574426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff61c102980 0x7ff61c1092a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff614009ca0 tx=0x7ff61402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 -- 172.21.15.94:0/256574426 shutdown_connections 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 --2- 172.21.15.94:0/256574426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff61c101fc0 0x7ff61c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 --2- 172.21.15.94:0/256574426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff61c102980 0x7ff61c1092a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 --2- 172.21.15.94:0/256574426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c100e70 0x7ff61c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 -- 172.21.15.94:0/256574426 >> 172.21.15.94:0/256574426 conn(0x7ff61c0fc3e0 msgr2=0x7ff61c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 -- 172.21.15.94:0/256574426 shutdown_connections 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 -- 172.21.15.94:0/256574426 wait complete. 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 Processor -- start 2024-04-21T08:38:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.553+0000 7ff623e43640 1 -- start start 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff623e43640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff61c100e70 0x7ff61c069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff623e43640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c101fc0 0x7ff61c19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff623e43640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff61c102980 0x7ff61c19f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff623e43640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c10c9a0 con 0x7ff61c100e70 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff623e43640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c10c820 con 0x7ff61c101fc0 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff623e43640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c10cb20 con 0x7ff61c102980 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff621bdf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff61c100e70 0x7ff61c069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff621bdf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff61c100e70 0x7ff61c069c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51782/0 (socket says 172.21.15.94:51782) 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff621bdf640 1 -- 172.21.15.94:0/113385017 learned_addr learned my addr 172.21.15.94:0/113385017 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff621bdf640 1 -- 172.21.15.94:0/113385017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff61c102980 msgr2=0x7ff61c19f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff621bdf640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff61c102980 0x7ff61c19f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff6213de640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c101fc0 0x7ff61c19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff621bdf640 1 -- 172.21.15.94:0/113385017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c101fc0 msgr2=0x7ff61c19cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff621bdf640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c101fc0 0x7ff61c19cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff621bdf640 1 -- 172.21.15.94:0/113385017 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61c19fce0 con 0x7ff61c100e70 2024-04-21T08:38:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff6213de640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c101fc0 0x7ff61c19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:38:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff621bdf640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff61c100e70 0x7ff61c069c70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff60c00c920 tx=0x7ff60c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff612ffd640 1 -- 172.21.15.94:0/113385017 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c007c70 con 0x7ff61c100e70 2024-04-21T08:38:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff612ffd640 1 -- 172.21.15.94:0/113385017 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff60c007e10 con 0x7ff61c100e70 2024-04-21T08:38:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff612ffd640 1 -- 172.21.15.94:0/113385017 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c018820 con 0x7ff61c100e70 2024-04-21T08:38:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff623e43640 1 -- 172.21.15.94:0/113385017 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff61c19ffd0 con 0x7ff61c100e70 2024-04-21T08:38:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.554+0000 7ff623e43640 1 -- 172.21.15.94:0/113385017 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff61c1a04e0 con 0x7ff61c100e70 2024-04-21T08:38:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.555+0000 7ff623e43640 1 -- 172.21.15.94:0/113385017 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff61c108120 con 0x7ff61c100e70 2024-04-21T08:38:00.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.556+0000 7ff612ffd640 1 -- 172.21.15.94:0/113385017 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff60c00f450 con 0x7ff61c100e70 2024-04-21T08:38:00.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.557+0000 7ff612ffd640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff6080a3cc0 0x7ff6080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:00.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.557+0000 7ff6213de640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff6080a3cc0 0x7ff6080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:00.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.557+0000 7ff612ffd640 1 -- 172.21.15.94:0/113385017 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff60c0d0940 con 0x7ff61c100e70 2024-04-21T08:38:00.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.557+0000 7ff6213de640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff6080a3cc0 0x7ff6080a6180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0699e0 tx=0x7ff618005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:00.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.561+0000 7ff612ffd640 1 -- 172.21.15.94:0/113385017 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff60c014030 con 0x7ff61c100e70 2024-04-21T08:38:00.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.737+0000 7ff623e43640 1 -- 172.21.15.94:0/113385017 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff61c062390 con 0x7ff6080a3cc0 2024-04-21T08:38:00.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.746+0000 7ff612ffd640 1 -- 172.21.15.94:0/113385017 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340506 (secure 0 0 0) 0x7ff61c062390 con 0x7ff6080a3cc0 2024-04-21T08:38:00.748 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:38:00.750 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 -- 172.21.15.94:0/113385017 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff6080a3cc0 msgr2=0x7ff6080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff6080a3cc0 0x7ff6080a6180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0699e0 tx=0x7ff618005f90 comp rx=0 tx=0).stop 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 -- 172.21.15.94:0/113385017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff61c100e70 msgr2=0x7ff61c069c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff61c100e70 0x7ff61c069c70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff60c00c920 tx=0x7ff60c00cde0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 -- 172.21.15.94:0/113385017 shutdown_connections 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff6080a3cc0 0x7ff6080a6180 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff61c102980 0x7ff61c19f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c101fc0 0x7ff61c19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 --2- 172.21.15.94:0/113385017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff61c100e70 0x7ff61c069c70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 -- 172.21.15.94:0/113385017 >> 172.21.15.94:0/113385017 conn(0x7ff61c0fc3e0 msgr2=0x7ff61c1052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 -- 172.21.15.94:0/113385017 shutdown_connections 2024-04-21T08:38:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:00.750+0000 7ff623e43640 1 -- 172.21.15.94:0/113385017 wait complete. 2024-04-21T08:38:00.759 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-04-21T08:37:59.586075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595098,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":142,"num_read_kb":127,"num_write":255,"num_write_kb":2958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":198,"ondisk_log_size":198,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373252,"kb_used_data":11588,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568572,"statfs":{"total":1151940427776,"available":1151558217728,"internally_reserved":0,"allocated":11866112,"data_stored":7515782,"data_compressed":284622,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2320,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":29,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327492+0000","last_change":"2024-04-21T08:37:31.448345+0000","last_active":"2024-04-21T08:37:37.327492+0000","last_peered":"2024-04-21T08:37:37.327492+0000","last_clean":"2024-04-21T08:37:37.327492+0000","last_became_active":"2024-04-21T08:37:31.447878+0000","last_became_peered":"2024-04-21T08:37:31.447878+0000","last_unstale":"2024-04-21T08:37:37.327492+0000","last_undegraded":"2024-04-21T08:37:37.327492+0000","last_fullsized":"2024-04-21T08:37:37.327492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:45.589165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644842+0000","last_change":"2024-04-21T08:37:31.452895+0000","last_active":"2024-04-21T08:37:36.644842+0000","last_peered":"2024-04-21T08:37:36.644842+0000","last_clean":"2024-04-21T08:37:36.644842+0000","last_became_active":"2024-04-21T08:37:31.452550+0000","last_became_peered":"2024-04-21T08:37:31.452550+0000","last_unstale":"2024-04-21T08:37:36.644842+0000","last_undegraded":"2024-04-21T08:37:36.644842+0000","last_fullsized":"2024-04-21T08:37:36.644842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:02.497802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644670+0000","last_change":"2024-04-21T08:37:31.447552+0000","last_active":"2024-04-21T08:37:36.644670+0000","last_peered":"2024-04-21T08:37:36.644670+0000","last_clean":"2024-04-21T08:37:36.644670+0000","last_became_active":"2024-04-21T08:37:31.446044+0000","last_became_peered":"2024-04-21T08:37:31.446044+0000","last_unstale":"2024-04-21T08:37:36.644670+0000","last_undegraded":"2024-04-21T08:37:36.644670+0000","last_fullsized":"2024-04-21T08:37:36.644670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:22.801977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.794319+0000","last_change":"2024-04-21T08:37:31.452400+0000","last_active":"2024-04-21T08:37:37.794319+0000","last_peered":"2024-04-21T08:37:37.794319+0000","last_clean":"2024-04-21T08:37:37.794319+0000","last_became_active":"2024-04-21T08:37:31.451462+0000","last_became_peered":"2024-04-21T08:37:31.451462+0000","last_unstale":"2024-04-21T08:37:37.794319+0000","last_undegraded":"2024-04-21T08:37:37.794319+0000","last_fullsized":"2024-04-21T08:37:37.794319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:22.463481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463505+0000","last_change":"2024-04-21T08:37:31.448942+0000","last_active":"2024-04-21T08:37:37.463505+0000","last_peered":"2024-04-21T08:37:37.463505+0000","last_clean":"2024-04-21T08:37:37.463505+0000","last_became_active":"2024-04-21T08:37:31.448526+0000","last_became_peered":"2024-04-21T08:37:31.448526+0000","last_unstale":"2024-04-21T08:37:37.463505+0000","last_undegraded":"2024-04-21T08:37:37.463505+0000","last_fullsized":"2024-04-21T08:37:37.463505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:29.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583194+0000","last_change":"2024-04-21T08:37:31.452165+0000","last_active":"2024-04-21T08:37:36.583194+0000","last_peered":"2024-04-21T08:37:36.583194+0000","last_clean":"2024-04-21T08:37:36.583194+0000","last_became_active":"2024-04-21T08:37:31.451802+0000","last_became_peered":"2024-04-21T08:37:31.451802+0000","last_unstale":"2024-04-21T08:37:36.583194+0000","last_undegraded":"2024-04-21T08:37:36.583194+0000","last_fullsized":"2024-04-21T08:37:36.583194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:42.210544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793621+0000","last_change":"2024-04-21T08:37:31.444798+0000","last_active":"2024-04-21T08:37:37.793621+0000","last_peered":"2024-04-21T08:37:37.793621+0000","last_clean":"2024-04-21T08:37:37.793621+0000","last_became_active":"2024-04-21T08:37:31.444313+0000","last_became_peered":"2024-04-21T08:37:31.444313+0000","last_unstale":"2024-04-21T08:37:37.793621+0000","last_undegraded":"2024-04-21T08:37:37.793621+0000","last_fullsized":"2024-04-21T08:37:37.793621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:22:03.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.009783+0000","last_change":"2024-04-21T08:37:31.448257+0000","last_active":"2024-04-21T08:37:37.009783+0000","last_peered":"2024-04-21T08:37:37.009783+0000","last_clean":"2024-04-21T08:37:37.009783+0000","last_became_active":"2024-04-21T08:37:31.447845+0000","last_became_peered":"2024-04-21T08:37:31.447845+0000","last_unstale":"2024-04-21T08:37:37.009783+0000","last_undegraded":"2024-04-21T08:37:37.009783+0000","last_fullsized":"2024-04-21T08:37:37.009783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:19.956699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.470394+0000","last_change":"2024-04-21T08:37:31.443022+0000","last_active":"2024-04-21T08:37:37.470394+0000","last_peered":"2024-04-21T08:37:37.470394+0000","last_clean":"2024-04-21T08:37:37.470394+0000","last_became_active":"2024-04-21T08:37:31.442645+0000","last_became_peered":"2024-04-21T08:37:31.442645+0000","last_unstale":"2024-04-21T08:37:37.470394+0000","last_undegraded":"2024-04-21T08:37:37.470394+0000","last_fullsized":"2024-04-21T08:37:37.470394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:18:10.861281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583635+0000","last_change":"2024-04-21T08:37:31.453427+0000","last_active":"2024-04-21T08:37:36.583635+0000","last_peered":"2024-04-21T08:37:36.583635+0000","last_clean":"2024-04-21T08:37:36.583635+0000","last_became_active":"2024-04-21T08:37:31.453096+0000","last_became_peered":"2024-04-21T08:37:31.453096+0000","last_unstale":"2024-04-21T08:37:36.583635+0000","last_undegraded":"2024-04-21T08:37:36.583635+0000","last_fullsized":"2024-04-21T08:37:36.583635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:49.450351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793834+0000","last_change":"2024-04-21T08:37:31.449908+0000","last_active":"2024-04-21T08:37:37.793834+0000","last_peered":"2024-04-21T08:37:37.793834+0000","last_clean":"2024-04-21T08:37:37.793834+0000","last_became_active":"2024-04-21T08:37:31.449558+0000","last_became_peered":"2024-04-21T08:37:31.449558+0000","last_unstale":"2024-04-21T08:37:37.793834+0000","last_undegraded":"2024-04-21T08:37:37.793834+0000","last_fullsized":"2024-04-21T08:37:37.793834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:21.538689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.794213+0000","last_change":"2024-04-21T08:37:31.452973+0000","last_active":"2024-04-21T08:37:37.794213+0000","last_peered":"2024-04-21T08:37:37.794213+0000","last_clean":"2024-04-21T08:37:37.794213+0000","last_became_active":"2024-04-21T08:37:31.452586+0000","last_became_peered":"2024-04-21T08:37:31.452586+0000","last_unstale":"2024-04-21T08:37:37.794213+0000","last_undegraded":"2024-04-21T08:37:37.794213+0000","last_fullsized":"2024-04-21T08:37:37.794213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:09:11.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792575+0000","last_change":"2024-04-21T08:37:31.445801+0000","last_active":"2024-04-21T08:37:37.792575+0000","last_peered":"2024-04-21T08:37:37.792575+0000","last_clean":"2024-04-21T08:37:37.792575+0000","last_became_active":"2024-04-21T08:37:31.445478+0000","last_became_peered":"2024-04-21T08:37:31.445478+0000","last_unstale":"2024-04-21T08:37:37.792575+0000","last_undegraded":"2024-04-21T08:37:37.792575+0000","last_fullsized":"2024-04-21T08:37:37.792575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:33.352739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644406+0000","last_change":"2024-04-21T08:37:31.450735+0000","last_active":"2024-04-21T08:37:36.644406+0000","last_peered":"2024-04-21T08:37:36.644406+0000","last_clean":"2024-04-21T08:37:36.644406+0000","last_became_active":"2024-04-21T08:37:31.450384+0000","last_became_peered":"2024-04-21T08:37:31.450384+0000","last_unstale":"2024-04-21T08:37:36.644406+0000","last_undegraded":"2024-04-21T08:37:36.644406+0000","last_fullsized":"2024-04-21T08:37:36.644406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:45.950489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644416+0000","last_change":"2024-04-21T08:37:31.454690+0000","last_active":"2024-04-21T08:37:36.644416+0000","last_peered":"2024-04-21T08:37:36.644416+0000","last_clean":"2024-04-21T08:37:36.644416+0000","last_became_active":"2024-04-21T08:37:31.454330+0000","last_became_peered":"2024-04-21T08:37:31.454330+0000","last_unstale":"2024-04-21T08:37:36.644416+0000","last_undegraded":"2024-04-21T08:37:36.644416+0000","last_fullsized":"2024-04-21T08:37:36.644416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:52.644317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.469900+0000","last_change":"2024-04-21T08:37:31.441389+0000","last_active":"2024-04-21T08:37:37.469900+0000","last_peered":"2024-04-21T08:37:37.469900+0000","last_clean":"2024-04-21T08:37:37.469900+0000","last_became_active":"2024-04-21T08:37:31.441065+0000","last_became_peered":"2024-04-21T08:37:31.441065+0000","last_unstale":"2024-04-21T08:37:37.469900+0000","last_undegraded":"2024-04-21T08:37:37.469900+0000","last_fullsized":"2024-04-21T08:37:37.469900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:56.528154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327357+0000","last_change":"2024-04-21T08:37:31.446722+0000","last_active":"2024-04-21T08:37:37.327357+0000","last_peered":"2024-04-21T08:37:37.327357+0000","last_clean":"2024-04-21T08:37:37.327357+0000","last_became_active":"2024-04-21T08:37:31.446396+0000","last_became_peered":"2024-04-21T08:37:31.446396+0000","last_unstale":"2024-04-21T08:37:37.327357+0000","last_undegraded":"2024-04-21T08:37:37.327357+0000","last_fullsized":"2024-04-21T08:37:37.327357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:17.488633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582923+0000","last_change":"2024-04-21T08:37:31.444388+0000","last_active":"2024-04-21T08:37:36.582923+0000","last_peered":"2024-04-21T08:37:36.582923+0000","last_clean":"2024-04-21T08:37:36.582923+0000","last_became_active":"2024-04-21T08:37:31.444023+0000","last_became_peered":"2024-04-21T08:37:31.444023+0000","last_unstale":"2024-04-21T08:37:36.582923+0000","last_undegraded":"2024-04-21T08:37:36.582923+0000","last_fullsized":"2024-04-21T08:37:36.582923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:07:46.238941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583622+0000","last_change":"2024-04-21T08:37:31.447850+0000","last_active":"2024-04-21T08:37:36.583622+0000","last_peered":"2024-04-21T08:37:36.583622+0000","last_clean":"2024-04-21T08:37:36.583622+0000","last_became_active":"2024-04-21T08:37:31.447505+0000","last_became_peered":"2024-04-21T08:37:31.447505+0000","last_unstale":"2024-04-21T08:37:36.583622+0000","last_undegraded":"2024-04-21T08:37:36.583622+0000","last_fullsized":"2024-04-21T08:37:36.583622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:55.399410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583175+0000","last_change":"2024-04-21T08:37:35.608747+0000","last_active":"2024-04-21T08:37:36.583175+0000","last_peered":"2024-04-21T08:37:36.583175+0000","last_clean":"2024-04-21T08:37:36.583175+0000","last_became_active":"2024-04-21T08:37:35.608157+0000","last_became_peered":"2024-04-21T08:37:35.608157+0000","last_unstale":"2024-04-21T08:37:36.583175+0000","last_undegraded":"2024-04-21T08:37:36.583175+0000","last_fullsized":"2024-04-21T08:37:36.583175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:53.384190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:56.668232+0000","last_change":"2024-04-21T08:37:30.414311+0000","last_active":"2024-04-21T08:37:56.668232+0000","last_peered":"2024-04-21T08:37:56.668232+0000","last_clean":"2024-04-21T08:37:56.668232+0000","last_became_active":"2024-04-21T08:37:30.413776+0000","last_became_peered":"2024-04-21T08:37:30.413776+0000","last_unstale":"2024-04-21T08:37:56.668232+0000","last_undegraded":"2024-04-21T08:37:56.668232+0000","last_fullsized":"2024-04-21T08:37:56.668232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:48.628006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010284+0000","last_change":"2024-04-21T08:37:31.451813+0000","last_active":"2024-04-21T08:37:37.010284+0000","last_peered":"2024-04-21T08:37:37.010284+0000","last_clean":"2024-04-21T08:37:37.010284+0000","last_became_active":"2024-04-21T08:37:31.451447+0000","last_became_peered":"2024-04-21T08:37:31.451447+0000","last_unstale":"2024-04-21T08:37:37.010284+0000","last_undegraded":"2024-04-21T08:37:37.010284+0000","last_fullsized":"2024-04-21T08:37:37.010284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:46:45.235326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793183+0000","last_change":"2024-04-21T08:37:35.603518+0000","last_active":"2024-04-21T08:37:37.793183+0000","last_peered":"2024-04-21T08:37:37.793183+0000","last_clean":"2024-04-21T08:37:37.793183+0000","last_became_active":"2024-04-21T08:37:35.603038+0000","last_became_peered":"2024-04-21T08:37:35.603038+0000","last_unstale":"2024-04-21T08:37:37.793183+0000","last_undegraded":"2024-04-21T08:37:37.793183+0000","last_fullsized":"2024-04-21T08:37:37.793183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:06:29.224025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010426+0000","last_change":"2024-04-21T08:37:30.424608+0000","last_active":"2024-04-21T08:37:37.010426+0000","last_peered":"2024-04-21T08:37:37.010426+0000","last_clean":"2024-04-21T08:37:37.010426+0000","last_became_active":"2024-04-21T08:37:30.424259+0000","last_became_peered":"2024-04-21T08:37:30.424259+0000","last_unstale":"2024-04-21T08:37:37.010426+0000","last_undegraded":"2024-04-21T08:37:37.010426+0000","last_fullsized":"2024-04-21T08:37:37.010426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:38.007095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644687+0000","last_change":"2024-04-21T08:37:31.444656+0000","last_active":"2024-04-21T08:37:36.644687+0000","last_peered":"2024-04-21T08:37:36.644687+0000","last_clean":"2024-04-21T08:37:36.644687+0000","last_became_active":"2024-04-21T08:37:31.444300+0000","last_became_peered":"2024-04-21T08:37:31.444300+0000","last_unstale":"2024-04-21T08:37:36.644687+0000","last_undegraded":"2024-04-21T08:37:36.644687+0000","last_fullsized":"2024-04-21T08:37:36.644687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:40.901674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793547+0000","last_change":"2024-04-21T08:37:35.611732+0000","last_active":"2024-04-21T08:37:37.793547+0000","last_peered":"2024-04-21T08:37:37.793547+0000","last_clean":"2024-04-21T08:37:37.793547+0000","last_became_active":"2024-04-21T08:37:35.611060+0000","last_became_peered":"2024-04-21T08:37:35.611060+0000","last_unstale":"2024-04-21T08:37:37.793547+0000","last_undegraded":"2024-04-21T08:37:37.793547+0000","last_fullsized":"2024-04-21T08:37:37.793547+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:36:01.491949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.464007+0000","last_change":"2024-04-21T08:37:30.422350+0000","last_active":"2024-04-21T08:37:37.464007+0000","last_peered":"2024-04-21T08:37:37.464007+0000","last_clean":"2024-04-21T08:37:37.464007+0000","last_became_active":"2024-04-21T08:37:30.421659+0000","last_became_peered":"2024-04-21T08:37:30.421659+0000","last_unstale":"2024-04-21T08:37:37.464007+0000","last_undegraded":"2024-04-21T08:37:37.464007+0000","last_fullsized":"2024-04-21T08:37:37.464007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:09:29.894205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.470215+0000","last_change":"2024-04-21T08:37:31.443896+0000","last_active":"2024-04-21T08:37:37.470215+0000","last_peered":"2024-04-21T08:37:37.470215+0000","last_clean":"2024-04-21T08:37:37.470215+0000","last_became_active":"2024-04-21T08:37:31.442684+0000","last_became_peered":"2024-04-21T08:37:31.442684+0000","last_unstale":"2024-04-21T08:37:37.470215+0000","last_undegraded":"2024-04-21T08:37:37.470215+0000","last_fullsized":"2024-04-21T08:37:37.470215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:14:18.649565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793303+0000","last_change":"2024-04-21T08:37:35.600468+0000","last_active":"2024-04-21T08:37:37.793303+0000","last_peered":"2024-04-21T08:37:37.793303+0000","last_clean":"2024-04-21T08:37:37.793303+0000","last_became_active":"2024-04-21T08:37:35.599940+0000","last_became_peered":"2024-04-21T08:37:35.599940+0000","last_unstale":"2024-04-21T08:37:37.793303+0000","last_undegraded":"2024-04-21T08:37:37.793303+0000","last_fullsized":"2024-04-21T08:37:37.793303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:48.125434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463767+0000","last_change":"2024-04-21T08:37:30.416480+0000","last_active":"2024-04-21T08:37:37.463767+0000","last_peered":"2024-04-21T08:37:37.463767+0000","last_clean":"2024-04-21T08:37:37.463767+0000","last_became_active":"2024-04-21T08:37:30.416072+0000","last_became_peered":"2024-04-21T08:37:30.416072+0000","last_unstale":"2024-04-21T08:37:37.463767+0000","last_undegraded":"2024-04-21T08:37:37.463767+0000","last_fullsized":"2024-04-21T08:37:37.463767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:28:22.400926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793203+0000","last_change":"2024-04-21T08:37:31.457216+0000","last_active":"2024-04-21T08:37:37.793203+0000","last_peered":"2024-04-21T08:37:37.793203+0000","last_clean":"2024-04-21T08:37:37.793203+0000","last_became_active":"2024-04-21T08:37:31.456856+0000","last_became_peered":"2024-04-21T08:37:31.456856+0000","last_unstale":"2024-04-21T08:37:37.793203+0000","last_undegraded":"2024-04-21T08:37:37.793203+0000","last_fullsized":"2024-04-21T08:37:37.793203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:10.272886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583515+0000","last_change":"2024-04-21T08:37:35.601715+0000","last_active":"2024-04-21T08:37:36.583515+0000","last_peered":"2024-04-21T08:37:36.583515+0000","last_clean":"2024-04-21T08:37:36.583515+0000","last_became_active":"2024-04-21T08:37:35.601269+0000","last_became_peered":"2024-04-21T08:37:35.601269+0000","last_unstale":"2024-04-21T08:37:36.583515+0000","last_undegraded":"2024-04-21T08:37:36.583515+0000","last_fullsized":"2024-04-21T08:37:36.583515+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:57.192918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793165+0000","last_change":"2024-04-21T08:37:30.415510+0000","last_active":"2024-04-21T08:37:37.793165+0000","last_peered":"2024-04-21T08:37:37.793165+0000","last_clean":"2024-04-21T08:37:37.793165+0000","last_became_active":"2024-04-21T08:37:30.415199+0000","last_became_peered":"2024-04-21T08:37:30.415199+0000","last_unstale":"2024-04-21T08:37:37.793165+0000","last_undegraded":"2024-04-21T08:37:37.793165+0000","last_fullsized":"2024-04-21T08:37:37.793165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:32.336247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463505+0000","last_change":"2024-04-21T08:37:31.444022+0000","last_active":"2024-04-21T08:37:37.463505+0000","last_peered":"2024-04-21T08:37:37.463505+0000","last_clean":"2024-04-21T08:37:37.463505+0000","last_became_active":"2024-04-21T08:37:31.443609+0000","last_became_peered":"2024-04-21T08:37:31.443609+0000","last_unstale":"2024-04-21T08:37:37.463505+0000","last_undegraded":"2024-04-21T08:37:37.463505+0000","last_fullsized":"2024-04-21T08:37:37.463505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:39.978817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583458+0000","last_change":"2024-04-21T08:37:35.608954+0000","last_active":"2024-04-21T08:37:36.583458+0000","last_peered":"2024-04-21T08:37:36.583458+0000","last_clean":"2024-04-21T08:37:36.583458+0000","last_became_active":"2024-04-21T08:37:35.608364+0000","last_became_peered":"2024-04-21T08:37:35.608364+0000","last_unstale":"2024-04-21T08:37:36.583458+0000","last_undegraded":"2024-04-21T08:37:36.583458+0000","last_fullsized":"2024-04-21T08:37:36.583458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:04.574382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:56.665863+0000","last_change":"2024-04-21T08:37:30.419877+0000","last_active":"2024-04-21T08:37:56.665863+0000","last_peered":"2024-04-21T08:37:56.665863+0000","last_clean":"2024-04-21T08:37:56.665863+0000","last_became_active":"2024-04-21T08:37:30.419572+0000","last_became_peered":"2024-04-21T08:37:30.419572+0000","last_unstale":"2024-04-21T08:37:56.665863+0000","last_undegraded":"2024-04-21T08:37:56.665863+0000","last_fullsized":"2024-04-21T08:37:56.665863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:32:35.524089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793298+0000","last_change":"2024-04-21T08:37:31.458590+0000","last_active":"2024-04-21T08:37:37.793298+0000","last_peered":"2024-04-21T08:37:37.793298+0000","last_clean":"2024-04-21T08:37:37.793298+0000","last_became_active":"2024-04-21T08:37:31.458196+0000","last_became_peered":"2024-04-21T08:37:31.458196+0000","last_unstale":"2024-04-21T08:37:37.793298+0000","last_undegraded":"2024-04-21T08:37:37.793298+0000","last_fullsized":"2024-04-21T08:37:37.793298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:00.890859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327672+0000","last_change":"2024-04-21T08:37:35.604555+0000","last_active":"2024-04-21T08:37:37.327672+0000","last_peered":"2024-04-21T08:37:37.327672+0000","last_clean":"2024-04-21T08:37:37.327672+0000","last_became_active":"2024-04-21T08:37:35.603847+0000","last_became_peered":"2024-04-21T08:37:35.603847+0000","last_unstale":"2024-04-21T08:37:37.327672+0000","last_undegraded":"2024-04-21T08:37:37.327672+0000","last_fullsized":"2024-04-21T08:37:37.327672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:41:59.194494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792381+0000","last_change":"2024-04-21T08:37:30.415786+0000","last_active":"2024-04-21T08:37:37.792381+0000","last_peered":"2024-04-21T08:37:37.792381+0000","last_clean":"2024-04-21T08:37:37.792381+0000","last_became_active":"2024-04-21T08:37:30.415472+0000","last_became_peered":"2024-04-21T08:37:30.415472+0000","last_unstale":"2024-04-21T08:37:37.792381+0000","last_undegraded":"2024-04-21T08:37:37.792381+0000","last_fullsized":"2024-04-21T08:37:37.792381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.417132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.009927+0000","last_change":"2024-04-21T08:37:31.455720+0000","last_active":"2024-04-21T08:37:37.009927+0000","last_peered":"2024-04-21T08:37:37.009927+0000","last_clean":"2024-04-21T08:37:37.009927+0000","last_became_active":"2024-04-21T08:37:31.455334+0000","last_became_peered":"2024-04-21T08:37:31.455334+0000","last_unstale":"2024-04-21T08:37:37.009927+0000","last_undegraded":"2024-04-21T08:37:37.009927+0000","last_fullsized":"2024-04-21T08:37:37.009927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:06:55.220656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644802+0000","last_change":"2024-04-21T08:37:35.603305+0000","last_active":"2024-04-21T08:37:36.644802+0000","last_peered":"2024-04-21T08:37:36.644802+0000","last_clean":"2024-04-21T08:37:36.644802+0000","last_became_active":"2024-04-21T08:37:35.602854+0000","last_became_peered":"2024-04-21T08:37:35.602854+0000","last_unstale":"2024-04-21T08:37:36.644802+0000","last_undegraded":"2024-04-21T08:37:36.644802+0000","last_fullsized":"2024-04-21T08:37:36.644802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:10.993864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792869+0000","last_change":"2024-04-21T08:37:30.420797+0000","last_active":"2024-04-21T08:37:37.792869+0000","last_peered":"2024-04-21T08:37:37.792869+0000","last_clean":"2024-04-21T08:37:37.792869+0000","last_became_active":"2024-04-21T08:37:30.420485+0000","last_became_peered":"2024-04-21T08:37:30.420485+0000","last_unstale":"2024-04-21T08:37:37.792869+0000","last_undegraded":"2024-04-21T08:37:37.792869+0000","last_fullsized":"2024-04-21T08:37:37.792869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:03.275677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582283+0000","last_change":"2024-04-21T08:37:31.448406+0000","last_active":"2024-04-21T08:37:36.582283+0000","last_peered":"2024-04-21T08:37:36.582283+0000","last_clean":"2024-04-21T08:37:36.582283+0000","last_became_active":"2024-04-21T08:37:31.448085+0000","last_became_peered":"2024-04-21T08:37:31.448085+0000","last_unstale":"2024-04-21T08:37:36.582283+0000","last_undegraded":"2024-04-21T08:37:36.582283+0000","last_fullsized":"2024-04-21T08:37:36.582283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:49.574064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010048+0000","last_change":"2024-04-21T08:37:35.601338+0000","last_active":"2024-04-21T08:37:37.010048+0000","last_peered":"2024-04-21T08:37:37.010048+0000","last_clean":"2024-04-21T08:37:37.010048+0000","last_became_active":"2024-04-21T08:37:35.600890+0000","last_became_peered":"2024-04-21T08:37:35.600890+0000","last_unstale":"2024-04-21T08:37:37.010048+0000","last_undegraded":"2024-04-21T08:37:37.010048+0000","last_fullsized":"2024-04-21T08:37:37.010048+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:30:21.755850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:55.661291+0000","last_change":"2024-04-21T08:37:30.423320+0000","last_active":"2024-04-21T08:37:55.661291+0000","last_peered":"2024-04-21T08:37:55.661291+0000","last_clean":"2024-04-21T08:37:55.661291+0000","last_became_active":"2024-04-21T08:37:30.423020+0000","last_became_peered":"2024-04-21T08:37:30.423020+0000","last_unstale":"2024-04-21T08:37:55.661291+0000","last_undegraded":"2024-04-21T08:37:55.661291+0000","last_fullsized":"2024-04-21T08:37:55.661291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:08:27.313196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.470485+0000","last_change":"2024-04-21T08:37:31.442425+0000","last_active":"2024-04-21T08:37:37.470485+0000","last_peered":"2024-04-21T08:37:37.470485+0000","last_clean":"2024-04-21T08:37:37.470485+0000","last_became_active":"2024-04-21T08:37:31.442028+0000","last_became_peered":"2024-04-21T08:37:31.442028+0000","last_unstale":"2024-04-21T08:37:37.470485+0000","last_undegraded":"2024-04-21T08:37:37.470485+0000","last_fullsized":"2024-04-21T08:37:37.470485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:45.498656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582551+0000","last_change":"2024-04-21T08:37:35.607829+0000","last_active":"2024-04-21T08:37:36.582551+0000","last_peered":"2024-04-21T08:37:36.582551+0000","last_clean":"2024-04-21T08:37:36.582551+0000","last_became_active":"2024-04-21T08:37:35.607150+0000","last_became_peered":"2024-04-21T08:37:35.607150+0000","last_unstale":"2024-04-21T08:37:36.582551+0000","last_undegraded":"2024-04-21T08:37:36.582551+0000","last_fullsized":"2024-04-21T08:37:36.582551+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:01:34.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793870+0000","last_change":"2024-04-21T08:37:30.418613+0000","last_active":"2024-04-21T08:37:37.793870+0000","last_peered":"2024-04-21T08:37:37.793870+0000","last_clean":"2024-04-21T08:37:37.793870+0000","last_became_active":"2024-04-21T08:37:30.418300+0000","last_became_peered":"2024-04-21T08:37:30.418300+0000","last_unstale":"2024-04-21T08:37:37.793870+0000","last_undegraded":"2024-04-21T08:37:37.793870+0000","last_fullsized":"2024-04-21T08:37:37.793870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:17:35.370753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644684+0000","last_change":"2024-04-21T08:37:31.456902+0000","last_active":"2024-04-21T08:37:36.644684+0000","last_peered":"2024-04-21T08:37:36.644684+0000","last_clean":"2024-04-21T08:37:36.644684+0000","last_became_active":"2024-04-21T08:37:31.453637+0000","last_became_peered":"2024-04-21T08:37:31.453637+0000","last_unstale":"2024-04-21T08:37:36.644684+0000","last_undegraded":"2024-04-21T08:37:36.644684+0000","last_fullsized":"2024-04-21T08:37:36.644684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:48:59.387349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793938+0000","last_change":"2024-04-21T08:37:35.608280+0000","last_active":"2024-04-21T08:37:37.793938+0000","last_peered":"2024-04-21T08:37:37.793938+0000","last_clean":"2024-04-21T08:37:37.793938+0000","last_became_active":"2024-04-21T08:37:35.607898+0000","last_became_peered":"2024-04-21T08:37:35.607898+0000","last_unstale":"2024-04-21T08:37:37.793938+0000","last_undegraded":"2024-04-21T08:37:37.793938+0000","last_fullsized":"2024-04-21T08:37:37.793938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:55.689519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793121+0000","last_change":"2024-04-21T08:37:30.417352+0000","last_active":"2024-04-21T08:37:37.793121+0000","last_peered":"2024-04-21T08:37:37.793121+0000","last_clean":"2024-04-21T08:37:37.793121+0000","last_became_active":"2024-04-21T08:37:30.417000+0000","last_became_peered":"2024-04-21T08:37:30.417000+0000","last_unstale":"2024-04-21T08:37:37.793121+0000","last_undegraded":"2024-04-21T08:37:37.793121+0000","last_fullsized":"2024-04-21T08:37:37.793121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:56.087334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.470354+0000","last_change":"2024-04-21T08:37:31.458229+0000","last_active":"2024-04-21T08:37:37.470354+0000","last_peered":"2024-04-21T08:37:37.470354+0000","last_clean":"2024-04-21T08:37:37.470354+0000","last_became_active":"2024-04-21T08:37:31.456897+0000","last_became_peered":"2024-04-21T08:37:31.456897+0000","last_unstale":"2024-04-21T08:37:37.470354+0000","last_undegraded":"2024-04-21T08:37:37.470354+0000","last_fullsized":"2024-04-21T08:37:37.470354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:25:58.253081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582556+0000","last_change":"2024-04-21T08:37:35.601077+0000","last_active":"2024-04-21T08:37:36.582556+0000","last_peered":"2024-04-21T08:37:36.582556+0000","last_clean":"2024-04-21T08:37:36.582556+0000","last_became_active":"2024-04-21T08:37:35.600673+0000","last_became_peered":"2024-04-21T08:37:35.600673+0000","last_unstale":"2024-04-21T08:37:36.582556+0000","last_undegraded":"2024-04-21T08:37:36.582556+0000","last_fullsized":"2024-04-21T08:37:36.582556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:04.554208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:56.704816+0000","last_change":"2024-04-21T08:37:30.421440+0000","last_active":"2024-04-21T08:37:56.704816+0000","last_peered":"2024-04-21T08:37:56.704816+0000","last_clean":"2024-04-21T08:37:56.704816+0000","last_became_active":"2024-04-21T08:37:30.420988+0000","last_became_peered":"2024-04-21T08:37:30.420988+0000","last_unstale":"2024-04-21T08:37:56.704816+0000","last_undegraded":"2024-04-21T08:37:56.704816+0000","last_fullsized":"2024-04-21T08:37:56.704816+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:38:03.850705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463776+0000","last_change":"2024-04-21T08:37:31.447208+0000","last_active":"2024-04-21T08:37:37.463776+0000","last_peered":"2024-04-21T08:37:37.463776+0000","last_clean":"2024-04-21T08:37:37.463776+0000","last_became_active":"2024-04-21T08:37:31.444464+0000","last_became_peered":"2024-04-21T08:37:31.444464+0000","last_unstale":"2024-04-21T08:37:37.463776+0000","last_undegraded":"2024-04-21T08:37:37.463776+0000","last_fullsized":"2024-04-21T08:37:37.463776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:33.805676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793579+0000","last_change":"2024-04-21T08:37:35.603937+0000","last_active":"2024-04-21T08:37:37.793579+0000","last_peered":"2024-04-21T08:37:37.793579+0000","last_clean":"2024-04-21T08:37:37.793579+0000","last_became_active":"2024-04-21T08:37:35.603477+0000","last_became_peered":"2024-04-21T08:37:35.603477+0000","last_unstale":"2024-04-21T08:37:37.793579+0000","last_undegraded":"2024-04-21T08:37:37.793579+0000","last_fullsized":"2024-04-21T08:37:37.793579+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:40:33.901570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:55.661103+0000","last_change":"2024-04-21T08:37:30.416941+0000","last_active":"2024-04-21T08:37:55.661103+0000","last_peered":"2024-04-21T08:37:55.661103+0000","last_clean":"2024-04-21T08:37:55.661103+0000","last_became_active":"2024-04-21T08:37:30.416625+0000","last_became_peered":"2024-04-21T08:37:30.416625+0000","last_unstale":"2024-04-21T08:37:55.661103+0000","last_undegraded":"2024-04-21T08:37:55.661103+0000","last_fullsized":"2024-04-21T08:37:55.661103+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:07.782779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792496+0000","last_change":"2024-04-21T08:37:31.441707+0000","last_active":"2024-04-21T08:37:37.792496+0000","last_peered":"2024-04-21T08:37:37.792496+0000","last_clean":"2024-04-21T08:37:37.792496+0000","last_became_active":"2024-04-21T08:37:31.441366+0000","last_became_peered":"2024-04-21T08:37:31.441366+0000","last_unstale":"2024-04-21T08:37:37.792496+0000","last_undegraded":"2024-04-21T08:37:37.792496+0000","last_fullsized":"2024-04-21T08:37:37.792496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:16.573804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327512+0000","last_change":"2024-04-21T08:37:35.596214+0000","last_active":"2024-04-21T08:37:37.327512+0000","last_peered":"2024-04-21T08:37:37.327512+0000","last_clean":"2024-04-21T08:37:37.327512+0000","last_became_active":"2024-04-21T08:37:35.595773+0000","last_became_peered":"2024-04-21T08:37:35.595773+0000","last_unstale":"2024-04-21T08:37:37.327512+0000","last_undegraded":"2024-04-21T08:37:37.327512+0000","last_fullsized":"2024-04-21T08:37:37.327512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:07.538190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793056+0000","last_change":"2024-04-21T08:37:30.416324+0000","last_active":"2024-04-21T08:37:37.793056+0000","last_peered":"2024-04-21T08:37:37.793056+0000","last_clean":"2024-04-21T08:37:37.793056+0000","last_became_active":"2024-04-21T08:37:30.416024+0000","last_became_peered":"2024-04-21T08:37:30.416024+0000","last_unstale":"2024-04-21T08:37:37.793056+0000","last_undegraded":"2024-04-21T08:37:37.793056+0000","last_fullsized":"2024-04-21T08:37:37.793056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:54.111986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.794218+0000","last_change":"2024-04-21T08:37:31.453680+0000","last_active":"2024-04-21T08:37:37.794218+0000","last_peered":"2024-04-21T08:37:37.794218+0000","last_clean":"2024-04-21T08:37:37.794218+0000","last_became_active":"2024-04-21T08:37:31.453319+0000","last_became_peered":"2024-04-21T08:37:31.453319+0000","last_unstale":"2024-04-21T08:37:37.794218+0000","last_undegraded":"2024-04-21T08:37:37.794218+0000","last_fullsized":"2024-04-21T08:37:37.794218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:43.139789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582873+0000","last_change":"2024-04-21T08:37:35.608906+0000","last_active":"2024-04-21T08:37:36.582873+0000","last_peered":"2024-04-21T08:37:36.582873+0000","last_clean":"2024-04-21T08:37:36.582873+0000","last_became_active":"2024-04-21T08:37:35.608535+0000","last_became_peered":"2024-04-21T08:37:35.608535+0000","last_unstale":"2024-04-21T08:37:36.582873+0000","last_undegraded":"2024-04-21T08:37:36.582873+0000","last_fullsized":"2024-04-21T08:37:36.582873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:58:01.422308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793158+0000","last_change":"2024-04-21T08:37:30.414865+0000","last_active":"2024-04-21T08:37:37.793158+0000","last_peered":"2024-04-21T08:37:37.793158+0000","last_clean":"2024-04-21T08:37:37.793158+0000","last_became_active":"2024-04-21T08:37:30.414471+0000","last_became_peered":"2024-04-21T08:37:30.414471+0000","last_unstale":"2024-04-21T08:37:37.793158+0000","last_undegraded":"2024-04-21T08:37:37.793158+0000","last_fullsized":"2024-04-21T08:37:37.793158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:46.943574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793278+0000","last_change":"2024-04-21T08:37:31.456634+0000","last_active":"2024-04-21T08:37:37.793278+0000","last_peered":"2024-04-21T08:37:37.793278+0000","last_clean":"2024-04-21T08:37:37.793278+0000","last_became_active":"2024-04-21T08:37:31.456278+0000","last_became_peered":"2024-04-21T08:37:31.456278+0000","last_unstale":"2024-04-21T08:37:37.793278+0000","last_undegraded":"2024-04-21T08:37:37.793278+0000","last_fullsized":"2024-04-21T08:37:37.793278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:04.870024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792829+0000","last_change":"2024-04-21T08:37:35.601760+0000","last_active":"2024-04-21T08:37:37.792829+0000","last_peered":"2024-04-21T08:37:37.792829+0000","last_clean":"2024-04-21T08:37:37.792829+0000","last_became_active":"2024-04-21T08:37:35.601332+0000","last_became_peered":"2024-04-21T08:37:35.601332+0000","last_unstale":"2024-04-21T08:37:37.792829+0000","last_undegraded":"2024-04-21T08:37:37.792829+0000","last_fullsized":"2024-04-21T08:37:37.792829+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:04:03.694581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463377+0000","last_change":"2024-04-21T08:37:30.422419+0000","last_active":"2024-04-21T08:37:37.463377+0000","last_peered":"2024-04-21T08:37:37.463377+0000","last_clean":"2024-04-21T08:37:37.463377+0000","last_became_active":"2024-04-21T08:37:30.422123+0000","last_became_peered":"2024-04-21T08:37:30.422123+0000","last_unstale":"2024-04-21T08:37:37.463377+0000","last_undegraded":"2024-04-21T08:37:37.463377+0000","last_fullsized":"2024-04-21T08:37:37.463377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:32:54.029041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792868+0000","last_change":"2024-04-21T08:37:31.458992+0000","last_active":"2024-04-21T08:37:37.792868+0000","last_peered":"2024-04-21T08:37:37.792868+0000","last_clean":"2024-04-21T08:37:37.792868+0000","last_became_active":"2024-04-21T08:37:31.458635+0000","last_became_peered":"2024-04-21T08:37:31.458635+0000","last_unstale":"2024-04-21T08:37:37.792868+0000","last_undegraded":"2024-04-21T08:37:37.792868+0000","last_fullsized":"2024-04-21T08:37:37.792868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:48:14.325571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792914+0000","last_change":"2024-04-21T08:37:35.603739+0000","last_active":"2024-04-21T08:37:37.792914+0000","last_peered":"2024-04-21T08:37:37.792914+0000","last_clean":"2024-04-21T08:37:37.792914+0000","last_became_active":"2024-04-21T08:37:35.603310+0000","last_became_peered":"2024-04-21T08:37:35.603310+0000","last_unstale":"2024-04-21T08:37:37.792914+0000","last_undegraded":"2024-04-21T08:37:37.792914+0000","last_fullsized":"2024-04-21T08:37:37.792914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:19.162520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463925+0000","last_change":"2024-04-21T08:37:30.418104+0000","last_active":"2024-04-21T08:37:37.463925+0000","last_peered":"2024-04-21T08:37:37.463925+0000","last_clean":"2024-04-21T08:37:37.463925+0000","last_became_active":"2024-04-21T08:37:30.417814+0000","last_became_peered":"2024-04-21T08:37:30.417814+0000","last_unstale":"2024-04-21T08:37:37.463925+0000","last_undegraded":"2024-04-21T08:37:37.463925+0000","last_fullsized":"2024-04-21T08:37:37.463925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:50:56.696513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327371+0000","last_change":"2024-04-21T08:37:31.451952+0000","last_active":"2024-04-21T08:37:37.327371+0000","last_peered":"2024-04-21T08:37:37.327371+0000","last_clean":"2024-04-21T08:37:37.327371+0000","last_became_active":"2024-04-21T08:37:31.451588+0000","last_became_peered":"2024-04-21T08:37:31.451588+0000","last_unstale":"2024-04-21T08:37:37.327371+0000","last_undegraded":"2024-04-21T08:37:37.327371+0000","last_fullsized":"2024-04-21T08:37:37.327371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:06.506769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463621+0000","last_change":"2024-04-21T08:37:35.760661+0000","last_active":"2024-04-21T08:37:37.463621+0000","last_peered":"2024-04-21T08:37:37.463621+0000","last_clean":"2024-04-21T08:37:37.463621+0000","last_became_active":"2024-04-21T08:37:35.759925+0000","last_became_peered":"2024-04-21T08:37:35.759925+0000","last_unstale":"2024-04-21T08:37:37.463621+0000","last_undegraded":"2024-04-21T08:37:37.463621+0000","last_fullsized":"2024-04-21T08:37:37.463621+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:11:53.062251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582616+0000","last_change":"2024-04-21T08:37:30.416487+0000","last_active":"2024-04-21T08:37:36.582616+0000","last_peered":"2024-04-21T08:37:36.582616+0000","last_clean":"2024-04-21T08:37:36.582616+0000","last_became_active":"2024-04-21T08:37:30.416076+0000","last_became_peered":"2024-04-21T08:37:30.416076+0000","last_unstale":"2024-04-21T08:37:36.582616+0000","last_undegraded":"2024-04-21T08:37:36.582616+0000","last_fullsized":"2024-04-21T08:37:36.582616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:18:33.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583419+0000","last_change":"2024-04-21T08:37:31.448824+0000","last_active":"2024-04-21T08:37:36.583419+0000","last_peered":"2024-04-21T08:37:36.583419+0000","last_clean":"2024-04-21T08:37:36.583419+0000","last_became_active":"2024-04-21T08:37:31.448432+0000","last_became_peered":"2024-04-21T08:37:31.448432+0000","last_unstale":"2024-04-21T08:37:36.583419+0000","last_undegraded":"2024-04-21T08:37:36.583419+0000","last_fullsized":"2024-04-21T08:37:36.583419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:41.111272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792582+0000","last_change":"2024-04-21T08:37:35.604539+0000","last_active":"2024-04-21T08:37:37.792582+0000","last_peered":"2024-04-21T08:37:37.792582+0000","last_clean":"2024-04-21T08:37:37.792582+0000","last_became_active":"2024-04-21T08:37:35.604132+0000","last_became_peered":"2024-04-21T08:37:35.604132+0000","last_unstale":"2024-04-21T08:37:37.792582+0000","last_undegraded":"2024-04-21T08:37:37.792582+0000","last_fullsized":"2024-04-21T08:37:37.792582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:12.180730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793615+0000","last_change":"2024-04-21T08:37:30.418854+0000","last_active":"2024-04-21T08:37:37.793615+0000","last_peered":"2024-04-21T08:37:37.793615+0000","last_clean":"2024-04-21T08:37:37.793615+0000","last_became_active":"2024-04-21T08:37:30.418511+0000","last_became_peered":"2024-04-21T08:37:30.418511+0000","last_unstale":"2024-04-21T08:37:37.793615+0000","last_undegraded":"2024-04-21T08:37:37.793615+0000","last_fullsized":"2024-04-21T08:37:37.793615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:26.751924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583178+0000","last_change":"2024-04-21T08:37:31.453365+0000","last_active":"2024-04-21T08:37:36.583178+0000","last_peered":"2024-04-21T08:37:36.583178+0000","last_clean":"2024-04-21T08:37:36.583178+0000","last_became_active":"2024-04-21T08:37:31.453042+0000","last_became_peered":"2024-04-21T08:37:31.453042+0000","last_unstale":"2024-04-21T08:37:36.583178+0000","last_undegraded":"2024-04-21T08:37:36.583178+0000","last_fullsized":"2024-04-21T08:37:36.583178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:49:46.648754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792773+0000","last_change":"2024-04-21T08:37:35.760626+0000","last_active":"2024-04-21T08:37:37.792773+0000","last_peered":"2024-04-21T08:37:37.792773+0000","last_clean":"2024-04-21T08:37:37.792773+0000","last_became_active":"2024-04-21T08:37:35.760125+0000","last_became_peered":"2024-04-21T08:37:35.760125+0000","last_unstale":"2024-04-21T08:37:37.792773+0000","last_undegraded":"2024-04-21T08:37:37.792773+0000","last_fullsized":"2024-04-21T08:37:37.792773+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:58.366557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793012+0000","last_change":"2024-04-21T08:37:31.457654+0000","last_active":"2024-04-21T08:37:37.793012+0000","last_peered":"2024-04-21T08:37:37.793012+0000","last_clean":"2024-04-21T08:37:37.793012+0000","last_became_active":"2024-04-21T08:37:31.456950+0000","last_became_peered":"2024-04-21T08:37:31.456950+0000","last_unstale":"2024-04-21T08:37:37.793012+0000","last_undegraded":"2024-04-21T08:37:37.793012+0000","last_fullsized":"2024-04-21T08:37:37.793012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:02.863000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792278+0000","last_change":"2024-04-21T08:37:30.422021+0000","last_active":"2024-04-21T08:37:37.792278+0000","last_peered":"2024-04-21T08:37:37.792278+0000","last_clean":"2024-04-21T08:37:37.792278+0000","last_became_active":"2024-04-21T08:37:30.421711+0000","last_became_peered":"2024-04-21T08:37:30.421711+0000","last_unstale":"2024-04-21T08:37:37.792278+0000","last_undegraded":"2024-04-21T08:37:37.792278+0000","last_fullsized":"2024-04-21T08:37:37.792278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:01.612205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793684+0000","last_change":"2024-04-21T08:37:35.598041+0000","last_active":"2024-04-21T08:37:37.793684+0000","last_peered":"2024-04-21T08:37:37.793684+0000","last_clean":"2024-04-21T08:37:37.793684+0000","last_became_active":"2024-04-21T08:37:35.597635+0000","last_became_peered":"2024-04-21T08:37:35.597635+0000","last_unstale":"2024-04-21T08:37:37.793684+0000","last_undegraded":"2024-04-21T08:37:37.793684+0000","last_fullsized":"2024-04-21T08:37:37.793684+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:35:27.452340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010021+0000","last_change":"2024-04-21T08:37:31.450423+0000","last_active":"2024-04-21T08:37:37.010021+0000","last_peered":"2024-04-21T08:37:37.010021+0000","last_clean":"2024-04-21T08:37:37.010021+0000","last_became_active":"2024-04-21T08:37:31.450112+0000","last_became_peered":"2024-04-21T08:37:31.450112+0000","last_unstale":"2024-04-21T08:37:37.010021+0000","last_undegraded":"2024-04-21T08:37:37.010021+0000","last_fullsized":"2024-04-21T08:37:37.010021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:10:47.006769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793736+0000","last_change":"2024-04-21T08:37:30.416519+0000","last_active":"2024-04-21T08:37:37.793736+0000","last_peered":"2024-04-21T08:37:37.793736+0000","last_clean":"2024-04-21T08:37:37.793736+0000","last_became_active":"2024-04-21T08:37:30.416131+0000","last_became_peered":"2024-04-21T08:37:30.416131+0000","last_unstale":"2024-04-21T08:37:37.793736+0000","last_undegraded":"2024-04-21T08:37:37.793736+0000","last_fullsized":"2024-04-21T08:37:37.793736+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:12.280847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793112+0000","last_change":"2024-04-21T08:36:20.257759+0000","last_active":"2024-04-21T08:37:37.793112+0000","last_peered":"2024-04-21T08:37:37.793112+0000","last_clean":"2024-04-21T08:37:37.793112+0000","last_became_active":"2024-04-21T08:36:20.257313+0000","last_became_peered":"2024-04-21T08:36:20.257313+0000","last_unstale":"2024-04-21T08:37:37.793112+0000","last_undegraded":"2024-04-21T08:37:37.793112+0000","last_fullsized":"2024-04-21T08:37:37.793112+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_clean_scrub_stamp":"2024-04-21T08:32:56.546295+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:42:44.393447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793548+0000","last_change":"2024-04-21T08:37:35.761562+0000","last_active":"2024-04-21T08:37:37.793548+0000","last_peered":"2024-04-21T08:37:37.793548+0000","last_clean":"2024-04-21T08:37:37.793548+0000","last_became_active":"2024-04-21T08:37:35.760703+0000","last_became_peered":"2024-04-21T08:37:35.760703+0000","last_unstale":"2024-04-21T08:37:37.793548+0000","last_undegraded":"2024-04-21T08:37:37.793548+0000","last_fullsized":"2024-04-21T08:37:37.793548+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:12:47.511117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792272+0000","last_change":"2024-04-21T08:37:31.453645+0000","last_active":"2024-04-21T08:37:37.792272+0000","last_peered":"2024-04-21T08:37:37.792272+0000","last_clean":"2024-04-21T08:37:37.792272+0000","last_became_active":"2024-04-21T08:37:31.453297+0000","last_became_peered":"2024-04-21T08:37:31.453297+0000","last_unstale":"2024-04-21T08:37:37.792272+0000","last_undegraded":"2024-04-21T08:37:37.792272+0000","last_fullsized":"2024-04-21T08:37:37.792272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:58.976327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793168+0000","last_change":"2024-04-21T08:37:30.420560+0000","last_active":"2024-04-21T08:37:37.793168+0000","last_peered":"2024-04-21T08:37:37.793168+0000","last_clean":"2024-04-21T08:37:37.793168+0000","last_became_active":"2024-04-21T08:37:30.420223+0000","last_became_peered":"2024-04-21T08:37:30.420223+0000","last_unstale":"2024-04-21T08:37:37.793168+0000","last_undegraded":"2024-04-21T08:37:37.793168+0000","last_fullsized":"2024-04-21T08:37:37.793168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:43:55.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327440+0000","last_change":"2024-04-21T08:37:35.598294+0000","last_active":"2024-04-21T08:37:37.327440+0000","last_peered":"2024-04-21T08:37:37.327440+0000","last_clean":"2024-04-21T08:37:37.327440+0000","last_became_active":"2024-04-21T08:37:35.597883+0000","last_became_peered":"2024-04-21T08:37:35.597883+0000","last_unstale":"2024-04-21T08:37:37.327440+0000","last_undegraded":"2024-04-21T08:37:37.327440+0000","last_fullsized":"2024-04-21T08:37:37.327440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:24:05.144763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463885+0000","last_change":"2024-04-21T08:37:30.423182+0000","last_active":"2024-04-21T08:37:37.463885+0000","last_peered":"2024-04-21T08:37:37.463885+0000","last_clean":"2024-04-21T08:37:37.463885+0000","last_became_active":"2024-04-21T08:37:30.422792+0000","last_became_peered":"2024-04-21T08:37:30.422792+0000","last_unstale":"2024-04-21T08:37:37.463885+0000","last_undegraded":"2024-04-21T08:37:37.463885+0000","last_fullsized":"2024-04-21T08:37:37.463885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:44:00.727269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792681+0000","last_change":"2024-04-21T08:37:31.444013+0000","last_active":"2024-04-21T08:37:37.792681+0000","last_peered":"2024-04-21T08:37:37.792681+0000","last_clean":"2024-04-21T08:37:37.792681+0000","last_became_active":"2024-04-21T08:37:31.443684+0000","last_became_peered":"2024-04-21T08:37:31.443684+0000","last_unstale":"2024-04-21T08:37:37.792681+0000","last_undegraded":"2024-04-21T08:37:37.792681+0000","last_fullsized":"2024-04-21T08:37:37.792681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:00:35.400818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.009982+0000","last_change":"2024-04-21T08:37:35.759817+0000","last_active":"2024-04-21T08:37:37.009982+0000","last_peered":"2024-04-21T08:37:37.009982+0000","last_clean":"2024-04-21T08:37:37.009982+0000","last_became_active":"2024-04-21T08:37:35.759396+0000","last_became_peered":"2024-04-21T08:37:35.759396+0000","last_unstale":"2024-04-21T08:37:37.009982+0000","last_undegraded":"2024-04-21T08:37:37.009982+0000","last_fullsized":"2024-04-21T08:37:37.009982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:20:36.170880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:56.668254+0000","last_change":"2024-04-21T08:37:30.414102+0000","last_active":"2024-04-21T08:37:56.668254+0000","last_peered":"2024-04-21T08:37:56.668254+0000","last_clean":"2024-04-21T08:37:56.668254+0000","last_became_active":"2024-04-21T08:37:30.413670+0000","last_became_peered":"2024-04-21T08:37:30.413670+0000","last_unstale":"2024-04-21T08:37:56.668254+0000","last_undegraded":"2024-04-21T08:37:56.668254+0000","last_fullsized":"2024-04-21T08:37:56.668254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:12:49.696516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010371+0000","last_change":"2024-04-21T08:37:31.457451+0000","last_active":"2024-04-21T08:37:37.010371+0000","last_peered":"2024-04-21T08:37:37.010371+0000","last_clean":"2024-04-21T08:37:37.010371+0000","last_became_active":"2024-04-21T08:37:31.457120+0000","last_became_peered":"2024-04-21T08:37:31.457120+0000","last_unstale":"2024-04-21T08:37:37.010371+0000","last_undegraded":"2024-04-21T08:37:37.010371+0000","last_fullsized":"2024-04-21T08:37:37.010371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:07:48.025843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583148+0000","last_change":"2024-04-21T08:37:35.600815+0000","last_active":"2024-04-21T08:37:36.583148+0000","last_peered":"2024-04-21T08:37:36.583148+0000","last_clean":"2024-04-21T08:37:36.583148+0000","last_became_active":"2024-04-21T08:37:35.600380+0000","last_became_peered":"2024-04-21T08:37:35.600380+0000","last_unstale":"2024-04-21T08:37:36.583148+0000","last_undegraded":"2024-04-21T08:37:36.583148+0000","last_fullsized":"2024-04-21T08:37:36.583148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:19:33.710431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793767+0000","last_change":"2024-04-21T08:37:30.425592+0000","last_active":"2024-04-21T08:37:37.793767+0000","last_peered":"2024-04-21T08:37:37.793767+0000","last_clean":"2024-04-21T08:37:37.793767+0000","last_became_active":"2024-04-21T08:37:30.425274+0000","last_became_peered":"2024-04-21T08:37:30.425274+0000","last_unstale":"2024-04-21T08:37:37.793767+0000","last_undegraded":"2024-04-21T08:37:37.793767+0000","last_fullsized":"2024-04-21T08:37:37.793767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:27:29.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644745+0000","last_change":"2024-04-21T08:37:31.455393+0000","last_active":"2024-04-21T08:37:36.644745+0000","last_peered":"2024-04-21T08:37:36.644745+0000","last_clean":"2024-04-21T08:37:36.644745+0000","last_became_active":"2024-04-21T08:37:31.455065+0000","last_became_peered":"2024-04-21T08:37:31.455065+0000","last_unstale":"2024-04-21T08:37:36.644745+0000","last_undegraded":"2024-04-21T08:37:36.644745+0000","last_fullsized":"2024-04-21T08:37:36.644745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:05:23.386790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.794208+0000","last_change":"2024-04-21T08:37:35.760767+0000","last_active":"2024-04-21T08:37:37.794208+0000","last_peered":"2024-04-21T08:37:37.794208+0000","last_clean":"2024-04-21T08:37:37.794208+0000","last_became_active":"2024-04-21T08:37:35.760187+0000","last_became_peered":"2024-04-21T08:37:35.760187+0000","last_unstale":"2024-04-21T08:37:37.794208+0000","last_undegraded":"2024-04-21T08:37:37.794208+0000","last_fullsized":"2024-04-21T08:37:37.794208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:52:11.823765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:55.659724+0000","last_change":"2024-04-21T08:37:30.416696+0000","last_active":"2024-04-21T08:37:55.659724+0000","last_peered":"2024-04-21T08:37:55.659724+0000","last_clean":"2024-04-21T08:37:55.659724+0000","last_became_active":"2024-04-21T08:37:30.415868+0000","last_became_peered":"2024-04-21T08:37:30.415868+0000","last_unstale":"2024-04-21T08:37:55.659724+0000","last_undegraded":"2024-04-21T08:37:55.659724+0000","last_fullsized":"2024-04-21T08:37:55.659724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:48:08.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463414+0000","last_change":"2024-04-21T08:37:31.453861+0000","last_active":"2024-04-21T08:37:37.463414+0000","last_peered":"2024-04-21T08:37:37.463414+0000","last_clean":"2024-04-21T08:37:37.463414+0000","last_became_active":"2024-04-21T08:37:31.453562+0000","last_became_peered":"2024-04-21T08:37:31.453562+0000","last_unstale":"2024-04-21T08:37:37.463414+0000","last_undegraded":"2024-04-21T08:37:37.463414+0000","last_fullsized":"2024-04-21T08:37:37.463414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:17:50.513917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583018+0000","last_change":"2024-04-21T08:37:35.609217+0000","last_active":"2024-04-21T08:37:36.583018+0000","last_peered":"2024-04-21T08:37:36.583018+0000","last_clean":"2024-04-21T08:37:36.583018+0000","last_became_active":"2024-04-21T08:37:35.608786+0000","last_became_peered":"2024-04-21T08:37:35.608786+0000","last_unstale":"2024-04-21T08:37:36.583018+0000","last_undegraded":"2024-04-21T08:37:36.583018+0000","last_fullsized":"2024-04-21T08:37:36.583018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:45:50.945845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.009929+0000","last_change":"2024-04-21T08:37:30.417156+0000","last_active":"2024-04-21T08:37:37.009929+0000","last_peered":"2024-04-21T08:37:37.009929+0000","last_clean":"2024-04-21T08:37:37.009929+0000","last_became_active":"2024-04-21T08:37:30.415717+0000","last_became_peered":"2024-04-21T08:37:30.415717+0000","last_unstale":"2024-04-21T08:37:37.009929+0000","last_undegraded":"2024-04-21T08:37:37.009929+0000","last_fullsized":"2024-04-21T08:37:37.009929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:37.819082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327169+0000","last_change":"2024-04-21T08:37:31.444948+0000","last_active":"2024-04-21T08:37:37.327169+0000","last_peered":"2024-04-21T08:37:37.327169+0000","last_clean":"2024-04-21T08:37:37.327169+0000","last_became_active":"2024-04-21T08:37:31.444604+0000","last_became_peered":"2024-04-21T08:37:31.444604+0000","last_unstale":"2024-04-21T08:37:37.327169+0000","last_undegraded":"2024-04-21T08:37:37.327169+0000","last_fullsized":"2024-04-21T08:37:37.327169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.820503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793995+0000","last_change":"2024-04-21T08:37:35.611541+0000","last_active":"2024-04-21T08:37:37.793995+0000","last_peered":"2024-04-21T08:37:37.793995+0000","last_clean":"2024-04-21T08:37:37.793995+0000","last_became_active":"2024-04-21T08:37:35.611136+0000","last_became_peered":"2024-04-21T08:37:35.611136+0000","last_unstale":"2024-04-21T08:37:37.793995+0000","last_undegraded":"2024-04-21T08:37:37.793995+0000","last_fullsized":"2024-04-21T08:37:37.793995+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:28.254529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583035+0000","last_change":"2024-04-21T08:37:30.414216+0000","last_active":"2024-04-21T08:37:36.583035+0000","last_peered":"2024-04-21T08:37:36.583035+0000","last_clean":"2024-04-21T08:37:36.583035+0000","last_became_active":"2024-04-21T08:37:30.413890+0000","last_became_peered":"2024-04-21T08:37:30.413890+0000","last_unstale":"2024-04-21T08:37:36.583035+0000","last_undegraded":"2024-04-21T08:37:36.583035+0000","last_fullsized":"2024-04-21T08:37:36.583035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:16.972618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010035+0000","last_change":"2024-04-21T08:37:31.449307+0000","last_active":"2024-04-21T08:37:37.010035+0000","last_peered":"2024-04-21T08:37:37.010035+0000","last_clean":"2024-04-21T08:37:37.010035+0000","last_became_active":"2024-04-21T08:37:31.448980+0000","last_became_peered":"2024-04-21T08:37:31.448980+0000","last_unstale":"2024-04-21T08:37:37.010035+0000","last_undegraded":"2024-04-21T08:37:37.010035+0000","last_fullsized":"2024-04-21T08:37:37.010035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:30:31.939274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793935+0000","last_change":"2024-04-21T08:37:35.760616+0000","last_active":"2024-04-21T08:37:37.793935+0000","last_peered":"2024-04-21T08:37:37.793935+0000","last_clean":"2024-04-21T08:37:37.793935+0000","last_became_active":"2024-04-21T08:37:35.760131+0000","last_became_peered":"2024-04-21T08:37:35.760131+0000","last_unstale":"2024-04-21T08:37:37.793935+0000","last_undegraded":"2024-04-21T08:37:37.793935+0000","last_fullsized":"2024-04-21T08:37:37.793935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:51.510259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.463508+0000","last_change":"2024-04-21T08:37:30.421935+0000","last_active":"2024-04-21T08:37:37.463508+0000","last_peered":"2024-04-21T08:37:37.463508+0000","last_clean":"2024-04-21T08:37:37.463508+0000","last_became_active":"2024-04-21T08:37:30.421600+0000","last_became_peered":"2024-04-21T08:37:30.421600+0000","last_unstale":"2024-04-21T08:37:37.463508+0000","last_undegraded":"2024-04-21T08:37:37.463508+0000","last_fullsized":"2024-04-21T08:37:37.463508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:28:03.160185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793806+0000","last_change":"2024-04-21T08:37:31.447234+0000","last_active":"2024-04-21T08:37:37.793806+0000","last_peered":"2024-04-21T08:37:37.793806+0000","last_clean":"2024-04-21T08:37:37.793806+0000","last_became_active":"2024-04-21T08:37:31.446876+0000","last_became_peered":"2024-04-21T08:37:31.446876+0000","last_unstale":"2024-04-21T08:37:37.793806+0000","last_undegraded":"2024-04-21T08:37:37.793806+0000","last_fullsized":"2024-04-21T08:37:37.793806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:51.784126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.469999+0000","last_change":"2024-04-21T08:37:35.598577+0000","last_active":"2024-04-21T08:37:37.469999+0000","last_peered":"2024-04-21T08:37:37.469999+0000","last_clean":"2024-04-21T08:37:37.469999+0000","last_became_active":"2024-04-21T08:37:35.598119+0000","last_became_peered":"2024-04-21T08:37:35.598119+0000","last_unstale":"2024-04-21T08:37:37.469999+0000","last_undegraded":"2024-04-21T08:37:37.469999+0000","last_fullsized":"2024-04-21T08:37:37.469999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:41.986674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583598+0000","last_change":"2024-04-21T08:37:30.425737+0000","last_active":"2024-04-21T08:37:36.583598+0000","last_peered":"2024-04-21T08:37:36.583598+0000","last_clean":"2024-04-21T08:37:36.583598+0000","last_became_active":"2024-04-21T08:37:30.425180+0000","last_became_peered":"2024-04-21T08:37:30.425180+0000","last_unstale":"2024-04-21T08:37:36.583598+0000","last_undegraded":"2024-04-21T08:37:36.583598+0000","last_fullsized":"2024-04-21T08:37:36.583598+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:08.646106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793988+0000","last_change":"2024-04-21T08:37:31.451505+0000","last_active":"2024-04-21T08:37:37.793988+0000","last_peered":"2024-04-21T08:37:37.793988+0000","last_clean":"2024-04-21T08:37:37.793988+0000","last_became_active":"2024-04-21T08:37:31.451146+0000","last_became_peered":"2024-04-21T08:37:31.451146+0000","last_unstale":"2024-04-21T08:37:37.793988+0000","last_undegraded":"2024-04-21T08:37:37.793988+0000","last_fullsized":"2024-04-21T08:37:37.793988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:27:01.793910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644629+0000","last_change":"2024-04-21T08:37:35.608995+0000","last_active":"2024-04-21T08:37:36.644629+0000","last_peered":"2024-04-21T08:37:36.644629+0000","last_clean":"2024-04-21T08:37:36.644629+0000","last_became_active":"2024-04-21T08:37:35.608587+0000","last_became_peered":"2024-04-21T08:37:35.608587+0000","last_unstale":"2024-04-21T08:37:36.644629+0000","last_undegraded":"2024-04-21T08:37:36.644629+0000","last_fullsized":"2024-04-21T08:37:36.644629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.829418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793277+0000","last_change":"2024-04-21T08:37:30.419026+0000","last_active":"2024-04-21T08:37:37.793277+0000","last_peered":"2024-04-21T08:37:37.793277+0000","last_clean":"2024-04-21T08:37:37.793277+0000","last_became_active":"2024-04-21T08:37:30.418706+0000","last_became_peered":"2024-04-21T08:37:30.418706+0000","last_unstale":"2024-04-21T08:37:37.793277+0000","last_undegraded":"2024-04-21T08:37:37.793277+0000","last_fullsized":"2024-04-21T08:37:37.793277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:24.441918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.793705+0000","last_change":"2024-04-21T08:37:31.446561+0000","last_active":"2024-04-21T08:37:37.793705+0000","last_peered":"2024-04-21T08:37:37.793705+0000","last_clean":"2024-04-21T08:37:37.793705+0000","last_became_active":"2024-04-21T08:37:31.446162+0000","last_became_peered":"2024-04-21T08:37:31.446162+0000","last_unstale":"2024-04-21T08:37:37.793705+0000","last_undegraded":"2024-04-21T08:37:37.793705+0000","last_fullsized":"2024-04-21T08:37:37.793705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:38:24.684537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792653+0000","last_change":"2024-04-21T08:37:35.598748+0000","last_active":"2024-04-21T08:37:37.792653+0000","last_peered":"2024-04-21T08:37:37.792653+0000","last_clean":"2024-04-21T08:37:37.792653+0000","last_became_active":"2024-04-21T08:37:35.598353+0000","last_became_peered":"2024-04-21T08:37:35.598353+0000","last_unstale":"2024-04-21T08:37:37.792653+0000","last_undegraded":"2024-04-21T08:37:37.792653+0000","last_fullsized":"2024-04-21T08:37:37.792653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:47:36.519694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792754+0000","last_change":"2024-04-21T08:37:30.422298+0000","last_active":"2024-04-21T08:37:37.792754+0000","last_peered":"2024-04-21T08:37:37.792754+0000","last_clean":"2024-04-21T08:37:37.792754+0000","last_became_active":"2024-04-21T08:37:30.421297+0000","last_became_peered":"2024-04-21T08:37:30.421297+0000","last_unstale":"2024-04-21T08:37:37.792754+0000","last_undegraded":"2024-04-21T08:37:37.792754+0000","last_fullsized":"2024-04-21T08:37:37.792754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:16.249586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.794146+0000","last_change":"2024-04-21T08:37:31.452367+0000","last_active":"2024-04-21T08:37:37.794146+0000","last_peered":"2024-04-21T08:37:37.794146+0000","last_clean":"2024-04-21T08:37:37.794146+0000","last_became_active":"2024-04-21T08:37:31.452031+0000","last_became_peered":"2024-04-21T08:37:31.452031+0000","last_unstale":"2024-04-21T08:37:37.794146+0000","last_undegraded":"2024-04-21T08:37:37.794146+0000","last_fullsized":"2024-04-21T08:37:37.794146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:58.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327228+0000","last_change":"2024-04-21T08:37:31.451222+0000","last_active":"2024-04-21T08:37:37.327228+0000","last_peered":"2024-04-21T08:37:37.327228+0000","last_clean":"2024-04-21T08:37:37.327228+0000","last_became_active":"2024-04-21T08:37:31.450869+0000","last_became_peered":"2024-04-21T08:37:31.450869+0000","last_unstale":"2024-04-21T08:37:37.327228+0000","last_undegraded":"2024-04-21T08:37:37.327228+0000","last_fullsized":"2024-04-21T08:37:37.327228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:17.051330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583538+0000","last_change":"2024-04-21T08:37:31.453415+0000","last_active":"2024-04-21T08:37:36.583538+0000","last_peered":"2024-04-21T08:37:36.583538+0000","last_clean":"2024-04-21T08:37:36.583538+0000","last_became_active":"2024-04-21T08:37:31.453089+0000","last_became_peered":"2024-04-21T08:37:31.453089+0000","last_unstale":"2024-04-21T08:37:36.583538+0000","last_undegraded":"2024-04-21T08:37:36.583538+0000","last_fullsized":"2024-04-21T08:37:36.583538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:25:44.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.582986+0000","last_change":"2024-04-21T08:37:31.446707+0000","last_active":"2024-04-21T08:37:36.582986+0000","last_peered":"2024-04-21T08:37:36.582986+0000","last_clean":"2024-04-21T08:37:36.582986+0000","last_became_active":"2024-04-21T08:37:31.446340+0000","last_became_peered":"2024-04-21T08:37:31.446340+0000","last_unstale":"2024-04-21T08:37:36.582986+0000","last_undegraded":"2024-04-21T08:37:36.582986+0000","last_fullsized":"2024-04-21T08:37:36.582986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:13.649607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.644552+0000","last_change":"2024-04-21T08:37:31.455532+0000","last_active":"2024-04-21T08:37:36.644552+0000","last_peered":"2024-04-21T08:37:36.644552+0000","last_clean":"2024-04-21T08:37:36.644552+0000","last_became_active":"2024-04-21T08:37:31.453547+0000","last_became_peered":"2024-04-21T08:37:31.453547+0000","last_unstale":"2024-04-21T08:37:36.644552+0000","last_undegraded":"2024-04-21T08:37:36.644552+0000","last_fullsized":"2024-04-21T08:37:36.644552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:39:47.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792742+0000","last_change":"2024-04-21T08:37:31.452535+0000","last_active":"2024-04-21T08:37:37.792742+0000","last_peered":"2024-04-21T08:37:37.792742+0000","last_clean":"2024-04-21T08:37:37.792742+0000","last_became_active":"2024-04-21T08:37:31.451637+0000","last_became_peered":"2024-04-21T08:37:31.451637+0000","last_unstale":"2024-04-21T08:37:37.792742+0000","last_undegraded":"2024-04-21T08:37:37.792742+0000","last_fullsized":"2024-04-21T08:37:37.792742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:38:38.162780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327676+0000","last_change":"2024-04-21T08:37:31.451806+0000","last_active":"2024-04-21T08:37:37.327676+0000","last_peered":"2024-04-21T08:37:37.327676+0000","last_clean":"2024-04-21T08:37:37.327676+0000","last_became_active":"2024-04-21T08:37:31.451442+0000","last_became_peered":"2024-04-21T08:37:31.451442+0000","last_unstale":"2024-04-21T08:37:37.327676+0000","last_undegraded":"2024-04-21T08:37:37.327676+0000","last_fullsized":"2024-04-21T08:37:37.327676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:07.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:36.583796+0000","last_change":"2024-04-21T08:37:31.444440+0000","last_active":"2024-04-21T08:37:36.583796+0000","last_peered":"2024-04-21T08:37:36.583796+0000","last_clean":"2024-04-21T08:37:36.583796+0000","last_became_active":"2024-04-21T08:37:31.444079+0000","last_became_peered":"2024-04-21T08:37:31.444079+0000","last_unstale":"2024-04-21T08:37:36.583796+0000","last_undegraded":"2024-04-21T08:37:36.583796+0000","last_fullsized":"2024-04-21T08:37:36.583796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:13.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010479+0000","last_change":"2024-04-21T08:37:31.453698+0000","last_active":"2024-04-21T08:37:37.010479+0000","last_peered":"2024-04-21T08:37:37.010479+0000","last_clean":"2024-04-21T08:37:37.010479+0000","last_became_active":"2024-04-21T08:37:31.453379+0000","last_became_peered":"2024-04-21T08:37:31.453379+0000","last_unstale":"2024-04-21T08:37:37.010479+0000","last_undegraded":"2024-04-21T08:37:37.010479+0000","last_fullsized":"2024-04-21T08:37:37.010479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:46:29.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792851+0000","last_change":"2024-04-21T08:37:31.444480+0000","last_active":"2024-04-21T08:37:37.792851+0000","last_peered":"2024-04-21T08:37:37.792851+0000","last_clean":"2024-04-21T08:37:37.792851+0000","last_became_active":"2024-04-21T08:37:31.444030+0000","last_became_peered":"2024-04-21T08:37:31.444030+0000","last_unstale":"2024-04-21T08:37:37.792851+0000","last_undegraded":"2024-04-21T08:37:37.792851+0000","last_fullsized":"2024-04-21T08:37:37.792851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:15:05.668885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.010031+0000","last_change":"2024-04-21T08:37:31.456602+0000","last_active":"2024-04-21T08:37:37.010031+0000","last_peered":"2024-04-21T08:37:37.010031+0000","last_clean":"2024-04-21T08:37:37.010031+0000","last_became_active":"2024-04-21T08:37:31.456286+0000","last_became_peered":"2024-04-21T08:37:31.456286+0000","last_unstale":"2024-04-21T08:37:37.010031+0000","last_undegraded":"2024-04-21T08:37:37.010031+0000","last_fullsized":"2024-04-21T08:37:37.010031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:48:12.462037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.470257+0000","last_change":"2024-04-21T08:37:31.447008+0000","last_active":"2024-04-21T08:37:37.470257+0000","last_peered":"2024-04-21T08:37:37.470257+0000","last_clean":"2024-04-21T08:37:37.470257+0000","last_became_active":"2024-04-21T08:37:31.445192+0000","last_became_peered":"2024-04-21T08:37:31.445192+0000","last_unstale":"2024-04-21T08:37:37.470257+0000","last_undegraded":"2024-04-21T08:37:37.470257+0000","last_fullsized":"2024-04-21T08:37:37.470257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:08.025647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.792578+0000","last_change":"2024-04-21T08:37:31.443288+0000","last_active":"2024-04-21T08:37:37.792578+0000","last_peered":"2024-04-21T08:37:37.792578+0000","last_clean":"2024-04-21T08:37:37.792578+0000","last_became_active":"2024-04-21T08:37:31.442586+0000","last_became_peered":"2024-04-21T08:37:31.442586+0000","last_unstale":"2024-04-21T08:37:37.792578+0000","last_undegraded":"2024-04-21T08:37:37.792578+0000","last_fullsized":"2024-04-21T08:37:37.792578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:01.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:37:37.327502+0000","last_change":"2024-04-21T08:37:31.453899+0000","last_active":"2024-04-21T08:37:37.327502+0000","last_peered":"2024-04-21T08:37:37.327502+0000","last_clean":"2024-04-21T08:37:37.327502+0000","last_became_active":"2024-04-21T08:37:31.453527+0000","last_became_peered":"2024-04-21T08:37:31.453527+0000","last_unstale":"2024-04-21T08:37:37.327502+0000","last_undegraded":"2024-04-21T08:37:37.327502+0000","last_fullsized":"2024-04-21T08:37:37.327502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:23:18.429618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283745,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4730,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":74,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":18132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1073003,"data_compressed":50780,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":477999,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070497,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":477819,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":50780,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":478693,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":481713,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920820,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14698,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50780,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:38:00.764 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-21T08:38:00.764 INFO:tasks.cephadm:Setting up client nodes... 2024-04-21T08:38:00.765 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-21T08:38:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:00 smithi111 ceph-mon[31493]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:00 smithi111 ceph-mon[31493]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:00 smithi111 ceph-mon[31493]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:00 smithi111 ceph-mon[31493]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 511 B/s wr, 2 op/s 2024-04-21T08:38:00.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:00 smithi111 ceph-mon[31493]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:38:00.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1566458981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:38:00.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2710731849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:38:01.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:00 smithi094 ceph-mon[27988]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:01.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:00 smithi094 ceph-mon[27988]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:01.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:00 smithi094 ceph-mon[27988]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:01.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:00 smithi094 ceph-mon[27988]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 511 B/s wr, 2 op/s 2024-04-21T08:38:01.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:00 smithi094 ceph-mon[27988]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:38:01.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1566458981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:38:01.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2710731849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:38:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:00 smithi173 ceph-mon[31428]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:00 smithi173 ceph-mon[31428]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:00 smithi173 ceph-mon[31428]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:00 smithi173 ceph-mon[31428]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 511 B/s wr, 2 op/s 2024-04-21T08:38:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:00 smithi173 ceph-mon[31428]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:38:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1566458981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:38:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2710731849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:38:01.037 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:01 smithi111 ceph-mon[31493]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:38:01.937 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:38:01 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b[61988]: 2024-04-21T08:38:01.683+0000 7f081fb22700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-21T08:38:02.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:01 smithi094 ceph-mon[27988]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:38:02.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:01 smithi173 ceph-mon[31428]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:38:02.756 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:02 smithi094 ceph-mon[27988]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:38:02.757 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:02 smithi094 ceph-mon[27988]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-21T08:38:02.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.873+0000 7f4227ab7700 1 -- 172.21.15.94:0/4163416014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42201047c0 msgr2=0x7f422010b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.873+0000 7f4227ab7700 1 --2- 172.21.15.94:0/4163416014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42201047c0 0x7f422010b070 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f421800b0d0 tx=0x7f421800b3e0 comp rx=0 tx=0).stop 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.873+0000 7f4227ab7700 1 -- 172.21.15.94:0/4163416014 shutdown_connections 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.873+0000 7f4227ab7700 1 --2- 172.21.15.94:0/4163416014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4220100da0 0x7f42201011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.873+0000 7f4227ab7700 1 --2- 172.21.15.94:0/4163416014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42201047c0 0x7f422010b070 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.873+0000 7f4227ab7700 1 --2- 172.21.15.94:0/4163416014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4220103e20 0x7f4220104280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.873+0000 7f4227ab7700 1 -- 172.21.15.94:0/4163416014 >> 172.21.15.94:0/4163416014 conn(0x7f42200fc6e0 msgr2=0x7f42200feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.873+0000 7f4227ab7700 1 -- 172.21.15.94:0/4163416014 shutdown_connections 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.873+0000 7f4227ab7700 1 -- 172.21.15.94:0/4163416014 wait complete. 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4227ab7700 1 Processor -- start 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4227ab7700 1 -- start start 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4227ab7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4220100da0 0x7f4220103780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4227ab7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4220103e20 0x7f4220101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4227ab7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42201047c0 0x7f4220102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:02.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4227ab7700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f422010e560 con 0x7f4220100da0 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4227ab7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f422010e3e0 con 0x7f42201047c0 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4227ab7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f422010e6e0 con 0x7f4220103e20 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4225853700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4220100da0 0x7f4220103780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4225853700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4220100da0 0x7f4220103780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51804/0 (socket says 172.21.15.94:51804) 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4225853700 1 -- 172.21.15.94:0/3437575462 learned_addr learned my addr 172.21.15.94:0/3437575462 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4226054700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42201047c0 0x7f4220102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4225052700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4220103e20 0x7f4220101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4225853700 1 -- 172.21.15.94:0/3437575462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4220103e20 msgr2=0x7f4220101dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4225853700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4220103e20 0x7f4220101dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4225853700 1 -- 172.21.15.94:0/3437575462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42201047c0 msgr2=0x7f4220102310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4225853700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42201047c0 0x7f4220102310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4225853700 1 -- 172.21.15.94:0/3437575462 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4220102bc0 con 0x7f4220100da0 2024-04-21T08:38:02.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.874+0000 7f4225853700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4220100da0 0x7f4220103780 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f421c00d980 tx=0x7f421c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.875+0000 7f4216ffd700 1 -- 172.21.15.94:0/3437575462 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c017030 con 0x7f4220100da0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.875+0000 7f4227ab7700 1 -- 172.21.15.94:0/3437575462 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42201a2c10 con 0x7f4220100da0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.875+0000 7f4216ffd700 1 -- 172.21.15.94:0/3437575462 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f421c0044d0 con 0x7f4220100da0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.875+0000 7f4216ffd700 1 -- 172.21.15.94:0/3437575462 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c003970 con 0x7f4220100da0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.875+0000 7f4227ab7700 1 -- 172.21.15.94:0/3437575462 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42201a3150 con 0x7f4220100da0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.876+0000 7f4227ab7700 1 -- 172.21.15.94:0/3437575462 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4220050f80 con 0x7f4220100da0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.877+0000 7f4216ffd700 1 -- 172.21.15.94:0/3437575462 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f421c01a070 con 0x7f4220100da0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.877+0000 7f4216ffd700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f420c0a3d80 0x7f420c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.877+0000 7f4225052700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f420c0a3d80 0x7f420c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.878+0000 7f4216ffd700 1 -- 172.21.15.94:0/3437575462 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f421c0d03f0 con 0x7f4220100da0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.878+0000 7f4225052700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f420c0a3d80 0x7f420c0a6240 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4210005950 tx=0x7f421000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:02.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:02.881+0000 7f4216ffd700 1 -- 172.21.15.94:0/3437575462 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f421c09ca30 con 0x7f4220100da0 2024-04-21T08:38:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:02 smithi111 ceph-mon[31493]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:38:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:02 smithi111 ceph-mon[31493]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-21T08:38:03.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:02 smithi173 ceph-mon[31428]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:38:03.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:02 smithi173 ceph-mon[31428]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-21T08:38:03.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.160+0000 7f4227ab7700 1 -- 172.21.15.94:0/3437575462 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f4220063650 con 0x7f4220100da0 2024-04-21T08:38:03.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.162+0000 7f4216ffd700 1 -- 172.21.15.94:0/3437575462 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f421c09c330 con 0x7f4220100da0 2024-04-21T08:38:03.161 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-04-21T08:38:03.161 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQBr0CRmeKSaCRAA3ZrFIqtq/TiM2IteZKlBuw== 2024-04-21T08:38:03.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.165+0000 7f4227ab7700 1 -- 172.21.15.94:0/3437575462 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f420c0a3d80 msgr2=0x7f420c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:03.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f420c0a3d80 0x7f420c0a6240 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4210005950 tx=0x7f421000a300 comp rx=0 tx=0).stop 2024-04-21T08:38:03.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 -- 172.21.15.94:0/3437575462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4220100da0 msgr2=0x7f4220103780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:03.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4220100da0 0x7f4220103780 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f421c00d980 tx=0x7f421c00dc90 comp rx=0 tx=0).stop 2024-04-21T08:38:03.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 -- 172.21.15.94:0/3437575462 shutdown_connections 2024-04-21T08:38:03.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f420c0a3d80 0x7f420c0a6240 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:03.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42201047c0 0x7f4220102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:03.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4220100da0 0x7f4220103780 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:03.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 --2- 172.21.15.94:0/3437575462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4220103e20 0x7f4220101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:03.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 -- 172.21.15.94:0/3437575462 >> 172.21.15.94:0/3437575462 conn(0x7f42200fc6e0 msgr2=0x7f4220107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:03.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 -- 172.21.15.94:0/3437575462 shutdown_connections 2024-04-21T08:38:03.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:03.166+0000 7f4227ab7700 1 -- 172.21.15.94:0/3437575462 wait complete. 2024-04-21T08:38:03.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:03 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3437575462' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:38:03.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:03 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3437575462' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:38:03.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:03 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3437575462' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:38:03.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:03 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3437575462' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:38:03.984 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:38:03.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-21T08:38:03.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-21T08:38:04.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:03 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3437575462' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:38:04.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:03 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3437575462' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:38:04.020 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-21T08:38:04.020 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-21T08:38:04.020 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-21T08:38:04.308 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:04 smithi111 ceph-mon[31493]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-04-21T08:38:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:04 smithi094 ceph-mon[27988]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-04-21T08:38:05.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:04 smithi173 ceph-mon[31428]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-04-21T08:38:06.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.339+0000 7f18a7156700 1 -- 172.21.15.94:0/1137017926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18a0107920 msgr2=0x7f18a0107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:06.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.339+0000 7f18a7156700 1 --2- 172.21.15.94:0/1137017926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18a0107920 0x7f18a0107d00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1898009990 tx=0x7f1898009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:06.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.339+0000 7f18a7156700 1 -- 172.21.15.94:0/1137017926 shutdown_connections 2024-04-21T08:38:06.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.339+0000 7f18a7156700 1 --2- 172.21.15.94:0/1137017926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18a0102330 0x7f18a010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:06.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.339+0000 7f18a7156700 1 --2- 172.21.15.94:0/1137017926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18a0107920 0x7f18a0107d00 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.339+0000 7f18a7156700 1 --2- 172.21.15.94:0/1137017926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18a0101970 0x7f18a0101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.339+0000 7f18a7156700 1 -- 172.21.15.94:0/1137017926 >> 172.21.15.94:0/1137017926 conn(0x7f18a00788f0 msgr2=0x7f18a0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.339+0000 7f18a7156700 1 -- 172.21.15.94:0/1137017926 shutdown_connections 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.339+0000 7f18a7156700 1 -- 172.21.15.94:0/1137017926 wait complete. 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.340+0000 7f18a7156700 1 Processor -- start 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.340+0000 7f18a7156700 1 -- start start 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.340+0000 7f18a7156700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18a0101970 0x7f18a0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.340+0000 7f18a7156700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18a0102330 0x7f18a01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.340+0000 7f18a7156700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18a0107920 0x7f18a019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.340+0000 7f18a7156700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f18a0111b30 con 0x7f18a0107920 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.340+0000 7f18a7156700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f18a01119b0 con 0x7f18a0101970 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.340+0000 7f18a7156700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f18a0111cb0 con 0x7f18a0102330 2024-04-21T08:38:06.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.340+0000 7f18a56f3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18a0107920 0x7f18a019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:06.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a4ef2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18a0101970 0x7f18a0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:06.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a56f3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18a0107920 0x7f18a019c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33680/0 (socket says 172.21.15.94:33680) 2024-04-21T08:38:06.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a4ef2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18a0101970 0x7f18a0198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:39606/0 (socket says 172.21.15.94:39606) 2024-04-21T08:38:06.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a56f3700 1 -- 172.21.15.94:0/1585452688 learned_addr learned my addr 172.21.15.94:0/1585452688 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:06.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f189ffff700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18a0102330 0x7f18a01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:06.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a56f3700 1 -- 172.21.15.94:0/1585452688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18a0102330 msgr2=0x7f18a01985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:06.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a56f3700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18a0102330 0x7f18a01985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:06.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a56f3700 1 -- 172.21.15.94:0/1585452688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18a0101970 msgr2=0x7f18a0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:06.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a56f3700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18a0101970 0x7f18a0198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:06.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a56f3700 1 -- 172.21.15.94:0/1585452688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a019d040 con 0x7f18a0107920 2024-04-21T08:38:06.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a56f3700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18a0107920 0x7f18a019c900 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1888007920 tx=0x7f1888007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:06.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f189dffb700 1 -- 172.21.15.94:0/1585452688 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1888019020 con 0x7f18a0107920 2024-04-21T08:38:06.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f18a7156700 1 -- 172.21.15.94:0/1585452688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a019d330 con 0x7f18a0107920 2024-04-21T08:38:06.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f189dffb700 1 -- 172.21.15.94:0/1585452688 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1888004ce0 con 0x7f18a0107920 2024-04-21T08:38:06.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.341+0000 7f189dffb700 1 -- 172.21.15.94:0/1585452688 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188800a440 con 0x7f18a0107920 2024-04-21T08:38:06.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.342+0000 7f18a7156700 1 -- 172.21.15.94:0/1585452688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a006b170 con 0x7f18a0107920 2024-04-21T08:38:06.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.342+0000 7f18a7156700 1 -- 172.21.15.94:0/1585452688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a0069730 con 0x7f18a0107920 2024-04-21T08:38:06.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.344+0000 7f189dffb700 1 -- 172.21.15.94:0/1585452688 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1888004750 con 0x7f18a0107920 2024-04-21T08:38:06.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.344+0000 7f189dffb700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f188c0a3cd0 0x7f188c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:06.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.344+0000 7f18a4ef2700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f188c0a3cd0 0x7f188c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:06.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.345+0000 7f189dffb700 1 -- 172.21.15.94:0/1585452688 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f18880d1af0 con 0x7f18a0107920 2024-04-21T08:38:06.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.345+0000 7f18a4ef2700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f188c0a3cd0 0x7f188c0a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f189800b5c0 tx=0x7f1898005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:06.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.349+0000 7f189dffb700 1 -- 172.21.15.94:0/1585452688 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1888023020 con 0x7f18a0107920 2024-04-21T08:38:06.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.604+0000 7f18a7156700 1 -- 172.21.15.94:0/1585452688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f18a006bba0 con 0x7f18a0107920 2024-04-21T08:38:06.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.609+0000 7f189dffb700 1 -- 172.21.15.94:0/1585452688 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264995 (secure 0 0 0) 0x7f188801b2b0 con 0x7f18a0107920 2024-04-21T08:38:06.612 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:38:06.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 -- 172.21.15.94:0/1585452688 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f188c0a3cd0 msgr2=0x7f188c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:06.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f188c0a3cd0 0x7f188c0a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f189800b5c0 tx=0x7f1898005960 comp rx=0 tx=0).stop 2024-04-21T08:38:06.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 -- 172.21.15.94:0/1585452688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18a0107920 msgr2=0x7f18a019c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:06.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18a0107920 0x7f18a019c900 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1888007920 tx=0x7f1888007c30 comp rx=0 tx=0).stop 2024-04-21T08:38:06.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 -- 172.21.15.94:0/1585452688 shutdown_connections 2024-04-21T08:38:06.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f188c0a3cd0 0x7f188c0a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:06.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18a0101970 0x7f18a0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:06.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18a0107920 0x7f18a019c900 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:06.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 --2- 172.21.15.94:0/1585452688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18a0102330 0x7f18a01985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:06.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 -- 172.21.15.94:0/1585452688 >> 172.21.15.94:0/1585452688 conn(0x7f18a00788f0 msgr2=0x7f18a00ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:06.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 -- 172.21.15.94:0/1585452688 shutdown_connections 2024-04-21T08:38:06.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:06.612+0000 7f18a7156700 1 -- 172.21.15.94:0/1585452688 wait complete. 2024-04-21T08:38:06.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:06 smithi111 ceph-mon[31493]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T08:38:06.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:06 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1585452688' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:38:06.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:06 smithi094 ceph-mon[27988]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T08:38:06.943 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:06 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1585452688' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:38:07.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:06 smithi173 ceph-mon[31428]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T08:38:07.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:06 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1585452688' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:38:07.372 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":3394521972},{"type":"v1","addr":"172.21.15.94:6801","nonce":3394521972}]},"active_addr":"172.21.15.94:6801/3394521972","active_change":"2024-04-21T08:30:29.483861+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.94:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":4286642695}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":1188403796}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":909646532}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":694152474}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":740344954}]}]} 2024-04-21T08:38:07.377 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-21T08:38:07.377 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-21T08:38:07.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:38:07.614 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:08.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:08 smithi111 ceph-mon[31493]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T08:38:08.985 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:08 smithi094 ceph-mon[27988]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T08:38:09.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:08 smithi173 ceph-mon[31428]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T08:38:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.406+0000 7fb869e88700 1 -- 172.21.15.94:0/3654445155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8641034d0 msgr2=0x7fb864103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.406+0000 7fb869e88700 1 --2- 172.21.15.94:0/3654445155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8641034d0 0x7fb864103950 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb858009990 tx=0x7fb858009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.406+0000 7fb869e88700 1 -- 172.21.15.94:0/3654445155 shutdown_connections 2024-04-21T08:38:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.406+0000 7fb869e88700 1 --2- 172.21.15.94:0/3654445155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864101d90 0x7fb864102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.406+0000 7fb869e88700 1 --2- 172.21.15.94:0/3654445155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb864103e90 0x7fb86410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.406+0000 7fb869e88700 1 --2- 172.21.15.94:0/3654445155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8641034d0 0x7fb864103950 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.406+0000 7fb869e88700 1 -- 172.21.15.94:0/3654445155 >> 172.21.15.94:0/3654445155 conn(0x7fb8640779e0 msgr2=0x7fb864077df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.406+0000 7fb869e88700 1 -- 172.21.15.94:0/3654445155 shutdown_connections 2024-04-21T08:38:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.406+0000 7fb869e88700 1 -- 172.21.15.94:0/3654445155 wait complete. 2024-04-21T08:38:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.407+0000 7fb869e88700 1 Processor -- start 2024-04-21T08:38:09.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.407+0000 7fb869e88700 1 -- start start 2024-04-21T08:38:09.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.407+0000 7fb869e88700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb864101d90 0x7fb86419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:09.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.407+0000 7fb869e88700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8641034d0 0x7fb86419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:09.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.407+0000 7fb869e88700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb864103e90 0x7fb8641a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:09.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.407+0000 7fb869e88700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb864111cf0 con 0x7fb864101d90 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.407+0000 7fb869e88700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb864111b70 con 0x7fb8641034d0 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb869e88700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb864111e70 con 0x7fb864103e90 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb8637fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb864101d90 0x7fb86419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb8637fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb864101d90 0x7fb86419a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33698/0 (socket says 172.21.15.94:33698) 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb8637fe700 1 -- 172.21.15.94:0/3170103762 learned_addr learned my addr 172.21.15.94:0/3170103762 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb862ffd700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8641034d0 0x7fb86419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb863fff700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb864103e90 0x7fb8641a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb8637fe700 1 -- 172.21.15.94:0/3170103762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb864103e90 msgr2=0x7fb8641a5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb8637fe700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb864103e90 0x7fb8641a5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb8637fe700 1 -- 172.21.15.94:0/3170103762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8641034d0 msgr2=0x7fb86419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb8637fe700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8641034d0 0x7fb86419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:09.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb8637fe700 1 -- 172.21.15.94:0/3170103762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8641a56b0 con 0x7fb864101d90 2024-04-21T08:38:09.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb862ffd700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8641034d0 0x7fb86419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:38:09.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb863fff700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb864103e90 0x7fb8641a5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:38:09.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb8637fe700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb864101d90 0x7fb86419a0f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb85400c9b0 tx=0x7fb85400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:09.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb860ff9700 1 -- 172.21.15.94:0/3170103762 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb854007b00 con 0x7fb864101d90 2024-04-21T08:38:09.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb869e88700 1 -- 172.21.15.94:0/3170103762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8641a5880 con 0x7fb864101d90 2024-04-21T08:38:09.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb860ff9700 1 -- 172.21.15.94:0/3170103762 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb854007ca0 con 0x7fb864101d90 2024-04-21T08:38:09.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb860ff9700 1 -- 172.21.15.94:0/3170103762 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb854018730 con 0x7fb864101d90 2024-04-21T08:38:09.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.408+0000 7fb869e88700 1 -- 172.21.15.94:0/3170103762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8641a5de0 con 0x7fb864101d90 2024-04-21T08:38:09.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.409+0000 7fb869e88700 1 -- 172.21.15.94:0/3170103762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb864069730 con 0x7fb864101d90 2024-04-21T08:38:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.410+0000 7fb860ff9700 1 -- 172.21.15.94:0/3170103762 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb85400f450 con 0x7fb864101d90 2024-04-21T08:38:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.411+0000 7fb860ff9700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb84c0a3cd0 0x7fb84c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.411+0000 7fb862ffd700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb84c0a3cd0 0x7fb84c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.411+0000 7fb860ff9700 1 -- 172.21.15.94:0/3170103762 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb8540d0860 con 0x7fb864101d90 2024-04-21T08:38:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.412+0000 7fb862ffd700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb84c0a3cd0 0x7fb84c0a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb858006010 tx=0x7fb85800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:09.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.416+0000 7fb860ff9700 1 -- 172.21.15.94:0/3170103762 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb854014030 con 0x7fb864101d90 2024-04-21T08:38:09.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.615+0000 7fb869e88700 1 -- 172.21.15.94:0/3170103762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb86406b770 con 0x7fb864101d90 2024-04-21T08:38:09.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.617+0000 7fb860ff9700 1 -- 172.21.15.94:0/3170103762 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24092 (secure 0 0 0) 0x7fb85409cea0 con 0x7fb864101d90 2024-04-21T08:38:09.617 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:38:09.617 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:37:35.939829+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 -- 172.21.15.94:0/3170103762 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb84c0a3cd0 msgr2=0x7fb84c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb84c0a3cd0 0x7fb84c0a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb858006010 tx=0x7fb85800b540 comp rx=0 tx=0).stop 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 -- 172.21.15.94:0/3170103762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb864101d90 msgr2=0x7fb86419a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb864101d90 0x7fb86419a0f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb85400c9b0 tx=0x7fb85400cd70 comp rx=0 tx=0).stop 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 -- 172.21.15.94:0/3170103762 shutdown_connections 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb84c0a3cd0 0x7fb84c0a6190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb864103e90 0x7fb8641a5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb864101d90 0x7fb86419a0f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 --2- 172.21.15.94:0/3170103762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8641034d0 0x7fb86419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 -- 172.21.15.94:0/3170103762 >> 172.21.15.94:0/3170103762 conn(0x7fb8640779e0 msgr2=0x7fb864106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:09.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 -- 172.21.15.94:0/3170103762 shutdown_connections 2024-04-21T08:38:09.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:09.619+0000 7fb869e88700 1 -- 172.21.15.94:0/3170103762 wait complete. 2024-04-21T08:38:09.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:09 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3170103762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:38:10.005 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:09 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3170103762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:38:10.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:09 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3170103762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:38:10.518 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-21T08:38:10.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:38:10.758 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:10.759 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:10 smithi094 ceph-mon[27988]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.7 KiB/s wr, 14 op/s 2024-04-21T08:38:10.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:10 smithi111 ceph-mon[31493]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.7 KiB/s wr, 14 op/s 2024-04-21T08:38:11.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:10 smithi173 ceph-mon[31428]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.7 KiB/s wr, 14 op/s 2024-04-21T08:38:12.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.527+0000 7f1eded18700 1 -- 172.21.15.94:0/3441536867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ed8101940 msgr2=0x7f1ed8101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:12.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.527+0000 7f1eded18700 1 --2- 172.21.15.94:0/3441536867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ed8101940 0x7f1ed8101dc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4009990 tx=0x7f1ec4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.527+0000 7f1eded18700 1 -- 172.21.15.94:0/3441536867 shutdown_connections 2024-04-21T08:38:12.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.527+0000 7f1eded18700 1 --2- 172.21.15.94:0/3441536867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed8102300 0x7f1ed810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.527+0000 7f1eded18700 1 --2- 172.21.15.94:0/3441536867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ed8107990 0x7f1ed8107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.527+0000 7f1eded18700 1 --2- 172.21.15.94:0/3441536867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ed8101940 0x7f1ed8101dc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.527+0000 7f1eded18700 1 -- 172.21.15.94:0/3441536867 >> 172.21.15.94:0/3441536867 conn(0x7f1ed80788f0 msgr2=0x7f1ed8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.527+0000 7f1eded18700 1 -- 172.21.15.94:0/3441536867 shutdown_connections 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.527+0000 7f1eded18700 1 -- 172.21.15.94:0/3441536867 wait complete. 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1eded18700 1 Processor -- start 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1eded18700 1 -- start start 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1eded18700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ed8101940 0x7f1ed81034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1eded18700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed8102300 0x7f1ed8103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1eded18700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ed8107990 0x7f1ed8103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1eded18700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed8111b00 con 0x7f1ed8107990 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1eded18700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed8111980 con 0x7f1ed8102300 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1eded18700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed8111c80 con 0x7f1ed8101940 2024-04-21T08:38:12.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1edd2b5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ed8107990 0x7f1ed8103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1edd2b5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ed8107990 0x7f1ed8103f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33712/0 (socket says 172.21.15.94:33712) 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1edd2b5700 1 -- 172.21.15.94:0/2045806950 learned_addr learned my addr 172.21.15.94:0/2045806950 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1edcab4700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ed8101940 0x7f1ed81034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1ed7fff700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed8102300 0x7f1ed8103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1edd2b5700 1 -- 172.21.15.94:0/2045806950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ed8101940 msgr2=0x7f1ed81034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1edd2b5700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ed8101940 0x7f1ed81034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1edd2b5700 1 -- 172.21.15.94:0/2045806950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed8102300 msgr2=0x7f1ed8103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1edd2b5700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed8102300 0x7f1ed8103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1edd2b5700 1 -- 172.21.15.94:0/2045806950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed81a7670 con 0x7f1ed8107990 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.528+0000 7f1edd2b5700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ed8107990 0x7f1ed8103f70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0007d70 tx=0x7f1ed000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.529+0000 7f1ed5ffb700 1 -- 172.21.15.94:0/2045806950 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0016020 con 0x7f1ed8107990 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.529+0000 7f1eded18700 1 -- 172.21.15.94:0/2045806950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed81a7960 con 0x7f1ed8107990 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.529+0000 7f1ed5ffb700 1 -- 172.21.15.94:0/2045806950 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ed000ad40 con 0x7f1ed8107990 2024-04-21T08:38:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.529+0000 7f1ed5ffb700 1 -- 172.21.15.94:0/2045806950 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00115e0 con 0x7f1ed8107990 2024-04-21T08:38:12.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.529+0000 7f1eded18700 1 -- 172.21.15.94:0/2045806950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed81a7ea0 con 0x7f1ed8107990 2024-04-21T08:38:12.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.530+0000 7f1eded18700 1 -- 172.21.15.94:0/2045806950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ed8069730 con 0x7f1ed8107990 2024-04-21T08:38:12.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.531+0000 7f1ed5ffb700 1 -- 172.21.15.94:0/2045806950 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1ed0019020 con 0x7f1ed8107990 2024-04-21T08:38:12.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.531+0000 7f1ed5ffb700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f1ec80a3cd0 0x7f1ec80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:12.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.531+0000 7f1edcab4700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f1ec80a3cd0 0x7f1ec80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:12.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.531+0000 7f1ed5ffb700 1 -- 172.21.15.94:0/2045806950 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1ed00d0630 con 0x7f1ed8107990 2024-04-21T08:38:12.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.532+0000 7f1edcab4700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f1ec80a3cd0 0x7f1ec80a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0045b0 tx=0x7f1ecc00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:12.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.536+0000 7f1ed5ffb700 1 -- 172.21.15.94:0/2045806950 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ed0020020 con 0x7f1ed8107990 2024-04-21T08:38:12.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.736+0000 7f1eded18700 1 -- 172.21.15.94:0/2045806950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ed802df80 con 0x7f1ed8107990 2024-04-21T08:38:12.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.737+0000 7f1ed5ffb700 1 -- 172.21.15.94:0/2045806950 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24092 (secure 0 0 0) 0x7f1ed009cd80 con 0x7f1ed8107990 2024-04-21T08:38:12.737 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:38:12.737 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:37:35.939829+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:38:12.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 -- 172.21.15.94:0/2045806950 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f1ec80a3cd0 msgr2=0x7f1ec80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:12.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f1ec80a3cd0 0x7f1ec80a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0045b0 tx=0x7f1ecc00b3c0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 -- 172.21.15.94:0/2045806950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ed8107990 msgr2=0x7f1ed8103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:12.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ed8107990 0x7f1ed8103f70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0007d70 tx=0x7f1ed000a340 comp rx=0 tx=0).stop 2024-04-21T08:38:12.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 -- 172.21.15.94:0/2045806950 shutdown_connections 2024-04-21T08:38:12.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f1ec80a3cd0 0x7f1ec80a6190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ed8107990 0x7f1ed8103f70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ed8101940 0x7f1ed81034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 --2- 172.21.15.94:0/2045806950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed8102300 0x7f1ed8103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:12.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 -- 172.21.15.94:0/2045806950 >> 172.21.15.94:0/2045806950 conn(0x7f1ed80788f0 msgr2=0x7f1ed806ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:12.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.739+0000 7f1eded18700 1 -- 172.21.15.94:0/2045806950 shutdown_connections 2024-04-21T08:38:12.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:12.740+0000 7f1eded18700 1 -- 172.21.15.94:0/2045806950 wait complete. 2024-04-21T08:38:12.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:12 smithi111 ceph-mon[31493]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.6 KiB/s wr, 14 op/s 2024-04-21T08:38:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:12 smithi094 ceph-mon[27988]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.6 KiB/s wr, 14 op/s 2024-04-21T08:38:13.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:12 smithi173 ceph-mon[31428]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.6 KiB/s wr, 14 op/s 2024-04-21T08:38:13.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-21T08:38:13.422 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-21T08:38:13.422 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-21T08:38:13.423 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-21T08:38:13.423 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-21T08:38:13.423 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-21T08:38:13.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-21T08:38:13.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-21T08:38:13.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-21T08:38:13.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-21T08:38:13.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-21T08:38:13.426 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-21T08:38:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:13 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2045806950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:38:14.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:13 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2045806950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:38:14.130 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:14.163 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:13 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2045806950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:38:14.272 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:14.272 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:14.273 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:14.273 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:14.276 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:14.277 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:14.277 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:14.895 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:14 smithi094 ceph-mon[27988]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 8 op/s 2024-04-21T08:38:15.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:14 smithi173 ceph-mon[31428]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 8 op/s 2024-04-21T08:38:15.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:14 smithi111 ceph-mon[31493]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 8 op/s 2024-04-21T08:38:15.437 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:15.953 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:15 smithi094 ceph-mon[27988]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T08:38:16.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:15 smithi173 ceph-mon[31428]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T08:38:16.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:15 smithi111 ceph-mon[31493]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T08:38:18.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:18 smithi111 ceph-mon[31493]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-21T08:38:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:18 smithi094 ceph-mon[27988]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-21T08:38:19.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:18 smithi173 ceph-mon[31428]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-21T08:38:20.930 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:20 smithi094 ceph-mon[27988]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-21T08:38:20.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:20 smithi111 ceph-mon[31493]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-21T08:38:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:20 smithi173 ceph-mon[31428]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-21T08:38:22.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:22 smithi111 ceph-mon[31493]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-21T08:38:23.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:22 smithi094 ceph-mon[27988]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-21T08:38:23.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:22 smithi173 ceph-mon[31428]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-21T08:38:24.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.574+0000 7f43ac5da700 1 -- 172.21.15.94:0/2291892283 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a4108ad0 msgr2=0x7f43a4108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:24.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.574+0000 7f43ac5da700 1 --2- 172.21.15.94:0/2291892283 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a4108ad0 0x7f43a4108eb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009990 tx=0x7f43a0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 -- 172.21.15.94:0/2291892283 shutdown_connections 2024-04-21T08:38:24.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 --2- 172.21.15.94:0/2291892283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43a41019d0 0x7f43a410f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 --2- 172.21.15.94:0/2291892283 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a4108ad0 0x7f43a4108eb0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 --2- 172.21.15.94:0/2291892283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a4101030 0x7f43a4101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 -- 172.21.15.94:0/2291892283 >> 172.21.15.94:0/2291892283 conn(0x7f43a4077fb0 msgr2=0x7f43a40ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:24.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 -- 172.21.15.94:0/2291892283 shutdown_connections 2024-04-21T08:38:24.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 -- 172.21.15.94:0/2291892283 wait complete. 2024-04-21T08:38:24.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 Processor -- start 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 -- start start 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43a4101030 0x7f43a41990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a41019d0 0x7f43a41995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a4108ad0 0x7f43a419d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f43a4112be0 con 0x7f43a4108ad0 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f43a4112a60 con 0x7f43a41019d0 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43ac5da700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f43a4112d60 con 0x7f43a4101030 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.575+0000 7f43aab77700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a4108ad0 0x7f43a419d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43aab77700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a4108ad0 0x7f43a419d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47616/0 (socket says 172.21.15.94:47616) 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43aab77700 1 -- 172.21.15.94:0/760318236 learned_addr learned my addr 172.21.15.94:0/760318236 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43a9b75700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a41019d0 0x7f43a41995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43aa376700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43a4101030 0x7f43a41990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43aab77700 1 -- 172.21.15.94:0/760318236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43a4101030 msgr2=0x7f43a41990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:24.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43aab77700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43a4101030 0x7f43a41990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43aab77700 1 -- 172.21.15.94:0/760318236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a41019d0 msgr2=0x7f43a41995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43aab77700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a41019d0 0x7f43a41995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43aab77700 1 -- 172.21.15.94:0/760318236 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a419e0c0 con 0x7f43a4108ad0 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43aab77700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a4108ad0 0x7f43a419d980 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f439c007ee0 tx=0x7f439c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f439b7fe700 1 -- 172.21.15.94:0/760318236 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439c004bb0 con 0x7f43a4108ad0 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f439b7fe700 1 -- 172.21.15.94:0/760318236 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f439c004d50 con 0x7f43a4108ad0 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a419e3b0 con 0x7f43a4108ad0 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f439b7fe700 1 -- 172.21.15.94:0/760318236 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439c005920 con 0x7f43a4108ad0 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.576+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a41a5c30 con 0x7f43a4108ad0 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.577+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f43a40631d0 con 0x7f43a4108ad0 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.578+0000 7f439b7fe700 1 -- 172.21.15.94:0/760318236 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f439c01a020 con 0x7f43a4108ad0 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.579+0000 7f439b7fe700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43900a3cd0 0x7f43900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.579+0000 7f43aa376700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43900a3cd0 0x7f43900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:24.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.579+0000 7f439b7fe700 1 -- 172.21.15.94:0/760318236 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f439c0d0450 con 0x7f43a4108ad0 2024-04-21T08:38:24.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.579+0000 7f439b7fe700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] conn(0x7f43900ae990 0x7f43900b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:24.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.579+0000 7f439b7fe700 1 -- 172.21.15.94:0/760318236 --> [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43900b14a0 con 0x7f43900ae990 2024-04-21T08:38:24.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.579+0000 7f439b7fe700 1 -- 172.21.15.94:0/760318236 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f439c0d08a0 con 0x7f43a4108ad0 2024-04-21T08:38:24.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.579+0000 7f43aa376700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43900a3cd0 0x7f43900a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b5c0 tx=0x7f43a0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:24.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.579+0000 7f43a9b75700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] conn(0x7f43900ae990 0x7f43900b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:24.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.580+0000 7f43a9b75700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] conn(0x7f43900ae990 0x7f43900b0dd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:24.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.581+0000 7f439b7fe700 1 -- 172.21.15.94:0/760318236 <== osd.8 v2:172.21.15.173:6816/1973310944 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f43900b14a0 con 0x7f43900ae990 2024-04-21T08:38:24.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.597+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 --> [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f43a4063650 con 0x7f43900ae990 2024-04-21T08:38:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.597+0000 7f439b7fe700 1 -- 172.21.15.94:0/760318236 <== osd.8 v2:172.21.15.173:6816/1973310944 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f43a4063650 con 0x7f43900ae990 2024-04-21T08:38:24.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 >> [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] conn(0x7f43900ae990 msgr2=0x7f43900b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:24.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43ac5da700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] conn(0x7f43900ae990 0x7f43900b0dd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43900a3cd0 msgr2=0x7f43900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:24.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43ac5da700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43900a3cd0 0x7f43900a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b5c0 tx=0x7f43a0005960 comp rx=0 tx=0).stop 2024-04-21T08:38:24.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a4108ad0 msgr2=0x7f43a419d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:24.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43ac5da700 1 --2- 172.21.15.94:0/760318236 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a4108ad0 0x7f43a419d980 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f439c007ee0 tx=0x7f439c00d3b0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43aab77700 1 -- 172.21.15.94:0/760318236 reap_dead start 2024-04-21T08:38:24.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 shutdown_connections 2024-04-21T08:38:24.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 >> 172.21.15.94:0/760318236 conn(0x7f43a4077fb0 msgr2=0x7f43a40ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:24.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 shutdown_connections 2024-04-21T08:38:24.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:24.598+0000 7f43ac5da700 1 -- 172.21.15.94:0/760318236 wait complete. 2024-04-21T08:38:24.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:24 smithi094 ceph-mon[27988]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-21T08:38:24.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:24 smithi111 ceph-mon[31493]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-21T08:38:25.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:24 smithi173 ceph-mon[31428]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-21T08:38:26.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.122+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2808311427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bb406bb70 msgr2=0x7f2bb410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:26.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.122+0000 7f2bb8aca700 1 --2- 172.21.15.94:0/2808311427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bb406bb70 0x7f2bb410f900 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00b0d0 tx=0x7f2bac00b3e0 comp rx=0 tx=0).stop 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.122+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2808311427 shutdown_connections 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.122+0000 7f2bb8aca700 1 --2- 172.21.15.94:0/2808311427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb4108a50 0x7f2bb4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.122+0000 7f2bb8aca700 1 --2- 172.21.15.94:0/2808311427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bb406bb70 0x7f2bb410f900 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.122+0000 7f2bb8aca700 1 --2- 172.21.15.94:0/2808311427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bb406b1b0 0x7f2bb406b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.122+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2808311427 >> 172.21.15.94:0/2808311427 conn(0x7f2bb4077990 msgr2=0x7f2bb4077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.122+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2808311427 shutdown_connections 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.122+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2808311427 wait complete. 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb8aca700 1 Processor -- start 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb8aca700 1 -- start start 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb8aca700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bb406b1b0 0x7f2bb4199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb8aca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb406bb70 0x7f2bb41995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:26.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb8aca700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bb4108a50 0x7f2bb419d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb8aca700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4112c40 con 0x7f2bb4108a50 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb8aca700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4112ac0 con 0x7f2bb406bb70 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb8aca700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4112dc0 con 0x7f2bb406b1b0 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb2d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bb4108a50 0x7f2bb419d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb2d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bb4108a50 0x7f2bb419d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47634/0 (socket says 172.21.15.94:47634) 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb2d9d700 1 -- 172.21.15.94:0/2814642386 learned_addr learned my addr 172.21.15.94:0/2814642386 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb259c700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bb406b1b0 0x7f2bb4199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.123+0000 7f2bb1d9b700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb406bb70 0x7f2bb41995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2bb2d9d700 1 -- 172.21.15.94:0/2814642386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bb406b1b0 msgr2=0x7f2bb4199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2bb2d9d700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bb406b1b0 0x7f2bb4199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2bb2d9d700 1 -- 172.21.15.94:0/2814642386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb406bb70 msgr2=0x7f2bb41995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2bb2d9d700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb406bb70 0x7f2bb41995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2bb2d9d700 1 -- 172.21.15.94:0/2814642386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb419e0a0 con 0x7f2bb4108a50 2024-04-21T08:38:26.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2bb2d9d700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bb4108a50 0x7f2bb419d960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0095c0 tx=0x7f2bac007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:26.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2ba37fe700 1 -- 172.21.15.94:0/2814642386 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac01f070 con 0x7f2bb4108a50 2024-04-21T08:38:26.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb419e330 con 0x7f2bb4108a50 2024-04-21T08:38:26.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2ba37fe700 1 -- 172.21.15.94:0/2814642386 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bac007e20 con 0x7f2bb4108a50 2024-04-21T08:38:26.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2ba37fe700 1 -- 172.21.15.94:0/2814642386 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac01b4a0 con 0x7f2bb4108a50 2024-04-21T08:38:26.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.124+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb41a5c10 con 0x7f2bb4108a50 2024-04-21T08:38:26.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.125+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2bb4063190 con 0x7f2bb4108a50 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.126+0000 7f2ba37fe700 1 -- 172.21.15.94:0/2814642386 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2bac022070 con 0x7f2bb4108a50 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.126+0000 7f2ba37fe700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b9c0a3cd0 0x7f2b9c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.127+0000 7f2bb259c700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b9c0a3cd0 0x7f2b9c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.127+0000 7f2ba37fe700 1 -- 172.21.15.94:0/2814642386 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2bac0d18e0 con 0x7f2bb4108a50 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.127+0000 7f2ba37fe700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] conn(0x7f2b9c0ae990 0x7f2b9c0b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.127+0000 7f2ba37fe700 1 -- 172.21.15.94:0/2814642386 --> [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bac02de50 con 0x7f2b9c0ae990 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.127+0000 7f2ba37fe700 1 -- 172.21.15.94:0/2814642386 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f2bac0d1d30 con 0x7f2bb4108a50 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.127+0000 7f2bb1d9b700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] conn(0x7f2b9c0ae990 0x7f2b9c0b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.127+0000 7f2bb259c700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b9c0a3cd0 0x7f2b9c0a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4005950 tx=0x7f2ba400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.127+0000 7f2bb1d9b700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] conn(0x7f2b9c0ae990 0x7f2b9c0b0dd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.128+0000 7f2ba37fe700 1 -- 172.21.15.94:0/2814642386 <== osd.9 v2:172.21.15.94:6826/3583446749 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2bac02de50 con 0x7f2b9c0ae990 2024-04-21T08:38:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.144+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 --> [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2bb4063610 con 0x7f2b9c0ae990 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.144+0000 7f2ba37fe700 1 -- 172.21.15.94:0/2814642386 <== osd.9 v2:172.21.15.94:6826/3583446749 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2bb4063610 con 0x7f2b9c0ae990 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.144+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] conn(0x7f2b9c0ae990 msgr2=0x7f2b9c0b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.144+0000 7f2bb8aca700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] conn(0x7f2b9c0ae990 0x7f2b9c0b0dd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.145+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b9c0a3cd0 msgr2=0x7f2b9c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.145+0000 7f2bb8aca700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b9c0a3cd0 0x7f2b9c0a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4005950 tx=0x7f2ba400a300 comp rx=0 tx=0).stop 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.145+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bb4108a50 msgr2=0x7f2bb419d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.145+0000 7f2bb8aca700 1 --2- 172.21.15.94:0/2814642386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bb4108a50 0x7f2bb419d960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0095c0 tx=0x7f2bac007a20 comp rx=0 tx=0).stop 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.145+0000 7f2bb2d9d700 1 -- 172.21.15.94:0/2814642386 reap_dead start 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.145+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 shutdown_connections 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.145+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 >> 172.21.15.94:0/2814642386 conn(0x7f2bb4077990 msgr2=0x7f2bb4100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.145+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 shutdown_connections 2024-04-21T08:38:26.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:26.145+0000 7f2bb8aca700 1 -- 172.21.15.94:0/2814642386 wait complete. 2024-04-21T08:38:26.839 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:26 smithi094 ceph-mon[27988]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-21T08:38:26.840 INFO:teuthology.orchestra.run.smithi094.stdout:287762808866 2024-04-21T08:38:26.841 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-21T08:38:26.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:26 smithi111 ceph-mon[31493]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-21T08:38:27.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:26 smithi173 ceph-mon[31428]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-21T08:38:28.750 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:28 smithi094 ceph-mon[27988]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-21T08:38:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:28 smithi111 ceph-mon[31493]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-21T08:38:29.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:28 smithi173 ceph-mon[31428]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-21T08:38:29.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2024-04-21T08:38:29.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-21T08:38:29.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2024-04-21T08:38:29.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-04-21T08:38:29.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-04-21T08:38:29.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-21T08:38:29.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8, 5, 10]}]: dispatch 2024-04-21T08:38:29.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:29 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-04-21T08:38:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2024-04-21T08:38:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-21T08:38:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2024-04-21T08:38:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-04-21T08:38:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-04-21T08:38:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-21T08:38:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8, 5, 10]}]: dispatch 2024-04-21T08:38:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:29 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-04-21T08:38:30.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2024-04-21T08:38:30.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-21T08:38:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2024-04-21T08:38:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-04-21T08:38:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-04-21T08:38:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-21T08:38:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8, 5, 10]}]: dispatch 2024-04-21T08:38:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:29 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-04-21T08:38:30.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.136+0000 7fcf6a99a700 1 -- 172.21.15.94:0/4068798120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf64108a20 msgr2=0x7fcf64108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.136+0000 7fcf6a99a700 1 --2- 172.21.15.94:0/4068798120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf64108a20 0x7fcf64108e00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009990 tx=0x7fcf54009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.136+0000 7fcf6a99a700 1 -- 172.21.15.94:0/4068798120 shutdown_connections 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.136+0000 7fcf6a99a700 1 --2- 172.21.15.94:0/4068798120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf6406b980 0x7fcf6410f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.136+0000 7fcf6a99a700 1 --2- 172.21.15.94:0/4068798120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf64108a20 0x7fcf64108e00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.136+0000 7fcf6a99a700 1 --2- 172.21.15.94:0/4068798120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf6406afe0 0x7fcf6406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.136+0000 7fcf6a99a700 1 -- 172.21.15.94:0/4068798120 >> 172.21.15.94:0/4068798120 conn(0x7fcf640788b0 msgr2=0x7fcf64078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.136+0000 7fcf6a99a700 1 -- 172.21.15.94:0/4068798120 shutdown_connections 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.136+0000 7fcf6a99a700 1 -- 172.21.15.94:0/4068798120 wait complete. 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf6a99a700 1 Processor -- start 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf6a99a700 1 -- start start 2024-04-21T08:38:30.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf6a99a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf6406afe0 0x7fcf64199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf6a99a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf6406b980 0x7fcf64199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf6a99a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf64108a20 0x7fcf6419d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf6a99a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64112bc0 con 0x7fcf6406afe0 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf6a99a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64112a40 con 0x7fcf6406b980 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf6a99a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64112d40 con 0x7fcf64108a20 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf63fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf6406afe0 0x7fcf64199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf63fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf6406afe0 0x7fcf64199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47688/0 (socket says 172.21.15.94:47688) 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf63fff700 1 -- 172.21.15.94:0/2308685873 learned_addr learned my addr 172.21.15.94:0/2308685873 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf637fe700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf6406b980 0x7fcf64199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf68f37700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf64108a20 0x7fcf6419d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf63fff700 1 -- 172.21.15.94:0/2308685873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf64108a20 msgr2=0x7fcf6419d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf63fff700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf64108a20 0x7fcf6419d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf63fff700 1 -- 172.21.15.94:0/2308685873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf6406b980 msgr2=0x7fcf64199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf63fff700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf6406b980 0x7fcf64199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf63fff700 1 -- 172.21.15.94:0/2308685873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6419e120 con 0x7fcf6406afe0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.137+0000 7fcf68f37700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf64108a20 0x7fcf6419d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.138+0000 7fcf63fff700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf6406afe0 0x7fcf64199110 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcf540057c0 tx=0x7fcf540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.138+0000 7fcf617fa700 1 -- 172.21.15.94:0/2308685873 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54027070 con 0x7fcf6406afe0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.138+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf6419e3b0 con 0x7fcf6406afe0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.138+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf641a5c90 con 0x7fcf6406afe0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.138+0000 7fcf617fa700 1 -- 172.21.15.94:0/2308685873 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf540095d0 con 0x7fcf6406afe0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.138+0000 7fcf617fa700 1 -- 172.21.15.94:0/2308685873 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54017630 con 0x7fcf6406afe0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.138+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcf50001030 con 0x7fcf6406afe0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.140+0000 7fcf617fa700 1 -- 172.21.15.94:0/2308685873 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcf5402a070 con 0x7fcf6406afe0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.140+0000 7fcf617fa700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcf4c0a3cd0 0x7fcf4c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.141+0000 7fcf637fe700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcf4c0a3cd0 0x7fcf4c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.141+0000 7fcf617fa700 1 -- 172.21.15.94:0/2308685873 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9050+0+0 (secure 0 0 0) 0x7fcf540d1470 con 0x7fcf6406afe0 2024-04-21T08:38:30.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.141+0000 7fcf617fa700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] conn(0x7fcf4c0aec00 0x7fcf4c0b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:30.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.141+0000 7fcf617fa700 1 -- 172.21.15.94:0/2308685873 --> [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf5400b0b0 con 0x7fcf4c0aec00 2024-04-21T08:38:30.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.141+0000 7fcf617fa700 1 -- 172.21.15.94:0/2308685873 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fcf540d18c0 con 0x7fcf6406afe0 2024-04-21T08:38:30.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.141+0000 7fcf637fe700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcf4c0a3cd0 0x7fcf4c0a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcf64078ff0 tx=0x7fcf5800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:30.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.141+0000 7fcf68f37700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] conn(0x7fcf4c0aec00 0x7fcf4c0b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:30.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.141+0000 7fcf68f37700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] conn(0x7fcf4c0aec00 0x7fcf4c0b10a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:30.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.142+0000 7fcf617fa700 1 -- 172.21.15.94:0/2308685873 <== osd.1 v2:172.21.15.111:6800/2507021892 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf5400b0b0 con 0x7fcf4c0aec00 2024-04-21T08:38:30.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.159+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 --> [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcf50002e60 con 0x7fcf4c0aec00 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.160+0000 7fcf617fa700 1 -- 172.21.15.94:0/2308685873 <== osd.1 v2:172.21.15.111:6800/2507021892 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcf50002e60 con 0x7fcf4c0aec00 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 >> [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] conn(0x7fcf4c0aec00 msgr2=0x7fcf4c0b10a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf6a99a700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] conn(0x7fcf4c0aec00 0x7fcf4c0b10a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcf4c0a3cd0 msgr2=0x7fcf4c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf6a99a700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcf4c0a3cd0 0x7fcf4c0a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcf64078ff0 tx=0x7fcf5800b410 comp rx=0 tx=0).stop 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf6406afe0 msgr2=0x7fcf64199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf6a99a700 1 --2- 172.21.15.94:0/2308685873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf6406afe0 0x7fcf64199110 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcf540057c0 tx=0x7fcf540057f0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf68f37700 1 -- 172.21.15.94:0/2308685873 reap_dead start 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 shutdown_connections 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 >> 172.21.15.94:0/2308685873 conn(0x7fcf640788b0 msgr2=0x7fcf640ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 shutdown_connections 2024-04-21T08:38:30.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.161+0000 7fcf6a99a700 1 -- 172.21.15.94:0/2308685873 wait complete. 2024-04-21T08:38:30.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.965+0000 7fba0ce44700 1 -- 172.21.15.94:0/1532246779 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba08107960 msgr2=0x7fba08107d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.965+0000 7fba0ce44700 1 --2- 172.21.15.94:0/1532246779 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba08107960 0x7fba08107d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009990 tx=0x7fb9f8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.965+0000 7fba0ce44700 1 -- 172.21.15.94:0/1532246779 shutdown_connections 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.965+0000 7fba0ce44700 1 --2- 172.21.15.94:0/1532246779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba08102320 0x7fba0810e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.965+0000 7fba0ce44700 1 --2- 172.21.15.94:0/1532246779 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba08107960 0x7fba08107d40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.965+0000 7fba0ce44700 1 --2- 172.21.15.94:0/1532246779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba08101960 0x7fba08101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.965+0000 7fba0ce44700 1 -- 172.21.15.94:0/1532246779 >> 172.21.15.94:0/1532246779 conn(0x7fba080789e0 msgr2=0x7fba08078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.965+0000 7fba0ce44700 1 -- 172.21.15.94:0/1532246779 shutdown_connections 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.965+0000 7fba0ce44700 1 -- 172.21.15.94:0/1532246779 wait complete. 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.966+0000 7fba0ce44700 1 Processor -- start 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.966+0000 7fba0ce44700 1 -- start start 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.966+0000 7fba0ce44700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba08101960 0x7fba08199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.966+0000 7fba0ce44700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba08102320 0x7fba0819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.966+0000 7fba0ce44700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba08107960 0x7fba081a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.966+0000 7fba0ce44700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fba08111b40 con 0x7fba08102320 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.966+0000 7fba0ce44700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fba081119c0 con 0x7fba08107960 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.966+0000 7fba0ce44700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fba08111cc0 con 0x7fba08101960 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba05d9b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba08102320 0x7fba0819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba05d9b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba08102320 0x7fba0819a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47656/0 (socket says 172.21.15.94:47656) 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba05d9b700 1 -- 172.21.15.94:0/1181378778 learned_addr learned my addr 172.21.15.94:0/1181378778 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba0659c700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba08101960 0x7fba08199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba06d9d700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba08107960 0x7fba081a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba05d9b700 1 -- 172.21.15.94:0/1181378778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba08101960 msgr2=0x7fba08199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba05d9b700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba08101960 0x7fba08199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba05d9b700 1 -- 172.21.15.94:0/1181378778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba08107960 msgr2=0x7fba081a4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba05d9b700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba08107960 0x7fba081a4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba05d9b700 1 -- 172.21.15.94:0/1181378778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba081a5400 con 0x7fba08102320 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba05d9b700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba08102320 0x7fba0819a380 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000c9b0 tx=0x7fb9f000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fb9ff7fe700 1 -- 172.21.15.94:0/1181378778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0007b00 con 0x7fba08102320 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba081a5630 con 0x7fba08102320 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fb9ff7fe700 1 -- 172.21.15.94:0/1181378778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f000ce50 con 0x7fba08102320 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fb9ff7fe700 1 -- 172.21.15.94:0/1181378778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0018660 con 0x7fba08102320 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.967+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba081a5b40 con 0x7fba08102320 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.968+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba08063150 con 0x7fba08102320 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.970+0000 7fb9ff7fe700 1 -- 172.21.15.94:0/1181378778 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb9f000f450 con 0x7fba08102320 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.970+0000 7fb9ff7fe700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9f40a3cd0 0x7fb9f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.970+0000 7fba0659c700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9f40a3cd0 0x7fb9f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.970+0000 7fb9ff7fe700 1 -- 172.21.15.94:0/1181378778 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9050+0+0 (secure 0 0 0) 0x7fb9f00d0c20 con 0x7fba08102320 2024-04-21T08:38:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.970+0000 7fb9ff7fe700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] conn(0x7fb9f40aed30 0x7fb9f40b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:30.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.970+0000 7fb9ff7fe700 1 -- 172.21.15.94:0/1181378778 --> [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9f40b18a0 con 0x7fb9f40aed30 2024-04-21T08:38:30.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.970+0000 7fb9ff7fe700 1 -- 172.21.15.94:0/1181378778 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb9f00d1070 con 0x7fba08102320 2024-04-21T08:38:30.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.971+0000 7fba0659c700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9f40a3cd0 0x7fb9f40a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80098e0 tx=0x7fb9f8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:30.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.971+0000 7fba06d9d700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] conn(0x7fb9f40aed30 0x7fb9f40b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:30.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.971+0000 7fba06d9d700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] conn(0x7fb9f40aed30 0x7fb9f40b11d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:30.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.972+0000 7fb9ff7fe700 1 -- 172.21.15.94:0/1181378778 <== osd.11 v2:172.21.15.173:6824/1600242692 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9f40b18a0 con 0x7fb9f40aed30 2024-04-21T08:38:30.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.988+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 --> [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fba08063550 con 0x7fb9f40aed30 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.989+0000 7fb9ff7fe700 1 -- 172.21.15.94:0/1181378778 <== osd.11 v2:172.21.15.173:6824/1600242692 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fba08063550 con 0x7fb9f40aed30 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.989+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 >> [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] conn(0x7fb9f40aed30 msgr2=0x7fb9f40b11d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.990+0000 7fba0ce44700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] conn(0x7fb9f40aed30 0x7fb9f40b11d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.990+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9f40a3cd0 msgr2=0x7fb9f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.990+0000 7fba0ce44700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb9f40a3cd0 0x7fb9f40a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80098e0 tx=0x7fb9f8005960 comp rx=0 tx=0).stop 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.990+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba08102320 msgr2=0x7fba0819a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.990+0000 7fba0ce44700 1 --2- 172.21.15.94:0/1181378778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba08102320 0x7fba0819a380 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000c9b0 tx=0x7fb9f000ccc0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.990+0000 7fba06d9d700 1 -- 172.21.15.94:0/1181378778 reap_dead start 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.990+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 shutdown_connections 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.990+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 >> 172.21.15.94:0/1181378778 conn(0x7fba080789e0 msgr2=0x7fba08104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.990+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 shutdown_connections 2024-04-21T08:38:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:29.990+0000 7fba0ce44700 1 -- 172.21.15.94:0/1181378778 wait complete. 2024-04-21T08:38:30.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-21T08:38:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2024-04-21T08:38:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-21T08:38:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2024-04-21T08:38:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-04-21T08:38:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-04-21T08:38:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-21T08:38:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8, 5, 10]}]': finished 2024-04-21T08:38:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-04-21T08:38:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:38:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:31.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-21T08:38:31.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2024-04-21T08:38:31.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-21T08:38:31.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2024-04-21T08:38:31.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-04-21T08:38:31.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-04-21T08:38:31.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-21T08:38:31.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8, 5, 10]}]': finished 2024-04-21T08:38:31.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-04-21T08:38:31.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:38:31.008 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:31.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-21T08:38:31.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2024-04-21T08:38:31.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-21T08:38:31.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2024-04-21T08:38:31.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-04-21T08:38:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-04-21T08:38:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-21T08:38:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8, 5, 10]}]': finished 2024-04-21T08:38:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-04-21T08:38:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:38:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:31.495 INFO:teuthology.orchestra.run.smithi094.stdout:343597383705 2024-04-21T08:38:31.495 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-21T08:38:31.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.050+0000 7f53539e4700 1 -- 172.21.15.94:0/2027460168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f534c101930 msgr2=0x7f534c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.050+0000 7f53539e4700 1 --2- 172.21.15.94:0/2027460168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f534c101930 0x7f534c101db0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f533c009990 tx=0x7f533c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.050+0000 7f53539e4700 1 -- 172.21.15.94:0/2027460168 shutdown_connections 2024-04-21T08:38:31.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.050+0000 7f53539e4700 1 --2- 172.21.15.94:0/2027460168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c107980 0x7f534c107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.050+0000 7f53539e4700 1 --2- 172.21.15.94:0/2027460168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f534c1022f0 0x7f534c10e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.050+0000 7f53539e4700 1 --2- 172.21.15.94:0/2027460168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f534c101930 0x7f534c101db0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.050+0000 7f53539e4700 1 -- 172.21.15.94:0/2027460168 >> 172.21.15.94:0/2027460168 conn(0x7f534c0788b0 msgr2=0x7f534c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.050+0000 7f53539e4700 1 -- 172.21.15.94:0/2027460168 shutdown_connections 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.050+0000 7f53539e4700 1 -- 172.21.15.94:0/2027460168 wait complete. 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f53539e4700 1 Processor -- start 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f53539e4700 1 -- start start 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f53539e4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f534c101930 0x7f534c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f53539e4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f534c1022f0 0x7f534c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f53539e4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c107980 0x7f534c19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f53539e4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f534c111b50 con 0x7f534c101930 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f53539e4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f534c1119d0 con 0x7f534c107980 2024-04-21T08:38:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f53539e4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f534c111cd0 con 0x7f534c1022f0 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f5351780700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f534c101930 0x7f534c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f5351780700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f534c101930 0x7f534c1980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47670/0 (socket says 172.21.15.94:47670) 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f5351780700 1 -- 172.21.15.94:0/2078819043 learned_addr learned my addr 172.21.15.94:0/2078819043 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f5350f7f700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f534c1022f0 0x7f534c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.051+0000 7f5351f81700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c107980 0x7f534c19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f5351780700 1 -- 172.21.15.94:0/2078819043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f534c1022f0 msgr2=0x7f534c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f5351780700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f534c1022f0 0x7f534c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f5351780700 1 -- 172.21.15.94:0/2078819043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c107980 msgr2=0x7f534c19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f5351780700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c107980 0x7f534c19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f5351780700 1 -- 172.21.15.94:0/2078819043 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534c19d060 con 0x7f534c101930 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f5351780700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f534c101930 0x7f534c1980a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f534800d9e0 tx=0x7f534800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f53427fc700 1 -- 172.21.15.94:0/2078819043 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534801b070 con 0x7f534c101930 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f53427fc700 1 -- 172.21.15.94:0/2078819043 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f534800eb70 con 0x7f534c101930 2024-04-21T08:38:31.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534c19d350 con 0x7f534c101930 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f53427fc700 1 -- 172.21.15.94:0/2078819043 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53480145a0 con 0x7f534c101930 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.052+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f534c06afb0 con 0x7f534c101930 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.053+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f534c06b5c0 con 0x7f534c101930 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.054+0000 7f53427fc700 1 -- 172.21.15.94:0/2078819043 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f534801e070 con 0x7f534c101930 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.055+0000 7f53427fc700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f53380a3cd0 0x7f53380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.055+0000 7f5350f7f700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f53380a3cd0 0x7f53380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.055+0000 7f53427fc700 1 -- 172.21.15.94:0/2078819043 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9050+0+0 (secure 0 0 0) 0x7f53480d28a0 con 0x7f534c101930 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.055+0000 7f53427fc700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] conn(0x7f53380aed30 0x7f53380b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.055+0000 7f53427fc700 1 -- 172.21.15.94:0/2078819043 --> [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53380b18a0 con 0x7f53380aed30 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.055+0000 7f53427fc700 1 -- 172.21.15.94:0/2078819043 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f53480d2cf0 con 0x7f534c101930 2024-04-21T08:38:31.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.055+0000 7f5350f7f700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f53380a3cd0 0x7f53380a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f533c00b5c0 tx=0x7f533c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.055+0000 7f5351f81700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] conn(0x7f53380aed30 0x7f53380b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.056+0000 7f5351f81700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] conn(0x7f53380aed30 0x7f53380b11d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.057+0000 7f53427fc700 1 -- 172.21.15.94:0/2078819043 <== osd.7 v2:172.21.15.111:6816/659250531 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53380b18a0 con 0x7f53380aed30 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.073+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 --> [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f534c06ba40 con 0x7f53380aed30 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.073+0000 7f53427fc700 1 -- 172.21.15.94:0/2078819043 <== osd.7 v2:172.21.15.111:6816/659250531 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f534c06ba40 con 0x7f53380aed30 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 >> [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] conn(0x7f53380aed30 msgr2=0x7f53380b11d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f53539e4700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] conn(0x7f53380aed30 0x7f53380b11d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f53380a3cd0 msgr2=0x7f53380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f53539e4700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f53380a3cd0 0x7f53380a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f533c00b5c0 tx=0x7f533c005960 comp rx=0 tx=0).stop 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f534c101930 msgr2=0x7f534c1980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f53539e4700 1 --2- 172.21.15.94:0/2078819043 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f534c101930 0x7f534c1980a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f534800d9e0 tx=0x7f534800dcf0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f5351f81700 1 -- 172.21.15.94:0/2078819043 reap_dead start 2024-04-21T08:38:31.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 shutdown_connections 2024-04-21T08:38:31.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 >> 172.21.15.94:0/2078819043 conn(0x7f534c0788b0 msgr2=0x7f534c0ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:31.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 shutdown_connections 2024-04-21T08:38:31.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:30.074+0000 7f53539e4700 1 -- 172.21.15.94:0/2078819043 wait complete. 2024-04-21T08:38:31.744 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:31 smithi094 ceph-mon[27988]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:38:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:31 smithi111 ceph-mon[31493]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:38:31.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 -- 172.21.15.94:0/3275365584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11c01034c0 msgr2=0x7f11c010f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 --2- 172.21.15.94:0/3275365584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11c01034c0 0x7f11c010f8e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f11b800b0a0 tx=0x7f11b800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 -- 172.21.15.94:0/3275365584 shutdown_connections 2024-04-21T08:38:31.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 --2- 172.21.15.94:0/3275365584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c0108a10 0x7f11c0108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 --2- 172.21.15.94:0/3275365584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11c01034c0 0x7f11c010f8e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 --2- 172.21.15.94:0/3275365584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11c0102b00 0x7f11c0102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 -- 172.21.15.94:0/3275365584 >> 172.21.15.94:0/3275365584 conn(0x7f11c0079c00 msgr2=0x7f11c007a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:31.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 -- 172.21.15.94:0/3275365584 shutdown_connections 2024-04-21T08:38:31.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 -- 172.21.15.94:0/3275365584 wait complete. 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 Processor -- start 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.963+0000 7f11c7788700 1 -- start start 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c7788700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11c0102b00 0x7f11c0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c7788700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11c01034c0 0x7f11c0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c7788700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c0108a10 0x7f11c00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c7788700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f11c0112c10 con 0x7f11c01034c0 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c7788700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f11c0112a90 con 0x7f11c0108a10 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c7788700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11c0112d90 con 0x7f11c0102b00 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c4d23700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11c01034c0 0x7f11c0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c4d23700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11c01034c0 0x7f11c0075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47718/0 (socket says 172.21.15.94:47718) 2024-04-21T08:38:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c4d23700 1 -- 172.21.15.94:0/1808474743 learned_addr learned my addr 172.21.15.94:0/1808474743 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c5d25700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c0108a10 0x7f11c00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c5524700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11c0102b00 0x7f11c0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c4d23700 1 -- 172.21.15.94:0/1808474743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11c0102b00 msgr2=0x7f11c0074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c4d23700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11c0102b00 0x7f11c0074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c4d23700 1 -- 172.21.15.94:0/1808474743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c0108a10 msgr2=0x7f11c00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c4d23700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c0108a10 0x7f11c00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c4d23700 1 -- 172.21.15.94:0/1808474743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c01a75a0 con 0x7f11c01034c0 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c4d23700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11c01034c0 0x7f11c0075280 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f11b000e9c0 tx=0x7f11b000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11b67fc700 1 -- 172.21.15.94:0/1808474743 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b000cbd0 con 0x7f11c01034c0 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c01a7890 con 0x7f11c01034c0 2024-04-21T08:38:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11b67fc700 1 -- 172.21.15.94:0/1808474743 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11b000ee60 con 0x7f11c01034c0 2024-04-21T08:38:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.964+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c01a7dd0 con 0x7f11c01034c0 2024-04-21T08:38:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.965+0000 7f11b67fc700 1 -- 172.21.15.94:0/1808474743 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b00055c0 con 0x7f11c01034c0 2024-04-21T08:38:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.965+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f11a4001030 con 0x7f11c01034c0 2024-04-21T08:38:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.967+0000 7f11b67fc700 1 -- 172.21.15.94:0/1808474743 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f11b0004750 con 0x7f11c01034c0 2024-04-21T08:38:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.968+0000 7f11b67fc700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f11ac0a3cd0 0x7f11ac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.968+0000 7f11c5524700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f11ac0a3cd0 0x7f11ac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.968+0000 7f11b67fc700 1 -- 172.21.15.94:0/1808474743 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f11b00d0eb0 con 0x7f11c01034c0 2024-04-21T08:38:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.968+0000 7f11b67fc700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] conn(0x7f11ac0aed30 0x7f11ac0b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.968+0000 7f11b67fc700 1 -- 172.21.15.94:0/1808474743 --> [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11b0011e50 con 0x7f11ac0aed30 2024-04-21T08:38:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.968+0000 7f11b67fc700 1 -- 172.21.15.94:0/1808474743 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f11b00d1260 con 0x7f11c01034c0 2024-04-21T08:38:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.968+0000 7f11c5524700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f11ac0a3cd0 0x7f11ac0a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f11bc005950 tx=0x7f11bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.968+0000 7f11c5d25700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] conn(0x7f11ac0aed30 0x7f11ac0b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:31.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.969+0000 7f11c5d25700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] conn(0x7f11ac0aed30 0x7f11ac0b11d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:31.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.969+0000 7f11b67fc700 1 -- 172.21.15.94:0/1808474743 <== osd.3 v2:172.21.15.94:6810/66052007 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f11b0011e50 con 0x7f11ac0aed30 2024-04-21T08:38:31.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.986+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 --> [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f11a4002e60 con 0x7f11ac0aed30 2024-04-21T08:38:31.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.986+0000 7f11b67fc700 1 -- 172.21.15.94:0/1808474743 <== osd.3 v2:172.21.15.94:6810/66052007 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f11a4002e60 con 0x7f11ac0aed30 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.986+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] conn(0x7f11ac0aed30 msgr2=0x7f11ac0b11d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.986+0000 7f11c7788700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] conn(0x7f11ac0aed30 0x7f11ac0b11d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.986+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f11ac0a3cd0 msgr2=0x7f11ac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.986+0000 7f11c7788700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f11ac0a3cd0 0x7f11ac0a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f11bc005950 tx=0x7f11bc00b410 comp rx=0 tx=0).stop 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.986+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11c01034c0 msgr2=0x7f11c0075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.986+0000 7f11c7788700 1 --2- 172.21.15.94:0/1808474743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11c01034c0 0x7f11c0075280 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f11b000e9c0 tx=0x7f11b000ecd0 comp rx=0 tx=0).stop 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.986+0000 7f11c5d25700 1 -- 172.21.15.94:0/1808474743 reap_dead start 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.986+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 shutdown_connections 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.987+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 >> 172.21.15.94:0/1808474743 conn(0x7f11c0079c00 msgr2=0x7f11c0100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.987+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 shutdown_connections 2024-04-21T08:38:31.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:31.987+0000 7f11c7788700 1 -- 172.21.15.94:0/1808474743 wait complete. 2024-04-21T08:38:32.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:31 smithi173 ceph-mon[31428]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:38:32.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 -- 172.21.15.94:0/150633796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f604c101da0 msgr2=0x7f604c10f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:32.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 --2- 172.21.15.94:0/150633796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f604c101da0 0x7f604c10f810 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f604400b0a0 tx=0x7f604400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:38:32.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 -- 172.21.15.94:0/150633796 shutdown_connections 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 --2- 172.21.15.94:0/150633796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f604c107b00 0x7f604c107ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 --2- 172.21.15.94:0/150633796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f604c101da0 0x7f604c10f810 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 --2- 172.21.15.94:0/150633796 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f604c1013e0 0x7f604c101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 -- 172.21.15.94:0/150633796 >> 172.21.15.94:0/150633796 conn(0x7f604c076c60 msgr2=0x7f604c077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 -- 172.21.15.94:0/150633796 shutdown_connections 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 -- 172.21.15.94:0/150633796 wait complete. 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 Processor -- start 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.730+0000 7f6053beb700 1 -- start start 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6053beb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f604c1013e0 0x7f604c199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6053beb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f604c101da0 0x7f604c199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6053beb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f604c107b00 0x7f604c19d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6053beb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f604c112c90 con 0x7f604c1013e0 2024-04-21T08:38:32.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6053beb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f604c112b10 con 0x7f604c107b00 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6053beb700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f604c112e10 con 0x7f604c101da0 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6051987700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f604c1013e0 0x7f604c199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6051987700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f604c1013e0 0x7f604c199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47736/0 (socket says 172.21.15.94:47736) 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6051987700 1 -- 172.21.15.94:0/1398666915 learned_addr learned my addr 172.21.15.94:0/1398666915 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6052188700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f604c107b00 0x7f604c19d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6051186700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f604c101da0 0x7f604c199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6051987700 1 -- 172.21.15.94:0/1398666915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f604c101da0 msgr2=0x7f604c199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6051987700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f604c101da0 0x7f604c199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6051987700 1 -- 172.21.15.94:0/1398666915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f604c107b00 msgr2=0x7f604c19d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6051987700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f604c107b00 0x7f604c19d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6051987700 1 -- 172.21.15.94:0/1398666915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604c19e030 con 0x7f604c1013e0 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6051987700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f604c1013e0 0x7f604c199020 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f604800d980 tx=0x7f604800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.731+0000 7f6042ffd700 1 -- 172.21.15.94:0/1398666915 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048017030 con 0x7f604c1013e0 2024-04-21T08:38:32.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.732+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f604c19e320 con 0x7f604c1013e0 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.732+0000 7f6042ffd700 1 -- 172.21.15.94:0/1398666915 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60480044d0 con 0x7f604c1013e0 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.732+0000 7f6042ffd700 1 -- 172.21.15.94:0/1398666915 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60480039e0 con 0x7f604c1013e0 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.732+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f604c1a5ba0 con 0x7f604c1013e0 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.732+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f604c0631d0 con 0x7f604c1013e0 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.734+0000 7f6042ffd700 1 -- 172.21.15.94:0/1398666915 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f604801a070 con 0x7f604c1013e0 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.734+0000 7f6042ffd700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f60380a3cd0 0x7f60380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.734+0000 7f6051186700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f60380a3cd0 0x7f60380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.734+0000 7f6042ffd700 1 -- 172.21.15.94:0/1398666915 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f60480d0930 con 0x7f604c1013e0 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.734+0000 7f6042ffd700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] conn(0x7f60380aed30 0x7f60380b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.734+0000 7f6042ffd700 1 -- 172.21.15.94:0/1398666915 --> [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6048013e50 con 0x7f60380aed30 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.734+0000 7f6042ffd700 1 -- 172.21.15.94:0/1398666915 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f60480d0d80 con 0x7f604c1013e0 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.735+0000 7f6051186700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f60380a3cd0 0x7f60380a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f604c0771f0 tx=0x7f603c00b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.735+0000 7f6052188700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] conn(0x7f60380aed30 0x7f60380b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.735+0000 7f6052188700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] conn(0x7f60380aed30 0x7f60380b11d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.736+0000 7f6042ffd700 1 -- 172.21.15.94:0/1398666915 <== osd.2 v2:172.21.15.173:6800/4262691909 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6048013e50 con 0x7f60380aed30 2024-04-21T08:38:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.754+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 --> [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f604c063600 con 0x7f60380aed30 2024-04-21T08:38:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.754+0000 7f6042ffd700 1 -- 172.21.15.94:0/1398666915 <== osd.2 v2:172.21.15.173:6800/4262691909 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f604c063600 con 0x7f60380aed30 2024-04-21T08:38:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.754+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 >> [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] conn(0x7f60380aed30 msgr2=0x7f60380b11d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.754+0000 7f6053beb700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] conn(0x7f60380aed30 0x7f60380b11d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.754+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f60380a3cd0 msgr2=0x7f60380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.754+0000 7f6053beb700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f60380a3cd0 0x7f60380a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f604c0771f0 tx=0x7f603c00b5b0 comp rx=0 tx=0).stop 2024-04-21T08:38:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.754+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f604c1013e0 msgr2=0x7f604c199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.754+0000 7f6053beb700 1 --2- 172.21.15.94:0/1398666915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f604c1013e0 0x7f604c199020 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f604800d980 tx=0x7f604800dc90 comp rx=0 tx=0).stop 2024-04-21T08:38:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.754+0000 7f6052188700 1 -- 172.21.15.94:0/1398666915 reap_dead start 2024-04-21T08:38:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.754+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 shutdown_connections 2024-04-21T08:38:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.755+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 >> 172.21.15.94:0/1398666915 conn(0x7f604c076c60 msgr2=0x7f604c0ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.755+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 shutdown_connections 2024-04-21T08:38:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.755+0000 7f6053beb700 1 -- 172.21.15.94:0/1398666915 wait complete. 2024-04-21T08:38:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:32 smithi111 ceph-mon[31493]: pgmap v322: 129 pgs: 6 peering, 123 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 383 B/s wr, 5 op/s 2024-04-21T08:38:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:32 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:32 smithi094 ceph-mon[27988]: pgmap v322: 129 pgs: 6 peering, 123 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 383 B/s wr, 5 op/s 2024-04-21T08:38:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:32 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:32 smithi173 ceph-mon[31428]: pgmap v322: 129 pgs: 6 peering, 123 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 383 B/s wr, 5 op/s 2024-04-21T08:38:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:33.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:32 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:38:33.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.230+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3602684806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ed4101970 msgr2=0x7f3ed4101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.230+0000 7f3edc6a0700 1 --2- 172.21.15.94:0/3602684806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ed4101970 0x7f3ed4101df0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009990 tx=0x7f3ec4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.230+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3602684806 shutdown_connections 2024-04-21T08:38:33.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.230+0000 7f3edc6a0700 1 --2- 172.21.15.94:0/3602684806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed4102330 0x7f3ed410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.230+0000 7f3edc6a0700 1 --2- 172.21.15.94:0/3602684806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ed4107920 0x7f3ed4107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.230+0000 7f3edc6a0700 1 --2- 172.21.15.94:0/3602684806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ed4101970 0x7f3ed4101df0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.230+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3602684806 >> 172.21.15.94:0/3602684806 conn(0x7f3ed40788f0 msgr2=0x7f3ed4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.230+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3602684806 shutdown_connections 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.230+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3602684806 wait complete. 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3edc6a0700 1 Processor -- start 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3edc6a0700 1 -- start start 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3edc6a0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ed4101970 0x7f3ed4199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3edc6a0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ed4102330 0x7f3ed4199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3edc6a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed4107920 0x7f3ed419da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3edc6a0700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed4111b60 con 0x7f3ed4101970 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3edc6a0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed41119e0 con 0x7f3ed4107920 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3edc6a0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed4111ce0 con 0x7f3ed4102330 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3eda43c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ed4101970 0x7f3ed4199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3eda43c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ed4101970 0x7f3ed4199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47750/0 (socket says 172.21.15.94:47750) 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3eda43c700 1 -- 172.21.15.94:0/3349941778 learned_addr learned my addr 172.21.15.94:0/3349941778 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3ed9c3b700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ed4102330 0x7f3ed4199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3eda43c700 1 -- 172.21.15.94:0/3349941778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ed4102330 msgr2=0x7f3ed4199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3eda43c700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ed4102330 0x7f3ed4199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3eda43c700 1 -- 172.21.15.94:0/3349941778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed4107920 msgr2=0x7f3ed419da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3edac3d700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed4107920 0x7f3ed419da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3eda43c700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed4107920 0x7f3ed419da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3eda43c700 1 -- 172.21.15.94:0/3349941778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed419e140 con 0x7f3ed4101970 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3ed9c3b700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ed4102330 0x7f3ed4199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.231+0000 7f3eda43c700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ed4101970 0x7f3ed4199130 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000d950 tx=0x7f3ed000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.232+0000 7f3ecb7fe700 1 -- 172.21.15.94:0/3349941778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0016070 con 0x7f3ed4101970 2024-04-21T08:38:33.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.232+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed419e430 con 0x7f3ed4101970 2024-04-21T08:38:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.232+0000 7f3ecb7fe700 1 -- 172.21.15.94:0/3349941778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ed0010460 con 0x7f3ed4101970 2024-04-21T08:38:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.232+0000 7f3ecb7fe700 1 -- 172.21.15.94:0/3349941778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed000f620 con 0x7f3ed4101970 2024-04-21T08:38:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.232+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed41a5cb0 con 0x7f3ed4101970 2024-04-21T08:38:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.232+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3ed4063150 con 0x7f3ed4101970 2024-04-21T08:38:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.234+0000 7f3ecb7fe700 1 -- 172.21.15.94:0/3349941778 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3ed0019070 con 0x7f3ed4101970 2024-04-21T08:38:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.234+0000 7f3ecb7fe700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3ec00a3cd0 0x7f3ec00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.234+0000 7f3ed9c3b700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3ec00a3cd0 0x7f3ec00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.234+0000 7f3ecb7fe700 1 -- 172.21.15.94:0/3349941778 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f3ed00d09c0 con 0x7f3ed4101970 2024-04-21T08:38:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.235+0000 7f3ecb7fe700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] conn(0x7f3ec00aed30 0x7f3ec00b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.235+0000 7f3ecb7fe700 1 -- 172.21.15.94:0/3349941778 --> [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ec00b18a0 con 0x7f3ec00aed30 2024-04-21T08:38:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.235+0000 7f3ecb7fe700 1 -- 172.21.15.94:0/3349941778 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3ed00d0e10 con 0x7f3ed4101970 2024-04-21T08:38:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.235+0000 7f3edac3d700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] conn(0x7f3ec00aed30 0x7f3ec00b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:33.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.235+0000 7f3ed9c3b700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3ec00a3cd0 0x7f3ec00a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400b5c0 tx=0x7f3ec4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:33.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.235+0000 7f3edac3d700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] conn(0x7f3ec00aed30 0x7f3ec00b11d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:33.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.236+0000 7f3ecb7fe700 1 -- 172.21.15.94:0/3349941778 <== osd.0 v2:172.21.15.94:6802/676652749 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ec00b18a0 con 0x7f3ec00aed30 2024-04-21T08:38:33.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.252+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 --> [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3ed4063550 con 0x7f3ec00aed30 2024-04-21T08:38:33.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.252+0000 7f3ecb7fe700 1 -- 172.21.15.94:0/3349941778 <== osd.0 v2:172.21.15.94:6802/676652749 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3ed4063550 con 0x7f3ec00aed30 2024-04-21T08:38:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] conn(0x7f3ec00aed30 msgr2=0x7f3ec00b11d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edc6a0700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] conn(0x7f3ec00aed30 0x7f3ec00b11d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3ec00a3cd0 msgr2=0x7f3ec00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edc6a0700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3ec00a3cd0 0x7f3ec00a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400b5c0 tx=0x7f3ec4005fb0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ed4101970 msgr2=0x7f3ed4199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edc6a0700 1 --2- 172.21.15.94:0/3349941778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ed4101970 0x7f3ed4199130 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000d950 tx=0x7f3ed000dd10 comp rx=0 tx=0).stop 2024-04-21T08:38:33.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edac3d700 1 -- 172.21.15.94:0/3349941778 reap_dead start 2024-04-21T08:38:33.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 shutdown_connections 2024-04-21T08:38:33.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 >> 172.21.15.94:0/3349941778 conn(0x7f3ed40788f0 msgr2=0x7f3ed406b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:33.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 shutdown_connections 2024-04-21T08:38:33.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.253+0000 7f3edc6a0700 1 -- 172.21.15.94:0/3349941778 wait complete. 2024-04-21T08:38:33.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.659+0000 7f43645a3700 1 -- 172.21.15.94:0/3456862765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f435c102d60 msgr2=0x7f435c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.659+0000 7f43645a3700 1 --2- 172.21.15.94:0/3456862765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f435c102d60 0x7f435c1031e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f434c009990 tx=0x7f434c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.659+0000 7f43645a3700 1 -- 172.21.15.94:0/3456862765 shutdown_connections 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.659+0000 7f43645a3700 1 --2- 172.21.15.94:0/3456862765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f435c108bc0 0x7f435c108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.659+0000 7f43645a3700 1 --2- 172.21.15.94:0/3456862765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f435c103720 0x7f435c10f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.659+0000 7f43645a3700 1 --2- 172.21.15.94:0/3456862765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f435c102d60 0x7f435c1031e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.659+0000 7f43645a3700 1 -- 172.21.15.94:0/3456862765 >> 172.21.15.94:0/3456862765 conn(0x7f435c07a9d0 msgr2=0x7f435c07ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.659+0000 7f43645a3700 1 -- 172.21.15.94:0/3456862765 shutdown_connections 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.659+0000 7f43645a3700 1 -- 172.21.15.94:0/3456862765 wait complete. 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f43645a3700 1 Processor -- start 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f43645a3700 1 -- start start 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f43645a3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f435c102d60 0x7f435c199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f43645a3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f435c103720 0x7f435c1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f43645a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f435c108bc0 0x7f435c19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:33.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f43645a3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f435c112dc0 con 0x7f435c102d60 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f43645a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f435c112c40 con 0x7f435c108bc0 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f43645a3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f435c112f40 con 0x7f435c103720 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f436233f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f435c102d60 0x7f435c199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f436233f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f435c102d60 0x7f435c199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40390/0 (socket says 172.21.15.94:40390) 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f436233f700 1 -- 172.21.15.94:0/3765320356 learned_addr learned my addr 172.21.15.94:0/3765320356 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f4362b40700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f435c108bc0 0x7f435c19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f4361b3e700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f435c103720 0x7f435c1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f436233f700 1 -- 172.21.15.94:0/3765320356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f435c103720 msgr2=0x7f435c1997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f436233f700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f435c103720 0x7f435c1997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f436233f700 1 -- 172.21.15.94:0/3765320356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f435c108bc0 msgr2=0x7f435c19db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f436233f700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f435c108bc0 0x7f435c19db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f436233f700 1 -- 172.21.15.94:0/3765320356 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c19e2a0 con 0x7f435c102d60 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f4362b40700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f435c108bc0 0x7f435c19db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:38:33.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.660+0000 7f436233f700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f435c102d60 0x7f435c199290 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f435800b9c0 tx=0x7f435800bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:33.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.661+0000 7f43537fe700 1 -- 172.21.15.94:0/3765320356 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4358019070 con 0x7f435c102d60 2024-04-21T08:38:33.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.661+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435c19e590 con 0x7f435c102d60 2024-04-21T08:38:33.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.661+0000 7f43537fe700 1 -- 172.21.15.94:0/3765320356 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4358014440 con 0x7f435c102d60 2024-04-21T08:38:33.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.661+0000 7f43537fe700 1 -- 172.21.15.94:0/3765320356 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43580125a0 con 0x7f435c102d60 2024-04-21T08:38:33.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.661+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f435c1a5e10 con 0x7f435c102d60 2024-04-21T08:38:33.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.662+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f435c063150 con 0x7f435c102d60 2024-04-21T08:38:33.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.663+0000 7f43537fe700 1 -- 172.21.15.94:0/3765320356 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f435801c070 con 0x7f435c102d60 2024-04-21T08:38:33.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.664+0000 7f43537fe700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3cd0 0x7f43480a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:33.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.664+0000 7f4361b3e700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3cd0 0x7f43480a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:33.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.664+0000 7f43537fe700 1 -- 172.21.15.94:0/3765320356 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f43580d0400 con 0x7f435c102d60 2024-04-21T08:38:33.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.664+0000 7f43537fe700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] conn(0x7f43480aed30 0x7f43480b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:33.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.664+0000 7f43537fe700 1 -- 172.21.15.94:0/3765320356 --> [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4358012e80 con 0x7f43480aed30 2024-04-21T08:38:33.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.664+0000 7f43537fe700 1 -- 172.21.15.94:0/3765320356 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f43580d0850 con 0x7f435c102d60 2024-04-21T08:38:33.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.664+0000 7f4362b40700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] conn(0x7f43480aed30 0x7f43480b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:33.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.664+0000 7f4361b3e700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3cd0 0x7f43480a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f434c006010 tx=0x7f434c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:33.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.665+0000 7f4362b40700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] conn(0x7f43480aed30 0x7f43480b11d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:33.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.665+0000 7f43537fe700 1 -- 172.21.15.94:0/3765320356 <== osd.6 v2:172.21.15.94:6818/234088342 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4358012e80 con 0x7f43480aed30 2024-04-21T08:38:33.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.684+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 --> [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f435c063550 con 0x7f43480aed30 2024-04-21T08:38:33.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43537fe700 1 -- 172.21.15.94:0/3765320356 <== osd.6 v2:172.21.15.94:6818/234088342 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f435c063550 con 0x7f43480aed30 2024-04-21T08:38:33.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] conn(0x7f43480aed30 msgr2=0x7f43480b11d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43645a3700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] conn(0x7f43480aed30 0x7f43480b11d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:33.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3cd0 msgr2=0x7f43480a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43645a3700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f43480a3cd0 0x7f43480a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f434c006010 tx=0x7f434c01a040 comp rx=0 tx=0).stop 2024-04-21T08:38:33.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f435c102d60 msgr2=0x7f435c199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:33.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43645a3700 1 --2- 172.21.15.94:0/3765320356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f435c102d60 0x7f435c199290 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f435800b9c0 tx=0x7f435800bd80 comp rx=0 tx=0).stop 2024-04-21T08:38:33.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f4362b40700 1 -- 172.21.15.94:0/3765320356 reap_dead start 2024-04-21T08:38:33.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 shutdown_connections 2024-04-21T08:38:33.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 >> 172.21.15.94:0/3765320356 conn(0x7f435c07a9d0 msgr2=0x7f435c1019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:33.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 shutdown_connections 2024-04-21T08:38:33.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:33.685+0000 7f43645a3700 1 -- 172.21.15.94:0/3765320356 wait complete. 2024-04-21T08:38:34.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.113+0000 7f609dbf1700 1 -- 172.21.15.94:0/3489228637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6098101030 msgr2=0x7f6098101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:34.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.113+0000 7f609dbf1700 1 --2- 172.21.15.94:0/3489228637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6098101030 0x7f6098101490 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f608c009990 tx=0x7f608c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.114+0000 7f609dbf1700 1 -- 172.21.15.94:0/3489228637 shutdown_connections 2024-04-21T08:38:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.114+0000 7f609dbf1700 1 --2- 172.21.15.94:0/3489228637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6098108ad0 0x7f6098108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.114+0000 7f609dbf1700 1 --2- 172.21.15.94:0/3489228637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60981019d0 0x7f609810f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.114+0000 7f609dbf1700 1 --2- 172.21.15.94:0/3489228637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6098101030 0x7f6098101490 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.114+0000 7f609dbf1700 1 -- 172.21.15.94:0/3489228637 >> 172.21.15.94:0/3489228637 conn(0x7f6098077fb0 msgr2=0x7f60980ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.114+0000 7f609dbf1700 1 -- 172.21.15.94:0/3489228637 shutdown_connections 2024-04-21T08:38:34.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.114+0000 7f609dbf1700 1 -- 172.21.15.94:0/3489228637 wait complete. 2024-04-21T08:38:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.114+0000 7f609dbf1700 1 Processor -- start 2024-04-21T08:38:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f609dbf1700 1 -- start start 2024-04-21T08:38:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f609dbf1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6098101030 0x7f6098199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f609dbf1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60981019d0 0x7f6098199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f609dbf1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6098108ad0 0x7f609819d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f609dbf1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6098112c50 con 0x7f6098108ad0 2024-04-21T08:38:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f609dbf1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6098112ad0 con 0x7f60981019d0 2024-04-21T08:38:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f609dbf1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6098112dd0 con 0x7f6098101030 2024-04-21T08:38:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6097fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6098108ad0 0x7f609819d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6097fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6098108ad0 0x7f609819d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40418/0 (socket says 172.21.15.94:40418) 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6097fff700 1 -- 172.21.15.94:0/3114066783 learned_addr learned my addr 172.21.15.94:0/3114066783 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6096ffd700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60981019d0 0x7f6098199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6097fff700 1 -- 172.21.15.94:0/3114066783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6098101030 msgr2=0x7f6098199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f60977fe700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6098101030 0x7f6098199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6097fff700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6098101030 0x7f6098199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6097fff700 1 -- 172.21.15.94:0/3114066783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60981019d0 msgr2=0x7f6098199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6097fff700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60981019d0 0x7f6098199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6097fff700 1 -- 172.21.15.94:0/3114066783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609819e130 con 0x7f6098108ad0 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6096ffd700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60981019d0 0x7f6098199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.115+0000 7f6097fff700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6098108ad0 0x7f609819d9f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6090007ee0 tx=0x7f609000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.116+0000 7f6094ff9700 1 -- 172.21.15.94:0/3114066783 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609000ddd0 con 0x7f6098108ad0 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.116+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609819e420 con 0x7f6098108ad0 2024-04-21T08:38:34.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.116+0000 7f6094ff9700 1 -- 172.21.15.94:0/3114066783 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6090004020 con 0x7f6098108ad0 2024-04-21T08:38:34.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.116+0000 7f6094ff9700 1 -- 172.21.15.94:0/3114066783 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6090015660 con 0x7f6098108ad0 2024-04-21T08:38:34.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.116+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60981a5ca0 con 0x7f6098108ad0 2024-04-21T08:38:34.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.116+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f60980631d0 con 0x7f6098108ad0 2024-04-21T08:38:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.118+0000 7f6094ff9700 1 -- 172.21.15.94:0/3114066783 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f60900044f0 con 0x7f6098108ad0 2024-04-21T08:38:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.118+0000 7f6094ff9700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f60800a3cd0 0x7f60800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.119+0000 7f60977fe700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f60800a3cd0 0x7f60800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.119+0000 7f6094ff9700 1 -- 172.21.15.94:0/3114066783 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f60900d0f40 con 0x7f6098108ad0 2024-04-21T08:38:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.119+0000 7f6094ff9700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] conn(0x7f60800aed30 0x7f60800b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.119+0000 7f6094ff9700 1 -- 172.21.15.94:0/3114066783 --> [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6090012ec0 con 0x7f60800aed30 2024-04-21T08:38:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.119+0000 7f6094ff9700 1 -- 172.21.15.94:0/3114066783 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f60900d1390 con 0x7f6098108ad0 2024-04-21T08:38:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.119+0000 7f60977fe700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f60800a3cd0 0x7f60800a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6088005950 tx=0x7f6088005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.119+0000 7f6096ffd700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] conn(0x7f60800aed30 0x7f60800b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.119+0000 7f6096ffd700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] conn(0x7f60800aed30 0x7f60800b11d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.120+0000 7f6094ff9700 1 -- 172.21.15.94:0/3114066783 <== osd.4 v2:172.21.15.111:6808/4084294051 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6090012ec0 con 0x7f60800aed30 2024-04-21T08:38:34.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.136+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 --> [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6098063650 con 0x7f60800aed30 2024-04-21T08:38:34.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.137+0000 7f6094ff9700 1 -- 172.21.15.94:0/3114066783 <== osd.4 v2:172.21.15.111:6808/4084294051 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6098063650 con 0x7f60800aed30 2024-04-21T08:38:34.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.137+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 >> [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] conn(0x7f60800aed30 msgr2=0x7f60800b11d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:34.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.137+0000 7f609dbf1700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] conn(0x7f60800aed30 0x7f60800b11d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:34.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.137+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f60800a3cd0 msgr2=0x7f60800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:34.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.137+0000 7f609dbf1700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f60800a3cd0 0x7f60800a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6088005950 tx=0x7f6088005c70 comp rx=0 tx=0).stop 2024-04-21T08:38:34.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.137+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6098108ad0 msgr2=0x7f609819d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.137+0000 7f609dbf1700 1 --2- 172.21.15.94:0/3114066783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6098108ad0 0x7f609819d9f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6090007ee0 tx=0x7f609000d3b0 comp rx=0 tx=0).stop 2024-04-21T08:38:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.137+0000 7f6097fff700 1 -- 172.21.15.94:0/3114066783 reap_dead start 2024-04-21T08:38:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.137+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 shutdown_connections 2024-04-21T08:38:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.137+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 >> 172.21.15.94:0/3114066783 conn(0x7f6098077fb0 msgr2=0x7f60980ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.138+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 shutdown_connections 2024-04-21T08:38:34.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:34.138+0000 7f609dbf1700 1 -- 172.21.15.94:0/3114066783 wait complete. 2024-04-21T08:38:34.718 INFO:teuthology.orchestra.run.smithi094.stdout:34359738448 2024-04-21T08:38:34.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-21T08:38:34.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:34 smithi111 ceph-mon[31493]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 6 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:38:35.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:34 smithi094 ceph-mon[27988]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 6 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:38:35.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:34 smithi173 ceph-mon[31428]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 6 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:38:35.279 INFO:teuthology.orchestra.run.smithi094.stdout:107374182466 2024-04-21T08:38:35.280 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-21T08:38:35.748 INFO:teuthology.orchestra.run.smithi094.stdout:77309411399 2024-04-21T08:38:35.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-21T08:38:36.352 INFO:teuthology.orchestra.run.smithi094.stdout:253403070503 2024-04-21T08:38:36.353 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-21T08:38:36.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.840+0000 7f624cf4b700 1 -- 172.21.15.94:0/1279838164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6248108a50 msgr2=0x7f6248108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:36.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.840+0000 7f624cf4b700 1 --2- 172.21.15.94:0/1279838164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6248108a50 0x7f6248108e30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6238009990 tx=0x7f6238009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.840+0000 7f624cf4b700 1 -- 172.21.15.94:0/1279838164 shutdown_connections 2024-04-21T08:38:36.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.840+0000 7f624cf4b700 1 --2- 172.21.15.94:0/1279838164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6248103450 0x7f624810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.840+0000 7f624cf4b700 1 --2- 172.21.15.94:0/1279838164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6248108a50 0x7f6248108e30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.840+0000 7f624cf4b700 1 --2- 172.21.15.94:0/1279838164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6248102a90 0x7f6248102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.840+0000 7f624cf4b700 1 -- 172.21.15.94:0/1279838164 >> 172.21.15.94:0/1279838164 conn(0x7f6248079ba0 msgr2=0x7f6248079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:36.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.840+0000 7f624cf4b700 1 -- 172.21.15.94:0/1279838164 shutdown_connections 2024-04-21T08:38:36.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.840+0000 7f624cf4b700 1 -- 172.21.15.94:0/1279838164 wait complete. 2024-04-21T08:38:36.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f624cf4b700 1 Processor -- start 2024-04-21T08:38:36.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f624cf4b700 1 -- start start 2024-04-21T08:38:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f624cf4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6248102a90 0x7f6248074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f624cf4b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6248103450 0x7f6248075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f624cf4b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6248108a50 0x7f62480757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f624cf4b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6248112c00 con 0x7f6248103450 2024-04-21T08:38:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f624cf4b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6248112a80 con 0x7f6248102a90 2024-04-21T08:38:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f624cf4b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6248112d80 con 0x7f6248108a50 2024-04-21T08:38:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f6245d9b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6248103450 0x7f6248075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f6245d9b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6248103450 0x7f6248075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47744/0 (socket says 172.21.15.94:47744) 2024-04-21T08:38:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f6245d9b700 1 -- 172.21.15.94:0/1477995461 learned_addr learned my addr 172.21.15.94:0/1477995461 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f624659c700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6248102a90 0x7f6248074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.841+0000 7f6246d9d700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6248108a50 0x7f62480757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f6245d9b700 1 -- 172.21.15.94:0/1477995461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6248108a50 msgr2=0x7f62480757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f6245d9b700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6248108a50 0x7f62480757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f6245d9b700 1 -- 172.21.15.94:0/1477995461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6248102a90 msgr2=0x7f6248074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f6245d9b700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6248102a90 0x7f6248074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f6245d9b700 1 -- 172.21.15.94:0/1477995461 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62481a7640 con 0x7f6248103450 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f624659c700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6248102a90 0x7f6248074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f6245d9b700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6248103450 0x7f6248075280 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f623000d980 tx=0x7f623000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f623f7fe700 1 -- 172.21.15.94:0/1477995461 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6230017070 con 0x7f6248103450 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62481a7930 con 0x7f6248103450 2024-04-21T08:38:36.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f623f7fe700 1 -- 172.21.15.94:0/1477995461 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6230005020 con 0x7f6248103450 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62481a7e70 con 0x7f6248103450 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.842+0000 7f623f7fe700 1 -- 172.21.15.94:0/1477995461 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623000f9b0 con 0x7f6248103450 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.843+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6248063110 con 0x7f6248103450 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.844+0000 7f623f7fe700 1 -- 172.21.15.94:0/1477995461 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6230004a30 con 0x7f6248103450 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.845+0000 7f623f7fe700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62340a3cd0 0x7f62340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.845+0000 7f624659c700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62340a3cd0 0x7f62340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.845+0000 7f623f7fe700 1 -- 172.21.15.94:0/1477995461 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f62300d0ce0 con 0x7f6248103450 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.845+0000 7f623f7fe700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] conn(0x7f62340aed30 0x7f62340b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.845+0000 7f623f7fe700 1 -- 172.21.15.94:0/1477995461 --> [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62340b18a0 con 0x7f62340aed30 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.845+0000 7f623f7fe700 1 -- 172.21.15.94:0/1477995461 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f62300d1130 con 0x7f6248103450 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.845+0000 7f624659c700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62340a3cd0 0x7f62340a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f623800b2e0 tx=0x7f623801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.845+0000 7f6246d9d700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] conn(0x7f62340aed30 0x7f62340b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.846+0000 7f6246d9d700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] conn(0x7f62340aed30 0x7f62340b11d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:36.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.846+0000 7f623f7fe700 1 -- 172.21.15.94:0/1477995461 <== osd.5 v2:172.21.15.173:6808/2394839325 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62340b18a0 con 0x7f62340aed30 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.863+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 --> [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6248063590 con 0x7f62340aed30 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.863+0000 7f623f7fe700 1 -- 172.21.15.94:0/1477995461 <== osd.5 v2:172.21.15.173:6808/2394839325 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6248063590 con 0x7f62340aed30 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 >> [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] conn(0x7f62340aed30 msgr2=0x7f62340b11d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f624cf4b700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] conn(0x7f62340aed30 0x7f62340b11d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62340a3cd0 msgr2=0x7f62340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f624cf4b700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62340a3cd0 0x7f62340a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f623800b2e0 tx=0x7f623801a040 comp rx=0 tx=0).stop 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6248103450 msgr2=0x7f6248075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f624cf4b700 1 --2- 172.21.15.94:0/1477995461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6248103450 0x7f6248075280 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f623000d980 tx=0x7f623000dc90 comp rx=0 tx=0).stop 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f6246d9d700 1 -- 172.21.15.94:0/1477995461 reap_dead start 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 shutdown_connections 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 >> 172.21.15.94:0/1477995461 conn(0x7f6248079ba0 msgr2=0x7f6248101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 shutdown_connections 2024-04-21T08:38:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:32.864+0000 7f624cf4b700 1 -- 172.21.15.94:0/1477995461 wait complete. 2024-04-21T08:38:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:36 smithi111 ceph-mon[31493]: pgmap v324: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T08:38:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:36 smithi094 ceph-mon[27988]: pgmap v324: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T08:38:37.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:36 smithi173 ceph-mon[31428]: pgmap v324: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T08:38:37.390 INFO:teuthology.orchestra.run.smithi094.stdout:141733920828 2024-04-21T08:38:37.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-21T08:38:37.778 INFO:teuthology.orchestra.run.smithi094.stdout:55834574923 2024-04-21T08:38:37.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-21T08:38:38.240 INFO:teuthology.orchestra.run.smithi094.stdout:197568495666 2024-04-21T08:38:38.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-21T08:38:38.312 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:38.802 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:38 smithi094 ceph-mon[27988]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T08:38:38.828 INFO:teuthology.orchestra.run.smithi094.stdout:163208757304 2024-04-21T08:38:38.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-21T08:38:38.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:38 smithi111 ceph-mon[31493]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T08:38:39.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:38 smithi173 ceph-mon[31428]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T08:38:39.301 INFO:teuthology.orchestra.run.smithi094.stdout:219043332140 2024-04-21T08:38:39.302 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-21T08:38:39.945 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:39.945 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:39.946 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:39.946 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:40.229 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:40.229 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:40.726 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:40 smithi094 ceph-mon[27988]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 58 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T08:38:40.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:40 smithi111 ceph-mon[31493]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 58 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T08:38:41.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:40 smithi173 ceph-mon[31428]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 58 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T08:38:42.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:42 smithi094 ceph-mon[27988]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 53 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:38:43.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:42 smithi173 ceph-mon[31428]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 53 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:38:43.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:42 smithi111 ceph-mon[31493]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 53 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:38:44.865 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:44 smithi094 ceph-mon[27988]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:38:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 -- 172.21.15.94:0/3298782823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b8102a90 msgr2=0x7f16b8102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 --2- 172.21.15.94:0/3298782823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b8102a90 0x7f16b8102f10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009990 tx=0x7f16a0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 -- 172.21.15.94:0/3298782823 shutdown_connections 2024-04-21T08:38:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 --2- 172.21.15.94:0/3298782823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8108a50 0x7f16b8108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 --2- 172.21.15.94:0/3298782823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16b8103450 0x7f16b810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 --2- 172.21.15.94:0/3298782823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b8102a90 0x7f16b8102f10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 -- 172.21.15.94:0/3298782823 >> 172.21.15.94:0/3298782823 conn(0x7f16b8079ba0 msgr2=0x7f16b8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 -- 172.21.15.94:0/3298782823 shutdown_connections 2024-04-21T08:38:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 -- 172.21.15.94:0/3298782823 wait complete. 2024-04-21T08:38:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 Processor -- start 2024-04-21T08:38:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 -- start start 2024-04-21T08:38:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16b8102a90 0x7f16b8199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8103450 0x7f16b8199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b8108a50 0x7f16b819d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f16b8112c20 con 0x7f16b8108a50 2024-04-21T08:38:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f16b8112aa0 con 0x7f16b8103450 2024-04-21T08:38:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.943+0000 7f16bdea1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f16b8112da0 con 0x7f16b8102a90 2024-04-21T08:38:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b8108a50 0x7f16b819d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:44.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b8108a50 0x7f16b819d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45076/0 (socket says 172.21.15.94:45076) 2024-04-21T08:38:44.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b7fff700 1 -- 172.21.15.94:0/3296642978 learned_addr learned my addr 172.21.15.94:0/3296642978 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:44.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b77fe700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16b8102a90 0x7f16b8199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:44.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b7fff700 1 -- 172.21.15.94:0/3296642978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16b8102a90 msgr2=0x7f16b8199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:44.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b7fff700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16b8102a90 0x7f16b8199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:44.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b7fff700 1 -- 172.21.15.94:0/3296642978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8103450 msgr2=0x7f16b8199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:44.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b7fff700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8103450 0x7f16b8199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:44.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b7fff700 1 -- 172.21.15.94:0/3296642978 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b819e120 con 0x7f16b8108a50 2024-04-21T08:38:44.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b7fff700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b8108a50 0x7f16b819d9e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f16b000ef30 tx=0x7f16b000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:44.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b57fa700 1 -- 172.21.15.94:0/3296642978 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b00092e0 con 0x7f16b8108a50 2024-04-21T08:38:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b57fa700 1 -- 172.21.15.94:0/3296642978 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b0004bb0 con 0x7f16b8108a50 2024-04-21T08:38:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16bdea1700 1 -- 172.21.15.94:0/3296642978 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b819e3b0 con 0x7f16b8108a50 2024-04-21T08:38:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16b57fa700 1 -- 172.21.15.94:0/3296642978 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0004280 con 0x7f16b8108a50 2024-04-21T08:38:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.944+0000 7f16bdea1700 1 -- 172.21.15.94:0/3296642978 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b81a5c90 con 0x7f16b8108a50 2024-04-21T08:38:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.945+0000 7f16bdea1700 1 -- 172.21.15.94:0/3296642978 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b8069730 con 0x7f16b8108a50 2024-04-21T08:38:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.946+0000 7f16b57fa700 1 -- 172.21.15.94:0/3296642978 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f16b001b030 con 0x7f16b8108a50 2024-04-21T08:38:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.947+0000 7f16b57fa700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16a40a3cd0 0x7f16a40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.947+0000 7f16b77fe700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16a40a3cd0 0x7f16a40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.947+0000 7f16b57fa700 1 -- 172.21.15.94:0/3296642978 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f16b00d0db0 con 0x7f16b8108a50 2024-04-21T08:38:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.947+0000 7f16b77fe700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16a40a3cd0 0x7f16a40a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f16a8005950 tx=0x7f16a800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:44.951+0000 7f16b57fa700 1 -- 172.21.15.94:0/3296642978 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16b0022020 con 0x7f16b8108a50 2024-04-21T08:38:45.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:44 smithi173 ceph-mon[31428]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:38:45.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.155+0000 7f16bdea1700 1 -- 172.21.15.94:0/3296642978 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f16b802df80 con 0x7f16b8108a50 2024-04-21T08:38:45.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.156+0000 7f16b57fa700 1 -- 172.21.15.94:0/3296642978 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f16b0010040 con 0x7f16b8108a50 2024-04-21T08:38:45.156 INFO:teuthology.orchestra.run.smithi094.stdout:107374182468 2024-04-21T08:38:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 -- 172.21.15.94:0/3296642978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16a40a3cd0 msgr2=0x7f16a40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16a40a3cd0 0x7f16a40a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f16a8005950 tx=0x7f16a800b410 comp rx=0 tx=0).stop 2024-04-21T08:38:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 -- 172.21.15.94:0/3296642978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b8108a50 msgr2=0x7f16b819d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b8108a50 0x7f16b819d9e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f16b000ef30 tx=0x7f16b000c3a0 comp rx=0 tx=0).stop 2024-04-21T08:38:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 -- 172.21.15.94:0/3296642978 shutdown_connections 2024-04-21T08:38:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16a40a3cd0 0x7f16a40a6190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b8108a50 0x7f16b819d9e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16b8102a90 0x7f16b8199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 --2- 172.21.15.94:0/3296642978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8103450 0x7f16b8199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 -- 172.21.15.94:0/3296642978 >> 172.21.15.94:0/3296642978 conn(0x7f16b8079ba0 msgr2=0x7f16b8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:45.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 -- 172.21.15.94:0/3296642978 shutdown_connections 2024-04-21T08:38:45.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:45.158+0000 7f16bdea1700 1 -- 172.21.15.94:0/3296642978 wait complete. 2024-04-21T08:38:45.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:44 smithi111 ceph-mon[31493]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:38:45.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:45 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3296642978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:38:46.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:45 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3296642978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:38:46.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:45 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3296642978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:38:46.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.255+0000 7fc2575b7700 1 -- 172.21.15.94:0/2580447314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc250103760 msgr2=0x7fc250103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:46.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.255+0000 7fc2575b7700 1 --2- 172.21.15.94:0/2580447314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc250103760 0x7fc250103b40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009990 tx=0x7fc24c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:46.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 -- 172.21.15.94:0/2580447314 shutdown_connections 2024-04-21T08:38:46.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 --2- 172.21.15.94:0/2580447314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc250102330 0x7fc25010e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:46.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 --2- 172.21.15.94:0/2580447314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc250103760 0x7fc250103b40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:46.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 --2- 172.21.15.94:0/2580447314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc250101970 0x7fc250101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:46.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 -- 172.21.15.94:0/2580447314 >> 172.21.15.94:0/2580447314 conn(0x7fc2500779b0 msgr2=0x7fc250077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:46.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 -- 172.21.15.94:0/2580447314 shutdown_connections 2024-04-21T08:38:46.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 -- 172.21.15.94:0/2580447314 wait complete. 2024-04-21T08:38:46.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 Processor -- start 2024-04-21T08:38:46.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 -- start start 2024-04-21T08:38:46.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc250101970 0x7fc250199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:46.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.256+0000 7fc2575b7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc250102330 0x7fc25019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc2575b7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc250103760 0x7fc2501a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc2575b7700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc250111bd0 con 0x7fc250102330 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc2575b7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc250111a50 con 0x7fc250103760 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc2575b7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc250111d50 con 0x7fc250101970 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc254b52700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc250102330 0x7fc25019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc254b52700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc250102330 0x7fc25019a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45088/0 (socket says 172.21.15.94:45088) 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc254b52700 1 -- 172.21.15.94:0/1124859598 learned_addr learned my addr 172.21.15.94:0/1124859598 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc255353700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc250101970 0x7fc250199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc255b54700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc250103760 0x7fc2501a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc254b52700 1 -- 172.21.15.94:0/1124859598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc250101970 msgr2=0x7fc250199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc254b52700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc250101970 0x7fc250199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc254b52700 1 -- 172.21.15.94:0/1124859598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc250103760 msgr2=0x7fc2501a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:46.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc254b52700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc250103760 0x7fc2501a4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:46.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc254b52700 1 -- 172.21.15.94:0/1124859598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2501a5510 con 0x7fc250102330 2024-04-21T08:38:46.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc255b54700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc250103760 0x7fc2501a4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:38:46.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc254b52700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc250102330 0x7fc25019a490 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc24000c920 tx=0x7fc24000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:46.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc2467fc700 1 -- 172.21.15.94:0/1124859598 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240007b00 con 0x7fc250102330 2024-04-21T08:38:46.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc2467fc700 1 -- 172.21.15.94:0/1124859598 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc240004ce0 con 0x7fc250102330 2024-04-21T08:38:46.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc2467fc700 1 -- 172.21.15.94:0/1124859598 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240005550 con 0x7fc250102330 2024-04-21T08:38:46.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2501a5740 con 0x7fc250102330 2024-04-21T08:38:46.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.257+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2501a5c50 con 0x7fc250102330 2024-04-21T08:38:46.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.258+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc250107090 con 0x7fc250102330 2024-04-21T08:38:46.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.259+0000 7fc2467fc700 1 -- 172.21.15.94:0/1124859598 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc240007ca0 con 0x7fc250102330 2024-04-21T08:38:46.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.260+0000 7fc2467fc700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc23c0a3cd0 0x7fc23c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:46.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.260+0000 7fc255353700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc23c0a3cd0 0x7fc23c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.260+0000 7fc2467fc700 1 -- 172.21.15.94:0/1124859598 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7fc2400d0910 con 0x7fc250102330 2024-04-21T08:38:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.260+0000 7fc2467fc700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] conn(0x7fc23c0aed30 0x7fc23c0b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.260+0000 7fc2467fc700 1 -- 172.21.15.94:0/1124859598 --> [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc23c0b18a0 con 0x7fc23c0aed30 2024-04-21T08:38:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.260+0000 7fc2467fc700 1 -- 172.21.15.94:0/1124859598 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fc2400d0d60 con 0x7fc250102330 2024-04-21T08:38:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.260+0000 7fc255353700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc23c0a3cd0 0x7fc23c0a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00b5c0 tx=0x7fc24c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.261+0000 7fc255b54700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] conn(0x7fc23c0aed30 0x7fc23c0b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.261+0000 7fc255b54700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] conn(0x7fc23c0aed30 0x7fc23c0b11d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.262+0000 7fc2467fc700 1 -- 172.21.15.94:0/1124859598 <== osd.10 v2:172.21.15.111:6824/176065870 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc23c0b18a0 con 0x7fc23c0aed30 2024-04-21T08:38:46.276 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:46.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.283+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 --> [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc250103060 con 0x7fc23c0aed30 2024-04-21T08:38:46.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.284+0000 7fc2467fc700 1 -- 172.21.15.94:0/1124859598 <== osd.10 v2:172.21.15.111:6824/176065870 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc250103060 con 0x7fc23c0aed30 2024-04-21T08:38:46.284 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:46.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.284+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 >> [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] conn(0x7fc23c0aed30 msgr2=0x7fc23c0b11d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:46.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.284+0000 7fc2575b7700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] conn(0x7fc23c0aed30 0x7fc23c0b11d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:46.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.285+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc23c0a3cd0 msgr2=0x7fc23c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:46.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.285+0000 7fc2575b7700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc23c0a3cd0 0x7fc23c0a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00b5c0 tx=0x7fc24c005960 comp rx=0 tx=0).stop 2024-04-21T08:38:46.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.285+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc250102330 msgr2=0x7fc25019a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:46.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.285+0000 7fc2575b7700 1 --2- 172.21.15.94:0/1124859598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc250102330 0x7fc25019a490 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc24000c920 tx=0x7fc24000cc30 comp rx=0 tx=0).stop 2024-04-21T08:38:46.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.285+0000 7fc255b54700 1 -- 172.21.15.94:0/1124859598 reap_dead start 2024-04-21T08:38:46.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.285+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 shutdown_connections 2024-04-21T08:38:46.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.285+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 >> 172.21.15.94:0/1124859598 conn(0x7fc2500779b0 msgr2=0x7fc250106630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:46.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.286+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 shutdown_connections 2024-04-21T08:38:46.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:46.286+0000 7fc2575b7700 1 -- 172.21.15.94:0/1124859598 wait complete. 2024-04-21T08:38:47.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:46 smithi094 ceph-mon[27988]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-21T08:38:47.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:46 smithi173 ceph-mon[31428]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-21T08:38:47.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:46 smithi111 ceph-mon[31493]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-21T08:38:49.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:48 smithi173 ceph-mon[31428]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:38:49.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:48 smithi094 ceph-mon[27988]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:38:49.153 INFO:teuthology.orchestra.run.smithi094.stdout:309237645345 2024-04-21T08:38:49.154 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-21T08:38:49.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:48 smithi111 ceph-mon[31493]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:38:50.788 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:50 smithi094 ceph-mon[27988]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:38:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.010+0000 7f330f76e700 1 -- 172.21.15.94:0/503655294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3308106500 msgr2=0x7f330810e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.010+0000 7f330f76e700 1 --2- 172.21.15.94:0/503655294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3308106500 0x7f330810e040 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f330000b0a0 tx=0x7f330000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.011+0000 7f330f76e700 1 -- 172.21.15.94:0/503655294 shutdown_connections 2024-04-21T08:38:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.011+0000 7f330f76e700 1 --2- 172.21.15.94:0/503655294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3308106500 0x7f330810e040 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.011+0000 7f330f76e700 1 --2- 172.21.15.94:0/503655294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3308101660 0x7f3308101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.011+0000 7f330f76e700 1 --2- 172.21.15.94:0/503655294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308102010 0x7f3308105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.011+0000 7f330f76e700 1 -- 172.21.15.94:0/503655294 >> 172.21.15.94:0/503655294 conn(0x7f3308076c60 msgr2=0x7f3308077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.011+0000 7f330f76e700 1 -- 172.21.15.94:0/503655294 shutdown_connections 2024-04-21T08:38:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.011+0000 7f330f76e700 1 -- 172.21.15.94:0/503655294 wait complete. 2024-04-21T08:38:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.011+0000 7f330f76e700 1 Processor -- start 2024-04-21T08:38:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.011+0000 7f330f76e700 1 -- start start 2024-04-21T08:38:51.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330f76e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308101660 0x7f3308199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:51.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330f76e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3308102010 0x7f3308199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:51.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330f76e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3308106500 0x7f330819d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:51.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330f76e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3308110ab0 con 0x7f3308102010 2024-04-21T08:38:51.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330f76e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3308110930 con 0x7f3308101660 2024-04-21T08:38:51.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330f76e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3308110c30 con 0x7f3308106500 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330cd09700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3308102010 0x7f3308199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330cd09700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3308102010 0x7f3308199570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45106/0 (socket says 172.21.15.94:45106) 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330cd09700 1 -- 172.21.15.94:0/401350837 learned_addr learned my addr 172.21.15.94:0/401350837 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330d50a700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308101660 0x7f3308199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330dd0b700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3308106500 0x7f330819d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330cd09700 1 -- 172.21.15.94:0/401350837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3308106500 msgr2=0x7f330819d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330cd09700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3308106500 0x7f330819d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330cd09700 1 -- 172.21.15.94:0/401350837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308101660 msgr2=0x7f3308199030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330cd09700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308101660 0x7f3308199030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330cd09700 1 -- 172.21.15.94:0/401350837 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330819dff0 con 0x7f3308102010 2024-04-21T08:38:51.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330dd0b700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3308106500 0x7f330819d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:38:51.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330cd09700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3308102010 0x7f3308199570 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f32f800c9b0 tx=0x7f32f800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:51.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f32fe7fc700 1 -- 172.21.15.94:0/401350837 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8007aa0 con 0x7f3308102010 2024-04-21T08:38:51.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330f76e700 1 -- 172.21.15.94:0/401350837 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f330819e2e0 con 0x7f3308102010 2024-04-21T08:38:51.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f32fe7fc700 1 -- 172.21.15.94:0/401350837 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32f8007c40 con 0x7f3308102010 2024-04-21T08:38:51.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f32fe7fc700 1 -- 172.21.15.94:0/401350837 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f80186f0 con 0x7f3308102010 2024-04-21T08:38:51.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.012+0000 7f330f76e700 1 -- 172.21.15.94:0/401350837 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33081a5b60 con 0x7f3308102010 2024-04-21T08:38:51.016 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:50 smithi173 ceph-mon[31428]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:38:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.013+0000 7f330f76e700 1 -- 172.21.15.94:0/401350837 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3308069730 con 0x7f3308102010 2024-04-21T08:38:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.014+0000 7f32fe7fc700 1 -- 172.21.15.94:0/401350837 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f32f800f450 con 0x7f3308102010 2024-04-21T08:38:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.015+0000 7f32fe7fc700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f32f40a3cd0 0x7f32f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.015+0000 7f330d50a700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f32f40a3cd0 0x7f32f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.015+0000 7f32fe7fc700 1 -- 172.21.15.94:0/401350837 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f32f80d0810 con 0x7f3308102010 2024-04-21T08:38:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.015+0000 7f330d50a700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f32f40a3cd0 0x7f32f40a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3304005950 tx=0x7f33040058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:51.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.020+0000 7f32fe7fc700 1 -- 172.21.15.94:0/401350837 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f32f8014030 con 0x7f3308102010 2024-04-21T08:38:51.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:50 smithi111 ceph-mon[31493]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:38:51.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.222+0000 7f330f76e700 1 -- 172.21.15.94:0/401350837 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f330802df80 con 0x7f3308102010 2024-04-21T08:38:51.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.222+0000 7f32fe7fc700 1 -- 172.21.15.94:0/401350837 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f32f809cd60 con 0x7f3308102010 2024-04-21T08:38:51.221 INFO:teuthology.orchestra.run.smithi094.stdout:55834574927 2024-04-21T08:38:51.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.224+0000 7f330f76e700 1 -- 172.21.15.94:0/401350837 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f32f40a3cd0 msgr2=0x7f32f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:51.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.224+0000 7f330f76e700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f32f40a3cd0 0x7f32f40a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3304005950 tx=0x7f33040058e0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.224+0000 7f330f76e700 1 -- 172.21.15.94:0/401350837 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3308102010 msgr2=0x7f3308199570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.224+0000 7f330f76e700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3308102010 0x7f3308199570 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f32f800c9b0 tx=0x7f32f800cd70 comp rx=0 tx=0).stop 2024-04-21T08:38:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.225+0000 7f330f76e700 1 -- 172.21.15.94:0/401350837 shutdown_connections 2024-04-21T08:38:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.225+0000 7f330f76e700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f32f40a3cd0 0x7f32f40a6190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.225+0000 7f330f76e700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308101660 0x7f3308199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.225+0000 7f330f76e700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3308106500 0x7f330819d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.225+0000 7f330f76e700 1 --2- 172.21.15.94:0/401350837 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3308102010 0x7f3308199570 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.225+0000 7f330f76e700 1 -- 172.21.15.94:0/401350837 >> 172.21.15.94:0/401350837 conn(0x7f3308076c60 msgr2=0x7f33080ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.225+0000 7f330f76e700 1 -- 172.21.15.94:0/401350837 shutdown_connections 2024-04-21T08:38:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.225+0000 7f330f76e700 1 -- 172.21.15.94:0/401350837 wait complete. 2024-04-21T08:38:51.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.247+0000 7fb79eaa8700 1 -- 172.21.15.94:0/2158564550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb798101030 msgr2=0x7fb798101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:51.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.247+0000 7fb79eaa8700 1 --2- 172.21.15.94:0/2158564550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb798101030 0x7fb798101490 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb784009990 tx=0x7fb784009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.247+0000 7fb79eaa8700 1 -- 172.21.15.94:0/2158564550 shutdown_connections 2024-04-21T08:38:51.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.247+0000 7fb79eaa8700 1 --2- 172.21.15.94:0/2158564550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb798108ad0 0x7fb798108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.247+0000 7fb79eaa8700 1 --2- 172.21.15.94:0/2158564550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7981019d0 0x7fb79810f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.247+0000 7fb79eaa8700 1 --2- 172.21.15.94:0/2158564550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb798101030 0x7fb798101490 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.247+0000 7fb79eaa8700 1 -- 172.21.15.94:0/2158564550 >> 172.21.15.94:0/2158564550 conn(0x7fb798077fb0 msgr2=0x7fb7980ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.248+0000 7fb79eaa8700 1 -- 172.21.15.94:0/2158564550 shutdown_connections 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.248+0000 7fb79eaa8700 1 -- 172.21.15.94:0/2158564550 wait complete. 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.248+0000 7fb79eaa8700 1 Processor -- start 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.248+0000 7fb79eaa8700 1 -- start start 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.248+0000 7fb79eaa8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb798101030 0x7fb798102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.248+0000 7fb79eaa8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7981019d0 0x7fb798102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.248+0000 7fb79eaa8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb798108ad0 0x7fb798102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.248+0000 7fb79eaa8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb798112c20 con 0x7fb798108ad0 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.248+0000 7fb79eaa8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb798112aa0 con 0x7fb7981019d0 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.248+0000 7fb79eaa8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb798112da0 con 0x7fb798101030 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79d045700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb798108ad0 0x7fb798102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79d045700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb798108ad0 0x7fb798102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45118/0 (socket says 172.21.15.94:45118) 2024-04-21T08:38:51.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79d045700 1 -- 172.21.15.94:0/1895667138 learned_addr learned my addr 172.21.15.94:0/1895667138 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb797fff700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7981019d0 0x7fb798102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79d045700 1 -- 172.21.15.94:0/1895667138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb798101030 msgr2=0x7fb798102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79d045700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb798101030 0x7fb798102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79d045700 1 -- 172.21.15.94:0/1895667138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7981019d0 msgr2=0x7fb798102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79d045700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7981019d0 0x7fb798102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79d045700 1 -- 172.21.15.94:0/1895667138 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7981a7640 con 0x7fb798108ad0 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79d045700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb798108ad0 0x7fb798102c90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb79000ada0 tx=0x7fb79000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb795ffb700 1 -- 172.21.15.94:0/1895667138 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb790017020 con 0x7fb798108ad0 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb795ffb700 1 -- 172.21.15.94:0/1895667138 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb79000ce30 con 0x7fb798108ad0 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79eaa8700 1 -- 172.21.15.94:0/1895667138 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7981a7930 con 0x7fb798108ad0 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb795ffb700 1 -- 172.21.15.94:0/1895667138 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb790005350 con 0x7fb798108ad0 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.249+0000 7fb79eaa8700 1 -- 172.21.15.94:0/1895667138 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7981a7e40 con 0x7fb798108ad0 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.250+0000 7fb79eaa8700 1 -- 172.21.15.94:0/1895667138 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb798104350 con 0x7fb798108ad0 2024-04-21T08:38:51.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.251+0000 7fb795ffb700 1 -- 172.21.15.94:0/1895667138 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb79000c980 con 0x7fb798108ad0 2024-04-21T08:38:51.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.252+0000 7fb795ffb700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7880a3cd0 0x7fb7880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:51.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.252+0000 7fb79c844700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7880a3cd0 0x7fb7880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:51.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.252+0000 7fb795ffb700 1 -- 172.21.15.94:0/1895667138 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7fb7900cff90 con 0x7fb798108ad0 2024-04-21T08:38:51.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.253+0000 7fb79c844700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7880a3cd0 0x7fb7880a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00a8f0 tx=0x7fb78c005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:51.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.259+0000 7fb795ffb700 1 -- 172.21.15.94:0/1895667138 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb790021020 con 0x7fb798108ad0 2024-04-21T08:38:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.473+0000 7fb79eaa8700 1 -- 172.21.15.94:0/1895667138 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fb79802df80 con 0x7fb798108ad0 2024-04-21T08:38:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.474+0000 7fb795ffb700 1 -- 172.21.15.94:0/1895667138 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fb79009c570 con 0x7fb798108ad0 2024-04-21T08:38:51.473 INFO:teuthology.orchestra.run.smithi094.stdout:343597383709 2024-04-21T08:38:51.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 -- 172.21.15.94:0/1895667138 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7880a3cd0 msgr2=0x7fb7880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:51.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7880a3cd0 0x7fb7880a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00a8f0 tx=0x7fb78c005bf0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 -- 172.21.15.94:0/1895667138 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb798108ad0 msgr2=0x7fb798102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:51.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb798108ad0 0x7fb798102c90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb79000ada0 tx=0x7fb79000c330 comp rx=0 tx=0).stop 2024-04-21T08:38:51.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 -- 172.21.15.94:0/1895667138 shutdown_connections 2024-04-21T08:38:51.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb7880a3cd0 0x7fb7880a6190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb798108ad0 0x7fb798102c90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb798101030 0x7fb798102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 --2- 172.21.15.94:0/1895667138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7981019d0 0x7fb798102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:51.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 -- 172.21.15.94:0/1895667138 >> 172.21.15.94:0/1895667138 conn(0x7fb798077fb0 msgr2=0x7fb7980ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:51.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 -- 172.21.15.94:0/1895667138 shutdown_connections 2024-04-21T08:38:51.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:51.476+0000 7fb79eaa8700 1 -- 172.21.15.94:0/1895667138 wait complete. 2024-04-21T08:38:51.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:51 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/401350837' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:38:51.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:51 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1895667138' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:38:51.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:51 smithi094 ceph-mon[27988]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:38:52.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:51 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/401350837' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:38:52.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:51 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1895667138' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:38:52.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:51 smithi173 ceph-mon[31428]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:38:52.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:51 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/401350837' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:38:52.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:51 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1895667138' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:38:52.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:51 smithi111 ceph-mon[31493]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:38:53.368 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383705 got 343597383709 for osd.11 2024-04-21T08:38:53.368 DEBUG:teuthology.parallel:result is None 2024-04-21T08:38:53.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 -- 172.21.15.94:0/3124467868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0107990 msgr2=0x7f65a0107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:53.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 --2- 172.21.15.94:0/3124467868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0107990 0x7f65a0107d70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f659c00b0a0 tx=0x7f659c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 -- 172.21.15.94:0/3124467868 shutdown_connections 2024-04-21T08:38:53.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 --2- 172.21.15.94:0/3124467868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0102300 0x7f65a010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 --2- 172.21.15.94:0/3124467868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0107990 0x7f65a0107d70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 --2- 172.21.15.94:0/3124467868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a0101940 0x7f65a0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 -- 172.21.15.94:0/3124467868 >> 172.21.15.94:0/3124467868 conn(0x7f65a00788f0 msgr2=0x7f65a0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 -- 172.21.15.94:0/3124467868 shutdown_connections 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 -- 172.21.15.94:0/3124467868 wait complete. 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 Processor -- start 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.525+0000 7f65a8409700 1 -- start start 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a8409700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0101940 0x7f65a0199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a8409700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a0102300 0x7f65a01996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a8409700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0107990 0x7f65a019da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a8409700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f65a0111b60 con 0x7f65a0107990 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a8409700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01119e0 con 0x7f65a0101940 2024-04-21T08:38:53.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a8409700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65a0111ce0 con 0x7f65a0102300 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a69a6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0107990 0x7f65a019da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a69a6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0107990 0x7f65a019da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45136/0 (socket says 172.21.15.94:45136) 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a69a6700 1 -- 172.21.15.94:0/1252884586 learned_addr learned my addr 172.21.15.94:0/1252884586 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a61a5700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0101940 0x7f65a0199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a59a4700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a0102300 0x7f65a01996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a69a6700 1 -- 172.21.15.94:0/1252884586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a0102300 msgr2=0x7f65a01996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a69a6700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a0102300 0x7f65a01996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a69a6700 1 -- 172.21.15.94:0/1252884586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0101940 msgr2=0x7f65a0199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a69a6700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0101940 0x7f65a0199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a69a6700 1 -- 172.21.15.94:0/1252884586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a019e170 con 0x7f65a0107990 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a69a6700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0107990 0x7f65a019da30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f659800b550 tx=0x7f659800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65977fe700 1 -- 172.21.15.94:0/1252884586 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598015020 con 0x7f65a0107990 2024-04-21T08:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65a8409700 1 -- 172.21.15.94:0/1252884586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a019e460 con 0x7f65a0107990 2024-04-21T08:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65977fe700 1 -- 172.21.15.94:0/1252884586 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6598011ce0 con 0x7f65a0107990 2024-04-21T08:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.526+0000 7f65977fe700 1 -- 172.21.15.94:0/1252884586 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598010d40 con 0x7f65a0107990 2024-04-21T08:38:53.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.527+0000 7f65a8409700 1 -- 172.21.15.94:0/1252884586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a01a5ce0 con 0x7f65a0107990 2024-04-21T08:38:53.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.528+0000 7f65a8409700 1 -- 172.21.15.94:0/1252884586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a0069730 con 0x7f65a0107990 2024-04-21T08:38:53.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.528+0000 7f65977fe700 1 -- 172.21.15.94:0/1252884586 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6598018020 con 0x7f65a0107990 2024-04-21T08:38:53.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.529+0000 7f65977fe700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f658c0a3cd0 0x7f658c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:53.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.529+0000 7f65a61a5700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f658c0a3cd0 0x7f658c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:53.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.529+0000 7f65977fe700 1 -- 172.21.15.94:0/1252884586 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f65980cfe30 con 0x7f65a0107990 2024-04-21T08:38:53.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.529+0000 7f65a61a5700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f658c0a3cd0 0x7f658c0a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f659c006130 tx=0x7f659c0060a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:53.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.534+0000 7f65977fe700 1 -- 172.21.15.94:0/1252884586 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f659801f030 con 0x7f65a0107990 2024-04-21T08:38:53.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.740+0000 7f65a8409700 1 -- 172.21.15.94:0/1252884586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f65a01a5ee0 con 0x7f65a0107990 2024-04-21T08:38:53.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.741+0000 7f65977fe700 1 -- 172.21.15.94:0/1252884586 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f659809c490 con 0x7f65a0107990 2024-04-21T08:38:53.740 INFO:teuthology.orchestra.run.smithi094.stdout:253403070509 2024-04-21T08:38:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 -- 172.21.15.94:0/1252884586 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f658c0a3cd0 msgr2=0x7f658c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f658c0a3cd0 0x7f658c0a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f659c006130 tx=0x7f659c0060a0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 -- 172.21.15.94:0/1252884586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0107990 msgr2=0x7f65a019da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0107990 0x7f65a019da30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f659800b550 tx=0x7f659800b910 comp rx=0 tx=0).stop 2024-04-21T08:38:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 -- 172.21.15.94:0/1252884586 shutdown_connections 2024-04-21T08:38:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f658c0a3cd0 0x7f658c0a6190 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0101940 0x7f65a0199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0107990 0x7f65a019da30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 --2- 172.21.15.94:0/1252884586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a0102300 0x7f65a01996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:53.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 -- 172.21.15.94:0/1252884586 >> 172.21.15.94:0/1252884586 conn(0x7f65a00788f0 msgr2=0x7f65a0105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:53.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 -- 172.21.15.94:0/1252884586 shutdown_connections 2024-04-21T08:38:53.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:53.743+0000 7f65a8409700 1 -- 172.21.15.94:0/1252884586 wait complete. 2024-04-21T08:38:54.904 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:54 smithi094 ceph-mon[27988]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:38:54.904 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:54 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1252884586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:38:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:54 smithi111 ceph-mon[31493]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:38:54.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:54 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1252884586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:38:54.941 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070503 got 253403070509 for osd.8 2024-04-21T08:38:54.941 DEBUG:teuthology.parallel:result is None 2024-04-21T08:38:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:54 smithi173 ceph-mon[31428]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:38:55.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:54 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1252884586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:38:55.338 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.128+0000 7f7b440d9700 1 -- 172.21.15.94:0/830830990 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b3c102300 msgr2=0x7f7b3c10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.128+0000 7f7b440d9700 1 --2- 172.21.15.94:0/830830990 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b3c102300 0x7f7b3c10e7f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400b0a0 tx=0x7f7b3400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.128+0000 7f7b440d9700 1 -- 172.21.15.94:0/830830990 shutdown_connections 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.128+0000 7f7b440d9700 1 --2- 172.21.15.94:0/830830990 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b3c102300 0x7f7b3c10e7f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.128+0000 7f7b440d9700 1 --2- 172.21.15.94:0/830830990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c107990 0x7f7b3c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.128+0000 7f7b440d9700 1 --2- 172.21.15.94:0/830830990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c101940 0x7f7b3c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.128+0000 7f7b440d9700 1 -- 172.21.15.94:0/830830990 >> 172.21.15.94:0/830830990 conn(0x7f7b3c0788f0 msgr2=0x7f7b3c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.128+0000 7f7b440d9700 1 -- 172.21.15.94:0/830830990 shutdown_connections 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.128+0000 7f7b440d9700 1 -- 172.21.15.94:0/830830990 wait complete. 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b440d9700 1 Processor -- start 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b440d9700 1 -- start start 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b440d9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b3c101940 0x7f7b3c199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b440d9700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c102300 0x7f7b3c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:55.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b440d9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c107990 0x7f7b3c1a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b440d9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c111b00 con 0x7f7b3c101940 2024-04-21T08:38:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b440d9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c111980 con 0x7f7b3c107990 2024-04-21T08:38:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b440d9700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c111c80 con 0x7f7b3c102300 2024-04-21T08:38:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b41e75700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b3c101940 0x7f7b3c199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b41e75700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b3c101940 0x7f7b3c199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35964/0 (socket says 172.21.15.94:35964) 2024-04-21T08:38:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b41e75700 1 -- 172.21.15.94:0/2813405280 learned_addr learned my addr 172.21.15.94:0/2813405280 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b42676700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c107990 0x7f7b3c1a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.129+0000 7f7b41674700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c102300 0x7f7b3c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:55.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b41e75700 1 -- 172.21.15.94:0/2813405280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c102300 msgr2=0x7f7b3c19a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b41e75700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c102300 0x7f7b3c19a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b41e75700 1 -- 172.21.15.94:0/2813405280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c107990 msgr2=0x7f7b3c1a5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b41e75700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c107990 0x7f7b3c1a5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b41e75700 1 -- 172.21.15.94:0/2813405280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3c1a5540 con 0x7f7b3c101940 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b41e75700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b3c101940 0x7f7b3c199f80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800d980 tx=0x7f7b3800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b32ffd700 1 -- 172.21.15.94:0/2813405280 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38017070 con 0x7f7b3c101940 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b32ffd700 1 -- 172.21.15.94:0/2813405280 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b38005020 con 0x7f7b3c101940 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b440d9700 1 -- 172.21.15.94:0/2813405280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3c1a5710 con 0x7f7b3c101940 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b32ffd700 1 -- 172.21.15.94:0/2813405280 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3800fa10 con 0x7f7b3c101940 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.130+0000 7f7b440d9700 1 -- 172.21.15.94:0/2813405280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b3c1a5c20 con 0x7f7b3c101940 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.131+0000 7f7b440d9700 1 -- 172.21.15.94:0/2813405280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b3c109eb0 con 0x7f7b3c101940 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.132+0000 7f7b32ffd700 1 -- 172.21.15.94:0/2813405280 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7b38004a30 con 0x7f7b3c101940 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.133+0000 7f7b32ffd700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7b280a3d50 0x7f7b280a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:55.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.133+0000 7f7b41674700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7b280a3d50 0x7f7b280a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.133+0000 7f7b32ffd700 1 -- 172.21.15.94:0/2813405280 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f7b380d0a70 con 0x7f7b3c101940 2024-04-21T08:38:55.444 INFO:teuthology.orchestra.run.smithi094.stdout:77309411403 2024-04-21T08:38:55.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.133+0000 7f7b41674700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7b280a3d50 0x7f7b280a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c078d10 tx=0x7f7b2c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.136+0000 7f7b32ffd700 1 -- 172.21.15.94:0/2813405280 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7b38020020 con 0x7f7b3c101940 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.343+0000 7f7b440d9700 1 -- 172.21.15.94:0/2813405280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7b3c069730 con 0x7f7b3c101940 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.344+0000 7f7b32ffd700 1 -- 172.21.15.94:0/2813405280 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7b3c069730 con 0x7f7b3c101940 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 -- 172.21.15.94:0/2813405280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7b280a3d50 msgr2=0x7f7b280a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7b280a3d50 0x7f7b280a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c078d10 tx=0x7f7b2c00a300 comp rx=0 tx=0).stop 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 -- 172.21.15.94:0/2813405280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b3c101940 msgr2=0x7f7b3c199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b3c101940 0x7f7b3c199f80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800d980 tx=0x7f7b3800dc90 comp rx=0 tx=0).stop 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 -- 172.21.15.94:0/2813405280 shutdown_connections 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7b280a3d50 0x7f7b280a6210 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c107990 0x7f7b3c1a5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b3c101940 0x7f7b3c199f80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 --2- 172.21.15.94:0/2813405280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b3c102300 0x7f7b3c19a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:55.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 -- 172.21.15.94:0/2813405280 >> 172.21.15.94:0/2813405280 conn(0x7f7b3c0788f0 msgr2=0x7f7b3c0ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 -- 172.21.15.94:0/2813405280 shutdown_connections 2024-04-21T08:38:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:55.346+0000 7f7b440d9700 1 -- 172.21.15.94:0/2813405280 wait complete. 2024-04-21T08:38:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:55.913 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:55 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2813405280' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:38:55.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:55 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2813405280' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:38:56.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:55 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2813405280' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:38:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:56 smithi111 ceph-mon[31493]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:38:56.986 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:56 smithi094 ceph-mon[27988]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:38:57.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:56 smithi173 ceph-mon[31428]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:38:58.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.685+0000 7fada5fb8700 1 -- 172.21.15.94:0/2691891182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fada0107990 msgr2=0x7fada0107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.685+0000 7fada5fb8700 1 --2- 172.21.15.94:0/2691891182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fada0107990 0x7fada0107d70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fad90009990 tx=0x7fad90009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.685+0000 7fada5fb8700 1 -- 172.21.15.94:0/2691891182 shutdown_connections 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.685+0000 7fada5fb8700 1 --2- 172.21.15.94:0/2691891182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fada0102300 0x7fada010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.685+0000 7fada5fb8700 1 --2- 172.21.15.94:0/2691891182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fada0107990 0x7fada0107d70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.685+0000 7fada5fb8700 1 --2- 172.21.15.94:0/2691891182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada0101940 0x7fada0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.685+0000 7fada5fb8700 1 -- 172.21.15.94:0/2691891182 >> 172.21.15.94:0/2691891182 conn(0x7fada00788f0 msgr2=0x7fada0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.685+0000 7fada5fb8700 1 -- 172.21.15.94:0/2691891182 shutdown_connections 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.685+0000 7fada5fb8700 1 -- 172.21.15.94:0/2691891182 wait complete. 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.685+0000 7fada5fb8700 1 Processor -- start 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fada5fb8700 1 -- start start 2024-04-21T08:38:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fada5fb8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fada0101940 0x7fada0199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fada5fb8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada0102300 0x7fada0199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fada5fb8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fada0107990 0x7fada019d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fada5fb8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fada0111b60 con 0x7fada0107990 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fada5fb8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fada01119e0 con 0x7fada0102300 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fada5fb8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fada0111ce0 con 0x7fada0101940 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fad9ffff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fada0107990 0x7fada019d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fad9ffff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fada0107990 0x7fada019d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35974/0 (socket says 172.21.15.94:35974) 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fad9ffff700 1 -- 172.21.15.94:0/2928328437 learned_addr learned my addr 172.21.15.94:0/2928328437 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fad9f7fe700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fada0101940 0x7fada0199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fad9effd700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada0102300 0x7fada0199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fad9ffff700 1 -- 172.21.15.94:0/2928328437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fada0101940 msgr2=0x7fada0199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fad9ffff700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fada0101940 0x7fada0199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fad9ffff700 1 -- 172.21.15.94:0/2928328437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada0102300 msgr2=0x7fada0199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:58.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fad9ffff700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada0102300 0x7fada0199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.686+0000 7fad9ffff700 1 -- 172.21.15.94:0/2928328437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada019e120 con 0x7fada0107990 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.687+0000 7fad9ffff700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fada0107990 0x7fada019d9e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fad9800b970 tx=0x7fad9800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.687+0000 7fad9cff9700 1 -- 172.21.15.94:0/2928328437 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9800d860 con 0x7fada0107990 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.687+0000 7fad9cff9700 1 -- 172.21.15.94:0/2928328437 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad9800e430 con 0x7fada0107990 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.687+0000 7fada5fb8700 1 -- 172.21.15.94:0/2928328437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada019e410 con 0x7fada0107990 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.687+0000 7fad9cff9700 1 -- 172.21.15.94:0/2928328437 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad98017660 con 0x7fada0107990 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.687+0000 7fada5fb8700 1 -- 172.21.15.94:0/2928328437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada01a5c90 con 0x7fada0107990 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.688+0000 7fada5fb8700 1 -- 172.21.15.94:0/2928328437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fada0069730 con 0x7fada0107990 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.690+0000 7fad9cff9700 1 -- 172.21.15.94:0/2928328437 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fad9800da00 con 0x7fada0107990 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.690+0000 7fad9cff9700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fad880a3cd0 0x7fad880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.690+0000 7fad9f7fe700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fad880a3cd0 0x7fad880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.690+0000 7fad9cff9700 1 -- 172.21.15.94:0/2928328437 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7fad980d0840 con 0x7fada0107990 2024-04-21T08:38:58.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.691+0000 7fad9f7fe700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fad880a3cd0 0x7fad880a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fad900098e0 tx=0x7fad90005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:58.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.694+0000 7fad9cff9700 1 -- 172.21.15.94:0/2928328437 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad98013030 con 0x7fada0107990 2024-04-21T08:38:58.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.898+0000 7fada5fb8700 1 -- 172.21.15.94:0/2928328437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fada01a5ec0 con 0x7fada0107990 2024-04-21T08:38:58.897 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:58 smithi094 ceph-mon[27988]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:38:58.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.898+0000 7fad9cff9700 1 -- 172.21.15.94:0/2928328437 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fad9809cd90 con 0x7fada0107990 2024-04-21T08:38:58.897 INFO:teuthology.orchestra.run.smithi094.stdout:34359738453 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 -- 172.21.15.94:0/2928328437 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fad880a3cd0 msgr2=0x7fad880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fad880a3cd0 0x7fad880a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fad900098e0 tx=0x7fad90005960 comp rx=0 tx=0).stop 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 -- 172.21.15.94:0/2928328437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fada0107990 msgr2=0x7fada019d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fada0107990 0x7fada019d9e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fad9800b970 tx=0x7fad9800bc80 comp rx=0 tx=0).stop 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 -- 172.21.15.94:0/2928328437 shutdown_connections 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fad880a3cd0 0x7fad880a6190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fada0107990 0x7fada019d9e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fada0101940 0x7fada0199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 --2- 172.21.15.94:0/2928328437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada0102300 0x7fada0199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 -- 172.21.15.94:0/2928328437 >> 172.21.15.94:0/2928328437 conn(0x7fada00788f0 msgr2=0x7fada0105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 -- 172.21.15.94:0/2928328437 shutdown_connections 2024-04-21T08:38:58.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:58.901+0000 7fada5fb8700 1 -- 172.21.15.94:0/2928328437 wait complete. 2024-04-21T08:38:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:58 smithi111 ceph-mon[31493]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:38:59.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:58 smithi173 ceph-mon[31428]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:38:59.129 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182466 got 107374182468 for osd.3 2024-04-21T08:38:59.130 DEBUG:teuthology.parallel:result is None 2024-04-21T08:38:59.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 -- 172.21.15.94:0/204156553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1778102ac0 msgr2=0x7f1778102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:59.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 --2- 172.21.15.94:0/204156553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1778102ac0 0x7f1778102f40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1768009990 tx=0x7f1768009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 -- 172.21.15.94:0/204156553 shutdown_connections 2024-04-21T08:38:59.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 --2- 172.21.15.94:0/204156553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1778103480 0x7f177810f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 --2- 172.21.15.94:0/204156553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1778108ab0 0x7f1778108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 --2- 172.21.15.94:0/204156553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1778102ac0 0x7f1778102f40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 -- 172.21.15.94:0/204156553 >> 172.21.15.94:0/204156553 conn(0x7f1778079bf0 msgr2=0x7f177807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:59.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 -- 172.21.15.94:0/204156553 shutdown_connections 2024-04-21T08:38:59.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 -- 172.21.15.94:0/204156553 wait complete. 2024-04-21T08:38:59.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 Processor -- start 2024-04-21T08:38:59.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.762+0000 7f177e702700 1 -- start start 2024-04-21T08:38:59.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f177e702700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1778102ac0 0x7f1778074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:59.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f177e702700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1778103480 0x7f1778075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:59.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f177e702700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1778108ab0 0x7f17780757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:59.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f177e702700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1778112ca0 con 0x7f1778102ac0 2024-04-21T08:38:59.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f177e702700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1778112b20 con 0x7f1778108ab0 2024-04-21T08:38:59.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f177e702700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1778112e20 con 0x7f1778103480 2024-04-21T08:38:59.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f1777fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1778102ac0 0x7f1778074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:59.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f1777fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1778102ac0 0x7f1778074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36020/0 (socket says 172.21.15.94:36020) 2024-04-21T08:38:59.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f1777fff700 1 -- 172.21.15.94:0/607007275 learned_addr learned my addr 172.21.15.94:0/607007275 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:38:59.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f177cc9f700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1778108ab0 0x7f17780757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:59.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f17777fe700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1778103480 0x7f1778075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:59.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f1777fff700 1 -- 172.21.15.94:0/607007275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1778103480 msgr2=0x7f1778075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:59.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f1777fff700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1778103480 0x7f1778075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f1777fff700 1 -- 172.21.15.94:0/607007275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1778108ab0 msgr2=0x7f17780757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:59.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f1777fff700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1778108ab0 0x7f17780757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f1777fff700 1 -- 172.21.15.94:0/607007275 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17781a76b0 con 0x7f1778102ac0 2024-04-21T08:38:59.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f17777fe700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1778103480 0x7f1778075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:38:59.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.763+0000 7f177cc9f700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1778108ab0 0x7f17780757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:38:59.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.764+0000 7f1777fff700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1778102ac0 0x7f1778074d40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f176c00d950 tx=0x7f176c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:59.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.764+0000 7f17757fa700 1 -- 172.21.15.94:0/607007275 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c016070 con 0x7f1778102ac0 2024-04-21T08:38:59.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.764+0000 7f177e702700 1 -- 172.21.15.94:0/607007275 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17781a79a0 con 0x7f1778102ac0 2024-04-21T08:38:59.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.764+0000 7f17757fa700 1 -- 172.21.15.94:0/607007275 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f176c009df0 con 0x7f1778102ac0 2024-04-21T08:38:59.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.764+0000 7f17757fa700 1 -- 172.21.15.94:0/607007275 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c00f3c0 con 0x7f1778102ac0 2024-04-21T08:38:59.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.764+0000 7f177e702700 1 -- 172.21.15.94:0/607007275 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17781a7f30 con 0x7f1778102ac0 2024-04-21T08:38:59.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.765+0000 7f177e702700 1 -- 172.21.15.94:0/607007275 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1778069730 con 0x7f1778102ac0 2024-04-21T08:38:59.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.766+0000 7f17757fa700 1 -- 172.21.15.94:0/607007275 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f176c019070 con 0x7f1778102ac0 2024-04-21T08:38:59.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.766+0000 7f17757fa700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f17600a3c40 0x7f17600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:59.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.767+0000 7f17777fe700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f17600a3c40 0x7f17600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:59.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.767+0000 7f17757fa700 1 -- 172.21.15.94:0/607007275 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f176c0d0270 con 0x7f1778102ac0 2024-04-21T08:38:59.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.767+0000 7f17777fe700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f17600a3c40 0x7f17600a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f176800b5c0 tx=0x7f1768005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.771+0000 7f17757fa700 1 -- 172.21.15.94:0/607007275 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f176c09c920 con 0x7f1778102ac0 2024-04-21T08:38:59.884 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:38:59 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2928328437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:38:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:38:59 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2928328437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:38:59.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.982+0000 7f177e702700 1 -- 172.21.15.94:0/607007275 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f1778076220 con 0x7f1778102ac0 2024-04-21T08:38:59.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.982+0000 7f17757fa700 1 -- 172.21.15.94:0/607007275 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f176c09c220 con 0x7f1778102ac0 2024-04-21T08:38:59.981 INFO:teuthology.orchestra.run.smithi094.stdout:197568495671 2024-04-21T08:38:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.984+0000 7f177e702700 1 -- 172.21.15.94:0/607007275 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f17600a3c40 msgr2=0x7f17600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.984+0000 7f177e702700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f17600a3c40 0x7f17600a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f176800b5c0 tx=0x7f1768005fb0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.984+0000 7f177e702700 1 -- 172.21.15.94:0/607007275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1778102ac0 msgr2=0x7f1778074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.984+0000 7f177e702700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1778102ac0 0x7f1778074d40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f176c00d950 tx=0x7f176c00dc60 comp rx=0 tx=0).stop 2024-04-21T08:38:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.985+0000 7f177e702700 1 -- 172.21.15.94:0/607007275 shutdown_connections 2024-04-21T08:38:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.985+0000 7f177e702700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f17600a3c40 0x7f17600a6100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.985+0000 7f177e702700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1778108ab0 0x7f17780757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.985+0000 7f177e702700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1778102ac0 0x7f1778074d40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.985+0000 7f177e702700 1 --2- 172.21.15.94:0/607007275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1778103480 0x7f1778075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.985+0000 7f177e702700 1 -- 172.21.15.94:0/607007275 >> 172.21.15.94:0/607007275 conn(0x7f1778079bf0 msgr2=0x7f1778100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.985+0000 7f177e702700 1 -- 172.21.15.94:0/607007275 shutdown_connections 2024-04-21T08:38:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.985+0000 7f177e702700 1 -- 172.21.15.94:0/607007275 wait complete. 2024-04-21T08:39:00.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:38:59 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2928328437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:39:00.255 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574927 for osd.1 2024-04-21T08:39:00.255 DEBUG:teuthology.parallel:result is None 2024-04-21T08:39:00.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.657+0000 7f4b111fe700 1 -- 172.21.15.94:0/2617087778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b0c107990 msgr2=0x7f4b0c107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:00.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.657+0000 7f4b111fe700 1 --2- 172.21.15.94:0/2617087778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b0c107990 0x7f4b0c107d70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009990 tx=0x7f4afc009ca0 comp rx=0 tx=0).stop 2024-04-21T08:39:00.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.657+0000 7f4b111fe700 1 -- 172.21.15.94:0/2617087778 shutdown_connections 2024-04-21T08:39:00.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.657+0000 7f4b111fe700 1 --2- 172.21.15.94:0/2617087778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b0c102300 0x7f4b0c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.657+0000 7f4b111fe700 1 --2- 172.21.15.94:0/2617087778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b0c107990 0x7f4b0c107d70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.657+0000 7f4b111fe700 1 --2- 172.21.15.94:0/2617087778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b0c101940 0x7f4b0c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.657+0000 7f4b111fe700 1 -- 172.21.15.94:0/2617087778 >> 172.21.15.94:0/2617087778 conn(0x7f4b0c0788f0 msgr2=0x7f4b0c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.657+0000 7f4b111fe700 1 -- 172.21.15.94:0/2617087778 shutdown_connections 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.657+0000 7f4b111fe700 1 -- 172.21.15.94:0/2617087778 wait complete. 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.658+0000 7f4b111fe700 1 Processor -- start 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.658+0000 7f4b111fe700 1 -- start start 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b111fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b0c101940 0x7f4b0c199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b111fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b0c102300 0x7f4b0c19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b111fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b0c107990 0x7f4b0c1a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:00.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b111fe700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c111ae0 con 0x7f4b0c102300 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b111fe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c111960 con 0x7f4b0c101940 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b111fe700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c111c60 con 0x7f4b0c107990 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0a59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b0c102300 0x7f4b0c19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0a59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b0c102300 0x7f4b0c19a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36006/0 (socket says 172.21.15.94:36006) 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0a59c700 1 -- 172.21.15.94:0/3095721440 learned_addr learned my addr 172.21.15.94:0/3095721440 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0ad9d700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b0c101940 0x7f4b0c199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0b59e700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b0c107990 0x7f4b0c1a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0a59c700 1 -- 172.21.15.94:0/3095721440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b0c107990 msgr2=0x7f4b0c1a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0a59c700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b0c107990 0x7f4b0c1a4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0a59c700 1 -- 172.21.15.94:0/3095721440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b0c101940 msgr2=0x7f4b0c199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:00.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0a59c700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b0c101940 0x7f4b0c199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0a59c700 1 -- 172.21.15.94:0/3095721440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0c1a5520 con 0x7f4b0c102300 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0ad9d700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b0c101940 0x7f4b0c199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4b0a59c700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b0c102300 0x7f4b0c19a4a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4b0000d980 tx=0x7f4b0000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.659+0000 7f4af3fff700 1 -- 172.21.15.94:0/3095721440 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b00016070 con 0x7f4b0c102300 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.660+0000 7f4b111fe700 1 -- 172.21.15.94:0/3095721440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0c1a5750 con 0x7f4b0c102300 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.660+0000 7f4b111fe700 1 -- 172.21.15.94:0/3095721440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b0c1a5cb0 con 0x7f4b0c102300 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.660+0000 7f4af3fff700 1 -- 172.21.15.94:0/3095721440 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b000044d0 con 0x7f4b0c102300 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.661+0000 7f4af3fff700 1 -- 172.21.15.94:0/3095721440 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b00004ca0 con 0x7f4b0c102300 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.661+0000 7f4b111fe700 1 -- 172.21.15.94:0/3095721440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af8005330 con 0x7f4b0c102300 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.662+0000 7f4af3fff700 1 -- 172.21.15.94:0/3095721440 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4b0000bb60 con 0x7f4b0c102300 2024-04-21T08:39:00.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.662+0000 7f4af3fff700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4af40a3cd0 0x7f4af40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:00.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.663+0000 7f4b0ad9d700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4af40a3cd0 0x7f4af40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:00.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.663+0000 7f4af3fff700 1 -- 172.21.15.94:0/3095721440 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f4b000d0b80 con 0x7f4b0c102300 2024-04-21T08:39:00.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.663+0000 7f4b0ad9d700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4af40a3cd0 0x7f4af40a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4afc006010 tx=0x7f4afc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:00.577 INFO:teuthology.orchestra.run.smithi094.stdout:141733920833 2024-04-21T08:39:00.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.670+0000 7f4af3fff700 1 -- 172.21.15.94:0/3095721440 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b0001f020 con 0x7f4b0c102300 2024-04-21T08:39:00.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.692+0000 7f4af3fff700 1 -- 172.21.15.94:0/3095721440 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4b00099040 con 0x7f4b0c102300 2024-04-21T08:39:00.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.885+0000 7f4b111fe700 1 -- 172.21.15.94:0/3095721440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f4af80058f0 con 0x7f4b0c102300 2024-04-21T08:39:00.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.885+0000 7f4af3fff700 1 -- 172.21.15.94:0/3095721440 <== mon.0 v2:172.21.15.94:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4b00003680 con 0x7f4b0c102300 2024-04-21T08:39:00.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.887+0000 7f4b111fe700 1 -- 172.21.15.94:0/3095721440 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4af40a3cd0 msgr2=0x7f4af40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:00.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.887+0000 7f4b111fe700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4af40a3cd0 0x7f4af40a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4afc006010 tx=0x7f4afc01a040 comp rx=0 tx=0).stop 2024-04-21T08:39:00.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.887+0000 7f4b111fe700 1 -- 172.21.15.94:0/3095721440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b0c102300 msgr2=0x7f4b0c19a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:00.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.887+0000 7f4b111fe700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b0c102300 0x7f4b0c19a4a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4b0000d980 tx=0x7f4b0000dd40 comp rx=0 tx=0).stop 2024-04-21T08:39:00.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.887+0000 7f4b111fe700 1 -- 172.21.15.94:0/3095721440 shutdown_connections 2024-04-21T08:39:00.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.887+0000 7f4b111fe700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4af40a3cd0 0x7f4af40a6190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:00.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.887+0000 7f4b111fe700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b0c101940 0x7f4b0c199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:00.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.887+0000 7f4b111fe700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b0c107990 0x7f4b0c1a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:00.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.887+0000 7f4b111fe700 1 --2- 172.21.15.94:0/3095721440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b0c102300 0x7f4b0c19a4a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:00.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.887+0000 7f4b111fe700 1 -- 172.21.15.94:0/3095721440 >> 172.21.15.94:0/3095721440 conn(0x7f4b0c0788f0 msgr2=0x7f4b0c100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:00.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.888+0000 7f4b111fe700 1 -- 172.21.15.94:0/3095721440 shutdown_connections 2024-04-21T08:39:00.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:38:59.888+0000 7f4b111fe700 1 -- 172.21.15.94:0/3095721440 wait complete. 2024-04-21T08:39:00.901 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:00 smithi094 ceph-mon[27988]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-21T08:39:00.901 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:00 smithi094 ceph-mon[27988]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-21T08:39:00.901 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3095721440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:39:00.901 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/607007275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:39:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:00 smithi111 ceph-mon[31493]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-21T08:39:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:00 smithi111 ceph-mon[31493]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-21T08:39:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3095721440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:39:00.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/607007275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:39:01.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:00 smithi173 ceph-mon[31428]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-21T08:39:01.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:00 smithi173 ceph-mon[31428]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-21T08:39:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3095721440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:39:01.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/607007275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:39:01.233 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738448 got 34359738453 for osd.0 2024-04-21T08:39:01.233 DEBUG:teuthology.parallel:result is None 2024-04-21T08:39:01.958 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411403 for osd.2 2024-04-21T08:39:01.958 DEBUG:teuthology.parallel:result is None 2024-04-21T08:39:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.651+0000 7f101535d700 1 -- 172.21.15.94:0/2674739536 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1010107990 msgr2=0x7f1010107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.651+0000 7f101535d700 1 --2- 172.21.15.94:0/2674739536 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1010107990 0x7f1010107d70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1000009990 tx=0x7f1000009ca0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.651+0000 7f101535d700 1 -- 172.21.15.94:0/2674739536 shutdown_connections 2024-04-21T08:39:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.651+0000 7f101535d700 1 --2- 172.21.15.94:0/2674739536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1010102300 0x7f101010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.651+0000 7f101535d700 1 --2- 172.21.15.94:0/2674739536 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1010107990 0x7f1010107d70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.651+0000 7f101535d700 1 --2- 172.21.15.94:0/2674739536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1010101940 0x7f1010101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.651+0000 7f101535d700 1 -- 172.21.15.94:0/2674739536 >> 172.21.15.94:0/2674739536 conn(0x7f10100788f0 msgr2=0x7f1010078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:02.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.651+0000 7f101535d700 1 -- 172.21.15.94:0/2674739536 shutdown_connections 2024-04-21T08:39:02.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.651+0000 7f101535d700 1 -- 172.21.15.94:0/2674739536 wait complete. 2024-04-21T08:39:02.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.652+0000 7f101535d700 1 Processor -- start 2024-04-21T08:39:02.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.652+0000 7f101535d700 1 -- start start 2024-04-21T08:39:02.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.652+0000 7f101535d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1010101940 0x7f1010199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.652+0000 7f101535d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1010102300 0x7f10101998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.652+0000 7f101535d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1010107990 0x7f101019d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.652+0000 7f101535d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1010111ae0 con 0x7f1010101940 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.652+0000 7f101535d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1010111960 con 0x7f1010102300 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.652+0000 7f101535d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1010111c60 con 0x7f1010107990 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.652+0000 7f100effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1010101940 0x7f1010199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.652+0000 7f100effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1010101940 0x7f1010199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36038/0 (socket says 172.21.15.94:36038) 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f100f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1010107990 0x7f101019d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f100effd700 1 -- 172.21.15.94:0/1712378823 learned_addr learned my addr 172.21.15.94:0/1712378823 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f100e7fc700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1010102300 0x7f10101998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f100effd700 1 -- 172.21.15.94:0/1712378823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1010107990 msgr2=0x7f101019d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f100effd700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1010107990 0x7f101019d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f100effd700 1 -- 172.21.15.94:0/1712378823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1010102300 msgr2=0x7f10101998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f100effd700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1010102300 0x7f10101998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f100effd700 1 -- 172.21.15.94:0/1712378823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101019e080 con 0x7f1010101940 2024-04-21T08:39:02.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f100effd700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1010101940 0x7f1010199390 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1000000c00 tx=0x7f10000047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:02.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f0ff7fff700 1 -- 172.21.15.94:0/1712378823 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000027070 con 0x7f1010101940 2024-04-21T08:39:02.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f101535d700 1 -- 172.21.15.94:0/1712378823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101019e310 con 0x7f1010101940 2024-04-21T08:39:02.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f0ff7fff700 1 -- 172.21.15.94:0/1712378823 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1000003fe0 con 0x7f1010101940 2024-04-21T08:39:02.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f0ff7fff700 1 -- 172.21.15.94:0/1712378823 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000022e30 con 0x7f1010101940 2024-04-21T08:39:02.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.653+0000 7f101535d700 1 -- 172.21.15.94:0/1712378823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10101a5bf0 con 0x7f1010101940 2024-04-21T08:39:02.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.654+0000 7f101535d700 1 -- 172.21.15.94:0/1712378823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1010069730 con 0x7f1010101940 2024-04-21T08:39:02.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.655+0000 7f0ff7fff700 1 -- 172.21.15.94:0/1712378823 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f100002a070 con 0x7f1010101940 2024-04-21T08:39:02.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.656+0000 7f0ff7fff700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ff80a3c40 0x7f0ff80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:02.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.656+0000 7f100e7fc700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ff80a3c40 0x7f0ff80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:02.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.656+0000 7f0ff7fff700 1 -- 172.21.15.94:0/1712378823 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f10000d1470 con 0x7f1010101940 2024-04-21T08:39:02.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.657+0000 7f100e7fc700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ff80a3c40 0x7f0ff80a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f10101013e0 tx=0x7f100400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:02.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.661+0000 7f0ff7fff700 1 -- 172.21.15.94:0/1712378823 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1000013070 con 0x7f1010101940 2024-04-21T08:39:02.682 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:02.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:02 smithi094 ceph-mon[27988]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-21T08:39:02.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.888+0000 7f101535d700 1 -- 172.21.15.94:0/1712378823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f10101a5dc0 con 0x7f1010101940 2024-04-21T08:39:02.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.888+0000 7f0ff7fff700 1 -- 172.21.15.94:0/1712378823 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f100009dbb0 con 0x7f1010101940 2024-04-21T08:39:02.887 INFO:teuthology.orchestra.run.smithi094.stdout:287762808873 2024-04-21T08:39:02.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 -- 172.21.15.94:0/1712378823 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ff80a3c40 msgr2=0x7f0ff80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:02.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ff80a3c40 0x7f0ff80a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f10101013e0 tx=0x7f100400a380 comp rx=0 tx=0).stop 2024-04-21T08:39:02.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 -- 172.21.15.94:0/1712378823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1010101940 msgr2=0x7f1010199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:02.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1010101940 0x7f1010199390 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1000000c00 tx=0x7f10000047b0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 -- 172.21.15.94:0/1712378823 shutdown_connections 2024-04-21T08:39:02.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f0ff80a3c40 0x7f0ff80a6100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1010107990 0x7f101019d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1010101940 0x7f1010199390 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 --2- 172.21.15.94:0/1712378823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1010102300 0x7f10101998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:02.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 -- 172.21.15.94:0/1712378823 >> 172.21.15.94:0/1712378823 conn(0x7f10100788f0 msgr2=0x7f10100ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:02.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 -- 172.21.15.94:0/1712378823 shutdown_connections 2024-04-21T08:39:02.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:02.890+0000 7f101535d700 1 -- 172.21.15.94:0/1712378823 wait complete. 2024-04-21T08:39:02.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:02 smithi111 ceph-mon[31493]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-21T08:39:03.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:02 smithi173 ceph-mon[31428]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-21T08:39:03.378 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495666 got 197568495671 for osd.6 2024-04-21T08:39:03.378 DEBUG:teuthology.parallel:result is None 2024-04-21T08:39:03.829 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:03 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1712378823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:39:03.829 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:03 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3000113710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:39:03.855 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920828 got 141733920833 for osd.4 2024-04-21T08:39:03.855 DEBUG:teuthology.parallel:result is None 2024-04-21T08:39:03.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:03 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1712378823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:39:03.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:03 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3000113710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:39:04.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:03 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1712378823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:39:04.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:03 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3000113710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:39:04.437 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808866 got 287762808873 for osd.9 2024-04-21T08:39:04.437 DEBUG:teuthology.parallel:result is None 2024-04-21T08:39:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.001+0000 7f406ed7e700 1 -- 172.21.15.94:0/2040526630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4068101940 msgr2=0x7f4068101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.001+0000 7f406ed7e700 1 --2- 172.21.15.94:0/2040526630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4068101940 0x7f4068101dc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4054009990 tx=0x7f4054009ca0 comp rx=0 tx=0).stop 2024-04-21T08:39:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.001+0000 7f406ed7e700 1 -- 172.21.15.94:0/2040526630 shutdown_connections 2024-04-21T08:39:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.001+0000 7f406ed7e700 1 --2- 172.21.15.94:0/2040526630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4068102300 0x7f406810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.001+0000 7f406ed7e700 1 --2- 172.21.15.94:0/2040526630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4068107990 0x7f4068107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.001+0000 7f406ed7e700 1 --2- 172.21.15.94:0/2040526630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4068101940 0x7f4068101dc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.001+0000 7f406ed7e700 1 -- 172.21.15.94:0/2040526630 >> 172.21.15.94:0/2040526630 conn(0x7f40680788f0 msgr2=0x7f4068078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.001+0000 7f406ed7e700 1 -- 172.21.15.94:0/2040526630 shutdown_connections 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.001+0000 7f406ed7e700 1 -- 172.21.15.94:0/2040526630 wait complete. 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406ed7e700 1 Processor -- start 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406ed7e700 1 -- start start 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406ed7e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4068101940 0x7f4068199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406ed7e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4068102300 0x7f40681995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406ed7e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4068107990 0x7f406819d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406ed7e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4068111ae0 con 0x7f4068107990 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406ed7e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4068111960 con 0x7f4068101940 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406ed7e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4068111c60 con 0x7f4068102300 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406d31b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4068107990 0x7f406819d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406d31b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4068107990 0x7f406819d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36042/0 (socket says 172.21.15.94:36042) 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406d31b700 1 -- 172.21.15.94:0/3000113710 learned_addr learned my addr 172.21.15.94:0/3000113710 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406cb1a700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4068101940 0x7f4068199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:04.528 INFO:teuthology.orchestra.run.smithi094.stdout:163208757310 2024-04-21T08:39:04.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f4067fff700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4068102300 0x7f40681995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:04.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406d31b700 1 -- 172.21.15.94:0/3000113710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4068102300 msgr2=0x7f40681995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406d31b700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4068102300 0x7f40681995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406d31b700 1 -- 172.21.15.94:0/3000113710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4068101940 msgr2=0x7f4068199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406d31b700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4068101940 0x7f4068199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406d31b700 1 -- 172.21.15.94:0/3000113710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406819e0a0 con 0x7f4068107990 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.002+0000 7f406d31b700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4068107990 0x7f406819d960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f406000ada0 tx=0x7f406000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.003+0000 7f4065ffb700 1 -- 172.21.15.94:0/3000113710 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4060017020 con 0x7f4068107990 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.003+0000 7f4065ffb700 1 -- 172.21.15.94:0/3000113710 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f406000cd80 con 0x7f4068107990 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.003+0000 7f406ed7e700 1 -- 172.21.15.94:0/3000113710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406819e390 con 0x7f4068107990 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.003+0000 7f4065ffb700 1 -- 172.21.15.94:0/3000113710 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4060005260 con 0x7f4068107990 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.003+0000 7f406ed7e700 1 -- 172.21.15.94:0/3000113710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40681a5c10 con 0x7f4068107990 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.004+0000 7f406ed7e700 1 -- 172.21.15.94:0/3000113710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4068069730 con 0x7f4068107990 2024-04-21T08:39:04.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.005+0000 7f4065ffb700 1 -- 172.21.15.94:0/3000113710 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f406000c8d0 con 0x7f4068107990 2024-04-21T08:39:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.006+0000 7f4065ffb700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f40580a3c40 0x7f40580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.006+0000 7f406cb1a700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f40580a3c40 0x7f40580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.006+0000 7f4065ffb700 1 -- 172.21.15.94:0/3000113710 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f40600cfec0 con 0x7f4068107990 2024-04-21T08:39:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.006+0000 7f406cb1a700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f40580a3c40 0x7f40580a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f405c00abd0 tx=0x7f405c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.010+0000 7f4065ffb700 1 -- 172.21.15.94:0/3000113710 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4060021020 con 0x7f4068107990 2024-04-21T08:39:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.216+0000 7f406ed7e700 1 -- 172.21.15.94:0/3000113710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f406802df80 con 0x7f4068107990 2024-04-21T08:39:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.216+0000 7f4065ffb700 1 -- 172.21.15.94:0/3000113710 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f406009c5a0 con 0x7f4068107990 2024-04-21T08:39:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 -- 172.21.15.94:0/3000113710 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f40580a3c40 msgr2=0x7f40580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f40580a3c40 0x7f40580a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f405c00abd0 tx=0x7f405c00a380 comp rx=0 tx=0).stop 2024-04-21T08:39:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 -- 172.21.15.94:0/3000113710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4068107990 msgr2=0x7f406819d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4068107990 0x7f406819d960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f406000ada0 tx=0x7f406000c330 comp rx=0 tx=0).stop 2024-04-21T08:39:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 -- 172.21.15.94:0/3000113710 shutdown_connections 2024-04-21T08:39:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f40580a3c40 0x7f40580a6100 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4068101940 0x7f4068199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4068107990 0x7f406819d960 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 --2- 172.21.15.94:0/3000113710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4068102300 0x7f40681995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 -- 172.21.15.94:0/3000113710 >> 172.21.15.94:0/3000113710 conn(0x7f40680788f0 msgr2=0x7f406806aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 -- 172.21.15.94:0/3000113710 shutdown_connections 2024-04-21T08:39:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:03.219+0000 7f406ed7e700 1 -- 172.21.15.94:0/3000113710 wait complete. 2024-04-21T08:39:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:04 smithi111 ceph-mon[31493]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:05.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:04 smithi094 ceph-mon[27988]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:05.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:04 smithi173 ceph-mon[31428]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:05.145 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757304 got 163208757310 for osd.5 2024-04-21T08:39:05.145 DEBUG:teuthology.parallel:result is None 2024-04-21T08:39:06.409 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:07.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:06 smithi094 ceph-mon[27988]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:07.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:06 smithi173 ceph-mon[31428]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:07.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:06 smithi111 ceph-mon[31493]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:07.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.713+0000 7ff569712700 1 -- 172.21.15.94:0/3459512977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff564102a90 msgr2=0x7ff564102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.713+0000 7ff569712700 1 --2- 172.21.15.94:0/3459512977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff564102a90 0x7ff564102f10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff558009990 tx=0x7ff558009ca0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.713+0000 7ff569712700 1 -- 172.21.15.94:0/3459512977 shutdown_connections 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.713+0000 7ff569712700 1 --2- 172.21.15.94:0/3459512977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff564108a50 0x7ff564108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.713+0000 7ff569712700 1 --2- 172.21.15.94:0/3459512977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff564103450 0x7ff56410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.713+0000 7ff569712700 1 --2- 172.21.15.94:0/3459512977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff564102a90 0x7ff564102f10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.713+0000 7ff569712700 1 -- 172.21.15.94:0/3459512977 >> 172.21.15.94:0/3459512977 conn(0x7ff564079ba0 msgr2=0x7ff564079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.713+0000 7ff569712700 1 -- 172.21.15.94:0/3459512977 shutdown_connections 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.713+0000 7ff569712700 1 -- 172.21.15.94:0/3459512977 wait complete. 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.714+0000 7ff569712700 1 Processor -- start 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.714+0000 7ff569712700 1 -- start start 2024-04-21T08:39:07.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff569712700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff564102a90 0x7ff564074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff569712700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff564103450 0x7ff564075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff569712700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff564108a50 0x7ff5640757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff569712700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff564112be0 con 0x7ff564103450 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff569712700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff564112a60 con 0x7ff564108a50 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff569712700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff564112d60 con 0x7ff564102a90 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5627fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff564103450 0x7ff564075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5627fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff564103450 0x7ff564075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46584/0 (socket says 172.21.15.94:46584) 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5627fc700 1 -- 172.21.15.94:0/3905131709 learned_addr learned my addr 172.21.15.94:0/3905131709 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5637fe700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff564108a50 0x7ff5640757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5627fc700 1 -- 172.21.15.94:0/3905131709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff564102a90 msgr2=0x7ff564074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5627fc700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff564102a90 0x7ff564074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5627fc700 1 -- 172.21.15.94:0/3905131709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff564108a50 msgr2=0x7ff5640757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5627fc700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff564108a50 0x7ff5640757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5627fc700 1 -- 172.21.15.94:0/3905131709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5641a76a0 con 0x7ff564103450 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5637fe700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff564108a50 0x7ff5640757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff5627fc700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff564103450 0x7ff564075280 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff5580057c0 tx=0x7ff5580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff54bfff700 1 -- 172.21.15.94:0/3905131709 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558027070 con 0x7ff564103450 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff54bfff700 1 -- 172.21.15.94:0/3905131709 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff558004020 con 0x7ff564103450 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff569712700 1 -- 172.21.15.94:0/3905131709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5641a7930 con 0x7ff564103450 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff54bfff700 1 -- 172.21.15.94:0/3905131709 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558022c40 con 0x7ff564103450 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.715+0000 7ff569712700 1 -- 172.21.15.94:0/3905131709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5641a7e10 con 0x7ff564103450 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.716+0000 7ff569712700 1 -- 172.21.15.94:0/3905131709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff564069730 con 0x7ff564103450 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.717+0000 7ff54bfff700 1 -- 172.21.15.94:0/3905131709 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff55802a070 con 0x7ff564103450 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.718+0000 7ff54bfff700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff54c0a3c40 0x7ff54c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.718+0000 7ff562ffd700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff54c0a3c40 0x7ff54c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.718+0000 7ff54bfff700 1 -- 172.21.15.94:0/3905131709 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7ff5580d1ea0 con 0x7ff564103450 2024-04-21T08:39:07.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.718+0000 7ff562ffd700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff54c0a3c40 0x7ff54c0a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff554005e90 tx=0x7ff554005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:07.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.722+0000 7ff54bfff700 1 -- 172.21.15.94:0/3905131709 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff558031020 con 0x7ff564103450 2024-04-21T08:39:07.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.926+0000 7ff569712700 1 -- 172.21.15.94:0/3905131709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff56402e140 con 0x7ff564103450 2024-04-21T08:39:07.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.926+0000 7ff54bfff700 1 -- 172.21.15.94:0/3905131709 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff558013070 con 0x7ff564103450 2024-04-21T08:39:07.926 INFO:teuthology.orchestra.run.smithi094.stdout:219043332148 2024-04-21T08:39:07.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.928+0000 7ff569712700 1 -- 172.21.15.94:0/3905131709 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff54c0a3c40 msgr2=0x7ff54c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:07.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.928+0000 7ff569712700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff54c0a3c40 0x7ff54c0a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff554005e90 tx=0x7ff554005d00 comp rx=0 tx=0).stop 2024-04-21T08:39:07.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.928+0000 7ff569712700 1 -- 172.21.15.94:0/3905131709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff564103450 msgr2=0x7ff564075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:07.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.928+0000 7ff569712700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff564103450 0x7ff564075280 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff5580057c0 tx=0x7ff5580057f0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.929+0000 7ff569712700 1 -- 172.21.15.94:0/3905131709 shutdown_connections 2024-04-21T08:39:07.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.929+0000 7ff569712700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff54c0a3c40 0x7ff54c0a6100 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.929+0000 7ff569712700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff564108a50 0x7ff5640757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.929+0000 7ff569712700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff564102a90 0x7ff564074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.929+0000 7ff569712700 1 --2- 172.21.15.94:0/3905131709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff564103450 0x7ff564075280 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:07.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.929+0000 7ff569712700 1 -- 172.21.15.94:0/3905131709 >> 172.21.15.94:0/3905131709 conn(0x7ff564079ba0 msgr2=0x7ff564101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:07.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.929+0000 7ff569712700 1 -- 172.21.15.94:0/3905131709 shutdown_connections 2024-04-21T08:39:07.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:07.929+0000 7ff569712700 1 -- 172.21.15.94:0/3905131709 wait complete. 2024-04-21T08:39:08.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.851+0000 7f2e034a1700 1 -- 172.21.15.94:0/2922076088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dfc1063f0 msgr2=0x7f2dfc10dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:08.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.851+0000 7f2e034a1700 1 --2- 172.21.15.94:0/2922076088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dfc1063f0 0x7f2dfc10dfc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2df400b0a0 tx=0x7f2df400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:39:08.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.851+0000 7f2e034a1700 1 -- 172.21.15.94:0/2922076088 shutdown_connections 2024-04-21T08:39:08.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.851+0000 7f2e034a1700 1 --2- 172.21.15.94:0/2922076088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dfc101360 0x7f2dfc101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:08.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.851+0000 7f2e034a1700 1 --2- 172.21.15.94:0/2922076088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dfc1063f0 0x7f2dfc10dfc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:08.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.851+0000 7f2e034a1700 1 --2- 172.21.15.94:0/2922076088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dfc101d10 0x7f2dfc105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:08.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.851+0000 7f2e034a1700 1 -- 172.21.15.94:0/2922076088 >> 172.21.15.94:0/2922076088 conn(0x7f2dfc077fb0 msgr2=0x7f2dfc0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:08.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.851+0000 7f2e034a1700 1 -- 172.21.15.94:0/2922076088 shutdown_connections 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.851+0000 7f2e034a1700 1 -- 172.21.15.94:0/2922076088 wait complete. 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e034a1700 1 Processor -- start 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e034a1700 1 -- start start 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e034a1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dfc101360 0x7f2dfc199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e034a1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dfc101d10 0x7f2dfc199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e034a1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dfc1063f0 0x7f2dfc19da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e034a1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc112ca0 con 0x7f2dfc101360 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e034a1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc112b20 con 0x7f2dfc1063f0 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e034a1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc112e20 con 0x7f2dfc101d10 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e0123d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dfc101360 0x7f2dfc199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:08.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e00a3c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dfc101d10 0x7f2dfc199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e0123d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dfc101360 0x7f2dfc199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46604/0 (socket says 172.21.15.94:46604) 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e00a3c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dfc101d10 0x7f2dfc199690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:60484/0 (socket says 172.21.15.94:60484) 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e0123d700 1 -- 172.21.15.94:0/2820970958 learned_addr learned my addr 172.21.15.94:0/2820970958 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.852+0000 7f2e01a3e700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dfc1063f0 0x7f2dfc19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2e0123d700 1 -- 172.21.15.94:0/2820970958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dfc101d10 msgr2=0x7f2dfc199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2e0123d700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dfc101d10 0x7f2dfc199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2e0123d700 1 -- 172.21.15.94:0/2820970958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dfc1063f0 msgr2=0x7f2dfc19da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2e0123d700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dfc1063f0 0x7f2dfc19da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2e0123d700 1 -- 172.21.15.94:0/2820970958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dfc19e160 con 0x7f2dfc101360 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2e0123d700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dfc101360 0x7f2dfc199150 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2df800b530 tx=0x7f2df800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2df27fc700 1 -- 172.21.15.94:0/2820970958 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df8015020 con 0x7f2dfc101360 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2df27fc700 1 -- 172.21.15.94:0/2820970958 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2df8011ce0 con 0x7f2dfc101360 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2e034a1700 1 -- 172.21.15.94:0/2820970958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dfc19e450 con 0x7f2dfc101360 2024-04-21T08:39:08.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2e034a1700 1 -- 172.21.15.94:0/2820970958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dfc1a5cd0 con 0x7f2dfc101360 2024-04-21T08:39:08.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.853+0000 7f2df27fc700 1 -- 172.21.15.94:0/2820970958 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df8010e40 con 0x7f2dfc101360 2024-04-21T08:39:08.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.854+0000 7f2e034a1700 1 -- 172.21.15.94:0/2820970958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de0005330 con 0x7f2dfc101360 2024-04-21T08:39:08.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.855+0000 7f2df27fc700 1 -- 172.21.15.94:0/2820970958 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2df8018020 con 0x7f2dfc101360 2024-04-21T08:39:08.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.856+0000 7f2df27fc700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2de80a3c40 0x7f2de80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:08.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.856+0000 7f2e00a3c700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2de80a3c40 0x7f2de80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:08.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.856+0000 7f2df27fc700 1 -- 172.21.15.94:0/2820970958 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f2df80cfe30 con 0x7f2dfc101360 2024-04-21T08:39:08.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.856+0000 7f2e00a3c700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2de80a3c40 0x7f2de80a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc103bb0 tx=0x7f2dec00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:08.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:08.860+0000 7f2df27fc700 1 -- 172.21.15.94:0/2820970958 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2df801f020 con 0x7f2dfc101360 2024-04-21T08:39:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:08 smithi094 ceph-mon[27988]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-21T08:39:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:08 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3905131709' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:39:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:39:09.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:08 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:39:09.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:08 smithi173 ceph-mon[31428]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-21T08:39:09.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:08 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3905131709' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:39:09.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:39:09.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:08 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:39:09.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.062+0000 7f2e034a1700 1 -- 172.21.15.94:0/2820970958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f2de0005620 con 0x7f2dfc101360 2024-04-21T08:39:09.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.063+0000 7f2df27fc700 1 -- 172.21.15.94:0/2820970958 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f2df8018310 con 0x7f2dfc101360 2024-04-21T08:39:09.062 INFO:teuthology.orchestra.run.smithi094.stdout:309237645349 2024-04-21T08:39:09.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 -- 172.21.15.94:0/2820970958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2de80a3c40 msgr2=0x7f2de80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:09.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2de80a3c40 0x7f2de80a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc103bb0 tx=0x7f2dec00b3c0 comp rx=0 tx=0).stop 2024-04-21T08:39:09.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 -- 172.21.15.94:0/2820970958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dfc101360 msgr2=0x7f2dfc199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:09.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dfc101360 0x7f2dfc199150 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2df800b530 tx=0x7f2df800b840 comp rx=0 tx=0).stop 2024-04-21T08:39:09.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 -- 172.21.15.94:0/2820970958 shutdown_connections 2024-04-21T08:39:09.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2de80a3c40 0x7f2de80a6100 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:09.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dfc1063f0 0x7f2dfc19da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:09.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dfc101360 0x7f2dfc199150 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:09.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 --2- 172.21.15.94:0/2820970958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dfc101d10 0x7f2dfc199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:09.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 -- 172.21.15.94:0/2820970958 >> 172.21.15.94:0/2820970958 conn(0x7f2dfc077fb0 msgr2=0x7f2dfc104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:09.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 -- 172.21.15.94:0/2820970958 shutdown_connections 2024-04-21T08:39:09.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:09.065+0000 7f2e034a1700 1 -- 172.21.15.94:0/2820970958 wait complete. 2024-04-21T08:39:09.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:08 smithi111 ceph-mon[31493]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-21T08:39:09.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:08 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3905131709' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:39:09.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:39:09.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:08 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:39:09.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:09 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2820970958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:39:09.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:09.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:09.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:09 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:39:09.875 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332140 got 219043332148 for osd.7 2024-04-21T08:39:09.876 DEBUG:teuthology.parallel:result is None 2024-04-21T08:39:10.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:09 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2820970958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:39:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:09 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:39:10.088 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:09 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2820970958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:39:10.089 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:10.089 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:10.089 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:09 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' 2024-04-21T08:39:10.437 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645345 got 309237645349 for osd.10 2024-04-21T08:39:10.437 DEBUG:teuthology.parallel:result is None 2024-04-21T08:39:10.437 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-21T08:39:10.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T08:39:10.677 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:10.949 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:10 smithi094 ceph-mon[27988]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:11.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:10 smithi173 ceph-mon[31428]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:11.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:10 smithi111 ceph-mon[31493]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:12.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.591+0000 7fd718b5c700 1 -- 172.21.15.94:0/1776225908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd714101940 msgr2=0x7fd714101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.591+0000 7fd718b5c700 1 --2- 172.21.15.94:0/1776225908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd714101940 0x7fd714101dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd708009990 tx=0x7fd708009ca0 comp rx=0 tx=0).stop 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.591+0000 7fd718b5c700 1 -- 172.21.15.94:0/1776225908 shutdown_connections 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.591+0000 7fd718b5c700 1 --2- 172.21.15.94:0/1776225908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714102300 0x7fd71410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.591+0000 7fd718b5c700 1 --2- 172.21.15.94:0/1776225908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd714107990 0x7fd714107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.591+0000 7fd718b5c700 1 --2- 172.21.15.94:0/1776225908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd714101940 0x7fd714101dc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.591+0000 7fd718b5c700 1 -- 172.21.15.94:0/1776225908 >> 172.21.15.94:0/1776225908 conn(0x7fd7140788f0 msgr2=0x7fd714078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.591+0000 7fd718b5c700 1 -- 172.21.15.94:0/1776225908 shutdown_connections 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.591+0000 7fd718b5c700 1 -- 172.21.15.94:0/1776225908 wait complete. 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.592+0000 7fd718b5c700 1 Processor -- start 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.592+0000 7fd718b5c700 1 -- start start 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.592+0000 7fd718b5c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714101940 0x7fd7141034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.592+0000 7fd718b5c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd714102300 0x7fd714103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.592+0000 7fd718b5c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd714107990 0x7fd714103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.592+0000 7fd718b5c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd714111bd0 con 0x7fd714107990 2024-04-21T08:39:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.592+0000 7fd718b5c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd714111a50 con 0x7fd714101940 2024-04-21T08:39:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd718b5c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd714111d50 con 0x7fd714102300 2024-04-21T08:39:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd712d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd714107990 0x7fd714103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd712d9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd714107990 0x7fd714103f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46622/0 (socket says 172.21.15.94:46622) 2024-04-21T08:39:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd712d9d700 1 -- 172.21.15.94:0/353566512 learned_addr learned my addr 172.21.15.94:0/353566512 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd711d9b700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd714102300 0x7fd714103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd71259c700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714101940 0x7fd7141034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd712d9d700 1 -- 172.21.15.94:0/353566512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd714102300 msgr2=0x7fd714103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd712d9d700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd714102300 0x7fd714103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd712d9d700 1 -- 172.21.15.94:0/353566512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714101940 msgr2=0x7fd7141034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd712d9d700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714101940 0x7fd7141034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd712d9d700 1 -- 172.21.15.94:0/353566512 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7141a77c0 con 0x7fd714107990 2024-04-21T08:39:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd712d9d700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd714107990 0x7fd714103f70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00b970 tx=0x7fd70c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd7037fe700 1 -- 172.21.15.94:0/353566512 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00d860 con 0x7fd714107990 2024-04-21T08:39:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd7037fe700 1 -- 172.21.15.94:0/353566512 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd70c00e430 con 0x7fd714107990 2024-04-21T08:39:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd718b5c700 1 -- 172.21.15.94:0/353566512 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7141a7ab0 con 0x7fd714107990 2024-04-21T08:39:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.593+0000 7fd7037fe700 1 -- 172.21.15.94:0/353566512 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c017660 con 0x7fd714107990 2024-04-21T08:39:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.594+0000 7fd718b5c700 1 -- 172.21.15.94:0/353566512 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7141a7fc0 con 0x7fd714107990 2024-04-21T08:39:12.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.594+0000 7fd718b5c700 1 -- 172.21.15.94:0/353566512 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd714069730 con 0x7fd714107990 2024-04-21T08:39:12.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.595+0000 7fd7037fe700 1 -- 172.21.15.94:0/353566512 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd70c00da00 con 0x7fd714107990 2024-04-21T08:39:12.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.596+0000 7fd7037fe700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6fc0a3c40 0x7fd6fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:12.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.596+0000 7fd71259c700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6fc0a3c40 0x7fd6fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:12.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.596+0000 7fd7037fe700 1 -- 172.21.15.94:0/353566512 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7fd70c0d0290 con 0x7fd714107990 2024-04-21T08:39:12.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.597+0000 7fd71259c700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6fc0a3c40 0x7fd6fc0a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd704009a10 tx=0x7fd704009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:12.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.601+0000 7fd7037fe700 1 -- 172.21.15.94:0/353566512 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd70c013030 con 0x7fd714107990 2024-04-21T08:39:12.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.797+0000 7fd718b5c700 1 -- 172.21.15.94:0/353566512 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd71406a720 con 0x7fd6fc0a3c40 2024-04-21T08:39:12.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.806+0000 7fd7037fe700 1 -- 172.21.15.94:0/353566512 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340788 (secure 0 0 0) 0x7fd71406a720 con 0x7fd6fc0a3c40 2024-04-21T08:39:12.811 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:12.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 -- 172.21.15.94:0/353566512 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6fc0a3c40 msgr2=0x7fd6fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:12.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6fc0a3c40 0x7fd6fc0a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd704009a10 tx=0x7fd704009190 comp rx=0 tx=0).stop 2024-04-21T08:39:12.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 -- 172.21.15.94:0/353566512 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd714107990 msgr2=0x7fd714103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:12.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd714107990 0x7fd714103f70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00b970 tx=0x7fd70c00bc80 comp rx=0 tx=0).stop 2024-04-21T08:39:12.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 -- 172.21.15.94:0/353566512 shutdown_connections 2024-04-21T08:39:12.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd6fc0a3c40 0x7fd6fc0a6100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:12.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714101940 0x7fd7141034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:12.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd714107990 0x7fd714103f70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:12.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 --2- 172.21.15.94:0/353566512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd714102300 0x7fd714103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:12.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 -- 172.21.15.94:0/353566512 >> 172.21.15.94:0/353566512 conn(0x7fd7140788f0 msgr2=0x7fd714105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:12.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 -- 172.21.15.94:0/353566512 shutdown_connections 2024-04-21T08:39:12.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:12.810+0000 7fd718b5c700 1 -- 172.21.15.94:0/353566512 wait complete. 2024-04-21T08:39:12.815 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-21T08:39:13.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:12 smithi094 ceph-mon[27988]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:13.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:12 smithi173 ceph-mon[31428]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:13.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:12 smithi111 ceph-mon[31493]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-21T08:39:13.584 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-04-21T08:39:11.606117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658349,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":600,"num_read_kb":422,"num_write":351,"num_write_kb":3061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377944,"kb_used_data":12152,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563880,"statfs":{"total":1151940427776,"available":1151553413120,"internally_reserved":0,"allocated":12443648,"data_stored":7959245,"data_compressed":361281,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":37,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003239"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.605355+0000","last_change":"2024-04-21T08:37:31.448345+0000","last_active":"2024-04-21T08:38:59.605355+0000","last_peered":"2024-04-21T08:38:59.605355+0000","last_clean":"2024-04-21T08:38:59.605355+0000","last_became_active":"2024-04-21T08:37:31.447878+0000","last_became_peered":"2024-04-21T08:37:31.447878+0000","last_unstale":"2024-04-21T08:38:59.605355+0000","last_undegraded":"2024-04-21T08:38:59.605355+0000","last_fullsized":"2024-04-21T08:38:59.605355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:45.589165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.595000+0000","last_change":"2024-04-21T08:37:31.452895+0000","last_active":"2024-04-21T08:38:59.595000+0000","last_peered":"2024-04-21T08:38:59.595000+0000","last_clean":"2024-04-21T08:38:59.595000+0000","last_became_active":"2024-04-21T08:37:31.452550+0000","last_became_peered":"2024-04-21T08:37:31.452550+0000","last_unstale":"2024-04-21T08:38:59.595000+0000","last_undegraded":"2024-04-21T08:38:59.595000+0000","last_fullsized":"2024-04-21T08:38:59.595000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:02.497802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.600232+0000","last_change":"2024-04-21T08:37:31.447552+0000","last_active":"2024-04-21T08:38:59.600232+0000","last_peered":"2024-04-21T08:38:59.600232+0000","last_clean":"2024-04-21T08:38:59.600232+0000","last_became_active":"2024-04-21T08:37:31.446044+0000","last_became_peered":"2024-04-21T08:37:31.446044+0000","last_unstale":"2024-04-21T08:38:59.600232+0000","last_undegraded":"2024-04-21T08:38:59.600232+0000","last_fullsized":"2024-04-21T08:38:59.600232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:22.801977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.588598+0000","last_change":"2024-04-21T08:37:31.452400+0000","last_active":"2024-04-21T08:38:59.588598+0000","last_peered":"2024-04-21T08:38:59.588598+0000","last_clean":"2024-04-21T08:38:59.588598+0000","last_became_active":"2024-04-21T08:37:31.451462+0000","last_became_peered":"2024-04-21T08:37:31.451462+0000","last_unstale":"2024-04-21T08:38:59.588598+0000","last_undegraded":"2024-04-21T08:38:59.588598+0000","last_fullsized":"2024-04-21T08:38:59.588598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:22.463481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.579416+0000","last_change":"2024-04-21T08:37:31.448942+0000","last_active":"2024-04-21T08:38:59.579416+0000","last_peered":"2024-04-21T08:38:59.579416+0000","last_clean":"2024-04-21T08:38:59.579416+0000","last_became_active":"2024-04-21T08:37:31.448526+0000","last_became_peered":"2024-04-21T08:37:31.448526+0000","last_unstale":"2024-04-21T08:38:59.579416+0000","last_undegraded":"2024-04-21T08:38:59.579416+0000","last_fullsized":"2024-04-21T08:38:59.579416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:29.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.592477+0000","last_change":"2024-04-21T08:37:31.452165+0000","last_active":"2024-04-21T08:38:59.592477+0000","last_peered":"2024-04-21T08:38:59.592477+0000","last_clean":"2024-04-21T08:38:59.592477+0000","last_became_active":"2024-04-21T08:37:31.451802+0000","last_became_peered":"2024-04-21T08:37:31.451802+0000","last_unstale":"2024-04-21T08:38:59.592477+0000","last_undegraded":"2024-04-21T08:38:59.592477+0000","last_fullsized":"2024-04-21T08:38:59.592477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:42.210544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.594832+0000","last_change":"2024-04-21T08:37:31.444798+0000","last_active":"2024-04-21T08:38:59.594832+0000","last_peered":"2024-04-21T08:38:59.594832+0000","last_clean":"2024-04-21T08:38:59.594832+0000","last_became_active":"2024-04-21T08:37:31.444313+0000","last_became_peered":"2024-04-21T08:37:31.444313+0000","last_unstale":"2024-04-21T08:38:59.594832+0000","last_undegraded":"2024-04-21T08:38:59.594832+0000","last_fullsized":"2024-04-21T08:38:59.594832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:22:03.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599340+0000","last_change":"2024-04-21T08:37:31.448257+0000","last_active":"2024-04-21T08:38:59.599340+0000","last_peered":"2024-04-21T08:38:59.599340+0000","last_clean":"2024-04-21T08:38:59.599340+0000","last_became_active":"2024-04-21T08:37:31.447845+0000","last_became_peered":"2024-04-21T08:37:31.447845+0000","last_unstale":"2024-04-21T08:38:59.599340+0000","last_undegraded":"2024-04-21T08:38:59.599340+0000","last_fullsized":"2024-04-21T08:38:59.599340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:19.956699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.604851+0000","last_change":"2024-04-21T08:37:31.443022+0000","last_active":"2024-04-21T08:38:59.604851+0000","last_peered":"2024-04-21T08:38:59.604851+0000","last_clean":"2024-04-21T08:38:59.604851+0000","last_became_active":"2024-04-21T08:37:31.442645+0000","last_became_peered":"2024-04-21T08:37:31.442645+0000","last_unstale":"2024-04-21T08:38:59.604851+0000","last_undegraded":"2024-04-21T08:38:59.604851+0000","last_fullsized":"2024-04-21T08:38:59.604851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:18:10.861281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.593772+0000","last_change":"2024-04-21T08:37:31.453427+0000","last_active":"2024-04-21T08:38:59.593772+0000","last_peered":"2024-04-21T08:38:59.593772+0000","last_clean":"2024-04-21T08:38:59.593772+0000","last_became_active":"2024-04-21T08:37:31.453096+0000","last_became_peered":"2024-04-21T08:37:31.453096+0000","last_unstale":"2024-04-21T08:38:59.593772+0000","last_undegraded":"2024-04-21T08:38:59.593772+0000","last_fullsized":"2024-04-21T08:38:59.593772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:49.450351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.608208+0000","last_change":"2024-04-21T08:37:31.449908+0000","last_active":"2024-04-21T08:38:59.608208+0000","last_peered":"2024-04-21T08:38:59.608208+0000","last_clean":"2024-04-21T08:38:59.608208+0000","last_became_active":"2024-04-21T08:37:31.449558+0000","last_became_peered":"2024-04-21T08:37:31.449558+0000","last_unstale":"2024-04-21T08:38:59.608208+0000","last_undegraded":"2024-04-21T08:38:59.608208+0000","last_fullsized":"2024-04-21T08:38:59.608208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:21.538689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598603+0000","last_change":"2024-04-21T08:37:31.452973+0000","last_active":"2024-04-21T08:38:59.598603+0000","last_peered":"2024-04-21T08:38:59.598603+0000","last_clean":"2024-04-21T08:38:59.598603+0000","last_became_active":"2024-04-21T08:37:31.452586+0000","last_became_peered":"2024-04-21T08:37:31.452586+0000","last_unstale":"2024-04-21T08:38:59.598603+0000","last_undegraded":"2024-04-21T08:38:59.598603+0000","last_fullsized":"2024-04-21T08:38:59.598603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:09:11.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.593552+0000","last_change":"2024-04-21T08:37:31.445801+0000","last_active":"2024-04-21T08:38:59.593552+0000","last_peered":"2024-04-21T08:38:59.593552+0000","last_clean":"2024-04-21T08:38:59.593552+0000","last_became_active":"2024-04-21T08:37:31.445478+0000","last_became_peered":"2024-04-21T08:37:31.445478+0000","last_unstale":"2024-04-21T08:38:59.593552+0000","last_undegraded":"2024-04-21T08:38:59.593552+0000","last_fullsized":"2024-04-21T08:38:59.593552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:33.352739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.593896+0000","last_change":"2024-04-21T08:37:31.450735+0000","last_active":"2024-04-21T08:38:59.593896+0000","last_peered":"2024-04-21T08:38:59.593896+0000","last_clean":"2024-04-21T08:38:59.593896+0000","last_became_active":"2024-04-21T08:37:31.450384+0000","last_became_peered":"2024-04-21T08:37:31.450384+0000","last_unstale":"2024-04-21T08:38:59.593896+0000","last_undegraded":"2024-04-21T08:38:59.593896+0000","last_fullsized":"2024-04-21T08:38:59.593896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:45.950489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599498+0000","last_change":"2024-04-21T08:37:31.454690+0000","last_active":"2024-04-21T08:38:59.599498+0000","last_peered":"2024-04-21T08:38:59.599498+0000","last_clean":"2024-04-21T08:38:59.599498+0000","last_became_active":"2024-04-21T08:37:31.454330+0000","last_became_peered":"2024-04-21T08:37:31.454330+0000","last_unstale":"2024-04-21T08:38:59.599498+0000","last_undegraded":"2024-04-21T08:38:59.599498+0000","last_fullsized":"2024-04-21T08:38:59.599498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:52.644317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.608553+0000","last_change":"2024-04-21T08:37:31.441389+0000","last_active":"2024-04-21T08:38:59.608553+0000","last_peered":"2024-04-21T08:38:59.608553+0000","last_clean":"2024-04-21T08:38:59.608553+0000","last_became_active":"2024-04-21T08:37:31.441065+0000","last_became_peered":"2024-04-21T08:37:31.441065+0000","last_unstale":"2024-04-21T08:38:59.608553+0000","last_undegraded":"2024-04-21T08:38:59.608553+0000","last_fullsized":"2024-04-21T08:38:59.608553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:56.528154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.605851+0000","last_change":"2024-04-21T08:37:31.446722+0000","last_active":"2024-04-21T08:38:59.605851+0000","last_peered":"2024-04-21T08:38:59.605851+0000","last_clean":"2024-04-21T08:38:59.605851+0000","last_became_active":"2024-04-21T08:37:31.446396+0000","last_became_peered":"2024-04-21T08:37:31.446396+0000","last_unstale":"2024-04-21T08:38:59.605851+0000","last_undegraded":"2024-04-21T08:38:59.605851+0000","last_fullsized":"2024-04-21T08:38:59.605851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:17.488633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.601347+0000","last_change":"2024-04-21T08:37:31.444388+0000","last_active":"2024-04-21T08:38:59.601347+0000","last_peered":"2024-04-21T08:38:59.601347+0000","last_clean":"2024-04-21T08:38:59.601347+0000","last_became_active":"2024-04-21T08:37:31.444023+0000","last_became_peered":"2024-04-21T08:37:31.444023+0000","last_unstale":"2024-04-21T08:38:59.601347+0000","last_undegraded":"2024-04-21T08:38:59.601347+0000","last_fullsized":"2024-04-21T08:38:59.601347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:07:46.238941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.597499+0000","last_change":"2024-04-21T08:37:31.447850+0000","last_active":"2024-04-21T08:38:59.597499+0000","last_peered":"2024-04-21T08:38:59.597499+0000","last_clean":"2024-04-21T08:38:59.597499+0000","last_became_active":"2024-04-21T08:37:31.447505+0000","last_became_peered":"2024-04-21T08:37:31.447505+0000","last_unstale":"2024-04-21T08:38:59.597499+0000","last_undegraded":"2024-04-21T08:38:59.597499+0000","last_fullsized":"2024-04-21T08:38:59.597499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:55.399410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.667080+0000","last_change":"2024-04-21T08:37:35.608747+0000","last_active":"2024-04-21T08:38:30.667080+0000","last_peered":"2024-04-21T08:38:30.667080+0000","last_clean":"2024-04-21T08:38:30.667080+0000","last_became_active":"2024-04-21T08:37:35.608157+0000","last_became_peered":"2024-04-21T08:37:35.608157+0000","last_unstale":"2024-04-21T08:38:30.667080+0000","last_undegraded":"2024-04-21T08:38:30.667080+0000","last_fullsized":"2024-04-21T08:38:30.667080+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:53.384190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.679843+0000","last_change":"2024-04-21T08:38:30.679843+0000","last_active":"2024-04-21T08:38:30.679843+0000","last_peered":"2024-04-21T08:38:30.679843+0000","last_clean":"2024-04-21T08:38:30.679843+0000","last_became_active":"2024-04-21T08:38:30.670964+0000","last_became_peered":"2024-04-21T08:38:30.670964+0000","last_unstale":"2024-04-21T08:38:30.679843+0000","last_undegraded":"2024-04-21T08:38:30.679843+0000","last_fullsized":"2024-04-21T08:38:30.679843+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:49.652571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598546+0000","last_change":"2024-04-21T08:37:31.451813+0000","last_active":"2024-04-21T08:38:59.598546+0000","last_peered":"2024-04-21T08:38:59.598546+0000","last_clean":"2024-04-21T08:38:59.598546+0000","last_became_active":"2024-04-21T08:37:31.451447+0000","last_became_peered":"2024-04-21T08:37:31.451447+0000","last_unstale":"2024-04-21T08:38:59.598546+0000","last_undegraded":"2024-04-21T08:38:59.598546+0000","last_fullsized":"2024-04-21T08:38:59.598546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:46:45.235326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.668976+0000","last_change":"2024-04-21T08:37:35.603518+0000","last_active":"2024-04-21T08:38:30.668976+0000","last_peered":"2024-04-21T08:38:30.668976+0000","last_clean":"2024-04-21T08:38:30.668976+0000","last_became_active":"2024-04-21T08:37:35.603038+0000","last_became_peered":"2024-04-21T08:37:35.603038+0000","last_unstale":"2024-04-21T08:38:30.668976+0000","last_undegraded":"2024-04-21T08:38:30.668976+0000","last_fullsized":"2024-04-21T08:38:30.668976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:06:29.224025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.673441+0000","last_change":"2024-04-21T08:37:30.424608+0000","last_active":"2024-04-21T08:38:30.673441+0000","last_peered":"2024-04-21T08:38:30.673441+0000","last_clean":"2024-04-21T08:38:30.673441+0000","last_became_active":"2024-04-21T08:37:30.424259+0000","last_became_peered":"2024-04-21T08:37:30.424259+0000","last_unstale":"2024-04-21T08:38:30.673441+0000","last_undegraded":"2024-04-21T08:38:30.673441+0000","last_fullsized":"2024-04-21T08:38:30.673441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:38.007095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.600543+0000","last_change":"2024-04-21T08:37:31.444656+0000","last_active":"2024-04-21T08:38:59.600543+0000","last_peered":"2024-04-21T08:38:59.600543+0000","last_clean":"2024-04-21T08:38:59.600543+0000","last_became_active":"2024-04-21T08:37:31.444300+0000","last_became_peered":"2024-04-21T08:37:31.444300+0000","last_unstale":"2024-04-21T08:38:59.600543+0000","last_undegraded":"2024-04-21T08:38:59.600543+0000","last_fullsized":"2024-04-21T08:38:59.600543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:40.901674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669700+0000","last_change":"2024-04-21T08:37:35.611732+0000","last_active":"2024-04-21T08:38:30.669700+0000","last_peered":"2024-04-21T08:38:30.669700+0000","last_clean":"2024-04-21T08:38:30.669700+0000","last_became_active":"2024-04-21T08:37:35.611060+0000","last_became_peered":"2024-04-21T08:37:35.611060+0000","last_unstale":"2024-04-21T08:38:30.669700+0000","last_undegraded":"2024-04-21T08:38:30.669700+0000","last_fullsized":"2024-04-21T08:38:30.669700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:36:01.491949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.689722+0000","last_change":"2024-04-21T08:38:30.689722+0000","last_active":"2024-04-21T08:38:30.689722+0000","last_peered":"2024-04-21T08:38:30.689722+0000","last_clean":"2024-04-21T08:38:30.689722+0000","last_became_active":"2024-04-21T08:38:30.676521+0000","last_became_peered":"2024-04-21T08:38:30.676521+0000","last_unstale":"2024-04-21T08:38:30.689722+0000","last_undegraded":"2024-04-21T08:38:30.689722+0000","last_fullsized":"2024-04-21T08:38:30.689722+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:24:15.799308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.605600+0000","last_change":"2024-04-21T08:37:31.443896+0000","last_active":"2024-04-21T08:38:59.605600+0000","last_peered":"2024-04-21T08:38:59.605600+0000","last_clean":"2024-04-21T08:38:59.605600+0000","last_became_active":"2024-04-21T08:37:31.442684+0000","last_became_peered":"2024-04-21T08:37:31.442684+0000","last_unstale":"2024-04-21T08:38:59.605600+0000","last_undegraded":"2024-04-21T08:38:59.605600+0000","last_fullsized":"2024-04-21T08:38:59.605600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:14:18.649565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669557+0000","last_change":"2024-04-21T08:37:35.600468+0000","last_active":"2024-04-21T08:38:30.669557+0000","last_peered":"2024-04-21T08:38:30.669557+0000","last_clean":"2024-04-21T08:38:30.669557+0000","last_became_active":"2024-04-21T08:37:35.599940+0000","last_became_peered":"2024-04-21T08:37:35.599940+0000","last_unstale":"2024-04-21T08:38:30.669557+0000","last_undegraded":"2024-04-21T08:38:30.669557+0000","last_fullsized":"2024-04-21T08:38:30.669557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:48.125434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669752+0000","last_change":"2024-04-21T08:37:30.416480+0000","last_active":"2024-04-21T08:38:30.669752+0000","last_peered":"2024-04-21T08:38:30.669752+0000","last_clean":"2024-04-21T08:38:30.669752+0000","last_became_active":"2024-04-21T08:37:30.416072+0000","last_became_peered":"2024-04-21T08:37:30.416072+0000","last_unstale":"2024-04-21T08:38:30.669752+0000","last_undegraded":"2024-04-21T08:38:30.669752+0000","last_fullsized":"2024-04-21T08:38:30.669752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:28:22.400926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.607612+0000","last_change":"2024-04-21T08:37:31.457216+0000","last_active":"2024-04-21T08:38:59.607612+0000","last_peered":"2024-04-21T08:38:59.607612+0000","last_clean":"2024-04-21T08:38:59.607612+0000","last_became_active":"2024-04-21T08:37:31.456856+0000","last_became_peered":"2024-04-21T08:37:31.456856+0000","last_unstale":"2024-04-21T08:38:59.607612+0000","last_undegraded":"2024-04-21T08:38:59.607612+0000","last_fullsized":"2024-04-21T08:38:59.607612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:10.272886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.668095+0000","last_change":"2024-04-21T08:37:35.601715+0000","last_active":"2024-04-21T08:38:30.668095+0000","last_peered":"2024-04-21T08:38:30.668095+0000","last_clean":"2024-04-21T08:38:30.668095+0000","last_became_active":"2024-04-21T08:37:35.601269+0000","last_became_peered":"2024-04-21T08:37:35.601269+0000","last_unstale":"2024-04-21T08:38:30.668095+0000","last_undegraded":"2024-04-21T08:38:30.668095+0000","last_fullsized":"2024-04-21T08:38:30.668095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:57.192918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:39:08.813596+0000","last_change":"2024-04-21T08:37:30.415510+0000","last_active":"2024-04-21T08:39:08.813596+0000","last_peered":"2024-04-21T08:39:08.813596+0000","last_clean":"2024-04-21T08:39:08.813596+0000","last_became_active":"2024-04-21T08:37:30.415199+0000","last_became_peered":"2024-04-21T08:37:30.415199+0000","last_unstale":"2024-04-21T08:39:08.813596+0000","last_undegraded":"2024-04-21T08:39:08.813596+0000","last_fullsized":"2024-04-21T08:39:08.813596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:32.336247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.586253+0000","last_change":"2024-04-21T08:37:31.444022+0000","last_active":"2024-04-21T08:38:59.586253+0000","last_peered":"2024-04-21T08:38:59.586253+0000","last_clean":"2024-04-21T08:38:59.586253+0000","last_became_active":"2024-04-21T08:37:31.443609+0000","last_became_peered":"2024-04-21T08:37:31.443609+0000","last_unstale":"2024-04-21T08:38:59.586253+0000","last_undegraded":"2024-04-21T08:38:59.586253+0000","last_fullsized":"2024-04-21T08:38:59.586253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:39.978817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.714455+0000","last_change":"2024-04-21T08:38:30.714455+0000","last_active":"2024-04-21T08:38:30.714455+0000","last_peered":"2024-04-21T08:38:30.714455+0000","last_clean":"2024-04-21T08:38:30.714455+0000","last_became_active":"2024-04-21T08:38:30.713910+0000","last_became_peered":"2024-04-21T08:38:30.713910+0000","last_unstale":"2024-04-21T08:38:30.714455+0000","last_undegraded":"2024-04-21T08:38:30.714455+0000","last_fullsized":"2024-04-21T08:38:30.714455+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.448652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,10],"acting":[2,6,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.712604+0000","last_change":"2024-04-21T08:37:30.419877+0000","last_active":"2024-04-21T08:38:30.712604+0000","last_peered":"2024-04-21T08:38:30.712604+0000","last_clean":"2024-04-21T08:38:30.712604+0000","last_became_active":"2024-04-21T08:37:30.419572+0000","last_became_peered":"2024-04-21T08:37:30.419572+0000","last_unstale":"2024-04-21T08:38:30.712604+0000","last_undegraded":"2024-04-21T08:38:30.712604+0000","last_fullsized":"2024-04-21T08:38:30.712604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:32:35.524089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.597003+0000","last_change":"2024-04-21T08:37:31.458590+0000","last_active":"2024-04-21T08:38:59.597003+0000","last_peered":"2024-04-21T08:38:59.597003+0000","last_clean":"2024-04-21T08:38:59.597003+0000","last_became_active":"2024-04-21T08:37:31.458196+0000","last_became_peered":"2024-04-21T08:37:31.458196+0000","last_unstale":"2024-04-21T08:38:59.597003+0000","last_undegraded":"2024-04-21T08:38:59.597003+0000","last_fullsized":"2024-04-21T08:38:59.597003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:00.890859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.803663+0000","last_change":"2024-04-21T08:38:30.803663+0000","last_active":"2024-04-21T08:38:30.803663+0000","last_peered":"2024-04-21T08:38:30.803663+0000","last_clean":"2024-04-21T08:38:30.803663+0000","last_became_active":"2024-04-21T08:38:30.803203+0000","last_became_peered":"2024-04-21T08:38:30.803203+0000","last_unstale":"2024-04-21T08:38:30.803663+0000","last_undegraded":"2024-04-21T08:38:30.803663+0000","last_fullsized":"2024-04-21T08:38:30.803663+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:06.085380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669367+0000","last_change":"2024-04-21T08:37:30.415786+0000","last_active":"2024-04-21T08:38:30.669367+0000","last_peered":"2024-04-21T08:38:30.669367+0000","last_clean":"2024-04-21T08:38:30.669367+0000","last_became_active":"2024-04-21T08:37:30.415472+0000","last_became_peered":"2024-04-21T08:37:30.415472+0000","last_unstale":"2024-04-21T08:38:30.669367+0000","last_undegraded":"2024-04-21T08:38:30.669367+0000","last_fullsized":"2024-04-21T08:38:30.669367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.417132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.597874+0000","last_change":"2024-04-21T08:37:31.455720+0000","last_active":"2024-04-21T08:38:59.597874+0000","last_peered":"2024-04-21T08:38:59.597874+0000","last_clean":"2024-04-21T08:38:59.597874+0000","last_became_active":"2024-04-21T08:37:31.455334+0000","last_became_peered":"2024-04-21T08:37:31.455334+0000","last_unstale":"2024-04-21T08:38:59.597874+0000","last_undegraded":"2024-04-21T08:38:59.597874+0000","last_fullsized":"2024-04-21T08:38:59.597874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:06:55.220656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.802537+0000","last_change":"2024-04-21T08:37:35.603305+0000","last_active":"2024-04-21T08:38:30.802537+0000","last_peered":"2024-04-21T08:38:30.802537+0000","last_clean":"2024-04-21T08:38:30.802537+0000","last_became_active":"2024-04-21T08:37:35.602854+0000","last_became_peered":"2024-04-21T08:37:35.602854+0000","last_unstale":"2024-04-21T08:38:30.802537+0000","last_undegraded":"2024-04-21T08:38:30.802537+0000","last_fullsized":"2024-04-21T08:38:30.802537+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:10.993864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.674921+0000","last_change":"2024-04-21T08:37:30.420797+0000","last_active":"2024-04-21T08:38:30.674921+0000","last_peered":"2024-04-21T08:38:30.674921+0000","last_clean":"2024-04-21T08:38:30.674921+0000","last_became_active":"2024-04-21T08:37:30.420485+0000","last_became_peered":"2024-04-21T08:37:30.420485+0000","last_unstale":"2024-04-21T08:38:30.674921+0000","last_undegraded":"2024-04-21T08:38:30.674921+0000","last_fullsized":"2024-04-21T08:38:30.674921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:03.275677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598515+0000","last_change":"2024-04-21T08:37:31.448406+0000","last_active":"2024-04-21T08:38:59.598515+0000","last_peered":"2024-04-21T08:38:59.598515+0000","last_clean":"2024-04-21T08:38:59.598515+0000","last_became_active":"2024-04-21T08:37:31.448085+0000","last_became_peered":"2024-04-21T08:37:31.448085+0000","last_unstale":"2024-04-21T08:38:59.598515+0000","last_undegraded":"2024-04-21T08:38:59.598515+0000","last_fullsized":"2024-04-21T08:38:59.598515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:49.574064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.672162+0000","last_change":"2024-04-21T08:37:35.601338+0000","last_active":"2024-04-21T08:38:30.672162+0000","last_peered":"2024-04-21T08:38:30.672162+0000","last_clean":"2024-04-21T08:38:30.672162+0000","last_became_active":"2024-04-21T08:37:35.600890+0000","last_became_peered":"2024-04-21T08:37:35.600890+0000","last_unstale":"2024-04-21T08:38:30.672162+0000","last_undegraded":"2024-04-21T08:38:30.672162+0000","last_fullsized":"2024-04-21T08:38:30.672162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:30:21.755850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:39:05.808153+0000","last_change":"2024-04-21T08:37:30.423320+0000","last_active":"2024-04-21T08:39:05.808153+0000","last_peered":"2024-04-21T08:39:05.808153+0000","last_clean":"2024-04-21T08:39:05.808153+0000","last_became_active":"2024-04-21T08:37:30.423020+0000","last_became_peered":"2024-04-21T08:37:30.423020+0000","last_unstale":"2024-04-21T08:39:05.808153+0000","last_undegraded":"2024-04-21T08:39:05.808153+0000","last_fullsized":"2024-04-21T08:39:05.808153+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:08:27.313196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.605814+0000","last_change":"2024-04-21T08:37:31.442425+0000","last_active":"2024-04-21T08:38:59.605814+0000","last_peered":"2024-04-21T08:38:59.605814+0000","last_clean":"2024-04-21T08:38:59.605814+0000","last_became_active":"2024-04-21T08:37:31.442028+0000","last_became_peered":"2024-04-21T08:37:31.442028+0000","last_unstale":"2024-04-21T08:38:59.605814+0000","last_undegraded":"2024-04-21T08:38:59.605814+0000","last_fullsized":"2024-04-21T08:38:59.605814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:45.498656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666252+0000","last_change":"2024-04-21T08:37:35.607829+0000","last_active":"2024-04-21T08:38:30.666252+0000","last_peered":"2024-04-21T08:38:30.666252+0000","last_clean":"2024-04-21T08:38:30.666252+0000","last_became_active":"2024-04-21T08:37:35.607150+0000","last_became_peered":"2024-04-21T08:37:35.607150+0000","last_unstale":"2024-04-21T08:38:30.666252+0000","last_undegraded":"2024-04-21T08:38:30.666252+0000","last_fullsized":"2024-04-21T08:38:30.666252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:01:34.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669039+0000","last_change":"2024-04-21T08:37:30.418613+0000","last_active":"2024-04-21T08:38:30.669039+0000","last_peered":"2024-04-21T08:38:30.669039+0000","last_clean":"2024-04-21T08:38:30.669039+0000","last_became_active":"2024-04-21T08:37:30.418300+0000","last_became_peered":"2024-04-21T08:37:30.418300+0000","last_unstale":"2024-04-21T08:38:30.669039+0000","last_undegraded":"2024-04-21T08:38:30.669039+0000","last_fullsized":"2024-04-21T08:38:30.669039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:17:35.370753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.596670+0000","last_change":"2024-04-21T08:37:31.456902+0000","last_active":"2024-04-21T08:38:59.596670+0000","last_peered":"2024-04-21T08:38:59.596670+0000","last_clean":"2024-04-21T08:38:59.596670+0000","last_became_active":"2024-04-21T08:37:31.453637+0000","last_became_peered":"2024-04-21T08:37:31.453637+0000","last_unstale":"2024-04-21T08:38:59.596670+0000","last_undegraded":"2024-04-21T08:38:59.596670+0000","last_fullsized":"2024-04-21T08:38:59.596670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:48:59.387349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.670000+0000","last_change":"2024-04-21T08:37:35.608280+0000","last_active":"2024-04-21T08:38:30.670000+0000","last_peered":"2024-04-21T08:38:30.670000+0000","last_clean":"2024-04-21T08:38:30.670000+0000","last_became_active":"2024-04-21T08:37:35.607898+0000","last_became_peered":"2024-04-21T08:37:35.607898+0000","last_unstale":"2024-04-21T08:38:30.670000+0000","last_undegraded":"2024-04-21T08:38:30.670000+0000","last_fullsized":"2024-04-21T08:38:30.670000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:55.689519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.675589+0000","last_change":"2024-04-21T08:37:30.417352+0000","last_active":"2024-04-21T08:38:30.675589+0000","last_peered":"2024-04-21T08:38:30.675589+0000","last_clean":"2024-04-21T08:38:30.675589+0000","last_became_active":"2024-04-21T08:37:30.417000+0000","last_became_peered":"2024-04-21T08:37:30.417000+0000","last_unstale":"2024-04-21T08:38:30.675589+0000","last_undegraded":"2024-04-21T08:38:30.675589+0000","last_fullsized":"2024-04-21T08:38:30.675589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:56.087334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.608925+0000","last_change":"2024-04-21T08:37:31.458229+0000","last_active":"2024-04-21T08:38:59.608925+0000","last_peered":"2024-04-21T08:38:59.608925+0000","last_clean":"2024-04-21T08:38:59.608925+0000","last_became_active":"2024-04-21T08:37:31.456897+0000","last_became_peered":"2024-04-21T08:37:31.456897+0000","last_unstale":"2024-04-21T08:38:59.608925+0000","last_undegraded":"2024-04-21T08:38:59.608925+0000","last_fullsized":"2024-04-21T08:38:59.608925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:25:58.253081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666373+0000","last_change":"2024-04-21T08:37:35.601077+0000","last_active":"2024-04-21T08:38:30.666373+0000","last_peered":"2024-04-21T08:38:30.666373+0000","last_clean":"2024-04-21T08:38:30.666373+0000","last_became_active":"2024-04-21T08:37:35.600673+0000","last_became_peered":"2024-04-21T08:37:35.600673+0000","last_unstale":"2024-04-21T08:38:30.666373+0000","last_undegraded":"2024-04-21T08:38:30.666373+0000","last_fullsized":"2024-04-21T08:38:30.666373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:04.554208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":214,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:39:06.809668+0000","last_change":"2024-04-21T08:37:30.421440+0000","last_active":"2024-04-21T08:39:06.809668+0000","last_peered":"2024-04-21T08:39:06.809668+0000","last_clean":"2024-04-21T08:39:06.809668+0000","last_became_active":"2024-04-21T08:37:30.420988+0000","last_became_peered":"2024-04-21T08:37:30.420988+0000","last_unstale":"2024-04-21T08:39:06.809668+0000","last_undegraded":"2024-04-21T08:39:06.809668+0000","last_fullsized":"2024-04-21T08:39:06.809668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:38:03.850705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":138,"num_write":37,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.595271+0000","last_change":"2024-04-21T08:37:31.447208+0000","last_active":"2024-04-21T08:38:59.595271+0000","last_peered":"2024-04-21T08:38:59.595271+0000","last_clean":"2024-04-21T08:38:59.595271+0000","last_became_active":"2024-04-21T08:37:31.444464+0000","last_became_peered":"2024-04-21T08:37:31.444464+0000","last_unstale":"2024-04-21T08:38:59.595271+0000","last_undegraded":"2024-04-21T08:38:59.595271+0000","last_fullsized":"2024-04-21T08:38:59.595271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:33.805676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.675797+0000","last_change":"2024-04-21T08:37:35.603937+0000","last_active":"2024-04-21T08:38:30.675797+0000","last_peered":"2024-04-21T08:38:30.675797+0000","last_clean":"2024-04-21T08:38:30.675797+0000","last_became_active":"2024-04-21T08:37:35.603477+0000","last_became_peered":"2024-04-21T08:37:35.603477+0000","last_unstale":"2024-04-21T08:38:30.675797+0000","last_undegraded":"2024-04-21T08:38:30.675797+0000","last_fullsized":"2024-04-21T08:38:30.675797+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:40:33.901570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.667786+0000","last_change":"2024-04-21T08:37:30.416941+0000","last_active":"2024-04-21T08:38:30.667786+0000","last_peered":"2024-04-21T08:38:30.667786+0000","last_clean":"2024-04-21T08:38:30.667786+0000","last_became_active":"2024-04-21T08:37:30.416625+0000","last_became_peered":"2024-04-21T08:37:30.416625+0000","last_unstale":"2024-04-21T08:38:30.667786+0000","last_undegraded":"2024-04-21T08:38:30.667786+0000","last_fullsized":"2024-04-21T08:38:30.667786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:07.782779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.592616+0000","last_change":"2024-04-21T08:37:31.441707+0000","last_active":"2024-04-21T08:38:59.592616+0000","last_peered":"2024-04-21T08:38:59.592616+0000","last_clean":"2024-04-21T08:38:59.592616+0000","last_became_active":"2024-04-21T08:37:31.441366+0000","last_became_peered":"2024-04-21T08:37:31.441366+0000","last_unstale":"2024-04-21T08:38:59.592616+0000","last_undegraded":"2024-04-21T08:38:59.592616+0000","last_fullsized":"2024-04-21T08:38:59.592616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:16.573804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669837+0000","last_change":"2024-04-21T08:37:35.596214+0000","last_active":"2024-04-21T08:38:30.669837+0000","last_peered":"2024-04-21T08:38:30.669837+0000","last_clean":"2024-04-21T08:38:30.669837+0000","last_became_active":"2024-04-21T08:37:35.595773+0000","last_became_peered":"2024-04-21T08:37:35.595773+0000","last_unstale":"2024-04-21T08:38:30.669837+0000","last_undegraded":"2024-04-21T08:38:30.669837+0000","last_fullsized":"2024-04-21T08:38:30.669837+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:07.538190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.684828+0000","last_change":"2024-04-21T08:38:30.684828+0000","last_active":"2024-04-21T08:38:30.684828+0000","last_peered":"2024-04-21T08:38:30.684828+0000","last_clean":"2024-04-21T08:38:30.684828+0000","last_became_active":"2024-04-21T08:38:30.673923+0000","last_became_peered":"2024-04-21T08:38:30.673923+0000","last_unstale":"2024-04-21T08:38:30.684828+0000","last_undegraded":"2024-04-21T08:38:30.684828+0000","last_fullsized":"2024-04-21T08:38:30.684828+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:51:37.906150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.603733+0000","last_change":"2024-04-21T08:37:31.453680+0000","last_active":"2024-04-21T08:38:59.603733+0000","last_peered":"2024-04-21T08:38:59.603733+0000","last_clean":"2024-04-21T08:38:59.603733+0000","last_became_active":"2024-04-21T08:37:31.453319+0000","last_became_peered":"2024-04-21T08:37:31.453319+0000","last_unstale":"2024-04-21T08:38:59.603733+0000","last_undegraded":"2024-04-21T08:38:59.603733+0000","last_fullsized":"2024-04-21T08:38:59.603733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:43.139789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666696+0000","last_change":"2024-04-21T08:37:35.608906+0000","last_active":"2024-04-21T08:38:30.666696+0000","last_peered":"2024-04-21T08:38:30.666696+0000","last_clean":"2024-04-21T08:38:30.666696+0000","last_became_active":"2024-04-21T08:37:35.608535+0000","last_became_peered":"2024-04-21T08:37:35.608535+0000","last_unstale":"2024-04-21T08:38:30.666696+0000","last_undegraded":"2024-04-21T08:38:30.666696+0000","last_fullsized":"2024-04-21T08:38:30.666696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:58:01.422308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:39:08.812983+0000","last_change":"2024-04-21T08:37:30.414865+0000","last_active":"2024-04-21T08:39:08.812983+0000","last_peered":"2024-04-21T08:39:08.812983+0000","last_clean":"2024-04-21T08:39:08.812983+0000","last_became_active":"2024-04-21T08:37:30.414471+0000","last_became_peered":"2024-04-21T08:37:30.414471+0000","last_unstale":"2024-04-21T08:39:08.812983+0000","last_undegraded":"2024-04-21T08:39:08.812983+0000","last_fullsized":"2024-04-21T08:39:08.812983+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:46.943574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.597393+0000","last_change":"2024-04-21T08:37:31.456634+0000","last_active":"2024-04-21T08:38:59.597393+0000","last_peered":"2024-04-21T08:38:59.597393+0000","last_clean":"2024-04-21T08:38:59.597393+0000","last_became_active":"2024-04-21T08:37:31.456278+0000","last_became_peered":"2024-04-21T08:37:31.456278+0000","last_unstale":"2024-04-21T08:38:59.597393+0000","last_undegraded":"2024-04-21T08:38:59.597393+0000","last_fullsized":"2024-04-21T08:38:59.597393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:04.870024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.672121+0000","last_change":"2024-04-21T08:37:35.601760+0000","last_active":"2024-04-21T08:38:30.672121+0000","last_peered":"2024-04-21T08:38:30.672121+0000","last_clean":"2024-04-21T08:38:30.672121+0000","last_became_active":"2024-04-21T08:37:35.601332+0000","last_became_peered":"2024-04-21T08:37:35.601332+0000","last_unstale":"2024-04-21T08:38:30.672121+0000","last_undegraded":"2024-04-21T08:38:30.672121+0000","last_fullsized":"2024-04-21T08:38:30.672121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:04:03.694581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669050+0000","last_change":"2024-04-21T08:37:30.422419+0000","last_active":"2024-04-21T08:38:30.669050+0000","last_peered":"2024-04-21T08:38:30.669050+0000","last_clean":"2024-04-21T08:38:30.669050+0000","last_became_active":"2024-04-21T08:37:30.422123+0000","last_became_peered":"2024-04-21T08:37:30.422123+0000","last_unstale":"2024-04-21T08:38:30.669050+0000","last_undegraded":"2024-04-21T08:38:30.669050+0000","last_fullsized":"2024-04-21T08:38:30.669050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:32:54.029041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.607287+0000","last_change":"2024-04-21T08:37:31.458992+0000","last_active":"2024-04-21T08:38:59.607287+0000","last_peered":"2024-04-21T08:38:59.607287+0000","last_clean":"2024-04-21T08:38:59.607287+0000","last_became_active":"2024-04-21T08:37:31.458635+0000","last_became_peered":"2024-04-21T08:37:31.458635+0000","last_unstale":"2024-04-21T08:38:59.607287+0000","last_undegraded":"2024-04-21T08:38:59.607287+0000","last_fullsized":"2024-04-21T08:38:59.607287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:48:14.325571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.674535+0000","last_change":"2024-04-21T08:37:35.603739+0000","last_active":"2024-04-21T08:38:30.674535+0000","last_peered":"2024-04-21T08:38:30.674535+0000","last_clean":"2024-04-21T08:38:30.674535+0000","last_became_active":"2024-04-21T08:37:35.603310+0000","last_became_peered":"2024-04-21T08:37:35.603310+0000","last_unstale":"2024-04-21T08:38:30.674535+0000","last_undegraded":"2024-04-21T08:38:30.674535+0000","last_fullsized":"2024-04-21T08:38:30.674535+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:19.162520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669889+0000","last_change":"2024-04-21T08:37:30.418104+0000","last_active":"2024-04-21T08:38:30.669889+0000","last_peered":"2024-04-21T08:38:30.669889+0000","last_clean":"2024-04-21T08:38:30.669889+0000","last_became_active":"2024-04-21T08:37:30.417814+0000","last_became_peered":"2024-04-21T08:37:30.417814+0000","last_unstale":"2024-04-21T08:38:30.669889+0000","last_undegraded":"2024-04-21T08:38:30.669889+0000","last_fullsized":"2024-04-21T08:38:30.669889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:50:56.696513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.601328+0000","last_change":"2024-04-21T08:37:31.451952+0000","last_active":"2024-04-21T08:38:59.601328+0000","last_peered":"2024-04-21T08:38:59.601328+0000","last_clean":"2024-04-21T08:38:59.601328+0000","last_became_active":"2024-04-21T08:37:31.451588+0000","last_became_peered":"2024-04-21T08:37:31.451588+0000","last_unstale":"2024-04-21T08:38:59.601328+0000","last_undegraded":"2024-04-21T08:38:59.601328+0000","last_fullsized":"2024-04-21T08:38:59.601328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:06.506769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669424+0000","last_change":"2024-04-21T08:37:35.760661+0000","last_active":"2024-04-21T08:38:30.669424+0000","last_peered":"2024-04-21T08:38:30.669424+0000","last_clean":"2024-04-21T08:38:30.669424+0000","last_became_active":"2024-04-21T08:37:35.759925+0000","last_became_peered":"2024-04-21T08:37:35.759925+0000","last_unstale":"2024-04-21T08:38:30.669424+0000","last_undegraded":"2024-04-21T08:38:30.669424+0000","last_fullsized":"2024-04-21T08:38:30.669424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:11:53.062251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666484+0000","last_change":"2024-04-21T08:37:30.416487+0000","last_active":"2024-04-21T08:38:30.666484+0000","last_peered":"2024-04-21T08:38:30.666484+0000","last_clean":"2024-04-21T08:38:30.666484+0000","last_became_active":"2024-04-21T08:37:30.416076+0000","last_became_peered":"2024-04-21T08:37:30.416076+0000","last_unstale":"2024-04-21T08:38:30.666484+0000","last_undegraded":"2024-04-21T08:38:30.666484+0000","last_fullsized":"2024-04-21T08:38:30.666484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:18:33.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599128+0000","last_change":"2024-04-21T08:37:31.448824+0000","last_active":"2024-04-21T08:38:59.599128+0000","last_peered":"2024-04-21T08:38:59.599128+0000","last_clean":"2024-04-21T08:38:59.599128+0000","last_became_active":"2024-04-21T08:37:31.448432+0000","last_became_peered":"2024-04-21T08:37:31.448432+0000","last_unstale":"2024-04-21T08:38:59.599128+0000","last_undegraded":"2024-04-21T08:38:59.599128+0000","last_fullsized":"2024-04-21T08:38:59.599128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:41.111272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.670091+0000","last_change":"2024-04-21T08:37:35.604539+0000","last_active":"2024-04-21T08:38:30.670091+0000","last_peered":"2024-04-21T08:38:30.670091+0000","last_clean":"2024-04-21T08:38:30.670091+0000","last_became_active":"2024-04-21T08:37:35.604132+0000","last_became_peered":"2024-04-21T08:37:35.604132+0000","last_unstale":"2024-04-21T08:38:30.670091+0000","last_undegraded":"2024-04-21T08:38:30.670091+0000","last_fullsized":"2024-04-21T08:38:30.670091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:12.180730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669445+0000","last_change":"2024-04-21T08:37:30.418854+0000","last_active":"2024-04-21T08:38:30.669445+0000","last_peered":"2024-04-21T08:38:30.669445+0000","last_clean":"2024-04-21T08:38:30.669445+0000","last_became_active":"2024-04-21T08:37:30.418511+0000","last_became_peered":"2024-04-21T08:37:30.418511+0000","last_unstale":"2024-04-21T08:38:30.669445+0000","last_undegraded":"2024-04-21T08:38:30.669445+0000","last_fullsized":"2024-04-21T08:38:30.669445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:26.751924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.587033+0000","last_change":"2024-04-21T08:37:31.453365+0000","last_active":"2024-04-21T08:38:59.587033+0000","last_peered":"2024-04-21T08:38:59.587033+0000","last_clean":"2024-04-21T08:38:59.587033+0000","last_became_active":"2024-04-21T08:37:31.453042+0000","last_became_peered":"2024-04-21T08:37:31.453042+0000","last_unstale":"2024-04-21T08:38:59.587033+0000","last_undegraded":"2024-04-21T08:38:59.587033+0000","last_fullsized":"2024-04-21T08:38:59.587033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:49:46.648754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.671486+0000","last_change":"2024-04-21T08:37:35.760626+0000","last_active":"2024-04-21T08:38:30.671486+0000","last_peered":"2024-04-21T08:38:30.671486+0000","last_clean":"2024-04-21T08:38:30.671486+0000","last_became_active":"2024-04-21T08:37:35.760125+0000","last_became_peered":"2024-04-21T08:37:35.760125+0000","last_unstale":"2024-04-21T08:38:30.671486+0000","last_undegraded":"2024-04-21T08:38:30.671486+0000","last_fullsized":"2024-04-21T08:38:30.671486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:58.366557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599773+0000","last_change":"2024-04-21T08:37:31.457654+0000","last_active":"2024-04-21T08:38:59.599773+0000","last_peered":"2024-04-21T08:38:59.599773+0000","last_clean":"2024-04-21T08:38:59.599773+0000","last_became_active":"2024-04-21T08:37:31.456950+0000","last_became_peered":"2024-04-21T08:37:31.456950+0000","last_unstale":"2024-04-21T08:38:59.599773+0000","last_undegraded":"2024-04-21T08:38:59.599773+0000","last_fullsized":"2024-04-21T08:38:59.599773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:02.863000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:39:08.812171+0000","last_change":"2024-04-21T08:37:30.422021+0000","last_active":"2024-04-21T08:39:08.812171+0000","last_peered":"2024-04-21T08:39:08.812171+0000","last_clean":"2024-04-21T08:39:08.812171+0000","last_became_active":"2024-04-21T08:37:30.421711+0000","last_became_peered":"2024-04-21T08:37:30.421711+0000","last_unstale":"2024-04-21T08:39:08.812171+0000","last_undegraded":"2024-04-21T08:39:08.812171+0000","last_fullsized":"2024-04-21T08:39:08.812171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:01.612205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669220+0000","last_change":"2024-04-21T08:37:35.598041+0000","last_active":"2024-04-21T08:38:30.669220+0000","last_peered":"2024-04-21T08:38:30.669220+0000","last_clean":"2024-04-21T08:38:30.669220+0000","last_became_active":"2024-04-21T08:37:35.597635+0000","last_became_peered":"2024-04-21T08:37:35.597635+0000","last_unstale":"2024-04-21T08:38:30.669220+0000","last_undegraded":"2024-04-21T08:38:30.669220+0000","last_fullsized":"2024-04-21T08:38:30.669220+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:35:27.452340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598020+0000","last_change":"2024-04-21T08:37:31.450423+0000","last_active":"2024-04-21T08:38:59.598020+0000","last_peered":"2024-04-21T08:38:59.598020+0000","last_clean":"2024-04-21T08:38:59.598020+0000","last_became_active":"2024-04-21T08:37:31.450112+0000","last_became_peered":"2024-04-21T08:37:31.450112+0000","last_unstale":"2024-04-21T08:38:59.598020+0000","last_undegraded":"2024-04-21T08:38:59.598020+0000","last_fullsized":"2024-04-21T08:38:59.598020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:10:47.006769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669874+0000","last_change":"2024-04-21T08:37:30.416519+0000","last_active":"2024-04-21T08:38:30.669874+0000","last_peered":"2024-04-21T08:38:30.669874+0000","last_clean":"2024-04-21T08:38:30.669874+0000","last_became_active":"2024-04-21T08:37:30.416131+0000","last_became_peered":"2024-04-21T08:37:30.416131+0000","last_unstale":"2024-04-21T08:38:30.669874+0000","last_undegraded":"2024-04-21T08:38:30.669874+0000","last_fullsized":"2024-04-21T08:38:30.669874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:12.280847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":362,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.668877+0000","last_change":"2024-04-21T08:36:20.257759+0000","last_active":"2024-04-21T08:38:30.668877+0000","last_peered":"2024-04-21T08:38:30.668877+0000","last_clean":"2024-04-21T08:38:30.668877+0000","last_became_active":"2024-04-21T08:36:20.257313+0000","last_became_peered":"2024-04-21T08:36:20.257313+0000","last_unstale":"2024-04-21T08:38:30.668877+0000","last_undegraded":"2024-04-21T08:38:30.668877+0000","last_fullsized":"2024-04-21T08:38:30.668877+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_clean_scrub_stamp":"2024-04-21T08:32:56.546295+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:42:44.393447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.675737+0000","last_change":"2024-04-21T08:37:35.761562+0000","last_active":"2024-04-21T08:38:30.675737+0000","last_peered":"2024-04-21T08:38:30.675737+0000","last_clean":"2024-04-21T08:38:30.675737+0000","last_became_active":"2024-04-21T08:37:35.760703+0000","last_became_peered":"2024-04-21T08:37:35.760703+0000","last_unstale":"2024-04-21T08:38:30.675737+0000","last_undegraded":"2024-04-21T08:38:30.675737+0000","last_fullsized":"2024-04-21T08:38:30.675737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:12:47.511117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.587203+0000","last_change":"2024-04-21T08:37:31.453645+0000","last_active":"2024-04-21T08:38:59.587203+0000","last_peered":"2024-04-21T08:38:59.587203+0000","last_clean":"2024-04-21T08:38:59.587203+0000","last_became_active":"2024-04-21T08:37:31.453297+0000","last_became_peered":"2024-04-21T08:37:31.453297+0000","last_unstale":"2024-04-21T08:38:59.587203+0000","last_undegraded":"2024-04-21T08:38:59.587203+0000","last_fullsized":"2024-04-21T08:38:59.587203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:58.976327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.668944+0000","last_change":"2024-04-21T08:37:30.420560+0000","last_active":"2024-04-21T08:38:30.668944+0000","last_peered":"2024-04-21T08:38:30.668944+0000","last_clean":"2024-04-21T08:38:30.668944+0000","last_became_active":"2024-04-21T08:37:30.420223+0000","last_became_peered":"2024-04-21T08:37:30.420223+0000","last_unstale":"2024-04-21T08:38:30.668944+0000","last_undegraded":"2024-04-21T08:38:30.668944+0000","last_fullsized":"2024-04-21T08:38:30.668944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:43:55.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669718+0000","last_change":"2024-04-21T08:37:35.598294+0000","last_active":"2024-04-21T08:38:30.669718+0000","last_peered":"2024-04-21T08:38:30.669718+0000","last_clean":"2024-04-21T08:38:30.669718+0000","last_became_active":"2024-04-21T08:37:35.597883+0000","last_became_peered":"2024-04-21T08:37:35.597883+0000","last_unstale":"2024-04-21T08:38:30.669718+0000","last_undegraded":"2024-04-21T08:38:30.669718+0000","last_fullsized":"2024-04-21T08:38:30.669718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:24:05.144763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.670262+0000","last_change":"2024-04-21T08:37:30.423182+0000","last_active":"2024-04-21T08:38:30.670262+0000","last_peered":"2024-04-21T08:38:30.670262+0000","last_clean":"2024-04-21T08:38:30.670262+0000","last_became_active":"2024-04-21T08:37:30.422792+0000","last_became_peered":"2024-04-21T08:37:30.422792+0000","last_unstale":"2024-04-21T08:38:30.670262+0000","last_undegraded":"2024-04-21T08:38:30.670262+0000","last_fullsized":"2024-04-21T08:38:30.670262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:44:00.727269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.587317+0000","last_change":"2024-04-21T08:37:31.444013+0000","last_active":"2024-04-21T08:38:59.587317+0000","last_peered":"2024-04-21T08:38:59.587317+0000","last_clean":"2024-04-21T08:38:59.587317+0000","last_became_active":"2024-04-21T08:37:31.443684+0000","last_became_peered":"2024-04-21T08:37:31.443684+0000","last_unstale":"2024-04-21T08:38:59.587317+0000","last_undegraded":"2024-04-21T08:38:59.587317+0000","last_fullsized":"2024-04-21T08:38:59.587317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:00:35.400818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.671757+0000","last_change":"2024-04-21T08:37:35.759817+0000","last_active":"2024-04-21T08:38:30.671757+0000","last_peered":"2024-04-21T08:38:30.671757+0000","last_clean":"2024-04-21T08:38:30.671757+0000","last_became_active":"2024-04-21T08:37:35.759396+0000","last_became_peered":"2024-04-21T08:37:35.759396+0000","last_unstale":"2024-04-21T08:38:30.671757+0000","last_undegraded":"2024-04-21T08:38:30.671757+0000","last_fullsized":"2024-04-21T08:38:30.671757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:20:36.170880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666871+0000","last_change":"2024-04-21T08:37:30.414102+0000","last_active":"2024-04-21T08:38:30.666871+0000","last_peered":"2024-04-21T08:38:30.666871+0000","last_clean":"2024-04-21T08:38:30.666871+0000","last_became_active":"2024-04-21T08:37:30.413670+0000","last_became_peered":"2024-04-21T08:37:30.413670+0000","last_unstale":"2024-04-21T08:38:30.666871+0000","last_undegraded":"2024-04-21T08:38:30.666871+0000","last_fullsized":"2024-04-21T08:38:30.666871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:12:49.696516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598334+0000","last_change":"2024-04-21T08:37:31.457451+0000","last_active":"2024-04-21T08:38:59.598334+0000","last_peered":"2024-04-21T08:38:59.598334+0000","last_clean":"2024-04-21T08:38:59.598334+0000","last_became_active":"2024-04-21T08:37:31.457120+0000","last_became_peered":"2024-04-21T08:37:31.457120+0000","last_unstale":"2024-04-21T08:38:59.598334+0000","last_undegraded":"2024-04-21T08:38:59.598334+0000","last_fullsized":"2024-04-21T08:38:59.598334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:07:48.025843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666861+0000","last_change":"2024-04-21T08:37:35.600815+0000","last_active":"2024-04-21T08:38:30.666861+0000","last_peered":"2024-04-21T08:38:30.666861+0000","last_clean":"2024-04-21T08:38:30.666861+0000","last_became_active":"2024-04-21T08:37:35.600380+0000","last_became_peered":"2024-04-21T08:37:35.600380+0000","last_unstale":"2024-04-21T08:38:30.666861+0000","last_undegraded":"2024-04-21T08:38:30.666861+0000","last_fullsized":"2024-04-21T08:38:30.666861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:19:33.710431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.670129+0000","last_change":"2024-04-21T08:37:30.425592+0000","last_active":"2024-04-21T08:38:30.670129+0000","last_peered":"2024-04-21T08:38:30.670129+0000","last_clean":"2024-04-21T08:38:30.670129+0000","last_became_active":"2024-04-21T08:37:30.425274+0000","last_became_peered":"2024-04-21T08:37:30.425274+0000","last_unstale":"2024-04-21T08:38:30.670129+0000","last_undegraded":"2024-04-21T08:38:30.670129+0000","last_fullsized":"2024-04-21T08:38:30.670129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:27:29.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.596706+0000","last_change":"2024-04-21T08:37:31.455393+0000","last_active":"2024-04-21T08:38:59.596706+0000","last_peered":"2024-04-21T08:38:59.596706+0000","last_clean":"2024-04-21T08:38:59.596706+0000","last_became_active":"2024-04-21T08:37:31.455065+0000","last_became_peered":"2024-04-21T08:37:31.455065+0000","last_unstale":"2024-04-21T08:38:59.596706+0000","last_undegraded":"2024-04-21T08:38:59.596706+0000","last_fullsized":"2024-04-21T08:38:59.596706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:05:23.386790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.713521+0000","last_change":"2024-04-21T08:38:30.713521+0000","last_active":"2024-04-21T08:38:30.713521+0000","last_peered":"2024-04-21T08:38:30.713521+0000","last_clean":"2024-04-21T08:38:30.713521+0000","last_became_active":"2024-04-21T08:38:30.713096+0000","last_became_peered":"2024-04-21T08:38:30.713096+0000","last_unstale":"2024-04-21T08:38:30.713521+0000","last_undegraded":"2024-04-21T08:38:30.713521+0000","last_fullsized":"2024-04-21T08:38:30.713521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:54:59.078776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666980+0000","last_change":"2024-04-21T08:37:30.416696+0000","last_active":"2024-04-21T08:38:30.666980+0000","last_peered":"2024-04-21T08:38:30.666980+0000","last_clean":"2024-04-21T08:38:30.666980+0000","last_became_active":"2024-04-21T08:37:30.415868+0000","last_became_peered":"2024-04-21T08:37:30.415868+0000","last_unstale":"2024-04-21T08:38:30.666980+0000","last_undegraded":"2024-04-21T08:38:30.666980+0000","last_fullsized":"2024-04-21T08:38:30.666980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:48:08.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.557882+0000","last_change":"2024-04-21T08:37:31.453861+0000","last_active":"2024-04-21T08:38:59.557882+0000","last_peered":"2024-04-21T08:38:59.557882+0000","last_clean":"2024-04-21T08:38:59.557882+0000","last_became_active":"2024-04-21T08:37:31.453562+0000","last_became_peered":"2024-04-21T08:37:31.453562+0000","last_unstale":"2024-04-21T08:38:59.557882+0000","last_undegraded":"2024-04-21T08:38:59.557882+0000","last_fullsized":"2024-04-21T08:38:59.557882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:17:50.513917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666749+0000","last_change":"2024-04-21T08:37:35.609217+0000","last_active":"2024-04-21T08:38:30.666749+0000","last_peered":"2024-04-21T08:38:30.666749+0000","last_clean":"2024-04-21T08:38:30.666749+0000","last_became_active":"2024-04-21T08:37:35.608786+0000","last_became_peered":"2024-04-21T08:37:35.608786+0000","last_unstale":"2024-04-21T08:38:30.666749+0000","last_undegraded":"2024-04-21T08:38:30.666749+0000","last_fullsized":"2024-04-21T08:38:30.666749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:45:50.945845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.672091+0000","last_change":"2024-04-21T08:37:30.417156+0000","last_active":"2024-04-21T08:38:30.672091+0000","last_peered":"2024-04-21T08:38:30.672091+0000","last_clean":"2024-04-21T08:38:30.672091+0000","last_became_active":"2024-04-21T08:37:30.415717+0000","last_became_peered":"2024-04-21T08:37:30.415717+0000","last_unstale":"2024-04-21T08:38:30.672091+0000","last_undegraded":"2024-04-21T08:38:30.672091+0000","last_fullsized":"2024-04-21T08:38:30.672091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:37.819082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.596691+0000","last_change":"2024-04-21T08:37:31.444948+0000","last_active":"2024-04-21T08:38:59.596691+0000","last_peered":"2024-04-21T08:38:59.596691+0000","last_clean":"2024-04-21T08:38:59.596691+0000","last_became_active":"2024-04-21T08:37:31.444604+0000","last_became_peered":"2024-04-21T08:37:31.444604+0000","last_unstale":"2024-04-21T08:38:59.596691+0000","last_undegraded":"2024-04-21T08:38:59.596691+0000","last_fullsized":"2024-04-21T08:38:59.596691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.820503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.670106+0000","last_change":"2024-04-21T08:37:35.611541+0000","last_active":"2024-04-21T08:38:30.670106+0000","last_peered":"2024-04-21T08:38:30.670106+0000","last_clean":"2024-04-21T08:38:30.670106+0000","last_became_active":"2024-04-21T08:37:35.611136+0000","last_became_peered":"2024-04-21T08:37:35.611136+0000","last_unstale":"2024-04-21T08:38:30.670106+0000","last_undegraded":"2024-04-21T08:38:30.670106+0000","last_fullsized":"2024-04-21T08:38:30.670106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:28.254529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.667288+0000","last_change":"2024-04-21T08:37:30.414216+0000","last_active":"2024-04-21T08:38:30.667288+0000","last_peered":"2024-04-21T08:38:30.667288+0000","last_clean":"2024-04-21T08:38:30.667288+0000","last_became_active":"2024-04-21T08:37:30.413890+0000","last_became_peered":"2024-04-21T08:37:30.413890+0000","last_unstale":"2024-04-21T08:38:30.667288+0000","last_undegraded":"2024-04-21T08:38:30.667288+0000","last_fullsized":"2024-04-21T08:38:30.667288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:16.972618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599236+0000","last_change":"2024-04-21T08:37:31.449307+0000","last_active":"2024-04-21T08:38:59.599236+0000","last_peered":"2024-04-21T08:38:59.599236+0000","last_clean":"2024-04-21T08:38:59.599236+0000","last_became_active":"2024-04-21T08:37:31.448980+0000","last_became_peered":"2024-04-21T08:37:31.448980+0000","last_unstale":"2024-04-21T08:38:59.599236+0000","last_undegraded":"2024-04-21T08:38:59.599236+0000","last_fullsized":"2024-04-21T08:38:59.599236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:30:31.939274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669722+0000","last_change":"2024-04-21T08:37:35.760616+0000","last_active":"2024-04-21T08:38:30.669722+0000","last_peered":"2024-04-21T08:38:30.669722+0000","last_clean":"2024-04-21T08:38:30.669722+0000","last_became_active":"2024-04-21T08:37:35.760131+0000","last_became_peered":"2024-04-21T08:37:35.760131+0000","last_unstale":"2024-04-21T08:38:30.669722+0000","last_undegraded":"2024-04-21T08:38:30.669722+0000","last_fullsized":"2024-04-21T08:38:30.669722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:51.510259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.682791+0000","last_change":"2024-04-21T08:38:30.682791+0000","last_active":"2024-04-21T08:38:30.682791+0000","last_peered":"2024-04-21T08:38:30.682791+0000","last_clean":"2024-04-21T08:38:30.682791+0000","last_became_active":"2024-04-21T08:38:30.671977+0000","last_became_peered":"2024-04-21T08:38:30.671977+0000","last_unstale":"2024-04-21T08:38:30.682791+0000","last_undegraded":"2024-04-21T08:38:30.682791+0000","last_fullsized":"2024-04-21T08:38:30.682791+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:55:15.899399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.607675+0000","last_change":"2024-04-21T08:37:31.447234+0000","last_active":"2024-04-21T08:38:59.607675+0000","last_peered":"2024-04-21T08:38:59.607675+0000","last_clean":"2024-04-21T08:38:59.607675+0000","last_became_active":"2024-04-21T08:37:31.446876+0000","last_became_peered":"2024-04-21T08:37:31.446876+0000","last_unstale":"2024-04-21T08:38:59.607675+0000","last_undegraded":"2024-04-21T08:38:59.607675+0000","last_fullsized":"2024-04-21T08:38:59.607675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:51.784126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.667256+0000","last_change":"2024-04-21T08:37:35.598577+0000","last_active":"2024-04-21T08:38:30.667256+0000","last_peered":"2024-04-21T08:38:30.667256+0000","last_clean":"2024-04-21T08:38:30.667256+0000","last_became_active":"2024-04-21T08:37:35.598119+0000","last_became_peered":"2024-04-21T08:37:35.598119+0000","last_unstale":"2024-04-21T08:38:30.667256+0000","last_undegraded":"2024-04-21T08:38:30.667256+0000","last_fullsized":"2024-04-21T08:38:30.667256+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:41.986674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.667632+0000","last_change":"2024-04-21T08:37:30.425737+0000","last_active":"2024-04-21T08:38:30.667632+0000","last_peered":"2024-04-21T08:38:30.667632+0000","last_clean":"2024-04-21T08:38:30.667632+0000","last_became_active":"2024-04-21T08:37:30.425180+0000","last_became_peered":"2024-04-21T08:37:30.425180+0000","last_unstale":"2024-04-21T08:38:30.667632+0000","last_undegraded":"2024-04-21T08:38:30.667632+0000","last_fullsized":"2024-04-21T08:38:30.667632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:08.646106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.608800+0000","last_change":"2024-04-21T08:37:31.451505+0000","last_active":"2024-04-21T08:38:59.608800+0000","last_peered":"2024-04-21T08:38:59.608800+0000","last_clean":"2024-04-21T08:38:59.608800+0000","last_became_active":"2024-04-21T08:37:31.451146+0000","last_became_peered":"2024-04-21T08:37:31.451146+0000","last_unstale":"2024-04-21T08:38:59.608800+0000","last_undegraded":"2024-04-21T08:38:59.608800+0000","last_fullsized":"2024-04-21T08:38:59.608800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:27:01.793910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.802256+0000","last_change":"2024-04-21T08:37:35.608995+0000","last_active":"2024-04-21T08:38:30.802256+0000","last_peered":"2024-04-21T08:38:30.802256+0000","last_clean":"2024-04-21T08:38:30.802256+0000","last_became_active":"2024-04-21T08:37:35.608587+0000","last_became_peered":"2024-04-21T08:37:35.608587+0000","last_unstale":"2024-04-21T08:38:30.802256+0000","last_undegraded":"2024-04-21T08:38:30.802256+0000","last_fullsized":"2024-04-21T08:38:30.802256+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.829418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.675692+0000","last_change":"2024-04-21T08:37:30.419026+0000","last_active":"2024-04-21T08:38:30.675692+0000","last_peered":"2024-04-21T08:38:30.675692+0000","last_clean":"2024-04-21T08:38:30.675692+0000","last_became_active":"2024-04-21T08:37:30.418706+0000","last_became_peered":"2024-04-21T08:37:30.418706+0000","last_unstale":"2024-04-21T08:38:30.675692+0000","last_undegraded":"2024-04-21T08:38:30.675692+0000","last_fullsized":"2024-04-21T08:38:30.675692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:24.441918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599755+0000","last_change":"2024-04-21T08:37:31.446561+0000","last_active":"2024-04-21T08:38:59.599755+0000","last_peered":"2024-04-21T08:38:59.599755+0000","last_clean":"2024-04-21T08:38:59.599755+0000","last_became_active":"2024-04-21T08:37:31.446162+0000","last_became_peered":"2024-04-21T08:37:31.446162+0000","last_unstale":"2024-04-21T08:38:59.599755+0000","last_undegraded":"2024-04-21T08:38:59.599755+0000","last_fullsized":"2024-04-21T08:38:59.599755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:38:24.684537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.674131+0000","last_change":"2024-04-21T08:37:35.598748+0000","last_active":"2024-04-21T08:38:30.674131+0000","last_peered":"2024-04-21T08:38:30.674131+0000","last_clean":"2024-04-21T08:38:30.674131+0000","last_became_active":"2024-04-21T08:37:35.598353+0000","last_became_peered":"2024-04-21T08:37:35.598353+0000","last_unstale":"2024-04-21T08:38:30.674131+0000","last_undegraded":"2024-04-21T08:38:30.674131+0000","last_fullsized":"2024-04-21T08:38:30.674131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:47:36.519694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.691292+0000","last_change":"2024-04-21T08:38:30.691292+0000","last_active":"2024-04-21T08:38:30.691292+0000","last_peered":"2024-04-21T08:38:30.691292+0000","last_clean":"2024-04-21T08:38:30.691292+0000","last_became_active":"2024-04-21T08:38:30.676580+0000","last_became_peered":"2024-04-21T08:38:30.676580+0000","last_unstale":"2024-04-21T08:38:30.691292+0000","last_undegraded":"2024-04-21T08:38:30.691292+0000","last_fullsized":"2024-04-21T08:38:30.691292+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:43:53.150369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.591583+0000","last_change":"2024-04-21T08:37:31.452367+0000","last_active":"2024-04-21T08:38:59.591583+0000","last_peered":"2024-04-21T08:38:59.591583+0000","last_clean":"2024-04-21T08:38:59.591583+0000","last_became_active":"2024-04-21T08:37:31.452031+0000","last_became_peered":"2024-04-21T08:37:31.452031+0000","last_unstale":"2024-04-21T08:38:59.591583+0000","last_undegraded":"2024-04-21T08:38:59.591583+0000","last_fullsized":"2024-04-21T08:38:59.591583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:58.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.596018+0000","last_change":"2024-04-21T08:37:31.451222+0000","last_active":"2024-04-21T08:38:59.596018+0000","last_peered":"2024-04-21T08:38:59.596018+0000","last_clean":"2024-04-21T08:38:59.596018+0000","last_became_active":"2024-04-21T08:37:31.450869+0000","last_became_peered":"2024-04-21T08:37:31.450869+0000","last_unstale":"2024-04-21T08:38:59.596018+0000","last_undegraded":"2024-04-21T08:38:59.596018+0000","last_fullsized":"2024-04-21T08:38:59.596018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:17.051330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.602242+0000","last_change":"2024-04-21T08:37:31.453415+0000","last_active":"2024-04-21T08:38:59.602242+0000","last_peered":"2024-04-21T08:38:59.602242+0000","last_clean":"2024-04-21T08:38:59.602242+0000","last_became_active":"2024-04-21T08:37:31.453089+0000","last_became_peered":"2024-04-21T08:37:31.453089+0000","last_unstale":"2024-04-21T08:38:59.602242+0000","last_undegraded":"2024-04-21T08:38:59.602242+0000","last_fullsized":"2024-04-21T08:38:59.602242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:25:44.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.592759+0000","last_change":"2024-04-21T08:37:31.446707+0000","last_active":"2024-04-21T08:38:59.592759+0000","last_peered":"2024-04-21T08:38:59.592759+0000","last_clean":"2024-04-21T08:38:59.592759+0000","last_became_active":"2024-04-21T08:37:31.446340+0000","last_became_peered":"2024-04-21T08:37:31.446340+0000","last_unstale":"2024-04-21T08:38:59.592759+0000","last_undegraded":"2024-04-21T08:38:59.592759+0000","last_fullsized":"2024-04-21T08:38:59.592759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:13.649607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.596749+0000","last_change":"2024-04-21T08:37:31.455532+0000","last_active":"2024-04-21T08:38:59.596749+0000","last_peered":"2024-04-21T08:38:59.596749+0000","last_clean":"2024-04-21T08:38:59.596749+0000","last_became_active":"2024-04-21T08:37:31.453547+0000","last_became_peered":"2024-04-21T08:37:31.453547+0000","last_unstale":"2024-04-21T08:38:59.596749+0000","last_undegraded":"2024-04-21T08:38:59.596749+0000","last_fullsized":"2024-04-21T08:38:59.596749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:39:47.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.588944+0000","last_change":"2024-04-21T08:37:31.452535+0000","last_active":"2024-04-21T08:38:59.588944+0000","last_peered":"2024-04-21T08:38:59.588944+0000","last_clean":"2024-04-21T08:38:59.588944+0000","last_became_active":"2024-04-21T08:37:31.451637+0000","last_became_peered":"2024-04-21T08:37:31.451637+0000","last_unstale":"2024-04-21T08:38:59.588944+0000","last_undegraded":"2024-04-21T08:38:59.588944+0000","last_fullsized":"2024-04-21T08:38:59.588944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:38:38.162780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.597546+0000","last_change":"2024-04-21T08:37:31.451806+0000","last_active":"2024-04-21T08:38:59.597546+0000","last_peered":"2024-04-21T08:38:59.597546+0000","last_clean":"2024-04-21T08:38:59.597546+0000","last_became_active":"2024-04-21T08:37:31.451442+0000","last_became_peered":"2024-04-21T08:37:31.451442+0000","last_unstale":"2024-04-21T08:38:59.597546+0000","last_undegraded":"2024-04-21T08:38:59.597546+0000","last_fullsized":"2024-04-21T08:38:59.597546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:07.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.594529+0000","last_change":"2024-04-21T08:37:31.444440+0000","last_active":"2024-04-21T08:38:59.594529+0000","last_peered":"2024-04-21T08:38:59.594529+0000","last_clean":"2024-04-21T08:38:59.594529+0000","last_became_active":"2024-04-21T08:37:31.444079+0000","last_became_peered":"2024-04-21T08:37:31.444079+0000","last_unstale":"2024-04-21T08:38:59.594529+0000","last_undegraded":"2024-04-21T08:38:59.594529+0000","last_fullsized":"2024-04-21T08:38:59.594529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:13.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.604115+0000","last_change":"2024-04-21T08:37:31.453698+0000","last_active":"2024-04-21T08:38:59.604115+0000","last_peered":"2024-04-21T08:38:59.604115+0000","last_clean":"2024-04-21T08:38:59.604115+0000","last_became_active":"2024-04-21T08:37:31.453379+0000","last_became_peered":"2024-04-21T08:37:31.453379+0000","last_unstale":"2024-04-21T08:38:59.604115+0000","last_undegraded":"2024-04-21T08:38:59.604115+0000","last_fullsized":"2024-04-21T08:38:59.604115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:46:29.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.595960+0000","last_change":"2024-04-21T08:37:31.444480+0000","last_active":"2024-04-21T08:38:59.595960+0000","last_peered":"2024-04-21T08:38:59.595960+0000","last_clean":"2024-04-21T08:38:59.595960+0000","last_became_active":"2024-04-21T08:37:31.444030+0000","last_became_peered":"2024-04-21T08:37:31.444030+0000","last_unstale":"2024-04-21T08:38:59.595960+0000","last_undegraded":"2024-04-21T08:38:59.595960+0000","last_fullsized":"2024-04-21T08:38:59.595960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:15:05.668885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598820+0000","last_change":"2024-04-21T08:37:31.456602+0000","last_active":"2024-04-21T08:38:59.598820+0000","last_peered":"2024-04-21T08:38:59.598820+0000","last_clean":"2024-04-21T08:38:59.598820+0000","last_became_active":"2024-04-21T08:37:31.456286+0000","last_became_peered":"2024-04-21T08:37:31.456286+0000","last_unstale":"2024-04-21T08:38:59.598820+0000","last_undegraded":"2024-04-21T08:38:59.598820+0000","last_fullsized":"2024-04-21T08:38:59.598820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:48:12.462037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.607738+0000","last_change":"2024-04-21T08:37:31.447008+0000","last_active":"2024-04-21T08:38:59.607738+0000","last_peered":"2024-04-21T08:38:59.607738+0000","last_clean":"2024-04-21T08:38:59.607738+0000","last_became_active":"2024-04-21T08:37:31.445192+0000","last_became_peered":"2024-04-21T08:37:31.445192+0000","last_unstale":"2024-04-21T08:38:59.607738+0000","last_undegraded":"2024-04-21T08:38:59.607738+0000","last_fullsized":"2024-04-21T08:38:59.607738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:08.025647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.593342+0000","last_change":"2024-04-21T08:37:31.443288+0000","last_active":"2024-04-21T08:38:59.593342+0000","last_peered":"2024-04-21T08:38:59.593342+0000","last_clean":"2024-04-21T08:38:59.593342+0000","last_became_active":"2024-04-21T08:37:31.442586+0000","last_became_peered":"2024-04-21T08:37:31.442586+0000","last_unstale":"2024-04-21T08:38:59.593342+0000","last_undegraded":"2024-04-21T08:38:59.593342+0000","last_fullsized":"2024-04-21T08:38:59.593342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:01.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.601468+0000","last_change":"2024-04-21T08:37:31.453899+0000","last_active":"2024-04-21T08:38:59.601468+0000","last_peered":"2024-04-21T08:38:59.601468+0000","last_clean":"2024-04-21T08:38:59.601468+0000","last_became_active":"2024-04-21T08:37:31.453527+0000","last_became_peered":"2024-04-21T08:37:31.453527+0000","last_unstale":"2024-04-21T08:38:59.601468+0000","last_undegraded":"2024-04-21T08:38:59.601468+0000","last_fullsized":"2024-04-21T08:38:59.601468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:23:18.429618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283745,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67747,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":464,"num_read_kb":303,"num_write":168,"num_write_kb":121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":203241,"data_compressed":4983,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":500689,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1138688,"data_stored":1152094,"data_compressed":58414,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":505021,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":500263,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1096895,"data_compressed":56753,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":496101,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":560198,"data_compressed":22332,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":497489,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":560966,"data_compressed":22332,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":496295,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":501283,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1091951,"data_compressed":56753,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64187,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64955,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65715,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:39:13.590 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T08:39:13.831 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:14.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:14 smithi094 ceph-mon[27988]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:39:14.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:14 smithi094 ceph-mon[27988]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:39:15.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:14 smithi173 ceph-mon[31428]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:39:15.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:14 smithi173 ceph-mon[31428]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:39:15.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:14 smithi111 ceph-mon[31493]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:39:15.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:14 smithi111 ceph-mon[31493]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:39:15.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.526+0000 7fceddf8a700 1 -- 172.21.15.94:0/2200994596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8103450 msgr2=0x7fced810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:15.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.526+0000 7fceddf8a700 1 --2- 172.21.15.94:0/2200994596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8103450 0x7fced810f8b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fced0009990 tx=0x7fced0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.526+0000 7fceddf8a700 1 -- 172.21.15.94:0/2200994596 shutdown_connections 2024-04-21T08:39:15.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.526+0000 7fceddf8a700 1 --2- 172.21.15.94:0/2200994596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8103450 0x7fced810f8b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.526+0000 7fceddf8a700 1 --2- 172.21.15.94:0/2200994596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fced8108a50 0x7fced8108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.526+0000 7fceddf8a700 1 --2- 172.21.15.94:0/2200994596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fced8102a90 0x7fced8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.526+0000 7fceddf8a700 1 -- 172.21.15.94:0/2200994596 >> 172.21.15.94:0/2200994596 conn(0x7fced8079ba0 msgr2=0x7fced8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.526+0000 7fceddf8a700 1 -- 172.21.15.94:0/2200994596 shutdown_connections 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.526+0000 7fceddf8a700 1 -- 172.21.15.94:0/2200994596 wait complete. 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fceddf8a700 1 Processor -- start 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fceddf8a700 1 -- start start 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fceddf8a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fced8102a90 0x7fced8199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fceddf8a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fced8103450 0x7fced8199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fceddf8a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8108a50 0x7fced819d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fceddf8a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fced8112c00 con 0x7fced8108a50 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fceddf8a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fced8112a80 con 0x7fced8102a90 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fceddf8a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fced8112d80 con 0x7fced8103450 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fced7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8108a50 0x7fced819d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:15.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fced7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8108a50 0x7fced819d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46484/0 (socket says 172.21.15.94:46484) 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fced7fff700 1 -- 172.21.15.94:0/604801147 learned_addr learned my addr 172.21.15.94:0/604801147 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fced77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fced8102a90 0x7fced8199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.527+0000 7fced6ffd700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fced8103450 0x7fced8199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fced7fff700 1 -- 172.21.15.94:0/604801147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fced8103450 msgr2=0x7fced8199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fced7fff700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fced8103450 0x7fced8199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fced7fff700 1 -- 172.21.15.94:0/604801147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fced8102a90 msgr2=0x7fced8199010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fced7fff700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fced8102a90 0x7fced8199010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fced7fff700 1 -- 172.21.15.94:0/604801147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced819dfd0 con 0x7fced8108a50 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fced7fff700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8108a50 0x7fced819d890 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fced000c010 tx=0x7fced00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fced4ff9700 1 -- 172.21.15.94:0/604801147 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced0027070 con 0x7fced8108a50 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fced4ff9700 1 -- 172.21.15.94:0/604801147 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fced0022470 con 0x7fced8108a50 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fced4ff9700 1 -- 172.21.15.94:0/604801147 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced0017650 con 0x7fced8108a50 2024-04-21T08:39:15.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fceddf8a700 1 -- 172.21.15.94:0/604801147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced819e260 con 0x7fced8108a50 2024-04-21T08:39:15.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.528+0000 7fceddf8a700 1 -- 172.21.15.94:0/604801147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced81a5b40 con 0x7fced8108a50 2024-04-21T08:39:15.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.529+0000 7fceddf8a700 1 -- 172.21.15.94:0/604801147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fced8104180 con 0x7fced8108a50 2024-04-21T08:39:15.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.530+0000 7fced4ff9700 1 -- 172.21.15.94:0/604801147 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fced002a070 con 0x7fced8108a50 2024-04-21T08:39:15.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.531+0000 7fced4ff9700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcec00a3c40 0x7fcec00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:15.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.531+0000 7fced77fe700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcec00a3c40 0x7fcec00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:15.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.531+0000 7fced4ff9700 1 -- 172.21.15.94:0/604801147 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7fced00d18e0 con 0x7fced8108a50 2024-04-21T08:39:15.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.532+0000 7fced77fe700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcec00a3c40 0x7fcec00a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcec800f540 tx=0x7fcec800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:15.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.535+0000 7fced4ff9700 1 -- 172.21.15.94:0/604801147 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fced0013070 con 0x7fced8108a50 2024-04-21T08:39:15.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.730+0000 7fceddf8a700 1 -- 172.21.15.94:0/604801147 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fced8062e80 con 0x7fcec00a3c40 2024-04-21T08:39:15.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.742+0000 7fced4ff9700 1 -- 172.21.15.94:0/604801147 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340786 (secure 0 0 0) 0x7fced8062e80 con 0x7fcec00a3c40 2024-04-21T08:39:15.746 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 -- 172.21.15.94:0/604801147 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcec00a3c40 msgr2=0x7fcec00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcec00a3c40 0x7fcec00a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcec800f540 tx=0x7fcec800b040 comp rx=0 tx=0).stop 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 -- 172.21.15.94:0/604801147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8108a50 msgr2=0x7fced819d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8108a50 0x7fced819d890 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fced000c010 tx=0x7fced00047b0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 -- 172.21.15.94:0/604801147 shutdown_connections 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fcec00a3c40 0x7fcec00a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fced8102a90 0x7fced8199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8108a50 0x7fced819d890 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 --2- 172.21.15.94:0/604801147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fced8103450 0x7fced8199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 -- 172.21.15.94:0/604801147 >> 172.21.15.94:0/604801147 conn(0x7fced8079ba0 msgr2=0x7fced81016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 -- 172.21.15.94:0/604801147 shutdown_connections 2024-04-21T08:39:15.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:15.745+0000 7fceddf8a700 1 -- 172.21.15.94:0/604801147 wait complete. 2024-04-21T08:39:15.750 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-21T08:39:16.431 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-04-21T08:39:15.607141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658349,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":630,"num_read_kb":437,"num_write":351,"num_write_kb":3061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377944,"kb_used_data":12152,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563880,"statfs":{"total":1151940427776,"available":1151553413120,"internally_reserved":0,"allocated":12443648,"data_stored":7959245,"data_compressed":361281,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003167"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.605355+0000","last_change":"2024-04-21T08:37:31.448345+0000","last_active":"2024-04-21T08:38:59.605355+0000","last_peered":"2024-04-21T08:38:59.605355+0000","last_clean":"2024-04-21T08:38:59.605355+0000","last_became_active":"2024-04-21T08:37:31.447878+0000","last_became_peered":"2024-04-21T08:37:31.447878+0000","last_unstale":"2024-04-21T08:38:59.605355+0000","last_undegraded":"2024-04-21T08:38:59.605355+0000","last_fullsized":"2024-04-21T08:38:59.605355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:45.589165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.595000+0000","last_change":"2024-04-21T08:37:31.452895+0000","last_active":"2024-04-21T08:38:59.595000+0000","last_peered":"2024-04-21T08:38:59.595000+0000","last_clean":"2024-04-21T08:38:59.595000+0000","last_became_active":"2024-04-21T08:37:31.452550+0000","last_became_peered":"2024-04-21T08:37:31.452550+0000","last_unstale":"2024-04-21T08:38:59.595000+0000","last_undegraded":"2024-04-21T08:38:59.595000+0000","last_fullsized":"2024-04-21T08:38:59.595000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:02.497802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.600232+0000","last_change":"2024-04-21T08:37:31.447552+0000","last_active":"2024-04-21T08:38:59.600232+0000","last_peered":"2024-04-21T08:38:59.600232+0000","last_clean":"2024-04-21T08:38:59.600232+0000","last_became_active":"2024-04-21T08:37:31.446044+0000","last_became_peered":"2024-04-21T08:37:31.446044+0000","last_unstale":"2024-04-21T08:38:59.600232+0000","last_undegraded":"2024-04-21T08:38:59.600232+0000","last_fullsized":"2024-04-21T08:38:59.600232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:22.801977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.588598+0000","last_change":"2024-04-21T08:37:31.452400+0000","last_active":"2024-04-21T08:38:59.588598+0000","last_peered":"2024-04-21T08:38:59.588598+0000","last_clean":"2024-04-21T08:38:59.588598+0000","last_became_active":"2024-04-21T08:37:31.451462+0000","last_became_peered":"2024-04-21T08:37:31.451462+0000","last_unstale":"2024-04-21T08:38:59.588598+0000","last_undegraded":"2024-04-21T08:38:59.588598+0000","last_fullsized":"2024-04-21T08:38:59.588598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:22.463481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.579416+0000","last_change":"2024-04-21T08:37:31.448942+0000","last_active":"2024-04-21T08:38:59.579416+0000","last_peered":"2024-04-21T08:38:59.579416+0000","last_clean":"2024-04-21T08:38:59.579416+0000","last_became_active":"2024-04-21T08:37:31.448526+0000","last_became_peered":"2024-04-21T08:37:31.448526+0000","last_unstale":"2024-04-21T08:38:59.579416+0000","last_undegraded":"2024-04-21T08:38:59.579416+0000","last_fullsized":"2024-04-21T08:38:59.579416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:29.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.592477+0000","last_change":"2024-04-21T08:37:31.452165+0000","last_active":"2024-04-21T08:38:59.592477+0000","last_peered":"2024-04-21T08:38:59.592477+0000","last_clean":"2024-04-21T08:38:59.592477+0000","last_became_active":"2024-04-21T08:37:31.451802+0000","last_became_peered":"2024-04-21T08:37:31.451802+0000","last_unstale":"2024-04-21T08:38:59.592477+0000","last_undegraded":"2024-04-21T08:38:59.592477+0000","last_fullsized":"2024-04-21T08:38:59.592477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:42.210544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.594832+0000","last_change":"2024-04-21T08:37:31.444798+0000","last_active":"2024-04-21T08:38:59.594832+0000","last_peered":"2024-04-21T08:38:59.594832+0000","last_clean":"2024-04-21T08:38:59.594832+0000","last_became_active":"2024-04-21T08:37:31.444313+0000","last_became_peered":"2024-04-21T08:37:31.444313+0000","last_unstale":"2024-04-21T08:38:59.594832+0000","last_undegraded":"2024-04-21T08:38:59.594832+0000","last_fullsized":"2024-04-21T08:38:59.594832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:22:03.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599340+0000","last_change":"2024-04-21T08:37:31.448257+0000","last_active":"2024-04-21T08:38:59.599340+0000","last_peered":"2024-04-21T08:38:59.599340+0000","last_clean":"2024-04-21T08:38:59.599340+0000","last_became_active":"2024-04-21T08:37:31.447845+0000","last_became_peered":"2024-04-21T08:37:31.447845+0000","last_unstale":"2024-04-21T08:38:59.599340+0000","last_undegraded":"2024-04-21T08:38:59.599340+0000","last_fullsized":"2024-04-21T08:38:59.599340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:19.956699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.604851+0000","last_change":"2024-04-21T08:37:31.443022+0000","last_active":"2024-04-21T08:38:59.604851+0000","last_peered":"2024-04-21T08:38:59.604851+0000","last_clean":"2024-04-21T08:38:59.604851+0000","last_became_active":"2024-04-21T08:37:31.442645+0000","last_became_peered":"2024-04-21T08:37:31.442645+0000","last_unstale":"2024-04-21T08:38:59.604851+0000","last_undegraded":"2024-04-21T08:38:59.604851+0000","last_fullsized":"2024-04-21T08:38:59.604851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:18:10.861281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.593772+0000","last_change":"2024-04-21T08:37:31.453427+0000","last_active":"2024-04-21T08:38:59.593772+0000","last_peered":"2024-04-21T08:38:59.593772+0000","last_clean":"2024-04-21T08:38:59.593772+0000","last_became_active":"2024-04-21T08:37:31.453096+0000","last_became_peered":"2024-04-21T08:37:31.453096+0000","last_unstale":"2024-04-21T08:38:59.593772+0000","last_undegraded":"2024-04-21T08:38:59.593772+0000","last_fullsized":"2024-04-21T08:38:59.593772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:49.450351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.608208+0000","last_change":"2024-04-21T08:37:31.449908+0000","last_active":"2024-04-21T08:38:59.608208+0000","last_peered":"2024-04-21T08:38:59.608208+0000","last_clean":"2024-04-21T08:38:59.608208+0000","last_became_active":"2024-04-21T08:37:31.449558+0000","last_became_peered":"2024-04-21T08:37:31.449558+0000","last_unstale":"2024-04-21T08:38:59.608208+0000","last_undegraded":"2024-04-21T08:38:59.608208+0000","last_fullsized":"2024-04-21T08:38:59.608208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:21.538689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598603+0000","last_change":"2024-04-21T08:37:31.452973+0000","last_active":"2024-04-21T08:38:59.598603+0000","last_peered":"2024-04-21T08:38:59.598603+0000","last_clean":"2024-04-21T08:38:59.598603+0000","last_became_active":"2024-04-21T08:37:31.452586+0000","last_became_peered":"2024-04-21T08:37:31.452586+0000","last_unstale":"2024-04-21T08:38:59.598603+0000","last_undegraded":"2024-04-21T08:38:59.598603+0000","last_fullsized":"2024-04-21T08:38:59.598603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:09:11.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.593552+0000","last_change":"2024-04-21T08:37:31.445801+0000","last_active":"2024-04-21T08:38:59.593552+0000","last_peered":"2024-04-21T08:38:59.593552+0000","last_clean":"2024-04-21T08:38:59.593552+0000","last_became_active":"2024-04-21T08:37:31.445478+0000","last_became_peered":"2024-04-21T08:37:31.445478+0000","last_unstale":"2024-04-21T08:38:59.593552+0000","last_undegraded":"2024-04-21T08:38:59.593552+0000","last_fullsized":"2024-04-21T08:38:59.593552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:33.352739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.593896+0000","last_change":"2024-04-21T08:37:31.450735+0000","last_active":"2024-04-21T08:38:59.593896+0000","last_peered":"2024-04-21T08:38:59.593896+0000","last_clean":"2024-04-21T08:38:59.593896+0000","last_became_active":"2024-04-21T08:37:31.450384+0000","last_became_peered":"2024-04-21T08:37:31.450384+0000","last_unstale":"2024-04-21T08:38:59.593896+0000","last_undegraded":"2024-04-21T08:38:59.593896+0000","last_fullsized":"2024-04-21T08:38:59.593896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:45.950489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599498+0000","last_change":"2024-04-21T08:37:31.454690+0000","last_active":"2024-04-21T08:38:59.599498+0000","last_peered":"2024-04-21T08:38:59.599498+0000","last_clean":"2024-04-21T08:38:59.599498+0000","last_became_active":"2024-04-21T08:37:31.454330+0000","last_became_peered":"2024-04-21T08:37:31.454330+0000","last_unstale":"2024-04-21T08:38:59.599498+0000","last_undegraded":"2024-04-21T08:38:59.599498+0000","last_fullsized":"2024-04-21T08:38:59.599498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:52.644317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.608553+0000","last_change":"2024-04-21T08:37:31.441389+0000","last_active":"2024-04-21T08:38:59.608553+0000","last_peered":"2024-04-21T08:38:59.608553+0000","last_clean":"2024-04-21T08:38:59.608553+0000","last_became_active":"2024-04-21T08:37:31.441065+0000","last_became_peered":"2024-04-21T08:37:31.441065+0000","last_unstale":"2024-04-21T08:38:59.608553+0000","last_undegraded":"2024-04-21T08:38:59.608553+0000","last_fullsized":"2024-04-21T08:38:59.608553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:56.528154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.605851+0000","last_change":"2024-04-21T08:37:31.446722+0000","last_active":"2024-04-21T08:38:59.605851+0000","last_peered":"2024-04-21T08:38:59.605851+0000","last_clean":"2024-04-21T08:38:59.605851+0000","last_became_active":"2024-04-21T08:37:31.446396+0000","last_became_peered":"2024-04-21T08:37:31.446396+0000","last_unstale":"2024-04-21T08:38:59.605851+0000","last_undegraded":"2024-04-21T08:38:59.605851+0000","last_fullsized":"2024-04-21T08:38:59.605851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:17.488633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.601347+0000","last_change":"2024-04-21T08:37:31.444388+0000","last_active":"2024-04-21T08:38:59.601347+0000","last_peered":"2024-04-21T08:38:59.601347+0000","last_clean":"2024-04-21T08:38:59.601347+0000","last_became_active":"2024-04-21T08:37:31.444023+0000","last_became_peered":"2024-04-21T08:37:31.444023+0000","last_unstale":"2024-04-21T08:38:59.601347+0000","last_undegraded":"2024-04-21T08:38:59.601347+0000","last_fullsized":"2024-04-21T08:38:59.601347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:07:46.238941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.597499+0000","last_change":"2024-04-21T08:37:31.447850+0000","last_active":"2024-04-21T08:38:59.597499+0000","last_peered":"2024-04-21T08:38:59.597499+0000","last_clean":"2024-04-21T08:38:59.597499+0000","last_became_active":"2024-04-21T08:37:31.447505+0000","last_became_peered":"2024-04-21T08:37:31.447505+0000","last_unstale":"2024-04-21T08:38:59.597499+0000","last_undegraded":"2024-04-21T08:38:59.597499+0000","last_fullsized":"2024-04-21T08:38:59.597499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:55.399410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.667080+0000","last_change":"2024-04-21T08:37:35.608747+0000","last_active":"2024-04-21T08:38:30.667080+0000","last_peered":"2024-04-21T08:38:30.667080+0000","last_clean":"2024-04-21T08:38:30.667080+0000","last_became_active":"2024-04-21T08:37:35.608157+0000","last_became_peered":"2024-04-21T08:37:35.608157+0000","last_unstale":"2024-04-21T08:38:30.667080+0000","last_undegraded":"2024-04-21T08:38:30.667080+0000","last_fullsized":"2024-04-21T08:38:30.667080+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:53.384190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.679843+0000","last_change":"2024-04-21T08:38:30.679843+0000","last_active":"2024-04-21T08:38:30.679843+0000","last_peered":"2024-04-21T08:38:30.679843+0000","last_clean":"2024-04-21T08:38:30.679843+0000","last_became_active":"2024-04-21T08:38:30.670964+0000","last_became_peered":"2024-04-21T08:38:30.670964+0000","last_unstale":"2024-04-21T08:38:30.679843+0000","last_undegraded":"2024-04-21T08:38:30.679843+0000","last_fullsized":"2024-04-21T08:38:30.679843+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:49.652571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598546+0000","last_change":"2024-04-21T08:37:31.451813+0000","last_active":"2024-04-21T08:38:59.598546+0000","last_peered":"2024-04-21T08:38:59.598546+0000","last_clean":"2024-04-21T08:38:59.598546+0000","last_became_active":"2024-04-21T08:37:31.451447+0000","last_became_peered":"2024-04-21T08:37:31.451447+0000","last_unstale":"2024-04-21T08:38:59.598546+0000","last_undegraded":"2024-04-21T08:38:59.598546+0000","last_fullsized":"2024-04-21T08:38:59.598546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:46:45.235326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.668976+0000","last_change":"2024-04-21T08:37:35.603518+0000","last_active":"2024-04-21T08:38:30.668976+0000","last_peered":"2024-04-21T08:38:30.668976+0000","last_clean":"2024-04-21T08:38:30.668976+0000","last_became_active":"2024-04-21T08:37:35.603038+0000","last_became_peered":"2024-04-21T08:37:35.603038+0000","last_unstale":"2024-04-21T08:38:30.668976+0000","last_undegraded":"2024-04-21T08:38:30.668976+0000","last_fullsized":"2024-04-21T08:38:30.668976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:06:29.224025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.673441+0000","last_change":"2024-04-21T08:37:30.424608+0000","last_active":"2024-04-21T08:38:30.673441+0000","last_peered":"2024-04-21T08:38:30.673441+0000","last_clean":"2024-04-21T08:38:30.673441+0000","last_became_active":"2024-04-21T08:37:30.424259+0000","last_became_peered":"2024-04-21T08:37:30.424259+0000","last_unstale":"2024-04-21T08:38:30.673441+0000","last_undegraded":"2024-04-21T08:38:30.673441+0000","last_fullsized":"2024-04-21T08:38:30.673441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:38.007095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.600543+0000","last_change":"2024-04-21T08:37:31.444656+0000","last_active":"2024-04-21T08:38:59.600543+0000","last_peered":"2024-04-21T08:38:59.600543+0000","last_clean":"2024-04-21T08:38:59.600543+0000","last_became_active":"2024-04-21T08:37:31.444300+0000","last_became_peered":"2024-04-21T08:37:31.444300+0000","last_unstale":"2024-04-21T08:38:59.600543+0000","last_undegraded":"2024-04-21T08:38:59.600543+0000","last_fullsized":"2024-04-21T08:38:59.600543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:40.901674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669700+0000","last_change":"2024-04-21T08:37:35.611732+0000","last_active":"2024-04-21T08:38:30.669700+0000","last_peered":"2024-04-21T08:38:30.669700+0000","last_clean":"2024-04-21T08:38:30.669700+0000","last_became_active":"2024-04-21T08:37:35.611060+0000","last_became_peered":"2024-04-21T08:37:35.611060+0000","last_unstale":"2024-04-21T08:38:30.669700+0000","last_undegraded":"2024-04-21T08:38:30.669700+0000","last_fullsized":"2024-04-21T08:38:30.669700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:36:01.491949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.689722+0000","last_change":"2024-04-21T08:38:30.689722+0000","last_active":"2024-04-21T08:38:30.689722+0000","last_peered":"2024-04-21T08:38:30.689722+0000","last_clean":"2024-04-21T08:38:30.689722+0000","last_became_active":"2024-04-21T08:38:30.676521+0000","last_became_peered":"2024-04-21T08:38:30.676521+0000","last_unstale":"2024-04-21T08:38:30.689722+0000","last_undegraded":"2024-04-21T08:38:30.689722+0000","last_fullsized":"2024-04-21T08:38:30.689722+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:24:15.799308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.605600+0000","last_change":"2024-04-21T08:37:31.443896+0000","last_active":"2024-04-21T08:38:59.605600+0000","last_peered":"2024-04-21T08:38:59.605600+0000","last_clean":"2024-04-21T08:38:59.605600+0000","last_became_active":"2024-04-21T08:37:31.442684+0000","last_became_peered":"2024-04-21T08:37:31.442684+0000","last_unstale":"2024-04-21T08:38:59.605600+0000","last_undegraded":"2024-04-21T08:38:59.605600+0000","last_fullsized":"2024-04-21T08:38:59.605600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:14:18.649565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669557+0000","last_change":"2024-04-21T08:37:35.600468+0000","last_active":"2024-04-21T08:38:30.669557+0000","last_peered":"2024-04-21T08:38:30.669557+0000","last_clean":"2024-04-21T08:38:30.669557+0000","last_became_active":"2024-04-21T08:37:35.599940+0000","last_became_peered":"2024-04-21T08:37:35.599940+0000","last_unstale":"2024-04-21T08:38:30.669557+0000","last_undegraded":"2024-04-21T08:38:30.669557+0000","last_fullsized":"2024-04-21T08:38:30.669557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:48.125434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669752+0000","last_change":"2024-04-21T08:37:30.416480+0000","last_active":"2024-04-21T08:38:30.669752+0000","last_peered":"2024-04-21T08:38:30.669752+0000","last_clean":"2024-04-21T08:38:30.669752+0000","last_became_active":"2024-04-21T08:37:30.416072+0000","last_became_peered":"2024-04-21T08:37:30.416072+0000","last_unstale":"2024-04-21T08:38:30.669752+0000","last_undegraded":"2024-04-21T08:38:30.669752+0000","last_fullsized":"2024-04-21T08:38:30.669752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:28:22.400926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.607612+0000","last_change":"2024-04-21T08:37:31.457216+0000","last_active":"2024-04-21T08:38:59.607612+0000","last_peered":"2024-04-21T08:38:59.607612+0000","last_clean":"2024-04-21T08:38:59.607612+0000","last_became_active":"2024-04-21T08:37:31.456856+0000","last_became_peered":"2024-04-21T08:37:31.456856+0000","last_unstale":"2024-04-21T08:38:59.607612+0000","last_undegraded":"2024-04-21T08:38:59.607612+0000","last_fullsized":"2024-04-21T08:38:59.607612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:10.272886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.668095+0000","last_change":"2024-04-21T08:37:35.601715+0000","last_active":"2024-04-21T08:38:30.668095+0000","last_peered":"2024-04-21T08:38:30.668095+0000","last_clean":"2024-04-21T08:38:30.668095+0000","last_became_active":"2024-04-21T08:37:35.601269+0000","last_became_peered":"2024-04-21T08:37:35.601269+0000","last_unstale":"2024-04-21T08:38:30.668095+0000","last_undegraded":"2024-04-21T08:38:30.668095+0000","last_fullsized":"2024-04-21T08:38:30.668095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:57.192918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:39:13.821916+0000","last_change":"2024-04-21T08:37:30.415510+0000","last_active":"2024-04-21T08:39:13.821916+0000","last_peered":"2024-04-21T08:39:13.821916+0000","last_clean":"2024-04-21T08:39:13.821916+0000","last_became_active":"2024-04-21T08:37:30.415199+0000","last_became_peered":"2024-04-21T08:37:30.415199+0000","last_unstale":"2024-04-21T08:39:13.821916+0000","last_undegraded":"2024-04-21T08:39:13.821916+0000","last_fullsized":"2024-04-21T08:39:13.821916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:32.336247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.586253+0000","last_change":"2024-04-21T08:37:31.444022+0000","last_active":"2024-04-21T08:38:59.586253+0000","last_peered":"2024-04-21T08:38:59.586253+0000","last_clean":"2024-04-21T08:38:59.586253+0000","last_became_active":"2024-04-21T08:37:31.443609+0000","last_became_peered":"2024-04-21T08:37:31.443609+0000","last_unstale":"2024-04-21T08:38:59.586253+0000","last_undegraded":"2024-04-21T08:38:59.586253+0000","last_fullsized":"2024-04-21T08:38:59.586253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:39.978817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.714455+0000","last_change":"2024-04-21T08:38:30.714455+0000","last_active":"2024-04-21T08:38:30.714455+0000","last_peered":"2024-04-21T08:38:30.714455+0000","last_clean":"2024-04-21T08:38:30.714455+0000","last_became_active":"2024-04-21T08:38:30.713910+0000","last_became_peered":"2024-04-21T08:38:30.713910+0000","last_unstale":"2024-04-21T08:38:30.714455+0000","last_undegraded":"2024-04-21T08:38:30.714455+0000","last_fullsized":"2024-04-21T08:38:30.714455+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.448652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,10],"acting":[2,6,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.712604+0000","last_change":"2024-04-21T08:37:30.419877+0000","last_active":"2024-04-21T08:38:30.712604+0000","last_peered":"2024-04-21T08:38:30.712604+0000","last_clean":"2024-04-21T08:38:30.712604+0000","last_became_active":"2024-04-21T08:37:30.419572+0000","last_became_peered":"2024-04-21T08:37:30.419572+0000","last_unstale":"2024-04-21T08:38:30.712604+0000","last_undegraded":"2024-04-21T08:38:30.712604+0000","last_fullsized":"2024-04-21T08:38:30.712604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:32:35.524089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.597003+0000","last_change":"2024-04-21T08:37:31.458590+0000","last_active":"2024-04-21T08:38:59.597003+0000","last_peered":"2024-04-21T08:38:59.597003+0000","last_clean":"2024-04-21T08:38:59.597003+0000","last_became_active":"2024-04-21T08:37:31.458196+0000","last_became_peered":"2024-04-21T08:37:31.458196+0000","last_unstale":"2024-04-21T08:38:59.597003+0000","last_undegraded":"2024-04-21T08:38:59.597003+0000","last_fullsized":"2024-04-21T08:38:59.597003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:00.890859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.803663+0000","last_change":"2024-04-21T08:38:30.803663+0000","last_active":"2024-04-21T08:38:30.803663+0000","last_peered":"2024-04-21T08:38:30.803663+0000","last_clean":"2024-04-21T08:38:30.803663+0000","last_became_active":"2024-04-21T08:38:30.803203+0000","last_became_peered":"2024-04-21T08:38:30.803203+0000","last_unstale":"2024-04-21T08:38:30.803663+0000","last_undegraded":"2024-04-21T08:38:30.803663+0000","last_fullsized":"2024-04-21T08:38:30.803663+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:06.085380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669367+0000","last_change":"2024-04-21T08:37:30.415786+0000","last_active":"2024-04-21T08:38:30.669367+0000","last_peered":"2024-04-21T08:38:30.669367+0000","last_clean":"2024-04-21T08:38:30.669367+0000","last_became_active":"2024-04-21T08:37:30.415472+0000","last_became_peered":"2024-04-21T08:37:30.415472+0000","last_unstale":"2024-04-21T08:38:30.669367+0000","last_undegraded":"2024-04-21T08:38:30.669367+0000","last_fullsized":"2024-04-21T08:38:30.669367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.417132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.597874+0000","last_change":"2024-04-21T08:37:31.455720+0000","last_active":"2024-04-21T08:38:59.597874+0000","last_peered":"2024-04-21T08:38:59.597874+0000","last_clean":"2024-04-21T08:38:59.597874+0000","last_became_active":"2024-04-21T08:37:31.455334+0000","last_became_peered":"2024-04-21T08:37:31.455334+0000","last_unstale":"2024-04-21T08:38:59.597874+0000","last_undegraded":"2024-04-21T08:38:59.597874+0000","last_fullsized":"2024-04-21T08:38:59.597874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:06:55.220656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.802537+0000","last_change":"2024-04-21T08:37:35.603305+0000","last_active":"2024-04-21T08:38:30.802537+0000","last_peered":"2024-04-21T08:38:30.802537+0000","last_clean":"2024-04-21T08:38:30.802537+0000","last_became_active":"2024-04-21T08:37:35.602854+0000","last_became_peered":"2024-04-21T08:37:35.602854+0000","last_unstale":"2024-04-21T08:38:30.802537+0000","last_undegraded":"2024-04-21T08:38:30.802537+0000","last_fullsized":"2024-04-21T08:38:30.802537+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:10.993864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.674921+0000","last_change":"2024-04-21T08:37:30.420797+0000","last_active":"2024-04-21T08:38:30.674921+0000","last_peered":"2024-04-21T08:38:30.674921+0000","last_clean":"2024-04-21T08:38:30.674921+0000","last_became_active":"2024-04-21T08:37:30.420485+0000","last_became_peered":"2024-04-21T08:37:30.420485+0000","last_unstale":"2024-04-21T08:38:30.674921+0000","last_undegraded":"2024-04-21T08:38:30.674921+0000","last_fullsized":"2024-04-21T08:38:30.674921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:03.275677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598515+0000","last_change":"2024-04-21T08:37:31.448406+0000","last_active":"2024-04-21T08:38:59.598515+0000","last_peered":"2024-04-21T08:38:59.598515+0000","last_clean":"2024-04-21T08:38:59.598515+0000","last_became_active":"2024-04-21T08:37:31.448085+0000","last_became_peered":"2024-04-21T08:37:31.448085+0000","last_unstale":"2024-04-21T08:38:59.598515+0000","last_undegraded":"2024-04-21T08:38:59.598515+0000","last_fullsized":"2024-04-21T08:38:59.598515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:49.574064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.672162+0000","last_change":"2024-04-21T08:37:35.601338+0000","last_active":"2024-04-21T08:38:30.672162+0000","last_peered":"2024-04-21T08:38:30.672162+0000","last_clean":"2024-04-21T08:38:30.672162+0000","last_became_active":"2024-04-21T08:37:35.600890+0000","last_became_peered":"2024-04-21T08:37:35.600890+0000","last_unstale":"2024-04-21T08:38:30.672162+0000","last_undegraded":"2024-04-21T08:38:30.672162+0000","last_fullsized":"2024-04-21T08:38:30.672162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:30:21.755850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:39:10.815962+0000","last_change":"2024-04-21T08:37:30.423320+0000","last_active":"2024-04-21T08:39:10.815962+0000","last_peered":"2024-04-21T08:39:10.815962+0000","last_clean":"2024-04-21T08:39:10.815962+0000","last_became_active":"2024-04-21T08:37:30.423020+0000","last_became_peered":"2024-04-21T08:37:30.423020+0000","last_unstale":"2024-04-21T08:39:10.815962+0000","last_undegraded":"2024-04-21T08:39:10.815962+0000","last_fullsized":"2024-04-21T08:39:10.815962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:08:27.313196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.605814+0000","last_change":"2024-04-21T08:37:31.442425+0000","last_active":"2024-04-21T08:38:59.605814+0000","last_peered":"2024-04-21T08:38:59.605814+0000","last_clean":"2024-04-21T08:38:59.605814+0000","last_became_active":"2024-04-21T08:37:31.442028+0000","last_became_peered":"2024-04-21T08:37:31.442028+0000","last_unstale":"2024-04-21T08:38:59.605814+0000","last_undegraded":"2024-04-21T08:38:59.605814+0000","last_fullsized":"2024-04-21T08:38:59.605814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:45.498656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666252+0000","last_change":"2024-04-21T08:37:35.607829+0000","last_active":"2024-04-21T08:38:30.666252+0000","last_peered":"2024-04-21T08:38:30.666252+0000","last_clean":"2024-04-21T08:38:30.666252+0000","last_became_active":"2024-04-21T08:37:35.607150+0000","last_became_peered":"2024-04-21T08:37:35.607150+0000","last_unstale":"2024-04-21T08:38:30.666252+0000","last_undegraded":"2024-04-21T08:38:30.666252+0000","last_fullsized":"2024-04-21T08:38:30.666252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:01:34.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669039+0000","last_change":"2024-04-21T08:37:30.418613+0000","last_active":"2024-04-21T08:38:30.669039+0000","last_peered":"2024-04-21T08:38:30.669039+0000","last_clean":"2024-04-21T08:38:30.669039+0000","last_became_active":"2024-04-21T08:37:30.418300+0000","last_became_peered":"2024-04-21T08:37:30.418300+0000","last_unstale":"2024-04-21T08:38:30.669039+0000","last_undegraded":"2024-04-21T08:38:30.669039+0000","last_fullsized":"2024-04-21T08:38:30.669039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:17:35.370753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.596670+0000","last_change":"2024-04-21T08:37:31.456902+0000","last_active":"2024-04-21T08:38:59.596670+0000","last_peered":"2024-04-21T08:38:59.596670+0000","last_clean":"2024-04-21T08:38:59.596670+0000","last_became_active":"2024-04-21T08:37:31.453637+0000","last_became_peered":"2024-04-21T08:37:31.453637+0000","last_unstale":"2024-04-21T08:38:59.596670+0000","last_undegraded":"2024-04-21T08:38:59.596670+0000","last_fullsized":"2024-04-21T08:38:59.596670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:48:59.387349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.670000+0000","last_change":"2024-04-21T08:37:35.608280+0000","last_active":"2024-04-21T08:38:30.670000+0000","last_peered":"2024-04-21T08:38:30.670000+0000","last_clean":"2024-04-21T08:38:30.670000+0000","last_became_active":"2024-04-21T08:37:35.607898+0000","last_became_peered":"2024-04-21T08:37:35.607898+0000","last_unstale":"2024-04-21T08:38:30.670000+0000","last_undegraded":"2024-04-21T08:38:30.670000+0000","last_fullsized":"2024-04-21T08:38:30.670000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:55.689519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.675589+0000","last_change":"2024-04-21T08:37:30.417352+0000","last_active":"2024-04-21T08:38:30.675589+0000","last_peered":"2024-04-21T08:38:30.675589+0000","last_clean":"2024-04-21T08:38:30.675589+0000","last_became_active":"2024-04-21T08:37:30.417000+0000","last_became_peered":"2024-04-21T08:37:30.417000+0000","last_unstale":"2024-04-21T08:38:30.675589+0000","last_undegraded":"2024-04-21T08:38:30.675589+0000","last_fullsized":"2024-04-21T08:38:30.675589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:56.087334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.608925+0000","last_change":"2024-04-21T08:37:31.458229+0000","last_active":"2024-04-21T08:38:59.608925+0000","last_peered":"2024-04-21T08:38:59.608925+0000","last_clean":"2024-04-21T08:38:59.608925+0000","last_became_active":"2024-04-21T08:37:31.456897+0000","last_became_peered":"2024-04-21T08:37:31.456897+0000","last_unstale":"2024-04-21T08:38:59.608925+0000","last_undegraded":"2024-04-21T08:38:59.608925+0000","last_fullsized":"2024-04-21T08:38:59.608925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:25:58.253081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666373+0000","last_change":"2024-04-21T08:37:35.601077+0000","last_active":"2024-04-21T08:38:30.666373+0000","last_peered":"2024-04-21T08:38:30.666373+0000","last_clean":"2024-04-21T08:38:30.666373+0000","last_became_active":"2024-04-21T08:37:35.600673+0000","last_became_peered":"2024-04-21T08:37:35.600673+0000","last_unstale":"2024-04-21T08:38:30.666373+0000","last_undegraded":"2024-04-21T08:38:30.666373+0000","last_fullsized":"2024-04-21T08:38:30.666373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:04.554208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":224,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:39:11.817653+0000","last_change":"2024-04-21T08:37:30.421440+0000","last_active":"2024-04-21T08:39:11.817653+0000","last_peered":"2024-04-21T08:39:11.817653+0000","last_clean":"2024-04-21T08:39:11.817653+0000","last_became_active":"2024-04-21T08:37:30.420988+0000","last_became_peered":"2024-04-21T08:37:30.420988+0000","last_unstale":"2024-04-21T08:39:11.817653+0000","last_undegraded":"2024-04-21T08:39:11.817653+0000","last_fullsized":"2024-04-21T08:39:11.817653+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:38:03.850705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":161,"num_read_kb":143,"num_write":37,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.595271+0000","last_change":"2024-04-21T08:37:31.447208+0000","last_active":"2024-04-21T08:38:59.595271+0000","last_peered":"2024-04-21T08:38:59.595271+0000","last_clean":"2024-04-21T08:38:59.595271+0000","last_became_active":"2024-04-21T08:37:31.444464+0000","last_became_peered":"2024-04-21T08:37:31.444464+0000","last_unstale":"2024-04-21T08:38:59.595271+0000","last_undegraded":"2024-04-21T08:38:59.595271+0000","last_fullsized":"2024-04-21T08:38:59.595271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:33.805676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.675797+0000","last_change":"2024-04-21T08:37:35.603937+0000","last_active":"2024-04-21T08:38:30.675797+0000","last_peered":"2024-04-21T08:38:30.675797+0000","last_clean":"2024-04-21T08:38:30.675797+0000","last_became_active":"2024-04-21T08:37:35.603477+0000","last_became_peered":"2024-04-21T08:37:35.603477+0000","last_unstale":"2024-04-21T08:38:30.675797+0000","last_undegraded":"2024-04-21T08:38:30.675797+0000","last_fullsized":"2024-04-21T08:38:30.675797+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:40:33.901570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.667786+0000","last_change":"2024-04-21T08:37:30.416941+0000","last_active":"2024-04-21T08:38:30.667786+0000","last_peered":"2024-04-21T08:38:30.667786+0000","last_clean":"2024-04-21T08:38:30.667786+0000","last_became_active":"2024-04-21T08:37:30.416625+0000","last_became_peered":"2024-04-21T08:37:30.416625+0000","last_unstale":"2024-04-21T08:38:30.667786+0000","last_undegraded":"2024-04-21T08:38:30.667786+0000","last_fullsized":"2024-04-21T08:38:30.667786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:07.782779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.592616+0000","last_change":"2024-04-21T08:37:31.441707+0000","last_active":"2024-04-21T08:38:59.592616+0000","last_peered":"2024-04-21T08:38:59.592616+0000","last_clean":"2024-04-21T08:38:59.592616+0000","last_became_active":"2024-04-21T08:37:31.441366+0000","last_became_peered":"2024-04-21T08:37:31.441366+0000","last_unstale":"2024-04-21T08:38:59.592616+0000","last_undegraded":"2024-04-21T08:38:59.592616+0000","last_fullsized":"2024-04-21T08:38:59.592616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:16.573804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669837+0000","last_change":"2024-04-21T08:37:35.596214+0000","last_active":"2024-04-21T08:38:30.669837+0000","last_peered":"2024-04-21T08:38:30.669837+0000","last_clean":"2024-04-21T08:38:30.669837+0000","last_became_active":"2024-04-21T08:37:35.595773+0000","last_became_peered":"2024-04-21T08:37:35.595773+0000","last_unstale":"2024-04-21T08:38:30.669837+0000","last_undegraded":"2024-04-21T08:38:30.669837+0000","last_fullsized":"2024-04-21T08:38:30.669837+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:07.538190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.684828+0000","last_change":"2024-04-21T08:38:30.684828+0000","last_active":"2024-04-21T08:38:30.684828+0000","last_peered":"2024-04-21T08:38:30.684828+0000","last_clean":"2024-04-21T08:38:30.684828+0000","last_became_active":"2024-04-21T08:38:30.673923+0000","last_became_peered":"2024-04-21T08:38:30.673923+0000","last_unstale":"2024-04-21T08:38:30.684828+0000","last_undegraded":"2024-04-21T08:38:30.684828+0000","last_fullsized":"2024-04-21T08:38:30.684828+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:51:37.906150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.603733+0000","last_change":"2024-04-21T08:37:31.453680+0000","last_active":"2024-04-21T08:38:59.603733+0000","last_peered":"2024-04-21T08:38:59.603733+0000","last_clean":"2024-04-21T08:38:59.603733+0000","last_became_active":"2024-04-21T08:37:31.453319+0000","last_became_peered":"2024-04-21T08:37:31.453319+0000","last_unstale":"2024-04-21T08:38:59.603733+0000","last_undegraded":"2024-04-21T08:38:59.603733+0000","last_fullsized":"2024-04-21T08:38:59.603733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:43.139789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666696+0000","last_change":"2024-04-21T08:37:35.608906+0000","last_active":"2024-04-21T08:38:30.666696+0000","last_peered":"2024-04-21T08:38:30.666696+0000","last_clean":"2024-04-21T08:38:30.666696+0000","last_became_active":"2024-04-21T08:37:35.608535+0000","last_became_peered":"2024-04-21T08:37:35.608535+0000","last_unstale":"2024-04-21T08:38:30.666696+0000","last_undegraded":"2024-04-21T08:38:30.666696+0000","last_fullsized":"2024-04-21T08:38:30.666696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:58:01.422308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:39:13.821054+0000","last_change":"2024-04-21T08:37:30.414865+0000","last_active":"2024-04-21T08:39:13.821054+0000","last_peered":"2024-04-21T08:39:13.821054+0000","last_clean":"2024-04-21T08:39:13.821054+0000","last_became_active":"2024-04-21T08:37:30.414471+0000","last_became_peered":"2024-04-21T08:37:30.414471+0000","last_unstale":"2024-04-21T08:39:13.821054+0000","last_undegraded":"2024-04-21T08:39:13.821054+0000","last_fullsized":"2024-04-21T08:39:13.821054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:46.943574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.597393+0000","last_change":"2024-04-21T08:37:31.456634+0000","last_active":"2024-04-21T08:38:59.597393+0000","last_peered":"2024-04-21T08:38:59.597393+0000","last_clean":"2024-04-21T08:38:59.597393+0000","last_became_active":"2024-04-21T08:37:31.456278+0000","last_became_peered":"2024-04-21T08:37:31.456278+0000","last_unstale":"2024-04-21T08:38:59.597393+0000","last_undegraded":"2024-04-21T08:38:59.597393+0000","last_fullsized":"2024-04-21T08:38:59.597393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:04.870024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.672121+0000","last_change":"2024-04-21T08:37:35.601760+0000","last_active":"2024-04-21T08:38:30.672121+0000","last_peered":"2024-04-21T08:38:30.672121+0000","last_clean":"2024-04-21T08:38:30.672121+0000","last_became_active":"2024-04-21T08:37:35.601332+0000","last_became_peered":"2024-04-21T08:37:35.601332+0000","last_unstale":"2024-04-21T08:38:30.672121+0000","last_undegraded":"2024-04-21T08:38:30.672121+0000","last_fullsized":"2024-04-21T08:38:30.672121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:04:03.694581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669050+0000","last_change":"2024-04-21T08:37:30.422419+0000","last_active":"2024-04-21T08:38:30.669050+0000","last_peered":"2024-04-21T08:38:30.669050+0000","last_clean":"2024-04-21T08:38:30.669050+0000","last_became_active":"2024-04-21T08:37:30.422123+0000","last_became_peered":"2024-04-21T08:37:30.422123+0000","last_unstale":"2024-04-21T08:38:30.669050+0000","last_undegraded":"2024-04-21T08:38:30.669050+0000","last_fullsized":"2024-04-21T08:38:30.669050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:32:54.029041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.607287+0000","last_change":"2024-04-21T08:37:31.458992+0000","last_active":"2024-04-21T08:38:59.607287+0000","last_peered":"2024-04-21T08:38:59.607287+0000","last_clean":"2024-04-21T08:38:59.607287+0000","last_became_active":"2024-04-21T08:37:31.458635+0000","last_became_peered":"2024-04-21T08:37:31.458635+0000","last_unstale":"2024-04-21T08:38:59.607287+0000","last_undegraded":"2024-04-21T08:38:59.607287+0000","last_fullsized":"2024-04-21T08:38:59.607287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:48:14.325571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.674535+0000","last_change":"2024-04-21T08:37:35.603739+0000","last_active":"2024-04-21T08:38:30.674535+0000","last_peered":"2024-04-21T08:38:30.674535+0000","last_clean":"2024-04-21T08:38:30.674535+0000","last_became_active":"2024-04-21T08:37:35.603310+0000","last_became_peered":"2024-04-21T08:37:35.603310+0000","last_unstale":"2024-04-21T08:38:30.674535+0000","last_undegraded":"2024-04-21T08:38:30.674535+0000","last_fullsized":"2024-04-21T08:38:30.674535+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:19.162520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669889+0000","last_change":"2024-04-21T08:37:30.418104+0000","last_active":"2024-04-21T08:38:30.669889+0000","last_peered":"2024-04-21T08:38:30.669889+0000","last_clean":"2024-04-21T08:38:30.669889+0000","last_became_active":"2024-04-21T08:37:30.417814+0000","last_became_peered":"2024-04-21T08:37:30.417814+0000","last_unstale":"2024-04-21T08:38:30.669889+0000","last_undegraded":"2024-04-21T08:38:30.669889+0000","last_fullsized":"2024-04-21T08:38:30.669889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:50:56.696513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.601328+0000","last_change":"2024-04-21T08:37:31.451952+0000","last_active":"2024-04-21T08:38:59.601328+0000","last_peered":"2024-04-21T08:38:59.601328+0000","last_clean":"2024-04-21T08:38:59.601328+0000","last_became_active":"2024-04-21T08:37:31.451588+0000","last_became_peered":"2024-04-21T08:37:31.451588+0000","last_unstale":"2024-04-21T08:38:59.601328+0000","last_undegraded":"2024-04-21T08:38:59.601328+0000","last_fullsized":"2024-04-21T08:38:59.601328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:06.506769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669424+0000","last_change":"2024-04-21T08:37:35.760661+0000","last_active":"2024-04-21T08:38:30.669424+0000","last_peered":"2024-04-21T08:38:30.669424+0000","last_clean":"2024-04-21T08:38:30.669424+0000","last_became_active":"2024-04-21T08:37:35.759925+0000","last_became_peered":"2024-04-21T08:37:35.759925+0000","last_unstale":"2024-04-21T08:38:30.669424+0000","last_undegraded":"2024-04-21T08:38:30.669424+0000","last_fullsized":"2024-04-21T08:38:30.669424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:11:53.062251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666484+0000","last_change":"2024-04-21T08:37:30.416487+0000","last_active":"2024-04-21T08:38:30.666484+0000","last_peered":"2024-04-21T08:38:30.666484+0000","last_clean":"2024-04-21T08:38:30.666484+0000","last_became_active":"2024-04-21T08:37:30.416076+0000","last_became_peered":"2024-04-21T08:37:30.416076+0000","last_unstale":"2024-04-21T08:38:30.666484+0000","last_undegraded":"2024-04-21T08:38:30.666484+0000","last_fullsized":"2024-04-21T08:38:30.666484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:18:33.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599128+0000","last_change":"2024-04-21T08:37:31.448824+0000","last_active":"2024-04-21T08:38:59.599128+0000","last_peered":"2024-04-21T08:38:59.599128+0000","last_clean":"2024-04-21T08:38:59.599128+0000","last_became_active":"2024-04-21T08:37:31.448432+0000","last_became_peered":"2024-04-21T08:37:31.448432+0000","last_unstale":"2024-04-21T08:38:59.599128+0000","last_undegraded":"2024-04-21T08:38:59.599128+0000","last_fullsized":"2024-04-21T08:38:59.599128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:41.111272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.670091+0000","last_change":"2024-04-21T08:37:35.604539+0000","last_active":"2024-04-21T08:38:30.670091+0000","last_peered":"2024-04-21T08:38:30.670091+0000","last_clean":"2024-04-21T08:38:30.670091+0000","last_became_active":"2024-04-21T08:37:35.604132+0000","last_became_peered":"2024-04-21T08:37:35.604132+0000","last_unstale":"2024-04-21T08:38:30.670091+0000","last_undegraded":"2024-04-21T08:38:30.670091+0000","last_fullsized":"2024-04-21T08:38:30.670091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:12.180730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669445+0000","last_change":"2024-04-21T08:37:30.418854+0000","last_active":"2024-04-21T08:38:30.669445+0000","last_peered":"2024-04-21T08:38:30.669445+0000","last_clean":"2024-04-21T08:38:30.669445+0000","last_became_active":"2024-04-21T08:37:30.418511+0000","last_became_peered":"2024-04-21T08:37:30.418511+0000","last_unstale":"2024-04-21T08:38:30.669445+0000","last_undegraded":"2024-04-21T08:38:30.669445+0000","last_fullsized":"2024-04-21T08:38:30.669445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:26.751924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.587033+0000","last_change":"2024-04-21T08:37:31.453365+0000","last_active":"2024-04-21T08:38:59.587033+0000","last_peered":"2024-04-21T08:38:59.587033+0000","last_clean":"2024-04-21T08:38:59.587033+0000","last_became_active":"2024-04-21T08:37:31.453042+0000","last_became_peered":"2024-04-21T08:37:31.453042+0000","last_unstale":"2024-04-21T08:38:59.587033+0000","last_undegraded":"2024-04-21T08:38:59.587033+0000","last_fullsized":"2024-04-21T08:38:59.587033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:49:46.648754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.671486+0000","last_change":"2024-04-21T08:37:35.760626+0000","last_active":"2024-04-21T08:38:30.671486+0000","last_peered":"2024-04-21T08:38:30.671486+0000","last_clean":"2024-04-21T08:38:30.671486+0000","last_became_active":"2024-04-21T08:37:35.760125+0000","last_became_peered":"2024-04-21T08:37:35.760125+0000","last_unstale":"2024-04-21T08:38:30.671486+0000","last_undegraded":"2024-04-21T08:38:30.671486+0000","last_fullsized":"2024-04-21T08:38:30.671486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:58.366557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599773+0000","last_change":"2024-04-21T08:37:31.457654+0000","last_active":"2024-04-21T08:38:59.599773+0000","last_peered":"2024-04-21T08:38:59.599773+0000","last_clean":"2024-04-21T08:38:59.599773+0000","last_became_active":"2024-04-21T08:37:31.456950+0000","last_became_peered":"2024-04-21T08:37:31.456950+0000","last_unstale":"2024-04-21T08:38:59.599773+0000","last_undegraded":"2024-04-21T08:38:59.599773+0000","last_fullsized":"2024-04-21T08:38:59.599773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:02.863000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:39:13.820137+0000","last_change":"2024-04-21T08:37:30.422021+0000","last_active":"2024-04-21T08:39:13.820137+0000","last_peered":"2024-04-21T08:39:13.820137+0000","last_clean":"2024-04-21T08:39:13.820137+0000","last_became_active":"2024-04-21T08:37:30.421711+0000","last_became_peered":"2024-04-21T08:37:30.421711+0000","last_unstale":"2024-04-21T08:39:13.820137+0000","last_undegraded":"2024-04-21T08:39:13.820137+0000","last_fullsized":"2024-04-21T08:39:13.820137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:01.612205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":79,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669220+0000","last_change":"2024-04-21T08:37:35.598041+0000","last_active":"2024-04-21T08:38:30.669220+0000","last_peered":"2024-04-21T08:38:30.669220+0000","last_clean":"2024-04-21T08:38:30.669220+0000","last_became_active":"2024-04-21T08:37:35.597635+0000","last_became_peered":"2024-04-21T08:37:35.597635+0000","last_unstale":"2024-04-21T08:38:30.669220+0000","last_undegraded":"2024-04-21T08:38:30.669220+0000","last_fullsized":"2024-04-21T08:38:30.669220+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:35:27.452340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598020+0000","last_change":"2024-04-21T08:37:31.450423+0000","last_active":"2024-04-21T08:38:59.598020+0000","last_peered":"2024-04-21T08:38:59.598020+0000","last_clean":"2024-04-21T08:38:59.598020+0000","last_became_active":"2024-04-21T08:37:31.450112+0000","last_became_peered":"2024-04-21T08:37:31.450112+0000","last_unstale":"2024-04-21T08:38:59.598020+0000","last_undegraded":"2024-04-21T08:38:59.598020+0000","last_fullsized":"2024-04-21T08:38:59.598020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:10:47.006769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669874+0000","last_change":"2024-04-21T08:37:30.416519+0000","last_active":"2024-04-21T08:38:30.669874+0000","last_peered":"2024-04-21T08:38:30.669874+0000","last_clean":"2024-04-21T08:38:30.669874+0000","last_became_active":"2024-04-21T08:37:30.416131+0000","last_became_peered":"2024-04-21T08:37:30.416131+0000","last_unstale":"2024-04-21T08:38:30.669874+0000","last_undegraded":"2024-04-21T08:38:30.669874+0000","last_fullsized":"2024-04-21T08:38:30.669874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:12.280847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":362,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.668877+0000","last_change":"2024-04-21T08:36:20.257759+0000","last_active":"2024-04-21T08:38:30.668877+0000","last_peered":"2024-04-21T08:38:30.668877+0000","last_clean":"2024-04-21T08:38:30.668877+0000","last_became_active":"2024-04-21T08:36:20.257313+0000","last_became_peered":"2024-04-21T08:36:20.257313+0000","last_unstale":"2024-04-21T08:38:30.668877+0000","last_undegraded":"2024-04-21T08:38:30.668877+0000","last_fullsized":"2024-04-21T08:38:30.668877+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_clean_scrub_stamp":"2024-04-21T08:32:56.546295+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:42:44.393447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.675737+0000","last_change":"2024-04-21T08:37:35.761562+0000","last_active":"2024-04-21T08:38:30.675737+0000","last_peered":"2024-04-21T08:38:30.675737+0000","last_clean":"2024-04-21T08:38:30.675737+0000","last_became_active":"2024-04-21T08:37:35.760703+0000","last_became_peered":"2024-04-21T08:37:35.760703+0000","last_unstale":"2024-04-21T08:38:30.675737+0000","last_undegraded":"2024-04-21T08:38:30.675737+0000","last_fullsized":"2024-04-21T08:38:30.675737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:12:47.511117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.587203+0000","last_change":"2024-04-21T08:37:31.453645+0000","last_active":"2024-04-21T08:38:59.587203+0000","last_peered":"2024-04-21T08:38:59.587203+0000","last_clean":"2024-04-21T08:38:59.587203+0000","last_became_active":"2024-04-21T08:37:31.453297+0000","last_became_peered":"2024-04-21T08:37:31.453297+0000","last_unstale":"2024-04-21T08:38:59.587203+0000","last_undegraded":"2024-04-21T08:38:59.587203+0000","last_fullsized":"2024-04-21T08:38:59.587203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:58.976327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.668944+0000","last_change":"2024-04-21T08:37:30.420560+0000","last_active":"2024-04-21T08:38:30.668944+0000","last_peered":"2024-04-21T08:38:30.668944+0000","last_clean":"2024-04-21T08:38:30.668944+0000","last_became_active":"2024-04-21T08:37:30.420223+0000","last_became_peered":"2024-04-21T08:37:30.420223+0000","last_unstale":"2024-04-21T08:38:30.668944+0000","last_undegraded":"2024-04-21T08:38:30.668944+0000","last_fullsized":"2024-04-21T08:38:30.668944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:43:55.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669718+0000","last_change":"2024-04-21T08:37:35.598294+0000","last_active":"2024-04-21T08:38:30.669718+0000","last_peered":"2024-04-21T08:38:30.669718+0000","last_clean":"2024-04-21T08:38:30.669718+0000","last_became_active":"2024-04-21T08:37:35.597883+0000","last_became_peered":"2024-04-21T08:37:35.597883+0000","last_unstale":"2024-04-21T08:38:30.669718+0000","last_undegraded":"2024-04-21T08:38:30.669718+0000","last_fullsized":"2024-04-21T08:38:30.669718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:24:05.144763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.670262+0000","last_change":"2024-04-21T08:37:30.423182+0000","last_active":"2024-04-21T08:38:30.670262+0000","last_peered":"2024-04-21T08:38:30.670262+0000","last_clean":"2024-04-21T08:38:30.670262+0000","last_became_active":"2024-04-21T08:37:30.422792+0000","last_became_peered":"2024-04-21T08:37:30.422792+0000","last_unstale":"2024-04-21T08:38:30.670262+0000","last_undegraded":"2024-04-21T08:38:30.670262+0000","last_fullsized":"2024-04-21T08:38:30.670262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:44:00.727269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.587317+0000","last_change":"2024-04-21T08:37:31.444013+0000","last_active":"2024-04-21T08:38:59.587317+0000","last_peered":"2024-04-21T08:38:59.587317+0000","last_clean":"2024-04-21T08:38:59.587317+0000","last_became_active":"2024-04-21T08:37:31.443684+0000","last_became_peered":"2024-04-21T08:37:31.443684+0000","last_unstale":"2024-04-21T08:38:59.587317+0000","last_undegraded":"2024-04-21T08:38:59.587317+0000","last_fullsized":"2024-04-21T08:38:59.587317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:00:35.400818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.671757+0000","last_change":"2024-04-21T08:37:35.759817+0000","last_active":"2024-04-21T08:38:30.671757+0000","last_peered":"2024-04-21T08:38:30.671757+0000","last_clean":"2024-04-21T08:38:30.671757+0000","last_became_active":"2024-04-21T08:37:35.759396+0000","last_became_peered":"2024-04-21T08:37:35.759396+0000","last_unstale":"2024-04-21T08:38:30.671757+0000","last_undegraded":"2024-04-21T08:38:30.671757+0000","last_fullsized":"2024-04-21T08:38:30.671757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:20:36.170880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666871+0000","last_change":"2024-04-21T08:37:30.414102+0000","last_active":"2024-04-21T08:38:30.666871+0000","last_peered":"2024-04-21T08:38:30.666871+0000","last_clean":"2024-04-21T08:38:30.666871+0000","last_became_active":"2024-04-21T08:37:30.413670+0000","last_became_peered":"2024-04-21T08:37:30.413670+0000","last_unstale":"2024-04-21T08:38:30.666871+0000","last_undegraded":"2024-04-21T08:38:30.666871+0000","last_fullsized":"2024-04-21T08:38:30.666871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:12:49.696516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598334+0000","last_change":"2024-04-21T08:37:31.457451+0000","last_active":"2024-04-21T08:38:59.598334+0000","last_peered":"2024-04-21T08:38:59.598334+0000","last_clean":"2024-04-21T08:38:59.598334+0000","last_became_active":"2024-04-21T08:37:31.457120+0000","last_became_peered":"2024-04-21T08:37:31.457120+0000","last_unstale":"2024-04-21T08:38:59.598334+0000","last_undegraded":"2024-04-21T08:38:59.598334+0000","last_fullsized":"2024-04-21T08:38:59.598334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:07:48.025843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666861+0000","last_change":"2024-04-21T08:37:35.600815+0000","last_active":"2024-04-21T08:38:30.666861+0000","last_peered":"2024-04-21T08:38:30.666861+0000","last_clean":"2024-04-21T08:38:30.666861+0000","last_became_active":"2024-04-21T08:37:35.600380+0000","last_became_peered":"2024-04-21T08:37:35.600380+0000","last_unstale":"2024-04-21T08:38:30.666861+0000","last_undegraded":"2024-04-21T08:38:30.666861+0000","last_fullsized":"2024-04-21T08:38:30.666861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:19:33.710431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.670129+0000","last_change":"2024-04-21T08:37:30.425592+0000","last_active":"2024-04-21T08:38:30.670129+0000","last_peered":"2024-04-21T08:38:30.670129+0000","last_clean":"2024-04-21T08:38:30.670129+0000","last_became_active":"2024-04-21T08:37:30.425274+0000","last_became_peered":"2024-04-21T08:37:30.425274+0000","last_unstale":"2024-04-21T08:38:30.670129+0000","last_undegraded":"2024-04-21T08:38:30.670129+0000","last_fullsized":"2024-04-21T08:38:30.670129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:27:29.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.596706+0000","last_change":"2024-04-21T08:37:31.455393+0000","last_active":"2024-04-21T08:38:59.596706+0000","last_peered":"2024-04-21T08:38:59.596706+0000","last_clean":"2024-04-21T08:38:59.596706+0000","last_became_active":"2024-04-21T08:37:31.455065+0000","last_became_peered":"2024-04-21T08:37:31.455065+0000","last_unstale":"2024-04-21T08:38:59.596706+0000","last_undegraded":"2024-04-21T08:38:59.596706+0000","last_fullsized":"2024-04-21T08:38:59.596706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:05:23.386790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.713521+0000","last_change":"2024-04-21T08:38:30.713521+0000","last_active":"2024-04-21T08:38:30.713521+0000","last_peered":"2024-04-21T08:38:30.713521+0000","last_clean":"2024-04-21T08:38:30.713521+0000","last_became_active":"2024-04-21T08:38:30.713096+0000","last_became_peered":"2024-04-21T08:38:30.713096+0000","last_unstale":"2024-04-21T08:38:30.713521+0000","last_undegraded":"2024-04-21T08:38:30.713521+0000","last_fullsized":"2024-04-21T08:38:30.713521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:54:59.078776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666980+0000","last_change":"2024-04-21T08:37:30.416696+0000","last_active":"2024-04-21T08:38:30.666980+0000","last_peered":"2024-04-21T08:38:30.666980+0000","last_clean":"2024-04-21T08:38:30.666980+0000","last_became_active":"2024-04-21T08:37:30.415868+0000","last_became_peered":"2024-04-21T08:37:30.415868+0000","last_unstale":"2024-04-21T08:38:30.666980+0000","last_undegraded":"2024-04-21T08:38:30.666980+0000","last_fullsized":"2024-04-21T08:38:30.666980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:48:08.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.557882+0000","last_change":"2024-04-21T08:37:31.453861+0000","last_active":"2024-04-21T08:38:59.557882+0000","last_peered":"2024-04-21T08:38:59.557882+0000","last_clean":"2024-04-21T08:38:59.557882+0000","last_became_active":"2024-04-21T08:37:31.453562+0000","last_became_peered":"2024-04-21T08:37:31.453562+0000","last_unstale":"2024-04-21T08:38:59.557882+0000","last_undegraded":"2024-04-21T08:38:59.557882+0000","last_fullsized":"2024-04-21T08:38:59.557882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:17:50.513917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.666749+0000","last_change":"2024-04-21T08:37:35.609217+0000","last_active":"2024-04-21T08:38:30.666749+0000","last_peered":"2024-04-21T08:38:30.666749+0000","last_clean":"2024-04-21T08:38:30.666749+0000","last_became_active":"2024-04-21T08:37:35.608786+0000","last_became_peered":"2024-04-21T08:37:35.608786+0000","last_unstale":"2024-04-21T08:38:30.666749+0000","last_undegraded":"2024-04-21T08:38:30.666749+0000","last_fullsized":"2024-04-21T08:38:30.666749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:45:50.945845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.672091+0000","last_change":"2024-04-21T08:37:30.417156+0000","last_active":"2024-04-21T08:38:30.672091+0000","last_peered":"2024-04-21T08:38:30.672091+0000","last_clean":"2024-04-21T08:38:30.672091+0000","last_became_active":"2024-04-21T08:37:30.415717+0000","last_became_peered":"2024-04-21T08:37:30.415717+0000","last_unstale":"2024-04-21T08:38:30.672091+0000","last_undegraded":"2024-04-21T08:38:30.672091+0000","last_fullsized":"2024-04-21T08:38:30.672091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:37.819082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.596691+0000","last_change":"2024-04-21T08:37:31.444948+0000","last_active":"2024-04-21T08:38:59.596691+0000","last_peered":"2024-04-21T08:38:59.596691+0000","last_clean":"2024-04-21T08:38:59.596691+0000","last_became_active":"2024-04-21T08:37:31.444604+0000","last_became_peered":"2024-04-21T08:37:31.444604+0000","last_unstale":"2024-04-21T08:38:59.596691+0000","last_undegraded":"2024-04-21T08:38:59.596691+0000","last_fullsized":"2024-04-21T08:38:59.596691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.820503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.670106+0000","last_change":"2024-04-21T08:37:35.611541+0000","last_active":"2024-04-21T08:38:30.670106+0000","last_peered":"2024-04-21T08:38:30.670106+0000","last_clean":"2024-04-21T08:38:30.670106+0000","last_became_active":"2024-04-21T08:37:35.611136+0000","last_became_peered":"2024-04-21T08:37:35.611136+0000","last_unstale":"2024-04-21T08:38:30.670106+0000","last_undegraded":"2024-04-21T08:38:30.670106+0000","last_fullsized":"2024-04-21T08:38:30.670106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:28.254529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.667288+0000","last_change":"2024-04-21T08:37:30.414216+0000","last_active":"2024-04-21T08:38:30.667288+0000","last_peered":"2024-04-21T08:38:30.667288+0000","last_clean":"2024-04-21T08:38:30.667288+0000","last_became_active":"2024-04-21T08:37:30.413890+0000","last_became_peered":"2024-04-21T08:37:30.413890+0000","last_unstale":"2024-04-21T08:38:30.667288+0000","last_undegraded":"2024-04-21T08:38:30.667288+0000","last_fullsized":"2024-04-21T08:38:30.667288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:16.972618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599236+0000","last_change":"2024-04-21T08:37:31.449307+0000","last_active":"2024-04-21T08:38:59.599236+0000","last_peered":"2024-04-21T08:38:59.599236+0000","last_clean":"2024-04-21T08:38:59.599236+0000","last_became_active":"2024-04-21T08:37:31.448980+0000","last_became_peered":"2024-04-21T08:37:31.448980+0000","last_unstale":"2024-04-21T08:38:59.599236+0000","last_undegraded":"2024-04-21T08:38:59.599236+0000","last_fullsized":"2024-04-21T08:38:59.599236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:30:31.939274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.669722+0000","last_change":"2024-04-21T08:37:35.760616+0000","last_active":"2024-04-21T08:38:30.669722+0000","last_peered":"2024-04-21T08:38:30.669722+0000","last_clean":"2024-04-21T08:38:30.669722+0000","last_became_active":"2024-04-21T08:37:35.760131+0000","last_became_peered":"2024-04-21T08:37:35.760131+0000","last_unstale":"2024-04-21T08:38:30.669722+0000","last_undegraded":"2024-04-21T08:38:30.669722+0000","last_fullsized":"2024-04-21T08:38:30.669722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:51.510259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.682791+0000","last_change":"2024-04-21T08:38:30.682791+0000","last_active":"2024-04-21T08:38:30.682791+0000","last_peered":"2024-04-21T08:38:30.682791+0000","last_clean":"2024-04-21T08:38:30.682791+0000","last_became_active":"2024-04-21T08:38:30.671977+0000","last_became_peered":"2024-04-21T08:38:30.671977+0000","last_unstale":"2024-04-21T08:38:30.682791+0000","last_undegraded":"2024-04-21T08:38:30.682791+0000","last_fullsized":"2024-04-21T08:38:30.682791+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:55:15.899399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.607675+0000","last_change":"2024-04-21T08:37:31.447234+0000","last_active":"2024-04-21T08:38:59.607675+0000","last_peered":"2024-04-21T08:38:59.607675+0000","last_clean":"2024-04-21T08:38:59.607675+0000","last_became_active":"2024-04-21T08:37:31.446876+0000","last_became_peered":"2024-04-21T08:37:31.446876+0000","last_unstale":"2024-04-21T08:38:59.607675+0000","last_undegraded":"2024-04-21T08:38:59.607675+0000","last_fullsized":"2024-04-21T08:38:59.607675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:51.784126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.667256+0000","last_change":"2024-04-21T08:37:35.598577+0000","last_active":"2024-04-21T08:38:30.667256+0000","last_peered":"2024-04-21T08:38:30.667256+0000","last_clean":"2024-04-21T08:38:30.667256+0000","last_became_active":"2024-04-21T08:37:35.598119+0000","last_became_peered":"2024-04-21T08:37:35.598119+0000","last_unstale":"2024-04-21T08:38:30.667256+0000","last_undegraded":"2024-04-21T08:38:30.667256+0000","last_fullsized":"2024-04-21T08:38:30.667256+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:41.986674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.667632+0000","last_change":"2024-04-21T08:37:30.425737+0000","last_active":"2024-04-21T08:38:30.667632+0000","last_peered":"2024-04-21T08:38:30.667632+0000","last_clean":"2024-04-21T08:38:30.667632+0000","last_became_active":"2024-04-21T08:37:30.425180+0000","last_became_peered":"2024-04-21T08:37:30.425180+0000","last_unstale":"2024-04-21T08:38:30.667632+0000","last_undegraded":"2024-04-21T08:38:30.667632+0000","last_fullsized":"2024-04-21T08:38:30.667632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:08.646106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.608800+0000","last_change":"2024-04-21T08:37:31.451505+0000","last_active":"2024-04-21T08:38:59.608800+0000","last_peered":"2024-04-21T08:38:59.608800+0000","last_clean":"2024-04-21T08:38:59.608800+0000","last_became_active":"2024-04-21T08:37:31.451146+0000","last_became_peered":"2024-04-21T08:37:31.451146+0000","last_unstale":"2024-04-21T08:38:59.608800+0000","last_undegraded":"2024-04-21T08:38:59.608800+0000","last_fullsized":"2024-04-21T08:38:59.608800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:27:01.793910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.802256+0000","last_change":"2024-04-21T08:37:35.608995+0000","last_active":"2024-04-21T08:38:30.802256+0000","last_peered":"2024-04-21T08:38:30.802256+0000","last_clean":"2024-04-21T08:38:30.802256+0000","last_became_active":"2024-04-21T08:37:35.608587+0000","last_became_peered":"2024-04-21T08:37:35.608587+0000","last_unstale":"2024-04-21T08:38:30.802256+0000","last_undegraded":"2024-04-21T08:38:30.802256+0000","last_fullsized":"2024-04-21T08:38:30.802256+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.829418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.675692+0000","last_change":"2024-04-21T08:37:30.419026+0000","last_active":"2024-04-21T08:38:30.675692+0000","last_peered":"2024-04-21T08:38:30.675692+0000","last_clean":"2024-04-21T08:38:30.675692+0000","last_became_active":"2024-04-21T08:37:30.418706+0000","last_became_peered":"2024-04-21T08:37:30.418706+0000","last_unstale":"2024-04-21T08:38:30.675692+0000","last_undegraded":"2024-04-21T08:38:30.675692+0000","last_fullsized":"2024-04-21T08:38:30.675692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:24.441918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.599755+0000","last_change":"2024-04-21T08:37:31.446561+0000","last_active":"2024-04-21T08:38:59.599755+0000","last_peered":"2024-04-21T08:38:59.599755+0000","last_clean":"2024-04-21T08:38:59.599755+0000","last_became_active":"2024-04-21T08:37:31.446162+0000","last_became_peered":"2024-04-21T08:37:31.446162+0000","last_unstale":"2024-04-21T08:38:59.599755+0000","last_undegraded":"2024-04-21T08:38:59.599755+0000","last_fullsized":"2024-04-21T08:38:59.599755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:38:24.684537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.674131+0000","last_change":"2024-04-21T08:37:35.598748+0000","last_active":"2024-04-21T08:38:30.674131+0000","last_peered":"2024-04-21T08:38:30.674131+0000","last_clean":"2024-04-21T08:38:30.674131+0000","last_became_active":"2024-04-21T08:37:35.598353+0000","last_became_peered":"2024-04-21T08:37:35.598353+0000","last_unstale":"2024-04-21T08:38:30.674131+0000","last_undegraded":"2024-04-21T08:38:30.674131+0000","last_fullsized":"2024-04-21T08:38:30.674131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:47:36.519694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:30.691292+0000","last_change":"2024-04-21T08:38:30.691292+0000","last_active":"2024-04-21T08:38:30.691292+0000","last_peered":"2024-04-21T08:38:30.691292+0000","last_clean":"2024-04-21T08:38:30.691292+0000","last_became_active":"2024-04-21T08:38:30.676580+0000","last_became_peered":"2024-04-21T08:38:30.676580+0000","last_unstale":"2024-04-21T08:38:30.691292+0000","last_undegraded":"2024-04-21T08:38:30.691292+0000","last_fullsized":"2024-04-21T08:38:30.691292+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:43:53.150369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.591583+0000","last_change":"2024-04-21T08:37:31.452367+0000","last_active":"2024-04-21T08:38:59.591583+0000","last_peered":"2024-04-21T08:38:59.591583+0000","last_clean":"2024-04-21T08:38:59.591583+0000","last_became_active":"2024-04-21T08:37:31.452031+0000","last_became_peered":"2024-04-21T08:37:31.452031+0000","last_unstale":"2024-04-21T08:38:59.591583+0000","last_undegraded":"2024-04-21T08:38:59.591583+0000","last_fullsized":"2024-04-21T08:38:59.591583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:58.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.596018+0000","last_change":"2024-04-21T08:37:31.451222+0000","last_active":"2024-04-21T08:38:59.596018+0000","last_peered":"2024-04-21T08:38:59.596018+0000","last_clean":"2024-04-21T08:38:59.596018+0000","last_became_active":"2024-04-21T08:37:31.450869+0000","last_became_peered":"2024-04-21T08:37:31.450869+0000","last_unstale":"2024-04-21T08:38:59.596018+0000","last_undegraded":"2024-04-21T08:38:59.596018+0000","last_fullsized":"2024-04-21T08:38:59.596018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:17.051330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.602242+0000","last_change":"2024-04-21T08:37:31.453415+0000","last_active":"2024-04-21T08:38:59.602242+0000","last_peered":"2024-04-21T08:38:59.602242+0000","last_clean":"2024-04-21T08:38:59.602242+0000","last_became_active":"2024-04-21T08:37:31.453089+0000","last_became_peered":"2024-04-21T08:37:31.453089+0000","last_unstale":"2024-04-21T08:38:59.602242+0000","last_undegraded":"2024-04-21T08:38:59.602242+0000","last_fullsized":"2024-04-21T08:38:59.602242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:25:44.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.592759+0000","last_change":"2024-04-21T08:37:31.446707+0000","last_active":"2024-04-21T08:38:59.592759+0000","last_peered":"2024-04-21T08:38:59.592759+0000","last_clean":"2024-04-21T08:38:59.592759+0000","last_became_active":"2024-04-21T08:37:31.446340+0000","last_became_peered":"2024-04-21T08:37:31.446340+0000","last_unstale":"2024-04-21T08:38:59.592759+0000","last_undegraded":"2024-04-21T08:38:59.592759+0000","last_fullsized":"2024-04-21T08:38:59.592759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:13.649607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.596749+0000","last_change":"2024-04-21T08:37:31.455532+0000","last_active":"2024-04-21T08:38:59.596749+0000","last_peered":"2024-04-21T08:38:59.596749+0000","last_clean":"2024-04-21T08:38:59.596749+0000","last_became_active":"2024-04-21T08:37:31.453547+0000","last_became_peered":"2024-04-21T08:37:31.453547+0000","last_unstale":"2024-04-21T08:38:59.596749+0000","last_undegraded":"2024-04-21T08:38:59.596749+0000","last_fullsized":"2024-04-21T08:38:59.596749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:39:47.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.588944+0000","last_change":"2024-04-21T08:37:31.452535+0000","last_active":"2024-04-21T08:38:59.588944+0000","last_peered":"2024-04-21T08:38:59.588944+0000","last_clean":"2024-04-21T08:38:59.588944+0000","last_became_active":"2024-04-21T08:37:31.451637+0000","last_became_peered":"2024-04-21T08:37:31.451637+0000","last_unstale":"2024-04-21T08:38:59.588944+0000","last_undegraded":"2024-04-21T08:38:59.588944+0000","last_fullsized":"2024-04-21T08:38:59.588944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:38:38.162780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.597546+0000","last_change":"2024-04-21T08:37:31.451806+0000","last_active":"2024-04-21T08:38:59.597546+0000","last_peered":"2024-04-21T08:38:59.597546+0000","last_clean":"2024-04-21T08:38:59.597546+0000","last_became_active":"2024-04-21T08:37:31.451442+0000","last_became_peered":"2024-04-21T08:37:31.451442+0000","last_unstale":"2024-04-21T08:38:59.597546+0000","last_undegraded":"2024-04-21T08:38:59.597546+0000","last_fullsized":"2024-04-21T08:38:59.597546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:07.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.594529+0000","last_change":"2024-04-21T08:37:31.444440+0000","last_active":"2024-04-21T08:38:59.594529+0000","last_peered":"2024-04-21T08:38:59.594529+0000","last_clean":"2024-04-21T08:38:59.594529+0000","last_became_active":"2024-04-21T08:37:31.444079+0000","last_became_peered":"2024-04-21T08:37:31.444079+0000","last_unstale":"2024-04-21T08:38:59.594529+0000","last_undegraded":"2024-04-21T08:38:59.594529+0000","last_fullsized":"2024-04-21T08:38:59.594529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:13.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.604115+0000","last_change":"2024-04-21T08:37:31.453698+0000","last_active":"2024-04-21T08:38:59.604115+0000","last_peered":"2024-04-21T08:38:59.604115+0000","last_clean":"2024-04-21T08:38:59.604115+0000","last_became_active":"2024-04-21T08:37:31.453379+0000","last_became_peered":"2024-04-21T08:37:31.453379+0000","last_unstale":"2024-04-21T08:38:59.604115+0000","last_undegraded":"2024-04-21T08:38:59.604115+0000","last_fullsized":"2024-04-21T08:38:59.604115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:46:29.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.595960+0000","last_change":"2024-04-21T08:37:31.444480+0000","last_active":"2024-04-21T08:38:59.595960+0000","last_peered":"2024-04-21T08:38:59.595960+0000","last_clean":"2024-04-21T08:38:59.595960+0000","last_became_active":"2024-04-21T08:37:31.444030+0000","last_became_peered":"2024-04-21T08:37:31.444030+0000","last_unstale":"2024-04-21T08:38:59.595960+0000","last_undegraded":"2024-04-21T08:38:59.595960+0000","last_fullsized":"2024-04-21T08:38:59.595960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:15:05.668885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.598820+0000","last_change":"2024-04-21T08:37:31.456602+0000","last_active":"2024-04-21T08:38:59.598820+0000","last_peered":"2024-04-21T08:38:59.598820+0000","last_clean":"2024-04-21T08:38:59.598820+0000","last_became_active":"2024-04-21T08:37:31.456286+0000","last_became_peered":"2024-04-21T08:37:31.456286+0000","last_unstale":"2024-04-21T08:38:59.598820+0000","last_undegraded":"2024-04-21T08:38:59.598820+0000","last_fullsized":"2024-04-21T08:38:59.598820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:48:12.462037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.607738+0000","last_change":"2024-04-21T08:37:31.447008+0000","last_active":"2024-04-21T08:38:59.607738+0000","last_peered":"2024-04-21T08:38:59.607738+0000","last_clean":"2024-04-21T08:38:59.607738+0000","last_became_active":"2024-04-21T08:37:31.445192+0000","last_became_peered":"2024-04-21T08:37:31.445192+0000","last_unstale":"2024-04-21T08:38:59.607738+0000","last_undegraded":"2024-04-21T08:38:59.607738+0000","last_fullsized":"2024-04-21T08:38:59.607738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:08.025647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.593342+0000","last_change":"2024-04-21T08:37:31.443288+0000","last_active":"2024-04-21T08:38:59.593342+0000","last_peered":"2024-04-21T08:38:59.593342+0000","last_clean":"2024-04-21T08:38:59.593342+0000","last_became_active":"2024-04-21T08:37:31.442586+0000","last_became_peered":"2024-04-21T08:37:31.442586+0000","last_unstale":"2024-04-21T08:38:59.593342+0000","last_undegraded":"2024-04-21T08:38:59.593342+0000","last_fullsized":"2024-04-21T08:38:59.593342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:01.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:38:59.601468+0000","last_change":"2024-04-21T08:37:31.453899+0000","last_active":"2024-04-21T08:38:59.601468+0000","last_peered":"2024-04-21T08:38:59.601468+0000","last_clean":"2024-04-21T08:38:59.601468+0000","last_became_active":"2024-04-21T08:37:31.453527+0000","last_became_peered":"2024-04-21T08:37:31.453527+0000","last_unstale":"2024-04-21T08:38:59.601468+0000","last_undegraded":"2024-04-21T08:38:59.601468+0000","last_fullsized":"2024-04-21T08:38:59.601468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:23:18.429618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283745,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67747,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":494,"num_read_kb":318,"num_write":168,"num_write_kb":121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":203241,"data_compressed":4983,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":500689,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1138688,"data_stored":1152094,"data_compressed":58414,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":505021,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":500263,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1096895,"data_compressed":56753,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":496101,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":560198,"data_compressed":22332,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":497489,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":560966,"data_compressed":22332,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":496295,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":501283,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1091951,"data_compressed":56753,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34445,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64187,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64955,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65715,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:39:16.436 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-21T08:39:16.436 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-21T08:39:16.436 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-21T08:39:16.436 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph health --format=json 2024-04-21T08:39:16.676 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:17.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:16 smithi094 ceph-mon[27988]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:17.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:16 smithi173 ceph-mon[31428]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:17.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:16 smithi111 ceph-mon[31493]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:18.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:17 smithi094 ceph-mon[27988]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:39:18.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:17 smithi173 ceph-mon[31428]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:39:18.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:17 smithi111 ceph-mon[31493]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:39:18.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.428+0000 7fdf61da4700 1 -- 172.21.15.94:0/3954487219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf5c101dc0 msgr2=0x7fdf5c10f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:18.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.428+0000 7fdf61da4700 1 --2- 172.21.15.94:0/3954487219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf5c101dc0 0x7fdf5c10f830 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400b0a0 tx=0x7fdf5400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:39:18.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.428+0000 7fdf61da4700 1 -- 172.21.15.94:0/3954487219 shutdown_connections 2024-04-21T08:39:18.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.428+0000 7fdf61da4700 1 --2- 172.21.15.94:0/3954487219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf5c101dc0 0x7fdf5c10f830 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:18.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.428+0000 7fdf61da4700 1 --2- 172.21.15.94:0/3954487219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf5c107b20 0x7fdf5c107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:18.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.428+0000 7fdf61da4700 1 --2- 172.21.15.94:0/3954487219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf5c101400 0x7fdf5c101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:18.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.428+0000 7fdf61da4700 1 -- 172.21.15.94:0/3954487219 >> 172.21.15.94:0/3954487219 conn(0x7fdf5c076c60 msgr2=0x7fdf5c077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.428+0000 7fdf61da4700 1 -- 172.21.15.94:0/3954487219 shutdown_connections 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.428+0000 7fdf61da4700 1 -- 172.21.15.94:0/3954487219 wait complete. 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.428+0000 7fdf61da4700 1 Processor -- start 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf61da4700 1 -- start start 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf61da4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf5c101400 0x7fdf5c199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf61da4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf5c101dc0 0x7fdf5c199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf61da4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf5c107b20 0x7fdf5c19d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf61da4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c112cb0 con 0x7fdf5c101400 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf61da4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c112b30 con 0x7fdf5c107b20 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf61da4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c112e30 con 0x7fdf5c101dc0 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf5b7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf5c101400 0x7fdf5c199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf5b7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf5c101400 0x7fdf5c199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46500/0 (socket says 172.21.15.94:46500) 2024-04-21T08:39:18.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf5b7fe700 1 -- 172.21.15.94:0/298394511 learned_addr learned my addr 172.21.15.94:0/298394511 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf5bfff700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf5c107b20 0x7fdf5c19d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf5affd700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf5c101dc0 0x7fdf5c199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf5b7fe700 1 -- 172.21.15.94:0/298394511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf5c101dc0 msgr2=0x7fdf5c199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf5b7fe700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf5c101dc0 0x7fdf5c199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf5b7fe700 1 -- 172.21.15.94:0/298394511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf5c107b20 msgr2=0x7fdf5c19d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf5b7fe700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf5c107b20 0x7fdf5c19d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.429+0000 7fdf5b7fe700 1 -- 172.21.15.94:0/298394511 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c19e030 con 0x7fdf5c101400 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.430+0000 7fdf5b7fe700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf5c101400 0x7fdf5c199020 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000d980 tx=0x7fdf5000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.430+0000 7fdf58ff9700 1 -- 172.21.15.94:0/298394511 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50017030 con 0x7fdf5c101400 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.430+0000 7fdf61da4700 1 -- 172.21.15.94:0/298394511 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5c19e320 con 0x7fdf5c101400 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.430+0000 7fdf58ff9700 1 -- 172.21.15.94:0/298394511 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf500044d0 con 0x7fdf5c101400 2024-04-21T08:39:18.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.430+0000 7fdf58ff9700 1 -- 172.21.15.94:0/298394511 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50003970 con 0x7fdf5c101400 2024-04-21T08:39:18.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.430+0000 7fdf61da4700 1 -- 172.21.15.94:0/298394511 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf5c1a5ba0 con 0x7fdf5c101400 2024-04-21T08:39:18.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.431+0000 7fdf61da4700 1 -- 172.21.15.94:0/298394511 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf5c069730 con 0x7fdf5c101400 2024-04-21T08:39:18.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.432+0000 7fdf58ff9700 1 -- 172.21.15.94:0/298394511 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdf5001a070 con 0x7fdf5c101400 2024-04-21T08:39:18.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.432+0000 7fdf58ff9700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fdf440a3c40 0x7fdf440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:18.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.432+0000 7fdf5affd700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fdf440a3c40 0x7fdf440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:18.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.432+0000 7fdf58ff9700 1 -- 172.21.15.94:0/298394511 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7fdf500cffa0 con 0x7fdf5c101400 2024-04-21T08:39:18.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.433+0000 7fdf5affd700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fdf440a3c40 0x7fdf440a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0771d0 tx=0x7fdf4c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:18.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.437+0000 7fdf58ff9700 1 -- 172.21.15.94:0/298394511 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf500232f0 con 0x7fdf5c101400 2024-04-21T08:39:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.683+0000 7fdf61da4700 1 -- 172.21.15.94:0/298394511 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fdf5c19a3b0 con 0x7fdf5c101400 2024-04-21T08:39:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.684+0000 7fdf58ff9700 1 -- 172.21.15.94:0/298394511 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fdf5c19a3b0 con 0x7fdf5c101400 2024-04-21T08:39:18.684 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:18.684 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-21T08:39:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 -- 172.21.15.94:0/298394511 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fdf440a3c40 msgr2=0x7fdf440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fdf440a3c40 0x7fdf440a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0771d0 tx=0x7fdf4c00b410 comp rx=0 tx=0).stop 2024-04-21T08:39:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 -- 172.21.15.94:0/298394511 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf5c101400 msgr2=0x7fdf5c199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf5c101400 0x7fdf5c199020 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000d980 tx=0x7fdf5000dc90 comp rx=0 tx=0).stop 2024-04-21T08:39:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 -- 172.21.15.94:0/298394511 shutdown_connections 2024-04-21T08:39:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fdf440a3c40 0x7fdf440a6100 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf5c107b20 0x7fdf5c19d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf5c101400 0x7fdf5c199020 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 --2- 172.21.15.94:0/298394511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf5c101dc0 0x7fdf5c199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:18.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 -- 172.21.15.94:0/298394511 >> 172.21.15.94:0/298394511 conn(0x7fdf5c076c60 msgr2=0x7fdf5c0fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:18.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 -- 172.21.15.94:0/298394511 shutdown_connections 2024-04-21T08:39:18.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:18.686+0000 7fdf61da4700 1 -- 172.21.15.94:0/298394511 wait complete. 2024-04-21T08:39:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:18 smithi094 ceph-mon[27988]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:39:19.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:18 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/298394511' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:39:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:18 smithi173 ceph-mon[31428]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:39:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:18 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/298394511' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:39:19.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:18 smithi111 ceph-mon[31493]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:39:19.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:18 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/298394511' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:39:19.557 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-21T08:39:19.557 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:39:19.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.665+0000 7f7c5b53d640 1 -- 172.21.15.94:0/2426968081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c54103c60 msgr2=0x7f7c5410a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:19.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.665+0000 7f7c5b53d640 1 --2- 172.21.15.94:0/2426968081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c54103c60 0x7f7c5410a4f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009cd0 tx=0x7f7c4c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.665+0000 7f7c5b53d640 1 -- 172.21.15.94:0/2426968081 shutdown_connections 2024-04-21T08:39:19.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.665+0000 7f7c5b53d640 1 --2- 172.21.15.94:0/2426968081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c54103c60 0x7f7c5410a4f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.665+0000 7f7c5b53d640 1 --2- 172.21.15.94:0/2426968081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c541032a0 0x7f7c54103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.665+0000 7f7c5b53d640 1 --2- 172.21.15.94:0/2426968081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c541002d0 0x7f7c541006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.665+0000 7f7c5b53d640 1 -- 172.21.15.94:0/2426968081 >> 172.21.15.94:0/2426968081 conn(0x7f7c540fbc00 msgr2=0x7f7c540fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:19.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.665+0000 7f7c5b53d640 1 -- 172.21.15.94:0/2426968081 shutdown_connections 2024-04-21T08:39:19.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.665+0000 7f7c5b53d640 1 -- 172.21.15.94:0/2426968081 wait complete. 2024-04-21T08:39:19.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c5b53d640 1 Processor -- start 2024-04-21T08:39:19.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c5b53d640 1 -- start start 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c5b53d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c541002d0 0x7f7c54102c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c5b53d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c541032a0 0x7f7c54101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c5b53d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103c60 0x7f7c54101790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c5b53d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5410dac0 con 0x7f7c541002d0 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c5b53d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5410d940 con 0x7f7c541032a0 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c5b53d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5410dc40 con 0x7f7c54103c60 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c592d9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c541002d0 0x7f7c54102c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c592d9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c541002d0 0x7f7c54102c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46518/0 (socket says 172.21.15.94:46518) 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c592d9640 1 -- 172.21.15.94:0/3165844801 learned_addr learned my addr 172.21.15.94:0/3165844801 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c58ad8640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c541032a0 0x7f7c54101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c59ada640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103c60 0x7f7c54101790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c592d9640 1 -- 172.21.15.94:0/3165844801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103c60 msgr2=0x7f7c54101790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c592d9640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103c60 0x7f7c54101790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c592d9640 1 -- 172.21.15.94:0/3165844801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c541032a0 msgr2=0x7f7c54101250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c592d9640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c541032a0 0x7f7c54101250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c592d9640 1 -- 172.21.15.94:0/3165844801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c54102020 con 0x7f7c541002d0 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c58ad8640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c541032a0 0x7f7c54101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.666+0000 7f7c59ada640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103c60 0x7f7c54101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.667+0000 7f7c592d9640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c541002d0 0x7f7c54102c10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7c5000d950 tx=0x7f7c5000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.667+0000 7f7c4a7fc640 1 -- 172.21.15.94:0/3165844801 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50017020 con 0x7f7c541002d0 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.667+0000 7f7c4a7fc640 1 -- 172.21.15.94:0/3165844801 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c50009c30 con 0x7f7c541002d0 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.667+0000 7f7c5b53d640 1 -- 172.21.15.94:0/3165844801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c541a2250 con 0x7f7c541002d0 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.667+0000 7f7c4a7fc640 1 -- 172.21.15.94:0/3165844801 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5000f690 con 0x7f7c541002d0 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.667+0000 7f7c5b53d640 1 -- 172.21.15.94:0/3165844801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c541a27a0 con 0x7f7c541002d0 2024-04-21T08:39:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.668+0000 7f7c5b53d640 1 -- 172.21.15.94:0/3165844801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c54109450 con 0x7f7c541002d0 2024-04-21T08:39:19.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.669+0000 7f7c4a7fc640 1 -- 172.21.15.94:0/3165844801 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7c5001a020 con 0x7f7c541002d0 2024-04-21T08:39:19.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.669+0000 7f7c4a7fc640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7c400a3bb0 0x7f7c400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:19.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.670+0000 7f7c58ad8640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7c400a3bb0 0x7f7c400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:19.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.670+0000 7f7c4a7fc640 1 -- 172.21.15.94:0/3165844801 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f7c500d0770 con 0x7f7c541002d0 2024-04-21T08:39:19.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.670+0000 7f7c58ad8640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7c400a3bb0 0x7f7c400a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7c44005fd0 tx=0x7f7c44005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:19.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.674+0000 7f7c4a7fc640 1 -- 172.21.15.94:0/3165844801 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c50021020 con 0x7f7c541002d0 2024-04-21T08:39:19.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.858+0000 7f7c5b53d640 1 -- 172.21.15.94:0/3165844801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c54062b50 con 0x7f7c541002d0 2024-04-21T08:39:19.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.859+0000 7f7c4a7fc640 1 -- 172.21.15.94:0/3165844801 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24563 (secure 0 0 0) 0x7f7c50009de0 con 0x7f7c541002d0 2024-04-21T08:39:19.859 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:19.859 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":91,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:38:30.663522+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:39:19.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.861+0000 7f7c5b53d640 1 -- 172.21.15.94:0/3165844801 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7c400a3bb0 msgr2=0x7f7c400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:19.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.861+0000 7f7c5b53d640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7c400a3bb0 0x7f7c400a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7c44005fd0 tx=0x7f7c44005850 comp rx=0 tx=0).stop 2024-04-21T08:39:19.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.862+0000 7f7c5b53d640 1 -- 172.21.15.94:0/3165844801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c541002d0 msgr2=0x7f7c54102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:19.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.862+0000 7f7c5b53d640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c541002d0 0x7f7c54102c10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7c5000d950 tx=0x7f7c5000de10 comp rx=0 tx=0).stop 2024-04-21T08:39:19.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.862+0000 7f7c5b53d640 1 -- 172.21.15.94:0/3165844801 shutdown_connections 2024-04-21T08:39:19.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.862+0000 7f7c5b53d640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f7c400a3bb0 0x7f7c400a6070 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.862+0000 7f7c5b53d640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103c60 0x7f7c54101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.862+0000 7f7c5b53d640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c541032a0 0x7f7c54101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.862+0000 7f7c5b53d640 1 --2- 172.21.15.94:0/3165844801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c541002d0 0x7f7c54102c10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.862+0000 7f7c5b53d640 1 -- 172.21.15.94:0/3165844801 >> 172.21.15.94:0/3165844801 conn(0x7f7c540fbc00 msgr2=0x7f7c54106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:19.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.862+0000 7f7c5b53d640 1 -- 172.21.15.94:0/3165844801 shutdown_connections 2024-04-21T08:39:19.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.862+0000 7f7c5b53d640 1 -- 172.21.15.94:0/3165844801 wait complete. 2024-04-21T08:39:19.872 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:19 smithi094 ceph-mon[27988]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:19.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:39:19.874 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:39:19.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.979+0000 7f2b44af1640 1 -- 172.21.15.94:0/1775176921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b40101fe0 msgr2=0x7f2b40102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:19.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.979+0000 7f2b44af1640 1 --2- 172.21.15.94:0/1775176921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b40101fe0 0x7f2b40102460 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009900 tx=0x7f2b3402bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:19.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.979+0000 7f2b44af1640 1 -- 172.21.15.94:0/1775176921 shutdown_connections 2024-04-21T08:39:19.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.979+0000 7f2b44af1640 1 --2- 172.21.15.94:0/1775176921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b401029a0 0x7f2b401092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.979+0000 7f2b44af1640 1 --2- 172.21.15.94:0/1775176921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b40101fe0 0x7f2b40102460 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.979+0000 7f2b44af1640 1 --2- 172.21.15.94:0/1775176921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b40100e90 0x7f2b401012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.979+0000 7f2b44af1640 1 -- 172.21.15.94:0/1775176921 >> 172.21.15.94:0/1775176921 conn(0x7f2b400fc400 msgr2=0x7f2b400fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.979+0000 7f2b44af1640 1 -- 172.21.15.94:0/1775176921 shutdown_connections 2024-04-21T08:39:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.979+0000 7f2b44af1640 1 -- 172.21.15.94:0/1775176921 wait complete. 2024-04-21T08:39:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b44af1640 1 Processor -- start 2024-04-21T08:39:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b44af1640 1 -- start start 2024-04-21T08:39:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b44af1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b40100e90 0x7f2b401970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b44af1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b40101fe0 0x7f2b401975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b44af1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b401029a0 0x7f2b4019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:19.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b44af1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4010c9d0 con 0x7f2b40101fe0 2024-04-21T08:39:19.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b44af1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4010c850 con 0x7f2b401029a0 2024-04-21T08:39:19.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b44af1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4010cb50 con 0x7f2b40100e90 2024-04-21T08:39:19.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b3dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b40101fe0 0x7f2b401975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:19.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b3dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b40101fe0 0x7f2b401975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46538/0 (socket says 172.21.15.94:46538) 2024-04-21T08:39:19.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b3dd9b640 1 -- 172.21.15.94:0/273808477 learned_addr learned my addr 172.21.15.94:0/273808477 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:19.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b3ed9d640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b401029a0 0x7f2b4019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:19.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b3dd9b640 1 -- 172.21.15.94:0/273808477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b40100e90 msgr2=0x7f2b401970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:19.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b3dd9b640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b40100e90 0x7f2b401970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b3dd9b640 1 -- 172.21.15.94:0/273808477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b401029a0 msgr2=0x7f2b4019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:19.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b3dd9b640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b401029a0 0x7f2b4019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:19.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b3dd9b640 1 -- 172.21.15.94:0/273808477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4019ecc0 con 0x7f2b40101fe0 2024-04-21T08:39:19.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.980+0000 7f2b3dd9b640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b40101fe0 0x7f2b401975e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2b34036040 tx=0x7f2b340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:19.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.981+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/273808477 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b34043020 con 0x7f2b40101fe0 2024-04-21T08:39:19.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.981+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/273808477 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b34004aa0 con 0x7f2b40101fe0 2024-04-21T08:39:19.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.981+0000 7f2b44af1640 1 -- 172.21.15.94:0/273808477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4019ef50 con 0x7f2b40101fe0 2024-04-21T08:39:19.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.981+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/273808477 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b34003de0 con 0x7f2b40101fe0 2024-04-21T08:39:19.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.981+0000 7f2b44af1640 1 -- 172.21.15.94:0/273808477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4019f3b0 con 0x7f2b40101fe0 2024-04-21T08:39:19.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.982+0000 7f2b44af1640 1 -- 172.21.15.94:0/273808477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b401081f0 con 0x7f2b40101fe0 2024-04-21T08:39:19.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.983+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/273808477 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2b34046020 con 0x7f2b40101fe0 2024-04-21T08:39:19.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.983+0000 7f2b2f7fe640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b280a3bb0 0x7f2b280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:19.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.983+0000 7f2b3e59c640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b280a3bb0 0x7f2b280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:19.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.983+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/273808477 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f2b340f2f20 con 0x7f2b40101fe0 2024-04-21T08:39:19.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.984+0000 7f2b3e59c640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b280a3bb0 0x7f2b280a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000a990 tx=0x7f2b30005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:19.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:19.987+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/273808477 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b3404d020 con 0x7f2b40101fe0 2024-04-21T08:39:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:19 smithi173 ceph-mon[31428]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:20.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.163+0000 7f2b44af1640 1 -- 172.21.15.94:0/273808477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2b4002d870 con 0x7f2b40101fe0 2024-04-21T08:39:20.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.164+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/273808477 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f2b340bf5d0 con 0x7f2b40101fe0 2024-04-21T08:39:20.164 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:39:20.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 -- 172.21.15.94:0/273808477 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b280a3bb0 msgr2=0x7f2b280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b280a3bb0 0x7f2b280a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000a990 tx=0x7f2b30005c70 comp rx=0 tx=0).stop 2024-04-21T08:39:20.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 -- 172.21.15.94:0/273808477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b40101fe0 msgr2=0x7f2b401975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b40101fe0 0x7f2b401975e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2b34036040 tx=0x7f2b340042f0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 -- 172.21.15.94:0/273808477 shutdown_connections 2024-04-21T08:39:20.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f2b280a3bb0 0x7f2b280a6070 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b401029a0 0x7f2b4019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b40101fe0 0x7f2b401975e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 --2- 172.21.15.94:0/273808477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b40100e90 0x7f2b401970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 -- 172.21.15.94:0/273808477 >> 172.21.15.94:0/273808477 conn(0x7f2b400fc400 msgr2=0x7f2b401053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:20.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 -- 172.21.15.94:0/273808477 shutdown_connections 2024-04-21T08:39:20.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.166+0000 7f2b44af1640 1 -- 172.21.15.94:0/273808477 wait complete. 2024-04-21T08:39:20.178 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:39:20.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:19 smithi111 ceph-mon[31493]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.283+0000 7f8d02ea5640 1 -- 172.21.15.94:0/1249029319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cfc100e60 msgr2=0x7f8cfc101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.283+0000 7f8d02ea5640 1 --2- 172.21.15.94:0/1249029319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cfc100e60 0x7f8cfc101280 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0009900 tx=0x7f8cf002bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.283+0000 7f8d02ea5640 1 -- 172.21.15.94:0/1249029319 shutdown_connections 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.283+0000 7f8d02ea5640 1 --2- 172.21.15.94:0/1249029319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8cfc102a00 0x7f8cfc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.283+0000 7f8d02ea5640 1 --2- 172.21.15.94:0/1249029319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc102040 0x7f8cfc1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.283+0000 7f8d02ea5640 1 --2- 172.21.15.94:0/1249029319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cfc100e60 0x7f8cfc101280 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.283+0000 7f8d02ea5640 1 -- 172.21.15.94:0/1249029319 >> 172.21.15.94:0/1249029319 conn(0x7f8cfc0fc440 msgr2=0x7f8cfc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.283+0000 7f8d02ea5640 1 -- 172.21.15.94:0/1249029319 shutdown_connections 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.283+0000 7f8d02ea5640 1 -- 172.21.15.94:0/1249029319 wait complete. 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d02ea5640 1 Processor -- start 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d02ea5640 1 -- start start 2024-04-21T08:39:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d02ea5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc100e60 0x7f8cfc1970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d02ea5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8cfc102040 0x7f8cfc197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d02ea5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cfc102a00 0x7f8cfc19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d02ea5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc10c9f0 con 0x7f8cfc102a00 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d02ea5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc10c870 con 0x7f8cfc100e60 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d02ea5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc10cb70 con 0x7f8cfc102040 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d01442640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cfc102a00 0x7f8cfc19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d01442640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cfc102a00 0x7f8cfc19e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46554/0 (socket says 172.21.15.94:46554) 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d01442640 1 -- 172.21.15.94:0/3227297049 learned_addr learned my addr 172.21.15.94:0/3227297049 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d00c41640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc100e60 0x7f8cfc1970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8cfbfff640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8cfc102040 0x7f8cfc197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d01442640 1 -- 172.21.15.94:0/3227297049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8cfc102040 msgr2=0x7f8cfc197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d01442640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8cfc102040 0x7f8cfc197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d01442640 1 -- 172.21.15.94:0/3227297049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc100e60 msgr2=0x7f8cfc1970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d01442640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc100e60 0x7f8cfc1970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d01442640 1 -- 172.21.15.94:0/3227297049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cfc19ed10 con 0x7f8cfc102a00 2024-04-21T08:39:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.284+0000 7f8d01442640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cfc102a00 0x7f8cfc19e6a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400eca0 tx=0x7f8cf400b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.285+0000 7f8cf9ffb640 1 -- 172.21.15.94:0/3227297049 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4004b20 con 0x7f8cfc102a00 2024-04-21T08:39:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.285+0000 7f8d02ea5640 1 -- 172.21.15.94:0/3227297049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cfc06a300 con 0x7f8cfc102a00 2024-04-21T08:39:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.285+0000 7f8cf9ffb640 1 -- 172.21.15.94:0/3227297049 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cf4004cc0 con 0x7f8cfc102a00 2024-04-21T08:39:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.285+0000 7f8cf9ffb640 1 -- 172.21.15.94:0/3227297049 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4017670 con 0x7f8cfc102a00 2024-04-21T08:39:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.285+0000 7f8d02ea5640 1 -- 172.21.15.94:0/3227297049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cfc06a8a0 con 0x7f8cfc102a00 2024-04-21T08:39:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.286+0000 7f8d02ea5640 1 -- 172.21.15.94:0/3227297049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cfc0684e0 con 0x7f8cfc102a00 2024-04-21T08:39:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.287+0000 7f8cf9ffb640 1 -- 172.21.15.94:0/3227297049 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8cf4004e70 con 0x7f8cfc102a00 2024-04-21T08:39:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.287+0000 7f8cf9ffb640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8cec0a3bb0 0x7f8cec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.287+0000 7f8cf9ffb640 1 -- 172.21.15.94:0/3227297049 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f8cf40d2290 con 0x7f8cfc102a00 2024-04-21T08:39:20.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.287+0000 7f8d00c41640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8cec0a3bb0 0x7f8cec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.288+0000 7f8d00c41640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8cec0a3bb0 0x7f8cec0a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0006010 tx=0x7f8cf002c8e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:20.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.292+0000 7f8cf9ffb640 1 -- 172.21.15.94:0/3227297049 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8cf409e940 con 0x7f8cfc102a00 2024-04-21T08:39:20.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.477+0000 7f8d02ea5640 1 -- 172.21.15.94:0/3227297049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8cfc19efa0 con 0x7f8cfc102a00 2024-04-21T08:39:20.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.478+0000 7f8cf9ffb640 1 -- 172.21.15.94:0/3227297049 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f8cf409e240 con 0x7f8cfc102a00 2024-04-21T08:39:20.478 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:39:20.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 -- 172.21.15.94:0/3227297049 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8cec0a3bb0 msgr2=0x7f8cec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8cec0a3bb0 0x7f8cec0a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0006010 tx=0x7f8cf002c8e0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 -- 172.21.15.94:0/3227297049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cfc102a00 msgr2=0x7f8cfc19e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cfc102a00 0x7f8cfc19e6a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400eca0 tx=0x7f8cf400b690 comp rx=0 tx=0).stop 2024-04-21T08:39:20.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 -- 172.21.15.94:0/3227297049 shutdown_connections 2024-04-21T08:39:20.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8cec0a3bb0 0x7f8cec0a6070 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8cfc102040 0x7f8cfc197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cfc102a00 0x7f8cfc19e6a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 --2- 172.21.15.94:0/3227297049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc100e60 0x7f8cfc1970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 -- 172.21.15.94:0/3227297049 >> 172.21.15.94:0/3227297049 conn(0x7f8cfc0fc440 msgr2=0x7f8cfc105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:20.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 -- 172.21.15.94:0/3227297049 shutdown_connections 2024-04-21T08:39:20.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.480+0000 7f8d02ea5640 1 -- 172.21.15.94:0/3227297049 wait complete. 2024-04-21T08:39:20.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:39:20.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.597+0000 7f16468e0640 1 -- 172.21.15.94:0/3247308665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1640107c70 msgr2=0x7f164010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.597+0000 7f16468e0640 1 --2- 172.21.15.94:0/3247308665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1640107c70 0x7f164010a0c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1638009ca0 tx=0x7f163802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.597+0000 7f16468e0640 1 -- 172.21.15.94:0/3247308665 shutdown_connections 2024-04-21T08:39:20.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.597+0000 7f16468e0640 1 --2- 172.21.15.94:0/3247308665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1640107c70 0x7f164010a0c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.597+0000 7f16468e0640 1 --2- 172.21.15.94:0/3247308665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1640105340 0x7f1640107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.597+0000 7f16468e0640 1 --2- 172.21.15.94:0/3247308665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1640102a10 0x7f1640104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.597+0000 7f16468e0640 1 -- 172.21.15.94:0/3247308665 >> 172.21.15.94:0/3247308665 conn(0x7f16400fc400 msgr2=0x7f16400fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:20.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.597+0000 7f16468e0640 1 -- 172.21.15.94:0/3247308665 shutdown_connections 2024-04-21T08:39:20.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.597+0000 7f16468e0640 1 -- 172.21.15.94:0/3247308665 wait complete. 2024-04-21T08:39:20.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f16468e0640 1 Processor -- start 2024-04-21T08:39:20.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f16468e0640 1 -- start start 2024-04-21T08:39:20.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f16468e0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1640102a10 0x7f1640198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f16468e0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1640105340 0x7f16401986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f16468e0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1640107c70 0x7f164019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f16468e0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f164010dae0 con 0x7f1640105340 2024-04-21T08:39:20.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f16468e0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f164010d960 con 0x7f1640107c70 2024-04-21T08:39:20.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f16468e0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f164010dc60 con 0x7f1640102a10 2024-04-21T08:39:20.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f163f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1640105340 0x7f16401986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f163ffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1640102a10 0x7f1640198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f163f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1640105340 0x7f16401986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46558/0 (socket says 172.21.15.94:46558) 2024-04-21T08:39:20.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f163f7fe640 1 -- 172.21.15.94:0/3233194932 learned_addr learned my addr 172.21.15.94:0/3233194932 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:20.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.598+0000 7f1644e7d640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1640107c70 0x7f164019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f163f7fe640 1 -- 172.21.15.94:0/3233194932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1640102a10 msgr2=0x7f1640198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f163f7fe640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1640102a10 0x7f1640198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f163f7fe640 1 -- 172.21.15.94:0/3233194932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1640107c70 msgr2=0x7f164019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f163f7fe640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1640107c70 0x7f164019f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f163f7fe640 1 -- 172.21.15.94:0/3233194932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164019fda0 con 0x7f1640105340 2024-04-21T08:39:20.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f163ffff640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1640102a10 0x7f1640198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:39:20.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f163f7fe640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1640105340 0x7f16401986c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f163400c970 tx=0x7f163400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:20.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f163d7fa640 1 -- 172.21.15.94:0/3233194932 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1634014070 con 0x7f1640105340 2024-04-21T08:39:20.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f163d7fa640 1 -- 172.21.15.94:0/3233194932 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1634004440 con 0x7f1640105340 2024-04-21T08:39:20.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f16468e0640 1 -- 172.21.15.94:0/3233194932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16401a0090 con 0x7f1640105340 2024-04-21T08:39:20.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f163d7fa640 1 -- 172.21.15.94:0/3233194932 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1634003e50 con 0x7f1640105340 2024-04-21T08:39:20.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.599+0000 7f16468e0640 1 -- 172.21.15.94:0/3233194932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16401a05e0 con 0x7f1640105340 2024-04-21T08:39:20.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.600+0000 7f16468e0640 1 -- 172.21.15.94:0/3233194932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16400fdef0 con 0x7f1640105340 2024-04-21T08:39:20.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.602+0000 7f163d7fa640 1 -- 172.21.15.94:0/3233194932 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1634017070 con 0x7f1640105340 2024-04-21T08:39:20.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.603+0000 7f163d7fa640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16280a3d50 0x7f16280a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.603+0000 7f163ffff640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16280a3d50 0x7f16280a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.603+0000 7f163d7fa640 1 -- 172.21.15.94:0/3233194932 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f16340d0200 con 0x7f1640105340 2024-04-21T08:39:20.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.603+0000 7f163ffff640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16280a3d50 0x7f16280a6210 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1630005fd0 tx=0x7f163000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:20.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.607+0000 7f163d7fa640 1 -- 172.21.15.94:0/3233194932 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f163401e030 con 0x7f1640105340 2024-04-21T08:39:20.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.796+0000 7f16468e0640 1 -- 172.21.15.94:0/3233194932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1640062b50 con 0x7f1640105340 2024-04-21T08:39:20.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.797+0000 7f163d7fa640 1 -- 172.21.15.94:0/3233194932 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f163409c8b0 con 0x7f1640105340 2024-04-21T08:39:20.797 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:39:20.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 -- 172.21.15.94:0/3233194932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16280a3d50 msgr2=0x7f16280a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16280a3d50 0x7f16280a6210 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1630005fd0 tx=0x7f163000b390 comp rx=0 tx=0).stop 2024-04-21T08:39:20.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 -- 172.21.15.94:0/3233194932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1640105340 msgr2=0x7f16401986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1640105340 0x7f16401986c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f163400c970 tx=0x7f163400ce30 comp rx=0 tx=0).stop 2024-04-21T08:39:20.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 -- 172.21.15.94:0/3233194932 shutdown_connections 2024-04-21T08:39:20.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f16280a3d50 0x7f16280a6210 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1640107c70 0x7f164019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1640105340 0x7f16401986c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 --2- 172.21.15.94:0/3233194932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1640102a10 0x7f1640198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 -- 172.21.15.94:0/3233194932 >> 172.21.15.94:0/3233194932 conn(0x7f16400fc400 msgr2=0x7f16401032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:20.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 -- 172.21.15.94:0/3233194932 shutdown_connections 2024-04-21T08:39:20.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.799+0000 7f16468e0640 1 -- 172.21.15.94:0/3233194932 wait complete. 2024-04-21T08:39:20.810 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:39:20.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.914+0000 7f217252b640 1 -- 172.21.15.94:0/1100290277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c107c70 msgr2=0x7f216c10a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.914+0000 7f217252b640 1 --2- 172.21.15.94:0/1100290277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c107c70 0x7f216c10a0c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2164009ca0 tx=0x7f216402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.914+0000 7f217252b640 1 -- 172.21.15.94:0/1100290277 shutdown_connections 2024-04-21T08:39:20.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.914+0000 7f217252b640 1 --2- 172.21.15.94:0/1100290277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c107c70 0x7f216c10a0c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.914+0000 7f217252b640 1 --2- 172.21.15.94:0/1100290277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f216c105340 0x7f216c107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.914+0000 7f217252b640 1 --2- 172.21.15.94:0/1100290277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f216c102a10 0x7f216c104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.914+0000 7f217252b640 1 -- 172.21.15.94:0/1100290277 >> 172.21.15.94:0/1100290277 conn(0x7f216c0fc400 msgr2=0x7f216c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.914+0000 7f217252b640 1 -- 172.21.15.94:0/1100290277 shutdown_connections 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.914+0000 7f217252b640 1 -- 172.21.15.94:0/1100290277 wait complete. 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.914+0000 7f217252b640 1 Processor -- start 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f217252b640 1 -- start start 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f217252b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c102a10 0x7f216c102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f217252b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f216c105340 0x7f216c1009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f217252b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f216c107c70 0x7f216c100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f217252b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f216c10dac0 con 0x7f216c102a10 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f217252b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f216c10d940 con 0x7f216c107c70 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f217252b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f216c10dc40 con 0x7f216c105340 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f216bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c102a10 0x7f216c102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f216bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c102a10 0x7f216c102380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46580/0 (socket says 172.21.15.94:46580) 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f216bfff640 1 -- 172.21.15.94:0/3756886741 learned_addr learned my addr 172.21.15.94:0/3756886741 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f2170ac8640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f216c107c70 0x7f216c100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f216bfff640 1 -- 172.21.15.94:0/3756886741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f216c105340 msgr2=0x7f216c1009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f216bfff640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f216c105340 0x7f216c1009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f216bfff640 1 -- 172.21.15.94:0/3756886741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f216c107c70 msgr2=0x7f216c100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f216bfff640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f216c107c70 0x7f216c100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f216bfff640 1 -- 172.21.15.94:0/3756886741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216c101790 con 0x7f216c102a10 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f216bfff640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c102a10 0x7f216c102380 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f215c00d950 tx=0x7f215c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.915+0000 7f21697fa640 1 -- 172.21.15.94:0/3756886741 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c017030 con 0x7f216c102a10 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.916+0000 7f217252b640 1 -- 172.21.15.94:0/3756886741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f216c1a22e0 con 0x7f216c102a10 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.916+0000 7f21697fa640 1 -- 172.21.15.94:0/3756886741 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f215c004020 con 0x7f216c102a10 2024-04-21T08:39:20.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.916+0000 7f21697fa640 1 -- 172.21.15.94:0/3756886741 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c0047a0 con 0x7f216c102a10 2024-04-21T08:39:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.916+0000 7f217252b640 1 -- 172.21.15.94:0/3756886741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f216c1a2880 con 0x7f216c102a10 2024-04-21T08:39:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.917+0000 7f217252b640 1 -- 172.21.15.94:0/3756886741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f216c0fdef0 con 0x7f216c102a10 2024-04-21T08:39:20.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.918+0000 7f21697fa640 1 -- 172.21.15.94:0/3756886741 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f215c009a30 con 0x7f216c102a10 2024-04-21T08:39:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.919+0000 7f21697fa640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21540a3bb0 0x7f21540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.919+0000 7f216b7fe640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21540a3bb0 0x7f21540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.919+0000 7f21697fa640 1 -- 172.21.15.94:0/3756886741 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f215c0d02f0 con 0x7f216c102a10 2024-04-21T08:39:20.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.919+0000 7f216b7fe640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21540a3bb0 0x7f21540a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f216c1020f0 tx=0x7f2160005b60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:20.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:20.923+0000 7f21697fa640 1 -- 172.21.15.94:0/3756886741 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f215c09c9a0 con 0x7f216c102a10 2024-04-21T08:39:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:20 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3165844801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:20 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/273808477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:21.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:20 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3227297049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:21.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:20 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3165844801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:20 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/273808477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:21.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:20 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3227297049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:21.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.104+0000 7f217252b640 1 -- 172.21.15.94:0/3756886741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f216c062b50 con 0x7f216c102a10 2024-04-21T08:39:21.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.105+0000 7f21697fa640 1 -- 172.21.15.94:0/3756886741 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f215c09c2a0 con 0x7f216c102a10 2024-04-21T08:39:21.105 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 -- 172.21.15.94:0/3756886741 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21540a3bb0 msgr2=0x7f21540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21540a3bb0 0x7f21540a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f216c1020f0 tx=0x7f2160005b60 comp rx=0 tx=0).stop 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 -- 172.21.15.94:0/3756886741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c102a10 msgr2=0x7f216c102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c102a10 0x7f216c102380 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f215c00d950 tx=0x7f215c00de10 comp rx=0 tx=0).stop 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 -- 172.21.15.94:0/3756886741 shutdown_connections 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21540a3bb0 0x7f21540a6070 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f216c105340 0x7f216c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f216c107c70 0x7f216c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 --2- 172.21.15.94:0/3756886741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c102a10 0x7f216c102380 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 -- 172.21.15.94:0/3756886741 >> 172.21.15.94:0/3756886741 conn(0x7f216c0fc400 msgr2=0x7f216c1033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 -- 172.21.15.94:0/3756886741 shutdown_connections 2024-04-21T08:39:21.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.107+0000 7f217252b640 1 -- 172.21.15.94:0/3756886741 wait complete. 2024-04-21T08:39:21.118 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:39:21.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:20 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3165844801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:21.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:20 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/273808477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:21.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:20 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3227297049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.218+0000 7f386c6e0640 1 -- 172.21.15.94:0/3135943225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3864102980 msgr2=0x7f38641092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.218+0000 7f386c6e0640 1 --2- 172.21.15.94:0/3135943225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3864102980 0x7f38641092a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f385c009ca0 tx=0x7f385c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.218+0000 7f386c6e0640 1 -- 172.21.15.94:0/3135943225 shutdown_connections 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.218+0000 7f386c6e0640 1 --2- 172.21.15.94:0/3135943225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3864101fc0 0x7f3864102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.218+0000 7f386c6e0640 1 --2- 172.21.15.94:0/3135943225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3864102980 0x7f38641092a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.218+0000 7f386c6e0640 1 --2- 172.21.15.94:0/3135943225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3864100e70 0x7f3864101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.218+0000 7f386c6e0640 1 -- 172.21.15.94:0/3135943225 >> 172.21.15.94:0/3135943225 conn(0x7f38640fc3e0 msgr2=0x7f38640fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.219+0000 7f386c6e0640 1 -- 172.21.15.94:0/3135943225 shutdown_connections 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.219+0000 7f386c6e0640 1 -- 172.21.15.94:0/3135943225 wait complete. 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.219+0000 7f386c6e0640 1 Processor -- start 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.219+0000 7f386c6e0640 1 -- start start 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.219+0000 7f386c6e0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3864100e70 0x7f3864197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.219+0000 7f386c6e0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3864101fc0 0x7f3864197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.219+0000 7f386c6e0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3864102980 0x7f386419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.219+0000 7f386c6e0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f386410c9d0 con 0x7f3864101fc0 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f386c6e0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f386410c850 con 0x7f3864100e70 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f386c6e0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f386410cb50 con 0x7f3864102980 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f3869c7b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3864101fc0 0x7f3864197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f3869c7b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3864101fc0 0x7f3864197560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46592/0 (socket says 172.21.15.94:46592) 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f3869c7b640 1 -- 172.21.15.94:0/3435540127 learned_addr learned my addr 172.21.15.94:0/3435540127 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f386a47c640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3864100e70 0x7f3864197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f386ac7d640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3864102980 0x7f386419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f3869c7b640 1 -- 172.21.15.94:0/3435540127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3864102980 msgr2=0x7f386419e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f3869c7b640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3864102980 0x7f386419e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f3869c7b640 1 -- 172.21.15.94:0/3435540127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3864100e70 msgr2=0x7f3864197020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f3869c7b640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3864100e70 0x7f3864197020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f3869c7b640 1 -- 172.21.15.94:0/3435540127 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386419ec40 con 0x7f3864101fc0 2024-04-21T08:39:21.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f386ac7d640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3864102980 0x7f386419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:21.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f3869c7b640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3864101fc0 0x7f3864197560 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3854009850 tx=0x7f3854009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:21.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f385b7fe640 1 -- 172.21.15.94:0/3435540127 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385401d070 con 0x7f3864101fc0 2024-04-21T08:39:21.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f385b7fe640 1 -- 172.21.15.94:0/3435540127 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3854007c50 con 0x7f3864101fc0 2024-04-21T08:39:21.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f386c6e0640 1 -- 172.21.15.94:0/3435540127 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3864069140 con 0x7f3864101fc0 2024-04-21T08:39:21.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f385b7fe640 1 -- 172.21.15.94:0/3435540127 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854018d50 con 0x7f3864101fc0 2024-04-21T08:39:21.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.220+0000 7f386c6e0640 1 -- 172.21.15.94:0/3435540127 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3864069640 con 0x7f3864101fc0 2024-04-21T08:39:21.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.221+0000 7f386c6e0640 1 -- 172.21.15.94:0/3435540127 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3864107bf0 con 0x7f3864101fc0 2024-04-21T08:39:21.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.222+0000 7f385b7fe640 1 -- 172.21.15.94:0/3435540127 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3854020070 con 0x7f3864101fc0 2024-04-21T08:39:21.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.223+0000 7f385b7fe640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38500a3bb0 0x7f38500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.223+0000 7f386a47c640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38500a3bb0 0x7f38500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.223+0000 7f385b7fe640 1 -- 172.21.15.94:0/3435540127 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f38540d2200 con 0x7f3864101fc0 2024-04-21T08:39:21.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.224+0000 7f386a47c640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38500a3bb0 0x7f38500a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3860005d00 tx=0x7f386000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:21.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.227+0000 7f385b7fe640 1 -- 172.21.15.94:0/3435540127 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3854027070 con 0x7f3864101fc0 2024-04-21T08:39:21.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.448+0000 7f386c6e0640 1 -- 172.21.15.94:0/3435540127 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3864108060 con 0x7f3864101fc0 2024-04-21T08:39:21.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.448+0000 7f385b7fe640 1 -- 172.21.15.94:0/3435540127 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11885 (secure 0 0 0) 0x7f3854007e00 con 0x7f3864101fc0 2024-04-21T08:39:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:21.448 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:37:56.995828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24446,24488]},"id":1}]} 2024-04-21T08:39:21.449 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-04-21T08:39:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 -- 172.21.15.94:0/3435540127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38500a3bb0 msgr2=0x7f38500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38500a3bb0 0x7f38500a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3860005d00 tx=0x7f386000b410 comp rx=0 tx=0).stop 2024-04-21T08:39:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 -- 172.21.15.94:0/3435540127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3864101fc0 msgr2=0x7f3864197560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3864101fc0 0x7f3864197560 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3854009850 tx=0x7f3854009d10 comp rx=0 tx=0).stop 2024-04-21T08:39:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 -- 172.21.15.94:0/3435540127 shutdown_connections 2024-04-21T08:39:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38500a3bb0 0x7f38500a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3864102980 0x7f386419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3864101fc0 0x7f3864197560 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 --2- 172.21.15.94:0/3435540127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3864100e70 0x7f3864197020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 -- 172.21.15.94:0/3435540127 >> 172.21.15.94:0/3435540127 conn(0x7f38640fc3e0 msgr2=0x7f3864105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:21.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 -- 172.21.15.94:0/3435540127 shutdown_connections 2024-04-21T08:39:21.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.451+0000 7f386c6e0640 1 -- 172.21.15.94:0/3435540127 wait complete. 2024-04-21T08:39:21.462 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:39:21.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 -- 172.21.15.94:0/177968183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5298102990 msgr2=0x7f52981092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 --2- 172.21.15.94:0/177968183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5298102990 0x7f52981092b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5280009900 tx=0x7f528002bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:21.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 -- 172.21.15.94:0/177968183 shutdown_connections 2024-04-21T08:39:21.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 --2- 172.21.15.94:0/177968183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5298102990 0x7f52981092b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 --2- 172.21.15.94:0/177968183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5298101fd0 0x7f5298102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 --2- 172.21.15.94:0/177968183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5298100e80 0x7f52981012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 -- 172.21.15.94:0/177968183 >> 172.21.15.94:0/177968183 conn(0x7f52980fc410 msgr2=0x7f52980fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 -- 172.21.15.94:0/177968183 shutdown_connections 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 -- 172.21.15.94:0/177968183 wait complete. 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 Processor -- start 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.560+0000 7f529ea9a640 1 -- start start 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f529ea9a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5298100e80 0x7f5298197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f529ea9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5298101fd0 0x7f5298197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f529ea9a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5298102990 0x7f529819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f529ea9a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f529810c9e0 con 0x7f5298101fd0 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f529ea9a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f529810c860 con 0x7f5298102990 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f529ea9a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f529810cb60 con 0x7f5298100e80 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5297fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5298101fd0 0x7f5298197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5297fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5298101fd0 0x7f5298197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46606/0 (socket says 172.21.15.94:46606) 2024-04-21T08:39:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5297fff640 1 -- 172.21.15.94:0/2445208427 learned_addr learned my addr 172.21.15.94:0/2445208427 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f529d037640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5298102990 0x7f529819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f529c836640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5298100e80 0x7f5298197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5297fff640 1 -- 172.21.15.94:0/2445208427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5298100e80 msgr2=0x7f5298197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5297fff640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5298100e80 0x7f5298197030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5297fff640 1 -- 172.21.15.94:0/2445208427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5298102990 msgr2=0x7f529819e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5297fff640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5298102990 0x7f529819e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5297fff640 1 -- 172.21.15.94:0/2445208427 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529819ec50 con 0x7f5298101fd0 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5297fff640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5298101fd0 0x7f5298197570 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f528800d770 tx=0x7f528800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5295ffb640 1 -- 172.21.15.94:0/2445208427 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288017070 con 0x7f5298101fd0 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f529ea9a640 1 -- 172.21.15.94:0/2445208427 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529819ef40 con 0x7f5298101fd0 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5295ffb640 1 -- 172.21.15.94:0/2445208427 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5288009c30 con 0x7f5298101fd0 2024-04-21T08:39:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.561+0000 7f5295ffb640 1 -- 172.21.15.94:0/2445208427 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288003cc0 con 0x7f5298101fd0 2024-04-21T08:39:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.562+0000 7f529ea9a640 1 -- 172.21.15.94:0/2445208427 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f529819f400 con 0x7f5298101fd0 2024-04-21T08:39:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.562+0000 7f529ea9a640 1 -- 172.21.15.94:0/2445208427 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5298108190 con 0x7f5298101fd0 2024-04-21T08:39:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.563+0000 7f5295ffb640 1 -- 172.21.15.94:0/2445208427 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5288003e60 con 0x7f5298101fd0 2024-04-21T08:39:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.564+0000 7f5295ffb640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52840a3bb0 0x7f52840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.564+0000 7f529c836640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52840a3bb0 0x7f52840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.564+0000 7f5295ffb640 1 -- 172.21.15.94:0/2445208427 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f52880d00d0 con 0x7f5298101fd0 2024-04-21T08:39:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.564+0000 7f529c836640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52840a3bb0 0x7f52840a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5290005fd0 tx=0x7f5290009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:21.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.568+0000 7f5295ffb640 1 -- 172.21.15.94:0/2445208427 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f528809c780 con 0x7f5298101fd0 2024-04-21T08:39:21.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.749+0000 7f529ea9a640 1 -- 172.21.15.94:0/2445208427 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5298062b50 con 0x7f5298101fd0 2024-04-21T08:39:21.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.751+0000 7f5295ffb640 1 -- 172.21.15.94:0/2445208427 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24563 (secure 0 0 0) 0x7f5288009de0 con 0x7f5298101fd0 2024-04-21T08:39:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:21.751 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":91,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:38:30.663522+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:39:21.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 -- 172.21.15.94:0/2445208427 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52840a3bb0 msgr2=0x7f52840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52840a3bb0 0x7f52840a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5290005fd0 tx=0x7f5290009870 comp rx=0 tx=0).stop 2024-04-21T08:39:21.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 -- 172.21.15.94:0/2445208427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5298101fd0 msgr2=0x7f5298197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5298101fd0 0x7f5298197570 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f528800d770 tx=0x7f528800dc30 comp rx=0 tx=0).stop 2024-04-21T08:39:21.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 -- 172.21.15.94:0/2445208427 shutdown_connections 2024-04-21T08:39:21.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52840a3bb0 0x7f52840a6070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5298102990 0x7f529819e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5298101fd0 0x7f5298197570 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 --2- 172.21.15.94:0/2445208427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5298100e80 0x7f5298197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 -- 172.21.15.94:0/2445208427 >> 172.21.15.94:0/2445208427 conn(0x7f52980fc410 msgr2=0x7f5298105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:21.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 -- 172.21.15.94:0/2445208427 shutdown_connections 2024-04-21T08:39:21.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.753+0000 7f529ea9a640 1 -- 172.21.15.94:0/2445208427 wait complete. 2024-04-21T08:39:21.764 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:39:21.765 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.867+0000 7f21aa384640 1 -- 172.21.15.94:0/3467345661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21a4100e90 msgr2=0x7f21a41012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.867+0000 7f21aa384640 1 --2- 172.21.15.94:0/3467345661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21a4100e90 0x7f21a41012b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f218c009900 tx=0x7f218c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.867+0000 7f21aa384640 1 -- 172.21.15.94:0/3467345661 shutdown_connections 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.867+0000 7f21aa384640 1 --2- 172.21.15.94:0/3467345661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a41029a0 0x7f21a41092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.867+0000 7f21aa384640 1 --2- 172.21.15.94:0/3467345661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21a4101fe0 0x7f21a4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.867+0000 7f21aa384640 1 --2- 172.21.15.94:0/3467345661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21a4100e90 0x7f21a41012b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.867+0000 7f21aa384640 1 -- 172.21.15.94:0/3467345661 >> 172.21.15.94:0/3467345661 conn(0x7f21a40fc440 msgr2=0x7f21a40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21aa384640 1 -- 172.21.15.94:0/3467345661 shutdown_connections 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21aa384640 1 -- 172.21.15.94:0/3467345661 wait complete. 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21aa384640 1 Processor -- start 2024-04-21T08:39:21.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21aa384640 1 -- start start 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21aa384640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21a4100e90 0x7f21a41981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21aa384640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a4101fe0 0x7f21a4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21aa384640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21a41029a0 0x7f21a419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21aa384640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f21a410db30 con 0x7f21a41029a0 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21aa384640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f21a410d9b0 con 0x7f21a4100e90 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21aa384640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f21a410dcb0 con 0x7f21a4101fe0 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.868+0000 7f21a8921640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21a41029a0 0x7f21a419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a8921640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21a41029a0 0x7f21a419f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46624/0 (socket says 172.21.15.94:46624) 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a8921640 1 -- 172.21.15.94:0/160287088 learned_addr learned my addr 172.21.15.94:0/160287088 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a37fe640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a4101fe0 0x7f21a4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a3fff640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21a4100e90 0x7f21a41981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a8921640 1 -- 172.21.15.94:0/160287088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a4101fe0 msgr2=0x7f21a4198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a8921640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a4101fe0 0x7f21a4198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a8921640 1 -- 172.21.15.94:0/160287088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21a4100e90 msgr2=0x7f21a41981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a8921640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21a4100e90 0x7f21a41981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a8921640 1 -- 172.21.15.94:0/160287088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a419fdf0 con 0x7f21a41029a0 2024-04-21T08:39:21.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a3fff640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21a4100e90 0x7f21a41981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:21.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a8921640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21a41029a0 0x7f21a419f780 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f219c00eaa0 tx=0x7f219c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:21.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a17fa640 1 -- 172.21.15.94:0/160287088 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c00cdb0 con 0x7f21a41029a0 2024-04-21T08:39:21.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21aa384640 1 -- 172.21.15.94:0/160287088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a41a00e0 con 0x7f21a41029a0 2024-04-21T08:39:21.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a17fa640 1 -- 172.21.15.94:0/160287088 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f219c005020 con 0x7f21a41029a0 2024-04-21T08:39:21.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21a17fa640 1 -- 172.21.15.94:0/160287088 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c0116d0 con 0x7f21a41029a0 2024-04-21T08:39:21.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.869+0000 7f21aa384640 1 -- 172.21.15.94:0/160287088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a41a0630 con 0x7f21a41029a0 2024-04-21T08:39:21.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.870+0000 7f21aa384640 1 -- 172.21.15.94:0/160287088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21a4108220 con 0x7f21a41029a0 2024-04-21T08:39:21.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.871+0000 7f21a17fa640 1 -- 172.21.15.94:0/160287088 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f219c0051d0 con 0x7f21a41029a0 2024-04-21T08:39:21.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.872+0000 7f21a17fa640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21900a3c30 0x7f21900a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:21.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.872+0000 7f21a3fff640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21900a3c30 0x7f21900a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:21.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.872+0000 7f21a17fa640 1 -- 172.21.15.94:0/160287088 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f219c0d09f0 con 0x7f21a41029a0 2024-04-21T08:39:21.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.872+0000 7f21a3fff640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21900a3c30 0x7f21900a60f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f218c0096c0 tx=0x7f218c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:21.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:21.876+0000 7f21a17fa640 1 -- 172.21.15.94:0/160287088 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f219c016030 con 0x7f21a41029a0 2024-04-21T08:39:22.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:21 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3233194932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:22.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:21 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3756886741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:22.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:21 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3435540127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:22.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:21 smithi094 ceph-mon[27988]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:22.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:21 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3233194932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:21 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3756886741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:21 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3435540127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:22.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:21 smithi173 ceph-mon[31428]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:22.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.053+0000 7f21aa384640 1 -- 172.21.15.94:0/160287088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f21a4062b50 con 0x7f21a41029a0 2024-04-21T08:39:22.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.053+0000 7f21a17fa640 1 -- 172.21.15.94:0/160287088 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f219c09d040 con 0x7f21a41029a0 2024-04-21T08:39:22.053 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:39:22.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 -- 172.21.15.94:0/160287088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21900a3c30 msgr2=0x7f21900a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21900a3c30 0x7f21900a60f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f218c0096c0 tx=0x7f218c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 -- 172.21.15.94:0/160287088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21a41029a0 msgr2=0x7f21a419f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21a41029a0 0x7f21a419f780 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f219c00eaa0 tx=0x7f219c00ef60 comp rx=0 tx=0).stop 2024-04-21T08:39:22.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 -- 172.21.15.94:0/160287088 shutdown_connections 2024-04-21T08:39:22.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21900a3c30 0x7f21900a60f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a4101fe0 0x7f21a4198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21a41029a0 0x7f21a419f780 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 --2- 172.21.15.94:0/160287088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21a4100e90 0x7f21a41981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 -- 172.21.15.94:0/160287088 >> 172.21.15.94:0/160287088 conn(0x7f21a40fc440 msgr2=0x7f21a41053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:22.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 -- 172.21.15.94:0/160287088 shutdown_connections 2024-04-21T08:39:22.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.055+0000 7f21aa384640 1 -- 172.21.15.94:0/160287088 wait complete. 2024-04-21T08:39:22.066 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:39:22.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.167+0000 7f98f525b640 1 -- 172.21.15.94:0/2069208449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98f0100e90 msgr2=0x7f98f01012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.167+0000 7f98f525b640 1 --2- 172.21.15.94:0/2069208449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98f0100e90 0x7f98f01012b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009900 tx=0x7f98e002bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:22.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.167+0000 7f98f525b640 1 -- 172.21.15.94:0/2069208449 shutdown_connections 2024-04-21T08:39:22.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.167+0000 7f98f525b640 1 --2- 172.21.15.94:0/2069208449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98f01029a0 0x7f98f01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.167+0000 7f98f525b640 1 --2- 172.21.15.94:0/2069208449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98f0101fe0 0x7f98f0102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.167+0000 7f98f525b640 1 --2- 172.21.15.94:0/2069208449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98f0100e90 0x7f98f01012b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.167+0000 7f98f525b640 1 -- 172.21.15.94:0/2069208449 >> 172.21.15.94:0/2069208449 conn(0x7f98f00fc400 msgr2=0x7f98f00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.167+0000 7f98f525b640 1 -- 172.21.15.94:0/2069208449 shutdown_connections 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.167+0000 7f98f525b640 1 -- 172.21.15.94:0/2069208449 wait complete. 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98f525b640 1 Processor -- start 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98f525b640 1 -- start start 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98f525b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98f0100e90 0x7f98f0069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98f525b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98f0101fe0 0x7f98f019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98f525b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98f01029a0 0x7f98f019f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98f525b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f98f010c9e0 con 0x7f98f0100e90 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98f525b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f98f010c860 con 0x7f98f01029a0 2024-04-21T08:39:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98f525b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f98f010cb60 con 0x7f98f0101fe0 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98eed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98f0100e90 0x7f98f0069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98eed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98f0100e90 0x7f98f0069c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46650/0 (socket says 172.21.15.94:46650) 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.168+0000 7f98eed9d640 1 -- 172.21.15.94:0/1086825855 learned_addr learned my addr 172.21.15.94:0/1086825855 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98ee59c640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98f0101fe0 0x7f98f019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98eed9d640 1 -- 172.21.15.94:0/1086825855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98f0101fe0 msgr2=0x7f98f019ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98eed9d640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98f0101fe0 0x7f98f019ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98eed9d640 1 -- 172.21.15.94:0/1086825855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98f01029a0 msgr2=0x7f98f019f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98ef59e640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98f01029a0 0x7f98f019f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98eed9d640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98f01029a0 0x7f98f019f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98eed9d640 1 -- 172.21.15.94:0/1086825855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f019fdb0 con 0x7f98f0100e90 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98eed9d640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98f0100e90 0x7f98f0069c70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f98e0006010 tx=0x7f98e00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98d7fff640 1 -- 172.21.15.94:0/1086825855 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e0043020 con 0x7f98f0100e90 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98d7fff640 1 -- 172.21.15.94:0/1086825855 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98e0004aa0 con 0x7f98f0100e90 2024-04-21T08:39:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98f525b640 1 -- 172.21.15.94:0/1086825855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98f01a0040 con 0x7f98f0100e90 2024-04-21T08:39:22.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98d7fff640 1 -- 172.21.15.94:0/1086825855 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e0003c70 con 0x7f98f0100e90 2024-04-21T08:39:22.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.169+0000 7f98f525b640 1 -- 172.21.15.94:0/1086825855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f01a0590 con 0x7f98f0100e90 2024-04-21T08:39:22.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.170+0000 7f98f525b640 1 -- 172.21.15.94:0/1086825855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98f01081a0 con 0x7f98f0100e90 2024-04-21T08:39:22.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.171+0000 7f98d7fff640 1 -- 172.21.15.94:0/1086825855 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f98e0046020 con 0x7f98f0100e90 2024-04-21T08:39:22.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.172+0000 7f98d7fff640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f98d80a3bb0 0x7f98d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.172+0000 7f98ee59c640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f98d80a3bb0 0x7f98d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.172+0000 7f98d7fff640 1 -- 172.21.15.94:0/1086825855 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f98e00f3240 con 0x7f98f0100e90 2024-04-21T08:39:22.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.172+0000 7f98ee59c640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f98d80a3bb0 0x7f98d80a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f98e40058c0 tx=0x7f98e4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:22.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.175+0000 7f98d7fff640 1 -- 172.21.15.94:0/1086825855 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98e004d020 con 0x7f98f0100e90 2024-04-21T08:39:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:21 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3233194932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:21 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3756886741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:21 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3435540127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:21 smithi111 ceph-mon[31493]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:22.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.352+0000 7f98f525b640 1 -- 172.21.15.94:0/1086825855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f98f002d830 con 0x7f98f0100e90 2024-04-21T08:39:22.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.352+0000 7f98d7fff640 1 -- 172.21.15.94:0/1086825855 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f98e00bf8f0 con 0x7f98f0100e90 2024-04-21T08:39:22.352 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:39:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 -- 172.21.15.94:0/1086825855 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f98d80a3bb0 msgr2=0x7f98d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f98d80a3bb0 0x7f98d80a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f98e40058c0 tx=0x7f98e4005850 comp rx=0 tx=0).stop 2024-04-21T08:39:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 -- 172.21.15.94:0/1086825855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98f0100e90 msgr2=0x7f98f0069c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98f0100e90 0x7f98f0069c70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f98e0006010 tx=0x7f98e00042f0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 -- 172.21.15.94:0/1086825855 shutdown_connections 2024-04-21T08:39:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f98d80a3bb0 0x7f98d80a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98f0101fe0 0x7f98f019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98f01029a0 0x7f98f019f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 --2- 172.21.15.94:0/1086825855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98f0100e90 0x7f98f0069c70 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 -- 172.21.15.94:0/1086825855 >> 172.21.15.94:0/1086825855 conn(0x7f98f00fc400 msgr2=0x7f98f0105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:22.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 -- 172.21.15.94:0/1086825855 shutdown_connections 2024-04-21T08:39:22.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.354+0000 7f98f525b640 1 -- 172.21.15.94:0/1086825855 wait complete. 2024-04-21T08:39:22.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.467+0000 7ff83a83f640 1 -- 172.21.15.94:0/2824754717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff834101200 msgr2=0x7ff834101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.467+0000 7ff83a83f640 1 --2- 172.21.15.94:0/2824754717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff834101200 0x7ff834101660 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff828009900 tx=0x7ff82802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.467+0000 7ff83a83f640 1 -- 172.21.15.94:0/2824754717 shutdown_connections 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.467+0000 7ff83a83f640 1 --2- 172.21.15.94:0/2824754717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff834101ba0 0x7ff834102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.467+0000 7ff83a83f640 1 --2- 172.21.15.94:0/2824754717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff834101200 0x7ff834101660 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.467+0000 7ff83a83f640 1 --2- 172.21.15.94:0/2824754717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8341008a0 0x7ff834100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.467+0000 7ff83a83f640 1 -- 172.21.15.94:0/2824754717 >> 172.21.15.94:0/2824754717 conn(0x7ff8340fc420 msgr2=0x7ff8340fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.467+0000 7ff83a83f640 1 -- 172.21.15.94:0/2824754717 shutdown_connections 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.467+0000 7ff83a83f640 1 -- 172.21.15.94:0/2824754717 wait complete. 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.468+0000 7ff83a83f640 1 Processor -- start 2024-04-21T08:39:22.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.468+0000 7ff83a83f640 1 -- start start 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.468+0000 7ff83a83f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8341008a0 0x7ff83419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.468+0000 7ff83a83f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff834101200 0x7ff83419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.468+0000 7ff83a83f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff834101ba0 0x7ff8341a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.468+0000 7ff83a83f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff83410fcf0 con 0x7ff834101ba0 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.468+0000 7ff83a83f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff83410fb70 con 0x7ff834101200 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.468+0000 7ff83a83f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff83410fe70 con 0x7ff8341008a0 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.468+0000 7ff838ddc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff834101ba0 0x7ff8341a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.468+0000 7ff838ddc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff834101ba0 0x7ff8341a1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46674/0 (socket says 172.21.15.94:46674) 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff838ddc640 1 -- 172.21.15.94:0/3756091410 learned_addr learned my addr 172.21.15.94:0/3756091410 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff833fff640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8341008a0 0x7ff83419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff8337fe640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff834101200 0x7ff83419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff838ddc640 1 -- 172.21.15.94:0/3756091410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8341008a0 msgr2=0x7ff83419a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff838ddc640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8341008a0 0x7ff83419a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff838ddc640 1 -- 172.21.15.94:0/3756091410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff834101200 msgr2=0x7ff83419a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff838ddc640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff834101200 0x7ff83419a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff838ddc640 1 -- 172.21.15.94:0/3756091410 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8341a2000 con 0x7ff834101ba0 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff838ddc640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff834101ba0 0x7ff8341a1990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00cce0 tx=0x7ff82c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff8317fa640 1 -- 172.21.15.94:0/3756091410 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00f450 con 0x7ff834101ba0 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff8317fa640 1 -- 172.21.15.94:0/3756091410 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c00faa0 con 0x7ff834101ba0 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff83a83f640 1 -- 172.21.15.94:0/3756091410 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8341a22f0 con 0x7ff834101ba0 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff8317fa640 1 -- 172.21.15.94:0/3756091410 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c018b70 con 0x7ff834101ba0 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.469+0000 7ff83a83f640 1 -- 172.21.15.94:0/3756091410 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8341a27f0 con 0x7ff834101ba0 2024-04-21T08:39:22.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.470+0000 7ff83a83f640 1 -- 172.21.15.94:0/3756091410 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff83410b830 con 0x7ff834101ba0 2024-04-21T08:39:22.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.471+0000 7ff8317fa640 1 -- 172.21.15.94:0/3756091410 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff82c018d10 con 0x7ff834101ba0 2024-04-21T08:39:22.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.472+0000 7ff8317fa640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff81c0a3bb0 0x7ff81c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.472+0000 7ff833fff640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff81c0a3bb0 0x7ff81c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.472+0000 7ff8317fa640 1 -- 172.21.15.94:0/3756091410 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7ff82c0d0c20 con 0x7ff834101ba0 2024-04-21T08:39:22.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.472+0000 7ff833fff640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff81c0a3bb0 0x7ff81c0a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff824005fd0 tx=0x7ff82400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:22.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.476+0000 7ff8317fa640 1 -- 172.21.15.94:0/3756091410 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff82c014030 con 0x7ff834101ba0 2024-04-21T08:39:22.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.653+0000 7ff83a83f640 1 -- 172.21.15.94:0/3756091410 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff834062b50 con 0x7ff834101ba0 2024-04-21T08:39:22.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.653+0000 7ff8317fa640 1 -- 172.21.15.94:0/3756091410 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7ff82c09d360 con 0x7ff834101ba0 2024-04-21T08:39:22.653 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:39:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.655+0000 7ff83a83f640 1 -- 172.21.15.94:0/3756091410 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff81c0a3bb0 msgr2=0x7ff81c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.655+0000 7ff83a83f640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff81c0a3bb0 0x7ff81c0a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff824005fd0 tx=0x7ff82400b390 comp rx=0 tx=0).stop 2024-04-21T08:39:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.655+0000 7ff83a83f640 1 -- 172.21.15.94:0/3756091410 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff834101ba0 msgr2=0x7ff8341a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.655+0000 7ff83a83f640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff834101ba0 0x7ff8341a1990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00cce0 tx=0x7ff82c00a620 comp rx=0 tx=0).stop 2024-04-21T08:39:22.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.655+0000 7ff83a83f640 1 -- 172.21.15.94:0/3756091410 shutdown_connections 2024-04-21T08:39:22.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.655+0000 7ff83a83f640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ff81c0a3bb0 0x7ff81c0a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.655+0000 7ff83a83f640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff834101ba0 0x7ff8341a1990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.655+0000 7ff83a83f640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff834101200 0x7ff83419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.655+0000 7ff83a83f640 1 --2- 172.21.15.94:0/3756091410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8341008a0 0x7ff83419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.655+0000 7ff83a83f640 1 -- 172.21.15.94:0/3756091410 >> 172.21.15.94:0/3756091410 conn(0x7ff8340fc420 msgr2=0x7ff8340fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:22.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.656+0000 7ff83a83f640 1 -- 172.21.15.94:0/3756091410 shutdown_connections 2024-04-21T08:39:22.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.656+0000 7ff83a83f640 1 -- 172.21.15.94:0/3756091410 wait complete. 2024-04-21T08:39:22.666 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:39:22.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 -- 172.21.15.94:0/57577942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97b4103b60 msgr2=0x7f97b410a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 --2- 172.21.15.94:0/57577942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97b4103b60 0x7f97b410a3f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009ca0 tx=0x7f97ac02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 -- 172.21.15.94:0/57577942 shutdown_connections 2024-04-21T08:39:22.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 --2- 172.21.15.94:0/57577942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b41031a0 0x7f97b4103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 --2- 172.21.15.94:0/57577942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97b4103b60 0x7f97b410a3f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 --2- 172.21.15.94:0/57577942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b4101fb0 0x7f97b41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 -- 172.21.15.94:0/57577942 >> 172.21.15.94:0/57577942 conn(0x7f97b40fd500 msgr2=0x7f97b40ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 -- 172.21.15.94:0/57577942 shutdown_connections 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 -- 172.21.15.94:0/57577942 wait complete. 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 Processor -- start 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 -- start start 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97b4101fb0 0x7f97b41981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b41031a0 0x7f97b41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b4103b60 0x7f97b419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f97b410daf0 con 0x7f97b4101fb0 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f97b410d970 con 0x7f97b4103b60 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.769+0000 7f97badca640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f97b410dc70 con 0x7f97b41031a0 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b8b66640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97b4101fb0 0x7f97b41981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b8b66640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97b4101fb0 0x7f97b41981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46696/0 (socket says 172.21.15.94:46696) 2024-04-21T08:39:22.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b8b66640 1 -- 172.21.15.94:0/1847230468 learned_addr learned my addr 172.21.15.94:0/1847230468 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b3fff640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b41031a0 0x7f97b41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b9367640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b4103b60 0x7f97b419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b8b66640 1 -- 172.21.15.94:0/1847230468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b41031a0 msgr2=0x7f97b41986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b8b66640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b41031a0 0x7f97b41986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b8b66640 1 -- 172.21.15.94:0/1847230468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b4103b60 msgr2=0x7f97b419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b8b66640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b4103b60 0x7f97b419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b8b66640 1 -- 172.21.15.94:0/1847230468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b419fdc0 con 0x7f97b4101fb0 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b3fff640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b41031a0 0x7f97b41986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b9367640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b4103b60 0x7f97b419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b8b66640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97b4101fb0 0x7f97b41981a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f97a800c970 tx=0x7f97a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b1ffb640 1 -- 172.21.15.94:0/1847230468 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8004220 con 0x7f97b4101fb0 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b1ffb640 1 -- 172.21.15.94:0/1847230468 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97a80043c0 con 0x7f97b4101fb0 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97badca640 1 -- 172.21.15.94:0/1847230468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b41a00b0 con 0x7f97b4101fb0 2024-04-21T08:39:22.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97b1ffb640 1 -- 172.21.15.94:0/1847230468 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a800f450 con 0x7f97b4101fb0 2024-04-21T08:39:22.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.770+0000 7f97badca640 1 -- 172.21.15.94:0/1847230468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b41a0600 con 0x7f97b4101fb0 2024-04-21T08:39:22.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.771+0000 7f97badca640 1 -- 172.21.15.94:0/1847230468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b4109320 con 0x7f97b4101fb0 2024-04-21T08:39:22.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.772+0000 7f97b1ffb640 1 -- 172.21.15.94:0/1847230468 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f97a8003610 con 0x7f97b4101fb0 2024-04-21T08:39:22.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.773+0000 7f97b1ffb640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97a40a3bb0 0x7f97a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:22.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.773+0000 7f97b3fff640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97a40a3bb0 0x7f97a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:22.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.773+0000 7f97b1ffb640 1 -- 172.21.15.94:0/1847230468 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f97a80d0660 con 0x7f97b4101fb0 2024-04-21T08:39:22.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.773+0000 7f97b3fff640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97a40a3bb0 0x7f97a40a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f97b41996b0 tx=0x7f97a0005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:22.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.777+0000 7f97b1ffb640 1 -- 172.21.15.94:0/1847230468 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97a8014030 con 0x7f97b4101fb0 2024-04-21T08:39:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.953+0000 7f97badca640 1 -- 172.21.15.94:0/1847230468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f97b402d5c0 con 0x7f97b4101fb0 2024-04-21T08:39:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.954+0000 7f97b1ffb640 1 -- 172.21.15.94:0/1847230468 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f97b402d5c0 con 0x7f97b4101fb0 2024-04-21T08:39:22.953 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:39:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.955+0000 7f97badca640 1 -- 172.21.15.94:0/1847230468 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97a40a3bb0 msgr2=0x7f97a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.955+0000 7f97badca640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97a40a3bb0 0x7f97a40a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f97b41996b0 tx=0x7f97a0005e50 comp rx=0 tx=0).stop 2024-04-21T08:39:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.955+0000 7f97badca640 1 -- 172.21.15.94:0/1847230468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97b4101fb0 msgr2=0x7f97b41981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.955+0000 7f97badca640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97b4101fb0 0x7f97b41981a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f97a800c970 tx=0x7f97a800ce30 comp rx=0 tx=0).stop 2024-04-21T08:39:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.955+0000 7f97badca640 1 -- 172.21.15.94:0/1847230468 shutdown_connections 2024-04-21T08:39:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.955+0000 7f97badca640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97a40a3bb0 0x7f97a40a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.956+0000 7f97badca640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b41031a0 0x7f97b41986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.956+0000 7f97badca640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b4103b60 0x7f97b419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.956+0000 7f97badca640 1 --2- 172.21.15.94:0/1847230468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97b4101fb0 0x7f97b41981a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.956+0000 7f97badca640 1 -- 172.21.15.94:0/1847230468 >> 172.21.15.94:0/1847230468 conn(0x7f97b40fd500 msgr2=0x7f97b41064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.956+0000 7f97badca640 1 -- 172.21.15.94:0/1847230468 shutdown_connections 2024-04-21T08:39:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:22.956+0000 7f97badca640 1 -- 172.21.15.94:0/1847230468 wait complete. 2024-04-21T08:39:22.966 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:22 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2445208427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:22.966 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:22 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/160287088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:22.966 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:22 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1086825855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:22.966 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:22 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3756091410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:22.967 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:39:23.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:22 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2445208427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:23.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:22 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/160287088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:23.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:22 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1086825855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:23.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:22 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3756091410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.065+0000 7f20bd92b640 1 -- 172.21.15.94:0/3413408485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20b8101fc0 msgr2=0x7f20b8102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.065+0000 7f20bd92b640 1 --2- 172.21.15.94:0/3413408485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20b8101fc0 0x7f20b8102440 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009900 tx=0x7f20ac02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.065+0000 7f20bd92b640 1 -- 172.21.15.94:0/3413408485 shutdown_connections 2024-04-21T08:39:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.065+0000 7f20bd92b640 1 --2- 172.21.15.94:0/3413408485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b8102980 0x7f20b81092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.065+0000 7f20bd92b640 1 --2- 172.21.15.94:0/3413408485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20b8101fc0 0x7f20b8102440 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.065+0000 7f20bd92b640 1 --2- 172.21.15.94:0/3413408485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20b8100e70 0x7f20b8101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.065+0000 7f20bd92b640 1 -- 172.21.15.94:0/3413408485 >> 172.21.15.94:0/3413408485 conn(0x7f20b80fc3e0 msgr2=0x7f20b80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.065+0000 7f20bd92b640 1 -- 172.21.15.94:0/3413408485 shutdown_connections 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.065+0000 7f20bd92b640 1 -- 172.21.15.94:0/3413408485 wait complete. 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20bd92b640 1 Processor -- start 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20bd92b640 1 -- start start 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20bd92b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20b8100e70 0x7f20b8104df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20bd92b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20b8101fc0 0x7f20b8103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20bd92b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b8102980 0x7f20b8103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20bd92b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f20b810c9c0 con 0x7f20b8101fc0 2024-04-21T08:39:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20bd92b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f20b810c840 con 0x7f20b8102980 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20bd92b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f20b810cb40 con 0x7f20b8100e70 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20b8101fc0 0x7f20b8103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20b8101fc0 0x7f20b8103430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46724/0 (socket says 172.21.15.94:46724) 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b67fc640 1 -- 172.21.15.94:0/1246559718 learned_addr learned my addr 172.21.15.94:0/1246559718 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b6ffd640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20b8100e70 0x7f20b8104df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b77fe640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b8102980 0x7f20b8103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b67fc640 1 -- 172.21.15.94:0/1246559718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20b8100e70 msgr2=0x7f20b8104df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b67fc640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20b8100e70 0x7f20b8104df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b67fc640 1 -- 172.21.15.94:0/1246559718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b8102980 msgr2=0x7f20b8103970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b67fc640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b8102980 0x7f20b8103970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b67fc640 1 -- 172.21.15.94:0/1246559718 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b8104200 con 0x7f20b8101fc0 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.066+0000 7f20b77fe640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b8102980 0x7f20b8103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.067+0000 7f20b67fc640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20b8101fc0 0x7f20b8103430 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f20ac036040 tx=0x7f20ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.067+0000 7f20bc929640 1 -- 172.21.15.94:0/1246559718 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac043020 con 0x7f20b8101fc0 2024-04-21T08:39:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.067+0000 7f20bc929640 1 -- 172.21.15.94:0/1246559718 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20ac004aa0 con 0x7f20b8101fc0 2024-04-21T08:39:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.067+0000 7f20bd92b640 1 -- 172.21.15.94:0/1246559718 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b81a2360 con 0x7f20b8101fc0 2024-04-21T08:39:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.067+0000 7f20bc929640 1 -- 172.21.15.94:0/1246559718 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac003de0 con 0x7f20b8101fc0 2024-04-21T08:39:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.067+0000 7f20bd92b640 1 -- 172.21.15.94:0/1246559718 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b81a28b0 con 0x7f20b8101fc0 2024-04-21T08:39:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.068+0000 7f20bd92b640 1 -- 172.21.15.94:0/1246559718 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20b81a2ab0 con 0x7f20b8101fc0 2024-04-21T08:39:23.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.069+0000 7f20bc929640 1 -- 172.21.15.94:0/1246559718 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f20ac046020 con 0x7f20b8101fc0 2024-04-21T08:39:23.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.069+0000 7f20bc929640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f20a00a3bb0 0x7f20a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.069+0000 7f20b6ffd640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f20a00a3bb0 0x7f20a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.070+0000 7f20bc929640 1 -- 172.21.15.94:0/1246559718 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f20ac0f2fe0 con 0x7f20b8101fc0 2024-04-21T08:39:23.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.070+0000 7f20b6ffd640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f20a00a3bb0 0x7f20a00a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f20a8005fd0 tx=0x7f20a8005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:23.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.073+0000 7f20bc929640 1 -- 172.21.15.94:0/1246559718 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20ac04d020 con 0x7f20b8101fc0 2024-04-21T08:39:23.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:22 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2445208427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:23.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:22 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/160287088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:23.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:22 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1086825855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:23.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:22 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3756091410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.288+0000 7f20bd92b640 1 -- 172.21.15.94:0/1246559718 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20b81a2da0 con 0x7f20b8101fc0 2024-04-21T08:39:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.288+0000 7f20bc929640 1 -- 172.21.15.94:0/1246559718 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11885 (secure 0 0 0) 0x7f20ac0bf690 con 0x7f20b8101fc0 2024-04-21T08:39:23.288 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:23.289 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:37:56.995828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24446,24488]},"id":1}]} 2024-04-21T08:39:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-04-21T08:39:23.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.290+0000 7f20bd92b640 1 -- 172.21.15.94:0/1246559718 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f20a00a3bb0 msgr2=0x7f20a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.290+0000 7f20bd92b640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f20a00a3bb0 0x7f20a00a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f20a8005fd0 tx=0x7f20a8005e20 comp rx=0 tx=0).stop 2024-04-21T08:39:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.290+0000 7f20bd92b640 1 -- 172.21.15.94:0/1246559718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20b8101fc0 msgr2=0x7f20b8103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.290+0000 7f20bd92b640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20b8101fc0 0x7f20b8103430 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f20ac036040 tx=0x7f20ac0042f0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.290+0000 7f20bd92b640 1 -- 172.21.15.94:0/1246559718 shutdown_connections 2024-04-21T08:39:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.290+0000 7f20bd92b640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f20a00a3bb0 0x7f20a00a6070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.290+0000 7f20bd92b640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b8102980 0x7f20b8103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.290+0000 7f20bd92b640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20b8101fc0 0x7f20b8103430 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.290+0000 7f20bd92b640 1 --2- 172.21.15.94:0/1246559718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20b8100e70 0x7f20b8104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.290+0000 7f20bd92b640 1 -- 172.21.15.94:0/1246559718 >> 172.21.15.94:0/1246559718 conn(0x7f20b80fc3e0 msgr2=0x7f20b8069190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.291+0000 7f20bd92b640 1 -- 172.21.15.94:0/1246559718 shutdown_connections 2024-04-21T08:39:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.291+0000 7f20bd92b640 1 -- 172.21.15.94:0/1246559718 wait complete. 2024-04-21T08:39:23.302 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:39:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.401+0000 7f0773e77640 1 -- 172.21.15.94:0/3801028272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f076c1053c0 msgr2=0x7f076c1077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.401+0000 7f0773e77640 1 --2- 172.21.15.94:0/3801028272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f076c1053c0 0x7f076c1077b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f075c009900 tx=0x7f075c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.401+0000 7f0773e77640 1 -- 172.21.15.94:0/3801028272 shutdown_connections 2024-04-21T08:39:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.401+0000 7f0773e77640 1 --2- 172.21.15.94:0/3801028272 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f076c107cf0 0x7f076c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.401+0000 7f0773e77640 1 --2- 172.21.15.94:0/3801028272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f076c1053c0 0x7f076c1077b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.401+0000 7f0773e77640 1 --2- 172.21.15.94:0/3801028272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f076c102a90 0x7f076c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.401+0000 7f0773e77640 1 -- 172.21.15.94:0/3801028272 >> 172.21.15.94:0/3801028272 conn(0x7f076c0fc3e0 msgr2=0x7f076c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0773e77640 1 -- 172.21.15.94:0/3801028272 shutdown_connections 2024-04-21T08:39:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0773e77640 1 -- 172.21.15.94:0/3801028272 wait complete. 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0773e77640 1 Processor -- start 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0773e77640 1 -- start start 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0773e77640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f076c102a90 0x7f076c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0773e77640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f076c1053c0 0x7f076c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0773e77640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f076c107cf0 0x7f076c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0773e77640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f076c10daa0 con 0x7f076c1053c0 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0773e77640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f076c10d920 con 0x7f076c102a90 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0773e77640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f076c10dc20 con 0x7f076c107cf0 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0771412640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f076c1053c0 0x7f076c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0771412640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f076c1053c0 0x7f076c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46736/0 (socket says 172.21.15.94:46736) 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0771412640 1 -- 172.21.15.94:0/4016169009 learned_addr learned my addr 172.21.15.94:0/4016169009 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.402+0000 7f0772414640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f076c107cf0 0x7f076c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0771c13640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f076c102a90 0x7f076c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0771412640 1 -- 172.21.15.94:0/4016169009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f076c107cf0 msgr2=0x7f076c19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0771412640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f076c107cf0 0x7f076c19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0771412640 1 -- 172.21.15.94:0/4016169009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f076c102a90 msgr2=0x7f076c198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0771412640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f076c102a90 0x7f076c198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0771412640 1 -- 172.21.15.94:0/4016169009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076c19fe30 con 0x7f076c1053c0 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0771c13640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f076c102a90 0x7f076c198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0772414640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f076c107cf0 0x7f076c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0771412640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f076c1053c0 0x7f076c198750 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f075c000c00 tx=0x7f075c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0762ffd640 1 -- 172.21.15.94:0/4016169009 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075c043020 con 0x7f076c1053c0 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0773e77640 1 -- 172.21.15.94:0/4016169009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076c1a00c0 con 0x7f076c1053c0 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0762ffd640 1 -- 172.21.15.94:0/4016169009 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f075c004aa0 con 0x7f076c1053c0 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0762ffd640 1 -- 172.21.15.94:0/4016169009 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075c003d00 con 0x7f076c1053c0 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.403+0000 7f0773e77640 1 -- 172.21.15.94:0/4016169009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f076c1a0610 con 0x7f076c1053c0 2024-04-21T08:39:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.404+0000 7f0773e77640 1 -- 172.21.15.94:0/4016169009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f076c0fded0 con 0x7f076c1053c0 2024-04-21T08:39:23.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.405+0000 7f0762ffd640 1 -- 172.21.15.94:0/4016169009 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f075c046020 con 0x7f076c1053c0 2024-04-21T08:39:23.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.405+0000 7f0762ffd640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f07580a3bb0 0x7f07580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.406+0000 7f0771c13640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f07580a3bb0 0x7f07580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.406+0000 7f0762ffd640 1 -- 172.21.15.94:0/4016169009 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f075c0f3220 con 0x7f076c1053c0 2024-04-21T08:39:23.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.406+0000 7f0771c13640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f07580a3bb0 0x7f07580a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0768005fd0 tx=0x7f07680093e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:23.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.409+0000 7f0762ffd640 1 -- 172.21.15.94:0/4016169009 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f075c04d020 con 0x7f076c1053c0 2024-04-21T08:39:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.592+0000 7f0773e77640 1 -- 172.21.15.94:0/4016169009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f076c02d810 con 0x7f076c1053c0 2024-04-21T08:39:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.594+0000 7f0762ffd640 1 -- 172.21.15.94:0/4016169009 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24563 (secure 0 0 0) 0x7f075c0bf8d0 con 0x7f076c1053c0 2024-04-21T08:39:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:23.594 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":91,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:38:30.663522+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:39:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 -- 172.21.15.94:0/4016169009 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f07580a3bb0 msgr2=0x7f07580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f07580a3bb0 0x7f07580a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0768005fd0 tx=0x7f07680093e0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 -- 172.21.15.94:0/4016169009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f076c1053c0 msgr2=0x7f076c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f076c1053c0 0x7f076c198750 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f075c000c00 tx=0x7f075c004270 comp rx=0 tx=0).stop 2024-04-21T08:39:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 -- 172.21.15.94:0/4016169009 shutdown_connections 2024-04-21T08:39:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f07580a3bb0 0x7f07580a6070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f076c107cf0 0x7f076c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f076c1053c0 0x7f076c198750 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 --2- 172.21.15.94:0/4016169009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f076c102a90 0x7f076c198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 -- 172.21.15.94:0/4016169009 >> 172.21.15.94:0/4016169009 conn(0x7f076c0fc3e0 msgr2=0x7f076c103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 -- 172.21.15.94:0/4016169009 shutdown_connections 2024-04-21T08:39:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.596+0000 7f0773e77640 1 -- 172.21.15.94:0/4016169009 wait complete. 2024-04-21T08:39:23.608 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:39:23.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.705+0000 7f384e4a8640 1 -- 172.21.15.94:0/3038728733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3848100e70 msgr2=0x7f3848101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.705+0000 7f384e4a8640 1 --2- 172.21.15.94:0/3038728733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3848100e70 0x7f3848101290 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f383c009900 tx=0x7f383c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:23.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.705+0000 7f384e4a8640 1 -- 172.21.15.94:0/3038728733 shutdown_connections 2024-04-21T08:39:23.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.705+0000 7f384e4a8640 1 --2- 172.21.15.94:0/3038728733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3848101fc0 0x7f3848102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.705+0000 7f384e4a8640 1 --2- 172.21.15.94:0/3038728733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3848102980 0x7f38481092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.705+0000 7f384e4a8640 1 --2- 172.21.15.94:0/3038728733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3848100e70 0x7f3848101290 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.705+0000 7f384e4a8640 1 -- 172.21.15.94:0/3038728733 >> 172.21.15.94:0/3038728733 conn(0x7f38480fc3e0 msgr2=0x7f38480fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:23.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.705+0000 7f384e4a8640 1 -- 172.21.15.94:0/3038728733 shutdown_connections 2024-04-21T08:39:23.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.705+0000 7f384e4a8640 1 -- 172.21.15.94:0/3038728733 wait complete. 2024-04-21T08:39:23.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f384e4a8640 1 Processor -- start 2024-04-21T08:39:23.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f384e4a8640 1 -- start start 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f384e4a8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3848100e70 0x7f384819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f384e4a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3848101fc0 0x7f384819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f384e4a8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3848102980 0x7f38481a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f384e4a8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f384810c9c0 con 0x7f3848100e70 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f384e4a8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f384810c840 con 0x7f3848102980 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f384e4a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f384810cb40 con 0x7f3848101fc0 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f3847fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3848100e70 0x7f384819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f3847fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3848100e70 0x7f384819a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58980/0 (socket says 172.21.15.94:58980) 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.706+0000 7f3847fff640 1 -- 172.21.15.94:0/680596675 learned_addr learned my addr 172.21.15.94:0/680596675 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f38477fe640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3848101fc0 0x7f384819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f384ca45640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3848102980 0x7f38481a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f3847fff640 1 -- 172.21.15.94:0/680596675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3848101fc0 msgr2=0x7f384819a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f3847fff640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3848101fc0 0x7f384819a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f3847fff640 1 -- 172.21.15.94:0/680596675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3848102980 msgr2=0x7f38481a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f3847fff640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3848102980 0x7f38481a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f3847fff640 1 -- 172.21.15.94:0/680596675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38481a1fa0 con 0x7f3848100e70 2024-04-21T08:39:23.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f3847fff640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3848100e70 0x7f384819a380 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f383c006010 tx=0x7f383c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:23.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f38457fa640 1 -- 172.21.15.94:0/680596675 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c043020 con 0x7f3848100e70 2024-04-21T08:39:23.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f38457fa640 1 -- 172.21.15.94:0/680596675 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f383c004aa0 con 0x7f3848100e70 2024-04-21T08:39:23.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f38457fa640 1 -- 172.21.15.94:0/680596675 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c003c70 con 0x7f3848100e70 2024-04-21T08:39:23.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f384e4a8640 1 -- 172.21.15.94:0/680596675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38481a2230 con 0x7f3848100e70 2024-04-21T08:39:23.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.707+0000 7f384e4a8640 1 -- 172.21.15.94:0/680596675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38481a2780 con 0x7f3848100e70 2024-04-21T08:39:23.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.708+0000 7f384e4a8640 1 -- 172.21.15.94:0/680596675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38481a2aa0 con 0x7f3848100e70 2024-04-21T08:39:23.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.709+0000 7f38457fa640 1 -- 172.21.15.94:0/680596675 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f383c046020 con 0x7f3848100e70 2024-04-21T08:39:23.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.710+0000 7f38457fa640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38300a3bb0 0x7f38300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:23.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.710+0000 7f38477fe640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38300a3bb0 0x7f38300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:23.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.710+0000 7f38457fa640 1 -- 172.21.15.94:0/680596675 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f383c0f2ac0 con 0x7f3848100e70 2024-04-21T08:39:23.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.710+0000 7f38477fe640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38300a3bb0 0x7f38300a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f384819b890 tx=0x7f383800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.714+0000 7f38457fa640 1 -- 172.21.15.94:0/680596675 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f383c04d020 con 0x7f3848100e70 2024-04-21T08:39:23.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.929+0000 7f384e4a8640 1 -- 172.21.15.94:0/680596675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38481a2c70 con 0x7f3848100e70 2024-04-21T08:39:23.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.930+0000 7f38457fa640 1 -- 172.21.15.94:0/680596675 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11885 (secure 0 0 0) 0x7f383c0bf170 con 0x7f3848100e70 2024-04-21T08:39:23.929 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:23.930 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:37:56.995828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24446,24488]},"id":1}]} 2024-04-21T08:39:23.930 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-04-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 -- 172.21.15.94:0/680596675 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38300a3bb0 msgr2=0x7f38300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38300a3bb0 0x7f38300a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f384819b890 tx=0x7f383800b390 comp rx=0 tx=0).stop 2024-04-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 -- 172.21.15.94:0/680596675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3848100e70 msgr2=0x7f384819a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3848100e70 0x7f384819a380 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f383c006010 tx=0x7f383c0042f0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 -- 172.21.15.94:0/680596675 shutdown_connections 2024-04-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f38300a3bb0 0x7f38300a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3848101fc0 0x7f384819a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3848102980 0x7f38481a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 --2- 172.21.15.94:0/680596675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3848100e70 0x7f384819a380 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 -- 172.21.15.94:0/680596675 >> 172.21.15.94:0/680596675 conn(0x7f38480fc3e0 msgr2=0x7f38480fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:23.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 -- 172.21.15.94:0/680596675 shutdown_connections 2024-04-21T08:39:23.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:23.932+0000 7f384e4a8640 1 -- 172.21.15.94:0/680596675 wait complete. 2024-04-21T08:39:23.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:23 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1847230468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:23.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:23 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1246559718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:23.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:23 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4016169009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:23.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:23 smithi094 ceph-mon[27988]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:23.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:37:30.753966+0000', 'modified': '2024-04-21T08:37:56.995828+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24488, 'mds_1': 24454, 'mds_2': 24446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24416': {'gid': 24416, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6835/3967081988', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 3967081988}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 3967081988}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24421': {'gid': 24421, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/3922099450', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 3922099450}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 3922099450}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24433': {'gid': 24433, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6837/1466520447', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 1466520447}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 1466520447}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.111:6837/550365815', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6836', 'nonce': 550365815}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 550365815}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.173:6839/2987518473', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 2987518473}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 2987518473}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3878872423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3878872423}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3878872423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24488, 'qdb_cluster': [24454, 24446, 24488]} 2024-04-21T08:39:23.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-21T08:39:23.943 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-21T08:39:24.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:23 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1847230468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:24.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:23 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1246559718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:24.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:23 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4016169009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:24.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:23 smithi173 ceph-mon[31428]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:24.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.042+0000 7f52fe2d2640 1 -- 172.21.15.94:0/246993743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52f8107c70 msgr2=0x7f52f810a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.042+0000 7f52fe2d2640 1 --2- 172.21.15.94:0/246993743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52f8107c70 0x7f52f810a0c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f52f0009ca0 tx=0x7f52f002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.042+0000 7f52fe2d2640 1 -- 172.21.15.94:0/246993743 shutdown_connections 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.042+0000 7f52fe2d2640 1 --2- 172.21.15.94:0/246993743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f8105340 0x7f52f8107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.042+0000 7f52fe2d2640 1 --2- 172.21.15.94:0/246993743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52f8107c70 0x7f52f810a0c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.042+0000 7f52fe2d2640 1 --2- 172.21.15.94:0/246993743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a10 0x7f52f8104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.042+0000 7f52fe2d2640 1 -- 172.21.15.94:0/246993743 >> 172.21.15.94:0/246993743 conn(0x7f52f80fc440 msgr2=0x7f52f80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.042+0000 7f52fe2d2640 1 -- 172.21.15.94:0/246993743 shutdown_connections 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.042+0000 7f52fe2d2640 1 -- 172.21.15.94:0/246993743 wait complete. 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52fe2d2640 1 Processor -- start 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52fe2d2640 1 -- start start 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52fe2d2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a10 0x7f52f8198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52fe2d2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52f8105340 0x7f52f8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52fe2d2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f8107c70 0x7f52f819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52fe2d2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f52f810db30 con 0x7f52f8105340 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52fe2d2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52f810d9b0 con 0x7f52f8102a10 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52fe2d2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f52f810dcb0 con 0x7f52f8107c70 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52f8105340 0x7f52f8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52f8105340 0x7f52f8198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58988/0 (socket says 172.21.15.94:58988) 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52f77fe640 1 -- 172.21.15.94:0/8733626 learned_addr learned my addr 172.21.15.94:0/8733626 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52f7fff640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a10 0x7f52f8198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52f77fe640 1 -- 172.21.15.94:0/8733626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f8107c70 msgr2=0x7f52f819f7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52f77fe640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f8107c70 0x7f52f819f7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52f77fe640 1 -- 172.21.15.94:0/8733626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a10 msgr2=0x7f52f8198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52f77fe640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a10 0x7f52f8198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.043+0000 7f52f77fe640 1 -- 172.21.15.94:0/8733626 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f819fe20 con 0x7f52f8105340 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.044+0000 7f52f77fe640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52f8105340 0x7f52f8198740 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00b450 tx=0x7f52ec00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.044+0000 7f52f57fa640 1 -- 172.21.15.94:0/8733626 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ec015020 con 0x7f52f8105340 2024-04-21T08:39:24.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.044+0000 7f52fe2d2640 1 -- 172.21.15.94:0/8733626 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f81a0110 con 0x7f52f8105340 2024-04-21T08:39:24.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.044+0000 7f52f57fa640 1 -- 172.21.15.94:0/8733626 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52ec011d40 con 0x7f52f8105340 2024-04-21T08:39:24.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.044+0000 7f52f57fa640 1 -- 172.21.15.94:0/8733626 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ec00f340 con 0x7f52f8105340 2024-04-21T08:39:24.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.044+0000 7f52fe2d2640 1 -- 172.21.15.94:0/8733626 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f81a0610 con 0x7f52f8105340 2024-04-21T08:39:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.046+0000 7f52f57fa640 1 -- 172.21.15.94:0/8733626 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f52ec018020 con 0x7f52f8105340 2024-04-21T08:39:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.046+0000 7f5300f9e740 1 Processor -- start 2024-04-21T08:39:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.046+0000 7f5300f9e740 1 -- start start 2024-04-21T08:39:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.046+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d26e40 0x5588f2d27260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.046+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d27cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.046+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d2eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.046+0000 7f5300f9e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5588f2c8bb10 con 0x5588f2d282d0 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.046+0000 7f5300f9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5588f2c29240 con 0x5588f2d26e40 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.046+0000 7f5300f9e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588f2c87ad0 con 0x5588f2d27830 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.046+0000 7f52f57fa640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52e00a3bb0 0x7f52e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d2eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52fc86f640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52e00a3bb0 0x7f52e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d2eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58990/0 (socket says 172.21.15.94:58990) 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f77fe640 1 -- 172.21.15.94:0/1311414414 learned_addr learned my addr 172.21.15.94:0/1311414414 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f7fff640 1 --2- 172.21.15.94:0/1311414414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d27cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f57fa640 1 -- 172.21.15.94:0/8733626 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f52ec0d0230 con 0x7f52f8105340 2024-04-21T08:39:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52fc86f640 1 --2- 172.21.15.94:0/1311414414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d26e40 0x5588f2d27260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f77fe640 1 -- 172.21.15.94:0/1311414414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 msgr2=0x5588f2d27cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f77fe640 1 --2- 172.21.15.94:0/1311414414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d27cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f77fe640 1 -- 172.21.15.94:0/1311414414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d26e40 msgr2=0x5588f2d27260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f77fe640 1 --2- 172.21.15.94:0/1311414414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d26e40 0x5588f2d27260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f77fe640 1 -- 172.21.15.94:0/1311414414 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f299e2d0 con 0x5588f2d282d0 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52fc86f640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52e00a3bb0 0x7f52e00a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f52f0004af0 tx=0x7f52f0005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52f77fe640 1 --2- 172.21.15.94:0/1311414414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d2eeb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f52ec09c170 tx=0x7f52ec0c72f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277333805aa80ec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52de7fc640 1 -- 172.21.15.94:0/1311414414 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ec09a7a0 con 0x5588f2d282d0 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52de7fc640 1 -- 172.21.15.94:0/1311414414 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52ec066d40 con 0x5588f2d282d0 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.047+0000 7f52de7fc640 1 -- 172.21.15.94:0/1311414414 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ec009280 con 0x5588f2d282d0 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 -- 172.21.15.94:0/1311414414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 msgr2=0x5588f2d2eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 --2- 172.21.15.94:0/1311414414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d2eeb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f52ec09c170 tx=0x7f52ec0c72f0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 -- 172.21.15.94:0/1311414414 shutdown_connections 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 --2- 172.21.15.94:0/1311414414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d27cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 --2- 172.21.15.94:0/1311414414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d2eeb0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 --2- 172.21.15.94:0/1311414414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d26e40 0x5588f2d27260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 -- 172.21.15.94:0/1311414414 >> 172.21.15.94:0/1311414414 conn(0x5588f2d22f40 msgr2=0x5588f2d21490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 -- 172.21.15.94:0/1311414414 shutdown_connections 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 -- 172.21.15.94:0/1311414414 wait complete. 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 -- start start 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.048+0000 7f5300f9e740 1 Processor -- start 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d26e40 0x5588f2d30e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d27830 0x5588f2d313a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 0x5588f2d383e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f5300f9e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5588f2d2f3f0 con 0x5588f2d27830 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f5300f9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5588f2c14190 con 0x5588f2d282d0 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f5300f9e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588f2d2f570 con 0x5588f2d26e40 2024-04-21T08:39:24.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f5300f9e740 10 client.0 _ll_register_callbacks cb 0x5588f2c0f8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d27830 0x5588f2d313a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d27830 0x5588f2d313a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58994/0 (socket says 172.21.15.94:58994) 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52f77fe640 1 -- 172.21.15.94:0/2766063660 learned_addr learned my addr 172.21.15.94:0/2766063660 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52fc86f640 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 0x5588f2d383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52f7fff640 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d26e40 0x5588f2d30e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52f77fe640 1 -- 172.21.15.94:0/2766063660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d26e40 msgr2=0x5588f2d30e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52f77fe640 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d26e40 0x5588f2d30e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52f77fe640 1 -- 172.21.15.94:0/2766063660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 msgr2=0x5588f2d383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52f77fe640 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 0x5588f2d383e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52f77fe640 1 -- 172.21.15.94:0/2766063660 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f2a686b0 con 0x5588f2d27830 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52fc86f640 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 0x5588f2d383e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52f77fe640 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d27830 0x5588f2d313a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0d05e0 tx=0x7f52ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458566de5c34a414 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:24.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52d77fe640 1 -- 172.21.15.94:0/2766063660 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ec0b5070 con 0x5588f2d27830 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52d77fe640 1 -- 172.21.15.94:0/2766063660 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f52ec09ad10 con 0x5588f2d27830 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f5300f9e740 1 -- 172.21.15.94:0/2766063660 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52ec066d40 con 0x5588f2d27830 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f52d77fe640 1 -- 172.21.15.94:0/2766063660 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f52ec009280 con 0x5588f2d27830 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f5300f9e740 10 client.15213 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.049+0000 7f5300f9e740 1 -- 172.21.15.94:0/2766063660 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5588f2d3b750 con 0x5588f2d27830 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.050+0000 7f52d77fe640 1 -- 172.21.15.94:0/2766063660 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52ec066d40 con 0x5588f2d27830 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.050+0000 7f52d77fe640 1 -- 172.21.15.94:0/2766063660 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f52ec0a0170 con 0x5588f2d27830 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.050+0000 7f5300f9e740 10 client.15213 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.050+0000 7f5300f9e740 10 client.15213 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.050+0000 7f5300f9e740 20 client.15213 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.050+0000 7f5300f9e740 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5588f2d3d8a0 0x5588f2d3ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.050+0000 7f5300f9e740 4 client.15213 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:39:24.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.050+0000 7f5300f9e740 1 -- 172.21.15.94:0/2766063660 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588f2c99530 con 0x5588f2d3d8a0 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.050+0000 7f52f7fff640 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5588f2d3d8a0 0x5588f2d3ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.051+0000 7f52d77fe640 10 client.15213 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.051+0000 7f52f7fff640 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5588f2d3d8a0 0x5588f2d3ed10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ecec901b0a233df7 server_cookie=cc75b76cd6e72eb9 in_seq=0 out_seq=0 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.051+0000 7f52d77fe640 1 -- 172.21.15.94:0/2766063660 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588f2c99530 con 0x5588f2d3d8a0 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.051+0000 7f52d77fe640 10 client.15213 handle_command_reply: tid=0 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 client.15213 shutdown 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 -- 172.21.15.94:0/2766063660 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d27830 msgr2=0x5588f2d313a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d27830 0x5588f2d313a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0d05e0 tx=0x7f52ec0046c0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 -- 172.21.15.94:0/2766063660 shutdown_connections 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5588f2d3d8a0 0x5588f2d3ed10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 0x5588f2d383e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d27830 0x5588f2d313a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f52d77fe640 0 client.15213 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 --2- 172.21.15.94:0/2766063660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d26e40 0x5588f2d30e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 -- 172.21.15.94:0/2766063660 >> 172.21.15.94:0/2766063660 conn(0x5588f2d22f40 msgr2=0x5588f2d20a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 -- 172.21.15.94:0/2766063660 shutdown_connections 2024-04-21T08:39:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 1 -- 172.21.15.94:0/2766063660 wait complete. 2024-04-21T08:39:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.052+0000 7f5300f9e740 20 client.15213 trim_cache size 0 max 16384 2024-04-21T08:39:24.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f5300f9e740 1 Processor -- start 2024-04-21T08:39:24.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f5300f9e740 1 -- start start 2024-04-21T08:39:24.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d26e40 0x5588f2d2eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d383a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 0x5588f2d2b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f5300f9e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5588f2d3d050 con 0x5588f2d26e40 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f5300f9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5588f2d3d720 con 0x5588f2d282d0 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f5300f9e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588f2d3ddf0 con 0x5588f2d27830 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52f7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d26e40 0x5588f2d2eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52f7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d26e40 0x5588f2d2eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59010/0 (socket says 172.21.15.94:59010) 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52f7fff640 1 -- 172.21.15.94:0/3831012293 learned_addr learned my addr 172.21.15.94:0/3831012293 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52f77fe640 1 --2- 172.21.15.94:0/3831012293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52fc86f640 1 --2- 172.21.15.94:0/3831012293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 0x5588f2d2b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52f7fff640 1 -- 172.21.15.94:0/3831012293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 msgr2=0x5588f2d383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52f7fff640 1 --2- 172.21.15.94:0/3831012293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d383a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52f7fff640 1 -- 172.21.15.94:0/3831012293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 msgr2=0x5588f2d2b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52f7fff640 1 --2- 172.21.15.94:0/3831012293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 0x5588f2d2b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52f7fff640 1 -- 172.21.15.94:0/3831012293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f2d22630 con 0x5588f2d26e40 2024-04-21T08:39:24.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.070+0000 7f52f7fff640 1 --2- 172.21.15.94:0/3831012293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d26e40 0x5588f2d2eef0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x5588f2d3fb20 tx=0x7f52e800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2fb5f243e82dd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f52ddffb640 1 -- 172.21.15.94:0/3831012293 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8003be0 con 0x5588f2d26e40 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f52ddffb640 1 -- 172.21.15.94:0/3831012293 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e8023020 con 0x5588f2d26e40 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f52ddffb640 1 -- 172.21.15.94:0/3831012293 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e80116a0 con 0x5588f2d26e40 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f5300f9e740 1 -- 172.21.15.94:0/3831012293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d26e40 msgr2=0x5588f2d2eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f5300f9e740 1 --2- 172.21.15.94:0/3831012293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d26e40 0x5588f2d2eef0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x5588f2d3fb20 tx=0x7f52e800b390 comp rx=0 tx=0).stop 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f5300f9e740 1 -- 172.21.15.94:0/3831012293 shutdown_connections 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f5300f9e740 1 --2- 172.21.15.94:0/3831012293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d383a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f5300f9e740 1 --2- 172.21.15.94:0/3831012293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d282d0 0x5588f2d2b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f5300f9e740 1 --2- 172.21.15.94:0/3831012293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d26e40 0x5588f2d2eef0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f5300f9e740 1 -- 172.21.15.94:0/3831012293 >> 172.21.15.94:0/3831012293 conn(0x5588f2d22f40 msgr2=0x5588f2d1c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f5300f9e740 1 -- 172.21.15.94:0/3831012293 shutdown_connections 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.071+0000 7f5300f9e740 1 -- 172.21.15.94:0/3831012293 wait complete. 2024-04-21T08:39:24.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f5300f9e740 1 -- start start 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f5300f9e740 1 Processor -- start 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d26e40 0x5588f2d34380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f5300f9e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d36e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f5300f9e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5588f2d3eb90 con 0x5588f2d282d0 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f5300f9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5588f2d3e4c0 con 0x5588f2d26e40 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f5300f9e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588f2d3f260 con 0x5588f2d27830 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f5300f9e740 10 client.0 _ll_register_callbacks cb 0x7f52f0037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f52fc86f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d36e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f52fc86f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d36e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59016/0 (socket says 172.21.15.94:59016) 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f52fc86f640 1 -- 172.21.15.94:0/4119161768 learned_addr learned my addr 172.21.15.94:0/4119161768 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f52f7fff640 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d26e40 0x5588f2d34380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f52f77fe640 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f52fc86f640 1 -- 172.21.15.94:0/4119161768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 msgr2=0x5588f2d368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f52fc86f640 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f52fc86f640 1 -- 172.21.15.94:0/4119161768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d26e40 msgr2=0x5588f2d34380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f52fc86f640 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d26e40 0x5588f2d34380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.072+0000 7f52fc86f640 1 -- 172.21.15.94:0/4119161768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f2d37640 con 0x5588f2d282d0 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f52fc86f640 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d36e10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f52f0009ca0 tx=0x7f52f0034e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a8afc296b6ade2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f52d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f52d77fe640 1 -- 172.21.15.94:0/4119161768 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f0039050 con 0x5588f2d282d0 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f52d77fe640 1 -- 172.21.15.94:0/4119161768 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f52f0004040 con 0x5588f2d282d0 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f5300f9e740 1 -- 172.21.15.94:0/4119161768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52e8023020 con 0x5588f2d282d0 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f52d77fe640 1 -- 172.21.15.94:0/4119161768 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f52f005b050 con 0x5588f2d282d0 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f5300f9e740 10 client.15219 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f5300f9e740 1 -- 172.21.15.94:0/4119161768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5588f2da38c0 con 0x5588f2d282d0 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f52d77fe640 1 -- 172.21.15.94:0/4119161768 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e8023020 con 0x5588f2d282d0 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f52d77fe640 1 -- 172.21.15.94:0/4119161768 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f52f0034890 con 0x5588f2d282d0 2024-04-21T08:39:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f5300f9e740 10 client.15219 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f5300f9e740 10 client.15219 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f5300f9e740 20 client.15219 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f5300f9e740 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5588f2da8560 0x5588f2da7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f5300f9e740 4 client.15219 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f5300f9e740 1 -- 172.21.15.94:0/4119161768 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "status"}) -- 0x5588f2c933a0 con 0x5588f2da8560 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.073+0000 7f52f7fff640 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5588f2da8560 0x5588f2da7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f52f7fff640 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5588f2da8560 0x5588f2da7270 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=482167f171d9f0e5 server_cookie=d9ec6987785a47b1 in_seq=0 out_seq=0 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f52d77fe640 10 client.15219 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f52d77fe640 1 -- 172.21.15.94:0/4119161768 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5588f2c933a0 con 0x5588f2da8560 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f52d77fe640 10 client.15219 handle_command_reply: tid=0 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 client.15219 shutdown 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 -- 172.21.15.94:0/4119161768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 msgr2=0x5588f2d36e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d36e10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f52f0009ca0 tx=0x7f52f0034e70 comp rx=0 tx=0).stop 2024-04-21T08:39:24.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 -- 172.21.15.94:0/4119161768 shutdown_connections 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5588f2da8560 0x5588f2da7270 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588f2d27830 0x5588f2d368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f52d77fe640 0 client.15219 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5588f2d282d0 0x5588f2d36e10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 --2- 172.21.15.94:0/4119161768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5588f2d26e40 0x5588f2d34380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 -- 172.21.15.94:0/4119161768 >> 172.21.15.94:0/4119161768 conn(0x5588f2d22f40 msgr2=0x5588f2d2eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 -- 172.21.15.94:0/4119161768 shutdown_connections 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 1 -- 172.21.15.94:0/4119161768 wait complete. 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.074+0000 7f5300f9e740 20 client.15219 trim_cache size 0 max 16384 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:39:24.081 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 2, 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24446, 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 87.418419279000005, 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 23, 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 91, 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 129.115926573 2024-04-21T08:39:24.082 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:39:24.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 -- 172.21.15.94:0/8733626 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52e00a3bb0 msgr2=0x7f52e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52e00a3bb0 0x7f52e00a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f52f0004af0 tx=0x7f52f0005ef0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 -- 172.21.15.94:0/8733626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52f8105340 msgr2=0x7f52f8198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52f8105340 0x7f52f8198740 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00b450 tx=0x7f52ec00b910 comp rx=0 tx=0).stop 2024-04-21T08:39:24.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 -- 172.21.15.94:0/8733626 shutdown_connections 2024-04-21T08:39:24.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f52e00a3bb0 0x7f52e00a6070 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52f8107c70 0x7f52f819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52f8105340 0x7f52f8198740 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 --2- 172.21.15.94:0/8733626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a10 0x7f52f8198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 -- 172.21.15.94:0/8733626 >> 172.21.15.94:0/8733626 conn(0x7f52f80fc440 msgr2=0x7f52f8103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 -- 172.21.15.94:0/8733626 shutdown_connections 2024-04-21T08:39:24.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.075+0000 7f52fe2d2640 1 -- 172.21.15.94:0/8733626 wait complete. 2024-04-21T08:39:24.086 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-21T08:39:24.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.185+0000 7fa8bb493640 1 -- 172.21.15.94:0/859434002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8b41029a0 msgr2=0x7fa8b41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.185+0000 7fa8bb493640 1 --2- 172.21.15.94:0/859434002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8b41029a0 0x7fa8b41092c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009900 tx=0x7fa8ac02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.185+0000 7fa8bb493640 1 -- 172.21.15.94:0/859434002 shutdown_connections 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.185+0000 7fa8bb493640 1 --2- 172.21.15.94:0/859434002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8b41029a0 0x7fa8b41092c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.185+0000 7fa8bb493640 1 --2- 172.21.15.94:0/859434002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b4101fe0 0x7fa8b4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.185+0000 7fa8bb493640 1 --2- 172.21.15.94:0/859434002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b4100e90 0x7fa8b41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.185+0000 7fa8bb493640 1 -- 172.21.15.94:0/859434002 >> 172.21.15.94:0/859434002 conn(0x7fa8b40fc400 msgr2=0x7fa8b40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.185+0000 7fa8bb493640 1 -- 172.21.15.94:0/859434002 shutdown_connections 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.185+0000 7fa8bb493640 1 -- 172.21.15.94:0/859434002 wait complete. 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8bb493640 1 Processor -- start 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8bb493640 1 -- start start 2024-04-21T08:39:24.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8bb493640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b4100e90 0x7fa8b4069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8bb493640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8b4101fe0 0x7fa8b419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8bb493640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b41029a0 0x7fa8b419f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8bb493640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b410c9f0 con 0x7fa8b4101fe0 2024-04-21T08:39:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8bb493640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b410c870 con 0x7fa8b41029a0 2024-04-21T08:39:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8bb493640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b410cb70 con 0x7fa8b4100e90 2024-04-21T08:39:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8b8a2e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8b4101fe0 0x7fa8b419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8b8a2e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8b4101fe0 0x7fa8b419ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59042/0 (socket says 172.21.15.94:59042) 2024-04-21T08:39:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8b8a2e640 1 -- 172.21.15.94:0/351559499 learned_addr learned my addr 172.21.15.94:0/351559499 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b41029a0 0x7fa8b419f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.186+0000 7fa8b922f640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b4100e90 0x7fa8b4069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8b8a2e640 1 -- 172.21.15.94:0/351559499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b4100e90 msgr2=0x7fa8b4069cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8b8a2e640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b4100e90 0x7fa8b4069cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8b8a2e640 1 -- 172.21.15.94:0/351559499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b41029a0 msgr2=0x7fa8b419f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8b8a2e640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b41029a0 0x7fa8b419f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8b8a2e640 1 -- 172.21.15.94:0/351559499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b419fde0 con 0x7fa8b4101fe0 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b41029a0 0x7fa8b419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8b8a2e640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8b4101fe0 0x7fa8b419ceb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400d770 tx=0x7fa8a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8aa7fc640 1 -- 172.21.15.94:0/351559499 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4017070 con 0x7fa8b4101fe0 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8bb493640 1 -- 172.21.15.94:0/351559499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b41a00d0 con 0x7fa8b4101fe0 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8aa7fc640 1 -- 172.21.15.94:0/351559499 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8a4009c30 con 0x7fa8b4101fe0 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8aa7fc640 1 -- 172.21.15.94:0/351559499 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4003cc0 con 0x7fa8b4101fe0 2024-04-21T08:39:24.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.187+0000 7fa8bb493640 1 -- 172.21.15.94:0/351559499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b41a0670 con 0x7fa8b4101fe0 2024-04-21T08:39:24.191 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:23 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1847230468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:24.191 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:23 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1246559718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:24.191 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:23 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4016169009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:24.191 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:23 smithi111 ceph-mon[31493]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:24.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.189+0000 7fa8aa7fc640 1 -- 172.21.15.94:0/351559499 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa8a4003e60 con 0x7fa8b4101fe0 2024-04-21T08:39:24.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.189+0000 7fa8be165740 1 Processor -- start 2024-04-21T08:39:24.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.189+0000 7fa8be165740 1 -- start start 2024-04-21T08:39:24.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8be165740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c70156260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8be165740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8be165740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c701572d0 0x562c7015deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8be165740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x562c700bab10 con 0x562c70156830 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8be165740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562c70058240 con 0x562c70155e40 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8be165740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562c700b6ad0 con 0x562c701572d0 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8aa7fc640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8a00a3bb0 0x7fa8a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b922f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b922f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59054/0 (socket says 172.21.15.94:59054) 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b922f640 1 -- 172.21.15.94:0/1678249742 learned_addr learned my addr 172.21.15.94:0/1678249742 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8a00a3bb0 0x7fa8a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/1678249742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c70156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8aa7fc640 1 -- 172.21.15.94:0/351559499 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7fa8a40d0170 con 0x7fa8b4101fe0 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b8a2e640 1 --2- 172.21.15.94:0/1678249742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c701572d0 0x562c7015deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b922f640 1 -- 172.21.15.94:0/1678249742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c701572d0 msgr2=0x562c7015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b922f640 1 --2- 172.21.15.94:0/1678249742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c701572d0 0x562c7015deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b922f640 1 -- 172.21.15.94:0/1678249742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 msgr2=0x562c70156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b922f640 1 --2- 172.21.15.94:0/1678249742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c70156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b922f640 1 -- 172.21.15.94:0/1678249742 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c6fdcd2d0 con 0x562c70156830 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b922f640 1 --2- 172.21.15.94:0/1678249742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70156cb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0010480 tx=0x7fa8b0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb12f3cce63fc10d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8a00a3bb0 0x7fa8a00a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac005c40 tx=0x7fa8ac02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa89f7fe640 1 -- 172.21.15.94:0/1678249742 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0003b70 con 0x562c70156830 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa89f7fe640 1 -- 172.21.15.94:0/1678249742 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8b0003d10 con 0x562c70156830 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.190+0000 7fa89f7fe640 1 -- 172.21.15.94:0/1678249742 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0011aa0 con 0x562c70156830 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.191+0000 7fa8be165740 1 -- 172.21.15.94:0/1678249742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 msgr2=0x562c70156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.191+0000 7fa8be165740 1 --2- 172.21.15.94:0/1678249742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70156cb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0010480 tx=0x7fa8b0009f90 comp rx=0 tx=0).stop 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.191+0000 7fa8be165740 1 -- 172.21.15.94:0/1678249742 shutdown_connections 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.191+0000 7fa8be165740 1 --2- 172.21.15.94:0/1678249742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c701572d0 0x562c7015deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.191+0000 7fa8be165740 1 --2- 172.21.15.94:0/1678249742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70156cb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.191+0000 7fa8be165740 1 --2- 172.21.15.94:0/1678249742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c70156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.191+0000 7fa8be165740 1 -- 172.21.15.94:0/1678249742 >> 172.21.15.94:0/1678249742 conn(0x562c70151f40 msgr2=0x562c70150490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.191+0000 7fa8be165740 1 -- 172.21.15.94:0/1678249742 shutdown_connections 2024-04-21T08:39:24.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.191+0000 7fa8be165740 1 -- 172.21.15.94:0/1678249742 wait complete. 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.191+0000 7fa8be165740 1 -- start start 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8be165740 1 Processor -- start 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8be165740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c7015fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8be165740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8be165740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c701572d0 0x562c70167460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8be165740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x562c7015e4b0 con 0x562c70156830 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8be165740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562c70043190 con 0x562c70155e40 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8be165740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562c7015e630 con 0x562c701572d0 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8be165740 10 client.0 _ll_register_callbacks cb 0x562c7003e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b8a2e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b8a2e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59066/0 (socket says 172.21.15.94:59066) 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b8a2e640 1 -- 172.21.15.94:0/3636895758 learned_addr learned my addr 172.21.15.94:0/3636895758 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b922f640 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c7015fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c701572d0 0x562c70167460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b8a2e640 1 -- 172.21.15.94:0/3636895758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c701572d0 msgr2=0x562c70167460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b8a2e640 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c701572d0 0x562c70167460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b8a2e640 1 -- 172.21.15.94:0/3636895758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 msgr2=0x562c7015fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b8a2e640 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c7015fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b8a2e640 1 -- 172.21.15.94:0/3636895758 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c6fe976b0 con 0x562c70156830 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa8b8a2e640 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70160420 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40cdcb0 tx=0x7fa8a409bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f984b9c28b700bcd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.192+0000 7fa89cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa89cff9640 1 -- 172.21.15.94:0/3636895758 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a40b5060 con 0x562c70156830 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa89cff9640 1 -- 172.21.15.94:0/3636895758 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7fa8a40a0170 con 0x562c70156830 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa8be165740 1 -- 172.21.15.94:0/3636895758 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8b0003d10 con 0x562c70156830 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa89cff9640 1 -- 172.21.15.94:0/3636895758 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa8a4020040 con 0x562c70156830 2024-04-21T08:39:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa8be165740 10 client.15231 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa8be165740 1 -- 172.21.15.94:0/3636895758 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562c7016a840 con 0x562c70156830 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa89cff9640 1 -- 172.21.15.94:0/3636895758 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8b0003d10 con 0x562c70156830 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa89cff9640 1 -- 172.21.15.94:0/3636895758 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fa8a4098790 con 0x562c70156830 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa8be165740 10 client.15231 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa8be165740 10 client.15231 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa8be165740 20 client.15231 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa8be165740 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x562c7016ce40 0x562c7016e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa8be165740 4 client.15231 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.193+0000 7fa8be165740 1 -- 172.21.15.94:0/3636895758 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c700c8530 con 0x562c7016ce40 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.194+0000 7fa8b922f640 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x562c7016ce40 0x562c7016e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.194+0000 7fa8b922f640 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x562c7016ce40 0x562c7016e2b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25d25562107e7e6c server_cookie=c5300fbc17f51b19 in_seq=0 out_seq=0 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.194+0000 7fa89cff9640 10 client.15231 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa89cff9640 1 -- 172.21.15.94:0/3636895758 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562c700c8530 con 0x562c7016ce40 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa89cff9640 10 client.15231 handle_command_reply: tid=0 2024-04-21T08:39:24.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 client.15231 shutdown 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 -- 172.21.15.94:0/3636895758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 msgr2=0x562c70160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70160420 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40cdcb0 tx=0x7fa8a409bd20 comp rx=0 tx=0).stop 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 -- 172.21.15.94:0/3636895758 shutdown_connections 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c701572d0 0x562c70167460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x562c7016ce40 0x562c7016e2b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c70156830 0x562c70160420 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 --2- 172.21.15.94:0/3636895758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c7015fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa89cff9640 0 client.15231 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 -- 172.21.15.94:0/3636895758 >> 172.21.15.94:0/3636895758 conn(0x562c70151f40 msgr2=0x562c7014fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 -- 172.21.15.94:0/3636895758 shutdown_connections 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 1 -- 172.21.15.94:0/3636895758 wait complete. 2024-04-21T08:39:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.195+0000 7fa8be165740 20 client.15231 trim_cache size 0 max 16384 2024-04-21T08:39:24.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.213+0000 7fa8be165740 1 Processor -- start 2024-04-21T08:39:24.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.213+0000 7fa8be165740 1 -- start start 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.213+0000 7fa8be165740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c7015a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.213+0000 7fa8be165740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c70156830 0x562c7015adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.213+0000 7fa8be165740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70167420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.213+0000 7fa8be165740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x562c7016c5f0 con 0x562c701572d0 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.213+0000 7fa8be165740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562c7016ccc0 con 0x562c70155e40 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.213+0000 7fa8be165740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562c7016d390 con 0x562c70156830 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b9a30640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70167420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b9a30640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70167420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59076/0 (socket says 172.21.15.94:59076) 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b9a30640 1 -- 172.21.15.94:0/3606182877 learned_addr learned my addr 172.21.15.94:0/3606182877 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b8a2e640 1 --2- 172.21.15.94:0/3606182877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c70156830 0x562c7015adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b922f640 1 --2- 172.21.15.94:0/3606182877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c7015a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b9a30640 1 -- 172.21.15.94:0/3606182877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c70156830 msgr2=0x562c7015adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/3606182877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c70156830 0x562c7015adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b9a30640 1 -- 172.21.15.94:0/3606182877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 msgr2=0x562c7015a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/3606182877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c7015a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b9a30640 1 -- 172.21.15.94:0/3606182877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c7015e760 con 0x562c701572d0 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/3606182877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70167420 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac035000 tx=0x7fa8ac039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a4624ba1d1b328 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa89effd640 1 -- 172.21.15.94:0/3606182877 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac054070 con 0x562c701572d0 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa89effd640 1 -- 172.21.15.94:0/3606182877 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8ac0043f0 con 0x562c701572d0 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.214+0000 7fa89effd640 1 -- 172.21.15.94:0/3606182877 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac003950 con 0x562c701572d0 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 -- 172.21.15.94:0/3606182877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 msgr2=0x562c70167420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 --2- 172.21.15.94:0/3606182877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70167420 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac035000 tx=0x7fa8ac039000 comp rx=0 tx=0).stop 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 -- 172.21.15.94:0/3606182877 shutdown_connections 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 --2- 172.21.15.94:0/3606182877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c70156830 0x562c7015adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 --2- 172.21.15.94:0/3606182877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70167420 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 --2- 172.21.15.94:0/3606182877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70155e40 0x562c7015a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 -- 172.21.15.94:0/3606182877 >> 172.21.15.94:0/3606182877 conn(0x562c70151f40 msgr2=0x562c7015e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 -- 172.21.15.94:0/3606182877 shutdown_connections 2024-04-21T08:39:24.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 -- 172.21.15.94:0/3606182877 wait complete. 2024-04-21T08:39:24.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 -- start start 2024-04-21T08:39:24.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 Processor -- start 2024-04-21T08:39:24.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.215+0000 7fa8be165740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c70155e40 0x562c70163400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8be165740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70156830 0x562c70165950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8be165740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8be165740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x562c7016e130 con 0x562c701572d0 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8be165740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562c7016da60 con 0x562c70156830 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8be165740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562c7016e800 con 0x562c70155e40 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8be165740 10 client.0 _ll_register_callbacks cb 0x7fa8a400dec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b9a30640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b9a30640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70165e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59088/0 (socket says 172.21.15.94:59088) 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b9a30640 1 -- 172.21.15.94:0/3537712141 learned_addr learned my addr 172.21.15.94:0/3537712141 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b922f640 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c70155e40 0x562c70163400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b9a30640 1 -- 172.21.15.94:0/3537712141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c70155e40 msgr2=0x562c70163400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c70155e40 0x562c70163400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b9a30640 1 -- 172.21.15.94:0/3537712141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70156830 msgr2=0x562c70165950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b8a2e640 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70156830 0x562c70165950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70156830 0x562c70165950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b9a30640 1 -- 172.21.15.94:0/3537712141 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c701663d0 con 0x562c701572d0 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8b9a30640 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70165e90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac036040 tx=0x7fa8ac02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f82c12bc4a725c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa89cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa89cff9640 1 -- 172.21.15.94:0/3537712141 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac05c060 con 0x562c701572d0 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa89cff9640 1 -- 172.21.15.94:0/3537712141 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7fa8ac0036d0 con 0x562c701572d0 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8be165740 1 -- 172.21.15.94:0/3537712141 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8ac0043f0 con 0x562c701572d0 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa89cff9640 1 -- 172.21.15.94:0/3537712141 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa8ac063050 con 0x562c701572d0 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8be165740 10 client.15237 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.216+0000 7fa8be165740 1 -- 172.21.15.94:0/3537712141 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562c701676d0 con 0x562c701572d0 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.217+0000 7fa89cff9640 1 -- 172.21.15.94:0/3537712141 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8ac0043f0 con 0x562c701572d0 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.217+0000 7fa89cff9640 1 -- 172.21.15.94:0/3537712141 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fa8ac040040 con 0x562c701572d0 2024-04-21T08:39:24.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.217+0000 7fa8be165740 10 client.15237 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.217+0000 7fa8be165740 10 client.15237 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.217+0000 7fa8be165740 20 client.15237 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.217+0000 7fa8be165740 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x562c701d7190 0x562c701d5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.217+0000 7fa8be165740 4 client.15237 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.217+0000 7fa8be165740 1 -- 172.21.15.94:0/3537712141 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "status"}) -- 0x562c700c23a0 con 0x562c701d7190 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.217+0000 7fa8b922f640 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x562c701d7190 0x562c701d5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa89cff9640 10 client.15237 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8b922f640 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x562c701d7190 0x562c701d5ea0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5a30c86908ab9d0 server_cookie=a849076303e6856d in_seq=0 out_seq=0 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa89cff9640 1 -- 172.21.15.94:0/3537712141 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x562c700c23a0 con 0x562c701d7190 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa89cff9640 10 client.15237 handle_command_reply: tid=0 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 client.15237 shutdown 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 -- 172.21.15.94:0/3537712141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 msgr2=0x562c70165e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70165e90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac036040 tx=0x7fa8ac02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:24.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 -- 172.21.15.94:0/3537712141 shutdown_connections 2024-04-21T08:39:24.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x562c701d7190 0x562c701d5ea0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562c701572d0 0x562c70165e90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c70156830 0x562c70165950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa89cff9640 0 client.15237 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:39:24.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 --2- 172.21.15.94:0/3537712141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c70155e40 0x562c70163400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 -- 172.21.15.94:0/3537712141 >> 172.21.15.94:0/3537712141 conn(0x562c70151f40 msgr2=0x562c7015e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 -- 172.21.15.94:0/3537712141 shutdown_connections 2024-04-21T08:39:24.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 1 -- 172.21.15.94:0/3537712141 wait complete. 2024-04-21T08:39:24.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.218+0000 7fa8be165740 20 client.15237 trim_cache size 0 max 16384 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 1, 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24454, 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 88.567610211000002, 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 23, 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 91, 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 126.459066972 2024-04-21T08:39:24.227 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:39:24.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 -- 172.21.15.94:0/351559499 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8a00a3bb0 msgr2=0x7fa8a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8a00a3bb0 0x7fa8a00a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac005c40 tx=0x7fa8ac02c7e0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 -- 172.21.15.94:0/351559499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8b4101fe0 msgr2=0x7fa8b419ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8b4101fe0 0x7fa8b419ceb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400d770 tx=0x7fa8a400dc30 comp rx=0 tx=0).stop 2024-04-21T08:39:24.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 -- 172.21.15.94:0/351559499 shutdown_connections 2024-04-21T08:39:24.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa8a00a3bb0 0x7fa8a00a6070 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b41029a0 0x7fa8b419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8b4101fe0 0x7fa8b419ceb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 --2- 172.21.15.94:0/351559499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b4100e90 0x7fa8b4069cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 -- 172.21.15.94:0/351559499 >> 172.21.15.94:0/351559499 conn(0x7fa8b40fc400 msgr2=0x7fa8b4105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 -- 172.21.15.94:0/351559499 shutdown_connections 2024-04-21T08:39:24.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.219+0000 7fa8bb493640 1 -- 172.21.15.94:0/351559499 wait complete. 2024-04-21T08:39:24.230 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:39:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.329+0000 7f83847db640 1 -- 172.21.15.94:0/1669575289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f837c0749b0 msgr2=0x7f837c074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.329+0000 7f83847db640 1 --2- 172.21.15.94:0/1669575289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f837c0749b0 0x7f837c074e30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f836c009900 tx=0x7f836c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 -- 172.21.15.94:0/1669575289 shutdown_connections 2024-04-21T08:39:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 --2- 172.21.15.94:0/1669575289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f837c109e50 0x7f837c10c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 --2- 172.21.15.94:0/1669575289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f837c0749b0 0x7f837c074e30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 --2- 172.21.15.94:0/1669575289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f837c075f90 0x7f837c0743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 -- 172.21.15.94:0/1669575289 >> 172.21.15.94:0/1669575289 conn(0x7f837c0fd500 msgr2=0x7f837c0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 -- 172.21.15.94:0/1669575289 shutdown_connections 2024-04-21T08:39:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 -- 172.21.15.94:0/1669575289 wait complete. 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 Processor -- start 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 -- start start 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f837c0749b0 0x7f837c19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f837c075f90 0x7f837c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f837c109e50 0x7f837c1a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f837c10fcc0 con 0x7f837c0749b0 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f837c10fb40 con 0x7f837c075f90 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f83847db640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f837c10fe40 con 0x7f837c109e50 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f8382577640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f837c0749b0 0x7f837c19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f8382577640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f837c0749b0 0x7f837c19a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59118/0 (socket says 172.21.15.94:59118) 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f8382577640 1 -- 172.21.15.94:0/3425743673 learned_addr learned my addr 172.21.15.94:0/3425743673 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f8381d76640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f837c075f90 0x7f837c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.330+0000 7f8382d78640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f837c109e50 0x7f837c1a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f8382577640 1 -- 172.21.15.94:0/3425743673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f837c109e50 msgr2=0x7f837c1a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f8382577640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f837c109e50 0x7f837c1a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f8382577640 1 -- 172.21.15.94:0/3425743673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f837c075f90 msgr2=0x7f837c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f8382577640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f837c075f90 0x7f837c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f8382577640 1 -- 172.21.15.94:0/3425743673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f837c1a2000 con 0x7f837c0749b0 2024-04-21T08:39:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f8381d76640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f837c075f90 0x7f837c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f8382d78640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f837c109e50 0x7f837c1a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f8382577640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f837c0749b0 0x7f837c19a430 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f837800ea80 tx=0x7f837800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f83737fe640 1 -- 172.21.15.94:0/3425743673 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837800ce40 con 0x7f837c0749b0 2024-04-21T08:39:24.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f83737fe640 1 -- 172.21.15.94:0/3425743673 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8378005020 con 0x7f837c0749b0 2024-04-21T08:39:24.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f83847db640 1 -- 172.21.15.94:0/3425743673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f837c1a22f0 con 0x7f837c0749b0 2024-04-21T08:39:24.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f83737fe640 1 -- 172.21.15.94:0/3425743673 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83780116d0 con 0x7f837c0749b0 2024-04-21T08:39:24.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.331+0000 7f83847db640 1 -- 172.21.15.94:0/3425743673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f837c1a27f0 con 0x7f837c0749b0 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f83737fe640 1 -- 172.21.15.94:0/3425743673 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f83780054d0 con 0x7f837c0749b0 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f83873c5740 1 Processor -- start 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f83873c5740 1 -- start start 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f83873c5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa27e40 0x55848fa28260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f83873c5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa28830 0x55848fa28cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f83873c5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa292d0 0x55848fa2feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f83873c5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55848f98cb10 con 0x55848fa292d0 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f83873c5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55848f92a240 con 0x55848fa27e40 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f83873c5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55848f988ad0 con 0x55848fa28830 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f8381d76640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa292d0 0x55848fa2feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f8381d76640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa292d0 0x55848fa2feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59124/0 (socket says 172.21.15.94:59124) 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f8381d76640 1 -- 172.21.15.94:0/2737160932 learned_addr learned my addr 172.21.15.94:0/2737160932 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f8382577640 1 --2- 172.21.15.94:0/2737160932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa28830 0x55848fa28cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.334+0000 7f8382d78640 1 --2- 172.21.15.94:0/2737160932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa27e40 0x55848fa28260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f83737fe640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83680a3bb0 0x7f83680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f8381d76640 1 -- 172.21.15.94:0/2737160932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa28830 msgr2=0x55848fa28cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f8381d76640 1 --2- 172.21.15.94:0/2737160932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa28830 0x55848fa28cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f8381d76640 1 -- 172.21.15.94:0/2737160932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa27e40 msgr2=0x55848fa28260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f8381d76640 1 --2- 172.21.15.94:0/2737160932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa27e40 0x55848fa28260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f8381d76640 1 -- 172.21.15.94:0/2737160932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55848f69f2d0 con 0x55848fa292d0 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f8382d78640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83680a3bb0 0x7f83680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f83737fe640 1 -- 172.21.15.94:0/3425743673 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f83780d0ab0 con 0x7f837c0749b0 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f8381d76640 1 --2- 172.21.15.94:0/2737160932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa292d0 0x55848fa2feb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f836c000c00 tx=0x7f836c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc5f448dd3cc6ed server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f8370ff9640 1 -- 172.21.15.94:0/2737160932 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c044020 con 0x55848fa292d0 2024-04-21T08:39:24.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f8370ff9640 1 -- 172.21.15.94:0/2737160932 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f836c005040 con 0x55848fa292d0 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.335+0000 7f8370ff9640 1 -- 172.21.15.94:0/2737160932 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c004920 con 0x55848fa292d0 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 -- 172.21.15.94:0/2737160932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa292d0 msgr2=0x55848fa2feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f8382d78640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83680a3bb0 0x7f83680a6070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x55848fa23200 tx=0x7f83740046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 --2- 172.21.15.94:0/2737160932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa292d0 0x55848fa2feb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f836c000c00 tx=0x7f836c005dd0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 -- 172.21.15.94:0/2737160932 shutdown_connections 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 --2- 172.21.15.94:0/2737160932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa28830 0x55848fa28cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 --2- 172.21.15.94:0/2737160932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa292d0 0x55848fa2feb0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 --2- 172.21.15.94:0/2737160932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa27e40 0x55848fa28260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 -- 172.21.15.94:0/2737160932 >> 172.21.15.94:0/2737160932 conn(0x55848fa23f40 msgr2=0x55848fa22490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 -- 172.21.15.94:0/2737160932 shutdown_connections 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 -- 172.21.15.94:0/2737160932 wait complete. 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 -- start start 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 Processor -- start 2024-04-21T08:39:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa27e40 0x55848fa31e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa32340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa292d0 0x55848fa39380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55848fa30450 con 0x55848fa28830 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55848f915190 con 0x55848fa292d0 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.336+0000 7f83873c5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55848fa305d0 con 0x55848fa27e40 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f83873c5740 10 client.0 _ll_register_callbacks cb 0x55848f9108b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8381d76640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa32340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8381d76640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa32340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59136/0 (socket says 172.21.15.94:59136) 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8381d76640 1 -- 172.21.15.94:0/2936607039 learned_addr learned my addr 172.21.15.94:0/2936607039 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8382577640 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa27e40 0x55848fa31e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8382d78640 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa292d0 0x55848fa39380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8381d76640 1 -- 172.21.15.94:0/2936607039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa27e40 msgr2=0x55848fa31e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8381d76640 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa27e40 0x55848fa31e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8381d76640 1 -- 172.21.15.94:0/2936607039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa292d0 msgr2=0x55848fa39380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8381d76640 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa292d0 0x55848fa39380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8381d76640 1 -- 172.21.15.94:0/2936607039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55848f7696b0 con 0x55848fa28830 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8381d76640 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa32340 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f836c0096c0 tx=0x7f836c004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e906e266e1dbe7f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8365ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8365ffb640 1 -- 172.21.15.94:0/2936607039 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c04d030 con 0x55848fa28830 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8365ffb640 1 -- 172.21.15.94:0/2936607039 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f836c02caa0 con 0x55848fa28830 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f83873c5740 1 -- 172.21.15.94:0/2936607039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f836c005040 con 0x55848fa28830 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8365ffb640 1 -- 172.21.15.94:0/2936607039 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f836c044040 con 0x55848fa28830 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f83873c5740 10 client.15249 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f83873c5740 1 -- 172.21.15.94:0/2936607039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55848fa3c6f0 con 0x55848fa28830 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.337+0000 7f8365ffb640 1 -- 172.21.15.94:0/2936607039 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f836c005040 con 0x55848fa28830 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.338+0000 7f8365ffb640 1 -- 172.21.15.94:0/2936607039 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f836c0544c0 con 0x55848fa28830 2024-04-21T08:39:24.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.338+0000 7f83873c5740 10 client.15249 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.338+0000 7f83873c5740 10 client.15249 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.338+0000 7f83873c5740 20 client.15249 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.338+0000 7f83873c5740 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55848fa3e880 0x55848fa3fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.338+0000 7f83873c5740 4 client.15249 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.338+0000 7f83873c5740 1 -- 172.21.15.94:0/2936607039 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55848f99a530 con 0x55848fa3e880 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.338+0000 7f8382577640 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55848fa3e880 0x55848fa3fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.338+0000 7f8365ffb640 10 client.15249 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.338+0000 7f8382577640 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55848fa3e880 0x55848fa3fcf0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd246039e9c4e1ae server_cookie=a43e839b776d94d in_seq=0 out_seq=0 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f8365ffb640 1 -- 172.21.15.94:0/2936607039 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55848f99a530 con 0x55848fa3e880 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f8365ffb640 10 client.15249 handle_command_reply: tid=0 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 client.15249 shutdown 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 -- 172.21.15.94:0/2936607039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 msgr2=0x55848fa32340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa32340 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f836c0096c0 tx=0x7f836c004970 comp rx=0 tx=0).stop 2024-04-21T08:39:24.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 -- 172.21.15.94:0/2936607039 shutdown_connections 2024-04-21T08:39:24.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55848fa3e880 0x55848fa3fcf0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa292d0 0x55848fa39380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f8365ffb640 0 client.15249 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:39:24.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa32340 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 --2- 172.21.15.94:0/2936607039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa27e40 0x55848fa31e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 -- 172.21.15.94:0/2936607039 >> 172.21.15.94:0/2936607039 conn(0x55848fa23f40 msgr2=0x55848fa21a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 -- 172.21.15.94:0/2936607039 shutdown_connections 2024-04-21T08:39:24.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 1 -- 172.21.15.94:0/2936607039 wait complete. 2024-04-21T08:39:24.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.339+0000 7f83873c5740 20 client.15249 trim_cache size 0 max 16384 2024-04-21T08:39:24.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f83873c5740 1 Processor -- start 2024-04-21T08:39:24.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f83873c5740 1 -- start start 2024-04-21T08:39:24.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f83873c5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa27e40 0x55848fa39dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f83873c5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa28830 0x55848fa39340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f83873c5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa292d0 0x55848fa31dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f83873c5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55848fa3e030 con 0x55848fa27e40 2024-04-21T08:39:24.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f83873c5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55848fa3e700 con 0x55848fa28830 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f83873c5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55848fa3edd0 con 0x55848fa292d0 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382577640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa27e40 0x55848fa39dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382577640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa27e40 0x55848fa39dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59152/0 (socket says 172.21.15.94:59152) 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382577640 1 -- 172.21.15.94:0/2856508822 learned_addr learned my addr 172.21.15.94:0/2856508822 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382d78640 1 --2- 172.21.15.94:0/2856508822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa292d0 0x55848fa31dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382577640 1 -- 172.21.15.94:0/2856508822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa292d0 msgr2=0x55848fa31dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382577640 1 --2- 172.21.15.94:0/2856508822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa292d0 0x55848fa31dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382577640 1 -- 172.21.15.94:0/2856508822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa28830 msgr2=0x55848fa39340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8381d76640 1 --2- 172.21.15.94:0/2856508822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa28830 0x55848fa39340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382577640 1 --2- 172.21.15.94:0/2856508822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa28830 0x55848fa39340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382577640 1 -- 172.21.15.94:0/2856508822 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55848fa2c2d0 con 0x55848fa27e40 2024-04-21T08:39:24.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382d78640 1 --2- 172.21.15.94:0/2856508822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa292d0 0x55848fa31dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.357+0000 7f8382577640 1 --2- 172.21.15.94:0/2856508822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa27e40 0x55848fa39dd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8378099430 tx=0x7f83780ce150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c5c89c8ed46598 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f8367fff640 1 -- 172.21.15.94:0/2856508822 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837801f040 con 0x55848fa27e40 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f8367fff640 1 -- 172.21.15.94:0/2856508822 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83780bd0b0 con 0x55848fa27e40 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f8367fff640 1 -- 172.21.15.94:0/2856508822 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837809ad60 con 0x55848fa27e40 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f83873c5740 1 -- 172.21.15.94:0/2856508822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa27e40 msgr2=0x55848fa39dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f83873c5740 1 --2- 172.21.15.94:0/2856508822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa27e40 0x55848fa39dd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8378099430 tx=0x7f83780ce150 comp rx=0 tx=0).stop 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f83873c5740 1 -- 172.21.15.94:0/2856508822 shutdown_connections 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f83873c5740 1 --2- 172.21.15.94:0/2856508822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa292d0 0x55848fa31dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f83873c5740 1 --2- 172.21.15.94:0/2856508822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa28830 0x55848fa39340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f83873c5740 1 --2- 172.21.15.94:0/2856508822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa27e40 0x55848fa39dd0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f83873c5740 1 -- 172.21.15.94:0/2856508822 >> 172.21.15.94:0/2856508822 conn(0x55848fa23f40 msgr2=0x55848fa22400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f83873c5740 1 -- 172.21.15.94:0/2856508822 shutdown_connections 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f83873c5740 1 -- 172.21.15.94:0/2856508822 wait complete. 2024-04-21T08:39:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.358+0000 7f83873c5740 1 -- start start 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 1 Processor -- start 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa27e40 0x55848fa33560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa33aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa292d0 0x55848fa38470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55848fa3fb70 con 0x55848fa28830 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55848fa3f4a0 con 0x55848fa292d0 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55848fa40240 con 0x55848fa27e40 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 10 client.0 _ll_register_callbacks cb 0x7f8378098d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8381d76640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa33aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8381d76640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa33aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59154/0 (socket says 172.21.15.94:59154) 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8381d76640 1 -- 172.21.15.94:0/1097423349 learned_addr learned my addr 172.21.15.94:0/1097423349 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:24.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8382577640 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa27e40 0x55848fa33560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8381d76640 1 -- 172.21.15.94:0/1097423349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa27e40 msgr2=0x55848fa33560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8381d76640 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa27e40 0x55848fa33560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8381d76640 1 -- 172.21.15.94:0/1097423349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa292d0 msgr2=0x55848fa38470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8381d76640 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa292d0 0x55848fa38470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8381d76640 1 -- 172.21.15.94:0/1097423349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55848fa38a30 con 0x55848fa28830 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8381d76640 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa33aa0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f836c02bc70 tx=0x7f836c02cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c245e63bef3f55 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8365ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8365ffb640 1 -- 172.21.15.94:0/1097423349 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c004020 con 0x55848fa28830 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8365ffb640 1 -- 172.21.15.94:0/1097423349 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f836c0041c0 con 0x55848fa28830 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 1 -- 172.21.15.94:0/1097423349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83780bd0b0 con 0x55848fa28830 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f8365ffb640 1 -- 172.21.15.94:0/1097423349 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f836c0043c0 con 0x55848fa28830 2024-04-21T08:39:24.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 10 client.15255 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:24.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.359+0000 7f83873c5740 1 -- 172.21.15.94:0/1097423349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55848faa4c50 con 0x55848fa28830 2024-04-21T08:39:24.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f8365ffb640 1 -- 172.21.15.94:0/1097423349 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83780bd0b0 con 0x55848fa28830 2024-04-21T08:39:24.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f8365ffb640 1 -- 172.21.15.94:0/1097423349 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f836c0035f0 con 0x55848fa28830 2024-04-21T08:39:24.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f83873c5740 10 client.15255 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:24.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f83873c5740 10 client.15255 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:39:24.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f83873c5740 20 client.15255 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:24.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f83873c5740 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55848faa97d0 0x55848faa84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:24.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f83873c5740 4 client.15255 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:39:24.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f83873c5740 1 -- 172.21.15.94:0/1097423349 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "status"}) -- 0x55848f9943a0 con 0x55848faa97d0 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f8382577640 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55848faa97d0 0x55848faa84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f8365ffb640 10 client.15255 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.360+0000 7f8382577640 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55848faa97d0 0x55848faa84e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af57f0f0afb97811 server_cookie=5cce7a457c424e62 in_seq=0 out_seq=0 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f8365ffb640 1 -- 172.21.15.94:0/1097423349 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55848f9943a0 con 0x55848faa97d0 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f8365ffb640 10 client.15255 handle_command_reply: tid=0 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 client.15255 shutdown 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 -- 172.21.15.94:0/1097423349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 msgr2=0x55848fa33aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa33aa0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f836c02bc70 tx=0x7f836c02cc10 comp rx=0 tx=0).stop 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 -- 172.21.15.94:0/1097423349 shutdown_connections 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55848faa97d0 0x55848faa84e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55848fa292d0 0x55848fa38470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f8365ffb640 0 client.15255 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55848fa28830 0x55848fa33aa0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 --2- 172.21.15.94:0/1097423349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55848fa27e40 0x55848fa33560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 -- 172.21.15.94:0/1097423349 >> 172.21.15.94:0/1097423349 conn(0x55848fa23f40 msgr2=0x55848fa31dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 -- 172.21.15.94:0/1097423349 shutdown_connections 2024-04-21T08:39:24.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 1 -- 172.21.15.94:0/1097423349 wait complete. 2024-04-21T08:39:24.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83873c5740 20 client.15255 trim_cache size 0 max 16384 2024-04-21T08:39:24.370 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:39:24.370 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:39:24.370 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-04-21T08:39:24.370 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24488, 2024-04-21T08:39:24.370 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-21T08:39:24.370 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-21T08:39:24.371 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-21T08:39:24.371 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 112.79207522999999, 2024-04-21T08:39:24.371 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 23, 2024-04-21T08:39:24.371 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 91, 2024-04-21T08:39:24.371 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:39:24.371 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 118.06390555 2024-04-21T08:39:24.371 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:39:24.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83847db640 1 -- 172.21.15.94:0/3425743673 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83680a3bb0 msgr2=0x7f83680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83847db640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83680a3bb0 0x7f83680a6070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x55848fa23200 tx=0x7f83740046c0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83847db640 1 -- 172.21.15.94:0/3425743673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f837c0749b0 msgr2=0x7f837c19a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83847db640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f837c0749b0 0x7f837c19a430 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f837800ea80 tx=0x7f837800ef40 comp rx=0 tx=0).stop 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83847db640 1 -- 172.21.15.94:0/3425743673 shutdown_connections 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83847db640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83680a3bb0 0x7f83680a6070 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83847db640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f837c109e50 0x7f837c1a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83847db640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f837c075f90 0x7f837c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83847db640 1 --2- 172.21.15.94:0/3425743673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f837c0749b0 0x7f837c19a430 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.361+0000 7f83847db640 1 -- 172.21.15.94:0/3425743673 >> 172.21.15.94:0/3425743673 conn(0x7f837c0fd500 msgr2=0x7f837c105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.362+0000 7f83847db640 1 -- 172.21.15.94:0/3425743673 shutdown_connections 2024-04-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:24.362+0000 7f83847db640 1 -- 172.21.15.94:0/3425743673 wait complete. 2024-04-21T08:39:24.374 INFO:tasks.cephadm:Setup complete, yielding 2024-04-21T08:39:24.374 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T08:39:24.387 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:39:24.387 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-21T08:39:24.625 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:24.997 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:24 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/680596675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:25.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:24 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/680596675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:25.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:24 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/680596675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:25 smithi094 ceph-mon[27988]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:25 smithi173 ceph-mon[31428]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:26.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:25 smithi111 ceph-mon[31493]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:26.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.410+0000 7f5de249e700 1 -- 172.21.15.94:0/3708470871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ddc102ac0 msgr2=0x7f5ddc102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:26.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.410+0000 7f5de249e700 1 --2- 172.21.15.94:0/3708470871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ddc102ac0 0x7f5ddc102ee0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0099c0 tx=0x7f5dcc009cd0 comp rx=0 tx=0).stop 2024-04-21T08:39:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.410+0000 7f5de249e700 1 -- 172.21.15.94:0/3708470871 shutdown_connections 2024-04-21T08:39:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.410+0000 7f5de249e700 1 --2- 172.21.15.94:0/3708470871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ddc104660 0x7f5ddc10af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.410+0000 7f5de249e700 1 --2- 172.21.15.94:0/3708470871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ddc102ac0 0x7f5ddc102ee0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.410+0000 7f5de249e700 1 --2- 172.21.15.94:0/3708470871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc103cc0 0x7f5ddc104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.410+0000 7f5de249e700 1 -- 172.21.15.94:0/3708470871 >> 172.21.15.94:0/3708470871 conn(0x7f5ddc0fe060 msgr2=0x7f5ddc1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.410+0000 7f5de249e700 1 -- 172.21.15.94:0/3708470871 shutdown_connections 2024-04-21T08:39:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.410+0000 7f5de249e700 1 -- 172.21.15.94:0/3708470871 wait complete. 2024-04-21T08:39:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de249e700 1 Processor -- start 2024-04-21T08:39:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de249e700 1 -- start start 2024-04-21T08:39:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de249e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ddc102ac0 0x7f5ddc198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de249e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc103cc0 0x7f5ddc199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de249e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ddc104660 0x7f5ddc1a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de249e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc10e550 con 0x7f5ddc104660 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de249e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc10e3d0 con 0x7f5ddc103cc0 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de249e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc10e6d0 con 0x7f5ddc102ac0 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de0a3b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ddc104660 0x7f5ddc1a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5ddbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ddc102ac0 0x7f5ddc198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de0a3b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ddc104660 0x7f5ddc1a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59166/0 (socket says 172.21.15.94:59166) 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.411+0000 7f5de0a3b700 1 -- 172.21.15.94:0/2241142399 learned_addr learned my addr 172.21.15.94:0/2241142399 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5ddb7fe700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc103cc0 0x7f5ddc199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5ddbfff700 1 -- 172.21.15.94:0/2241142399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc103cc0 msgr2=0x7f5ddc199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5ddbfff700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc103cc0 0x7f5ddc199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5ddbfff700 1 -- 172.21.15.94:0/2241142399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ddc104660 msgr2=0x7f5ddc1a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5ddbfff700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ddc104660 0x7f5ddc1a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5ddbfff700 1 -- 172.21.15.94:0/2241142399 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ddc1a0890 con 0x7f5ddc102ac0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5ddbfff700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ddc102ac0 0x7f5ddc198c40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0036f0 tx=0x7f5dcc00f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5dd97fa700 1 -- 172.21.15.94:0/2241142399 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc009e60 con 0x7f5ddc102ac0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5de249e700 1 -- 172.21.15.94:0/2241142399 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ddc1a0b20 con 0x7f5ddc102ac0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5dd97fa700 1 -- 172.21.15.94:0/2241142399 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5dcc005940 con 0x7f5ddc102ac0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5dd97fa700 1 -- 172.21.15.94:0/2241142399 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc0054e0 con 0x7f5ddc102ac0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.412+0000 7f5de249e700 1 -- 172.21.15.94:0/2241142399 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ddc1a1000 con 0x7f5ddc102ac0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.413+0000 7f5de249e700 1 -- 172.21.15.94:0/2241142399 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ddc069730 con 0x7f5ddc102ac0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.414+0000 7f5dd97fa700 1 -- 172.21.15.94:0/2241142399 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5dcc00fc00 con 0x7f5ddc102ac0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.415+0000 7f5dd97fa700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5dc40a3c40 0x7f5dc40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.415+0000 7f5ddb7fe700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5dc40a3c40 0x7f5dc40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.415+0000 7f5dd97fa700 1 -- 172.21.15.94:0/2241142399 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f5dcc0d13d0 con 0x7f5ddc102ac0 2024-04-21T08:39:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.415+0000 7f5ddb7fe700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5dc40a3c40 0x7f5dc40a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009bf0 tx=0x7f5dd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.419+0000 7f5dd97fa700 1 -- 172.21.15.94:0/2241142399 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5dcc013070 con 0x7f5ddc102ac0 2024-04-21T08:39:26.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.633+0000 7f5de249e700 1 -- 172.21.15.94:0/2241142399 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f5ddc062e80 con 0x7f5dc40a3c40 2024-04-21T08:39:26.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.634+0000 7f5dd97fa700 1 -- 172.21.15.94:0/2241142399 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f5ddc062e80 con 0x7f5dc40a3c40 2024-04-21T08:39:26.634 INFO:teuthology.orchestra.run.smithi094.stdout:Backend: cephadm 2024-04-21T08:39:26.634 INFO:teuthology.orchestra.run.smithi094.stdout:Available: Yes 2024-04-21T08:39:26.634 INFO:teuthology.orchestra.run.smithi094.stdout:Paused: No 2024-04-21T08:39:26.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.636+0000 7f5de249e700 1 -- 172.21.15.94:0/2241142399 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5dc40a3c40 msgr2=0x7f5dc40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:26.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.636+0000 7f5de249e700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5dc40a3c40 0x7f5dc40a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009bf0 tx=0x7f5dd0009260 comp rx=0 tx=0).stop 2024-04-21T08:39:26.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.636+0000 7f5de249e700 1 -- 172.21.15.94:0/2241142399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ddc102ac0 msgr2=0x7f5ddc198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:26.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.636+0000 7f5de249e700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ddc102ac0 0x7f5ddc198c40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0036f0 tx=0x7f5dcc00f710 comp rx=0 tx=0).stop 2024-04-21T08:39:26.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.636+0000 7f5de249e700 1 -- 172.21.15.94:0/2241142399 shutdown_connections 2024-04-21T08:39:26.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.636+0000 7f5de249e700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f5dc40a3c40 0x7f5dc40a6100 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:26.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.636+0000 7f5de249e700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ddc104660 0x7f5ddc1a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:26.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.636+0000 7f5de249e700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ddc102ac0 0x7f5ddc198c40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:26.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.636+0000 7f5de249e700 1 --2- 172.21.15.94:0/2241142399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc103cc0 0x7f5ddc199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:26.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.636+0000 7f5de249e700 1 -- 172.21.15.94:0/2241142399 >> 172.21.15.94:0/2241142399 conn(0x7f5ddc0fe060 msgr2=0x7f5ddc1070e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:26.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.637+0000 7f5de249e700 1 -- 172.21.15.94:0/2241142399 shutdown_connections 2024-04-21T08:39:26.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:26.637+0000 7f5de249e700 1 -- 172.21.15.94:0/2241142399 wait complete. 2024-04-21T08:39:26.938 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:26 smithi094 ceph-mon[27988]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:27.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:26 smithi173 ceph-mon[31428]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:27.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:26 smithi111 ceph-mon[31493]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:27.428 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-21T08:39:27.674 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:28.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:27 smithi094 ceph-mon[27988]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:39:28.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:27 smithi173 ceph-mon[31428]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:39:28.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:27 smithi111 ceph-mon[31493]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:39:29.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.478+0000 7f4f7c186700 1 -- 172.21.15.94:0/4047841473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f74102aa0 msgr2=0x7f4f74102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:29.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.478+0000 7f4f7c186700 1 --2- 172.21.15.94:0/4047841473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f74102aa0 0x7f4f74102ec0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4f700099c0 tx=0x7f4f70009cd0 comp rx=0 tx=0).stop 2024-04-21T08:39:29.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.478+0000 7f4f7c186700 1 -- 172.21.15.94:0/4047841473 shutdown_connections 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.478+0000 7f4f7c186700 1 --2- 172.21.15.94:0/4047841473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f74104640 0x7f4f7410aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.478+0000 7f4f7c186700 1 --2- 172.21.15.94:0/4047841473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f74102aa0 0x7f4f74102ec0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.478+0000 7f4f7c186700 1 --2- 172.21.15.94:0/4047841473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f74103ca0 0x7f4f74104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.478+0000 7f4f7c186700 1 -- 172.21.15.94:0/4047841473 >> 172.21.15.94:0/4047841473 conn(0x7f4f740fe040 msgr2=0x7f4f74100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.478+0000 7f4f7c186700 1 -- 172.21.15.94:0/4047841473 shutdown_connections 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.478+0000 7f4f7c186700 1 -- 172.21.15.94:0/4047841473 wait complete. 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f7c186700 1 Processor -- start 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f7c186700 1 -- start start 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f7c186700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f74102aa0 0x7f4f740767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f7c186700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f74103ca0 0x7f4f74074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f7c186700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f74104640 0x7f4f74075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f7c186700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7410e590 con 0x7f4f74102aa0 2024-04-21T08:39:29.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f7c186700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7410e410 con 0x7f4f74104640 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f7c186700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7410e710 con 0x7f4f74103ca0 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f79f22700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f74102aa0 0x7f4f740767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f79f22700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f74102aa0 0x7f4f740767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59180/0 (socket says 172.21.15.94:59180) 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f79f22700 1 -- 172.21.15.94:0/2911430169 learned_addr learned my addr 172.21.15.94:0/2911430169 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f7a723700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f74104640 0x7f4f74075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f79721700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f74103ca0 0x7f4f74074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f79f22700 1 -- 172.21.15.94:0/2911430169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f74103ca0 msgr2=0x7f4f74074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f79f22700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f74103ca0 0x7f4f74074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f79f22700 1 -- 172.21.15.94:0/2911430169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f74104640 msgr2=0x7f4f74075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.479+0000 7f4f79f22700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f74104640 0x7f4f74075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.480+0000 7f4f79f22700 1 -- 172.21.15.94:0/2911430169 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f74075c20 con 0x7f4f74102aa0 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.480+0000 7f4f7a723700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f74104640 0x7f4f74075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.480+0000 7f4f79f22700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f74102aa0 0x7f4f740767e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4f70009f50 tx=0x7f4f70003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.480+0000 7f4f6affd700 1 -- 172.21.15.94:0/2911430169 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f70027070 con 0x7f4f74102aa0 2024-04-21T08:39:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.480+0000 7f4f7c186700 1 -- 172.21.15.94:0/2911430169 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f741a2dc0 con 0x7f4f74102aa0 2024-04-21T08:39:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.480+0000 7f4f6affd700 1 -- 172.21.15.94:0/2911430169 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f70022470 con 0x7f4f74102aa0 2024-04-21T08:39:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.480+0000 7f4f6affd700 1 -- 172.21.15.94:0/2911430169 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7000f6c0 con 0x7f4f74102aa0 2024-04-21T08:39:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.480+0000 7f4f7c186700 1 -- 172.21.15.94:0/2911430169 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f741a31e0 con 0x7f4f74102aa0 2024-04-21T08:39:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.481+0000 7f4f7c186700 1 -- 172.21.15.94:0/2911430169 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f74069730 con 0x7f4f74102aa0 2024-04-21T08:39:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.482+0000 7f4f6affd700 1 -- 172.21.15.94:0/2911430169 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4f7002a070 con 0x7f4f74102aa0 2024-04-21T08:39:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.482+0000 7f4f6affd700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4f600a3c40 0x7f4f600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.483+0000 7f4f79721700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4f600a3c40 0x7f4f600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.483+0000 7f4f6affd700 1 -- 172.21.15.94:0/2911430169 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9050+0+0 (secure 0 0 0) 0x7f4f700d1a70 con 0x7f4f74102aa0 2024-04-21T08:39:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.483+0000 7f4f79721700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4f600a3c40 0x7f4f600a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4f64005fd0 tx=0x7f4f6400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:29.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.486+0000 7f4f6affd700 1 -- 172.21.15.94:0/2911430169 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f70013070 con 0x7f4f74102aa0 2024-04-21T08:39:29.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.698+0000 7f4f7c186700 1 -- 172.21.15.94:0/2911430169 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f4f74062e80 con 0x7f4f600a3c40 2024-04-21T08:39:29.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.711+0000 7f4f6affd700 1 -- 172.21.15.94:0/2911430169 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7f4f74062e80 con 0x7f4f600a3c40 2024-04-21T08:39:29.711 INFO:teuthology.orchestra.run.smithi094.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-21T08:39:29.711 INFO:teuthology.orchestra.run.smithi094.stdout:mds.a smithi094 running (2m) 21s ago 2m 16.1M - 19.0.0-3154-ga630765c b45a994450e2 a445d8ba8f2d 2024-04-21T08:39:29.711 INFO:teuthology.orchestra.run.smithi094.stdout:mds.b smithi111 running (2m) 57s ago 2m 25.2M - 19.0.0-3154-ga630765c b45a994450e2 e1ed5b9cf8df 2024-04-21T08:39:29.711 INFO:teuthology.orchestra.run.smithi094.stdout:mds.c smithi173 running (2m) 57s ago 2m 16.8M - 19.0.0-3154-ga630765c b45a994450e2 a35cf0a58fae 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mds.d smithi094 running (2m) 21s ago 2m 16.5M - 19.0.0-3154-ga630765c b45a994450e2 a9482c9c542f 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mds.e smithi111 running (2m) 57s ago 2m 17.6M - 19.0.0-3154-ga630765c b45a994450e2 53f3a3b8425c 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mds.f smithi173 running (2m) 57s ago 2m 15.9M - 19.0.0-3154-ga630765c b45a994450e2 da02ff2d0f1c 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mds.g smithi094 running (2m) 21s ago 2m 13.8M - 19.0.0-3154-ga630765c b45a994450e2 b8094153e360 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mds.h smithi111 running (2m) 57s ago 2m 18.8M - 19.0.0-3154-ga630765c b45a994450e2 e79d7014fd23 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mds.i smithi173 running (2m) 57s ago 2m 17.4M - 19.0.0-3154-ga630765c b45a994450e2 63bb39d51f32 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mds.j smithi094 running (2m) 21s ago 2m 14.1M - 19.0.0-3154-ga630765c b45a994450e2 fc928bce4bb9 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mds.k smithi111 running (2m) 57s ago 2m 13.8M - 19.0.0-3154-ga630765c b45a994450e2 85af0acb4c4a 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mds.l smithi173 running (2m) 57s ago 2m 13.9M - 19.0.0-3154-ga630765c b45a994450e2 ed554a819eb2 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mgr.x smithi094 *:9283,8765 running (9m) 21s ago 9m 515M - 19.0.0-3154-ga630765c b45a994450e2 a11d562ff490 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mgr.y smithi111 *:8443,8765 running (7m) 57s ago 7m 455M - 19.0.0-3154-ga630765c b45a994450e2 bfe91950e324 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mgr.z smithi173 *:8443,8765 running (7m) 57s ago 7m 452M - 19.0.0-3154-ga630765c b45a994450e2 d3f8281e890f 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mon.a smithi094 running (9m) 21s ago 9m 52.7M 2048M 19.0.0-3154-ga630765c b45a994450e2 6dd38797f9a6 2024-04-21T08:39:29.712 INFO:teuthology.orchestra.run.smithi094.stdout:mon.b smithi111 running (8m) 57s ago 8m 42.3M 2048M 19.0.0-3154-ga630765c b45a994450e2 5317e573d842 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:mon.c smithi173 running (8m) 57s ago 8m 39.8M 2048M 19.0.0-3154-ga630765c b45a994450e2 55d1db549a70 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.0 smithi094 running (7m) 21s ago 7m 56.8M 4272M 19.0.0-3154-ga630765c b45a994450e2 40033951fba9 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.1 smithi111 running (7m) 57s ago 7m 49.6M 4272M 19.0.0-3154-ga630765c b45a994450e2 7cf8d4b0fe34 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.2 smithi173 running (6m) 57s ago 6m 56.7M 4272M 19.0.0-3154-ga630765c b45a994450e2 fa1404fd60fa 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.3 smithi094 running (6m) 21s ago 6m 44.9M 4272M 19.0.0-3154-ga630765c b45a994450e2 0f8409ef8e01 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.4 smithi111 running (5m) 57s ago 5m 46.2M 4272M 19.0.0-3154-ga630765c b45a994450e2 ae63335f2104 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.5 smithi173 running (5m) 57s ago 5m 58.0M 4272M 19.0.0-3154-ga630765c b45a994450e2 5bea7cbcf3f4 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.6 smithi094 running (5m) 21s ago 5m 52.4M 4272M 19.0.0-3154-ga630765c b45a994450e2 ed77bb2cbccb 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.7 smithi111 running (4m) 57s ago 4m 44.8M 4272M 19.0.0-3154-ga630765c b45a994450e2 5a7782f6eb6f 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.8 smithi173 running (4m) 57s ago 4m 43.0M 4272M 19.0.0-3154-ga630765c b45a994450e2 a94143809106 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.9 smithi094 running (3m) 21s ago 3m 54.1M 4272M 19.0.0-3154-ga630765c b45a994450e2 11109b4ec88a 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.10 smithi111 running (3m) 57s ago 3m 55.1M 4272M 19.0.0-3154-ga630765c b45a994450e2 e7a41661a14a 2024-04-21T08:39:29.713 INFO:teuthology.orchestra.run.smithi094.stdout:osd.11 smithi173 running (2m) 57s ago 2m 41.3M 4272M 19.0.0-3154-ga630765c b45a994450e2 b826371eeeb6 2024-04-21T08:39:29.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.713+0000 7f4f7c186700 1 -- 172.21.15.94:0/2911430169 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4f600a3c40 msgr2=0x7f4f600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:29.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.713+0000 7f4f7c186700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4f600a3c40 0x7f4f600a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4f64005fd0 tx=0x7f4f6400a560 comp rx=0 tx=0).stop 2024-04-21T08:39:29.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.713+0000 7f4f7c186700 1 -- 172.21.15.94:0/2911430169 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f74102aa0 msgr2=0x7f4f740767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:29.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.713+0000 7f4f7c186700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f74102aa0 0x7f4f740767e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4f70009f50 tx=0x7f4f70003a10 comp rx=0 tx=0).stop 2024-04-21T08:39:29.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.714+0000 7f4f7c186700 1 -- 172.21.15.94:0/2911430169 shutdown_connections 2024-04-21T08:39:29.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.714+0000 7f4f7c186700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4f600a3c40 0x7f4f600a6100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:29.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.714+0000 7f4f7c186700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f74104640 0x7f4f74075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:29.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.714+0000 7f4f7c186700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f74102aa0 0x7f4f740767e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:29.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.714+0000 7f4f7c186700 1 --2- 172.21.15.94:0/2911430169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f74103ca0 0x7f4f74074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:29.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.714+0000 7f4f7c186700 1 -- 172.21.15.94:0/2911430169 >> 172.21.15.94:0/2911430169 conn(0x7f4f740fe040 msgr2=0x7f4f741070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:29.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.714+0000 7f4f7c186700 1 -- 172.21.15.94:0/2911430169 shutdown_connections 2024-04-21T08:39:29.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:29.714+0000 7f4f7c186700 1 -- 172.21.15.94:0/2911430169 wait complete. 2024-04-21T08:39:30.403 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-21T08:39:30.648 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:30.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:30 smithi111 ceph-mon[31493]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:30.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-21T08:39:30.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2024-04-21T08:39:30.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-04-21T08:39:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-21T08:39:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-04-21T08:39:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-21T08:39:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:30 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-04-21T08:39:30.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:30 smithi111 ceph-mon[31493]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:30.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:30 smithi094 ceph-mon[27988]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:30.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-21T08:39:30.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2024-04-21T08:39:30.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-04-21T08:39:30.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-21T08:39:30.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-04-21T08:39:30.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-21T08:39:30.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:30 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-04-21T08:39:30.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:30 smithi094 ceph-mon[27988]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:31.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:30 smithi173 ceph-mon[31428]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-21T08:39:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2024-04-21T08:39:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-04-21T08:39:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-21T08:39:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-04-21T08:39:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-21T08:39:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:30 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-04-21T08:39:31.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:30 smithi173 ceph-mon[31428]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-21T08:39:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2024-04-21T08:39:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-04-21T08:39:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-21T08:39:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-04-21T08:39:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-21T08:39:31.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:31 smithi094 ceph-mon[27988]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-04-21T08:39:31.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:31 smithi094 ceph-mon[27988]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:39:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-21T08:39:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2024-04-21T08:39:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-04-21T08:39:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-21T08:39:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-04-21T08:39:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-21T08:39:31.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:31 smithi111 ceph-mon[31493]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-04-21T08:39:31.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:31 smithi111 ceph-mon[31493]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:39:32.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-21T08:39:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2024-04-21T08:39:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-04-21T08:39:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-21T08:39:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-04-21T08:39:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-21T08:39:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:31 smithi173 ceph-mon[31428]: from='mgr.14152 172.21.15.94:0/3040837564' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-04-21T08:39:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:31 smithi173 ceph-mon[31428]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:39:32.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.462+0000 7f8418c2f700 1 -- 172.21.15.94:0/2868025476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8414103540 msgr2=0x7f8414109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:32.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.462+0000 7f8418c2f700 1 --2- 172.21.15.94:0/2868025476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8414103540 0x7f8414109df0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f840c00b0d0 tx=0x7f840c00b3e0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.462+0000 7f8418c2f700 1 -- 172.21.15.94:0/2868025476 shutdown_connections 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.462+0000 7f8418c2f700 1 --2- 172.21.15.94:0/2868025476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84141019a0 0x7f8414101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.462+0000 7f8418c2f700 1 --2- 172.21.15.94:0/2868025476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8414103540 0x7f8414109df0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.462+0000 7f8418c2f700 1 --2- 172.21.15.94:0/2868025476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8414102ba0 0x7f8414103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.462+0000 7f8418c2f700 1 -- 172.21.15.94:0/2868025476 >> 172.21.15.94:0/2868025476 conn(0x7f84140fcf40 msgr2=0x7f84140ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.462+0000 7f8418c2f700 1 -- 172.21.15.94:0/2868025476 shutdown_connections 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.462+0000 7f8418c2f700 1 -- 172.21.15.94:0/2868025476 wait complete. 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f8418c2f700 1 Processor -- start 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f8418c2f700 1 -- start start 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f8418c2f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84141019a0 0x7f841406b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f8418c2f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8414102ba0 0x7f841419d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f8418c2f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8414103540 0x7f841419ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f8418c2f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f841410d440 con 0x7f84141019a0 2024-04-21T08:39:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f8418c2f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f841410d2c0 con 0x7f8414103540 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f8418c2f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f841410d5c0 con 0x7f8414102ba0 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f841259c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84141019a0 0x7f841406b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f841259c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84141019a0 0x7f841406b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59198/0 (socket says 172.21.15.94:59198) 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.463+0000 7f841259c700 1 -- 172.21.15.94:0/1170131611 learned_addr learned my addr 172.21.15.94:0/1170131611 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f8412d9d700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8414103540 0x7f841419ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f8411d9b700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8414102ba0 0x7f841419d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f841259c700 1 -- 172.21.15.94:0/1170131611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8414102ba0 msgr2=0x7f841419d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f841259c700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8414102ba0 0x7f841419d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f841259c700 1 -- 172.21.15.94:0/1170131611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8414103540 msgr2=0x7f841419ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f841259c700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8414103540 0x7f841419ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f841259c700 1 -- 172.21.15.94:0/1170131611 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84141a0850 con 0x7f84141019a0 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f8412d9d700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8414103540 0x7f841419ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f841259c700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84141019a0 0x7f841406b280 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f840400d980 tx=0x7f840400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f84037fe700 1 -- 172.21.15.94:0/1170131611 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404017030 con 0x7f84141019a0 2024-04-21T08:39:32.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f84037fe700 1 -- 172.21.15.94:0/1170131611 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84040044d0 con 0x7f84141019a0 2024-04-21T08:39:32.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f8418c2f700 1 -- 172.21.15.94:0/1170131611 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84141a0b40 con 0x7f84141019a0 2024-04-21T08:39:32.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f84037fe700 1 -- 172.21.15.94:0/1170131611 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404003aa0 con 0x7f84141019a0 2024-04-21T08:39:32.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.464+0000 7f8418c2f700 1 -- 172.21.15.94:0/1170131611 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84141a1080 con 0x7f84141019a0 2024-04-21T08:39:32.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.465+0000 7f8418c2f700 1 -- 172.21.15.94:0/1170131611 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8414069730 con 0x7f84141019a0 2024-04-21T08:39:32.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.466+0000 7f84037fe700 1 -- 172.21.15.94:0/1170131611 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f840401a070 con 0x7f84141019a0 2024-04-21T08:39:32.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.467+0000 7f84037fe700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83fc0a3c40 0x7f83fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:32.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.467+0000 7f8411d9b700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83fc0a3c40 0x7f83fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:32.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.467+0000 7f84037fe700 1 -- 172.21.15.94:0/1170131611 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f84040d01c0 con 0x7f84141019a0 2024-04-21T08:39:32.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.467+0000 7f8411d9b700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83fc0a3c40 0x7f83fc0a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8408005950 tx=0x7f84080058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.471+0000 7f84037fe700 1 -- 172.21.15.94:0/1170131611 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f840409c7a0 con 0x7f84141019a0 2024-04-21T08:39:32.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.681+0000 7f8418c2f700 1 -- 172.21.15.94:0/1170131611 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f8414062e80 con 0x7f83fc0a3c40 2024-04-21T08:39:32.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.687+0000 7f84037fe700 1 -- 172.21.15.94:0/1170131611 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f8414062e80 con 0x7f83fc0a3c40 2024-04-21T08:39:32.687 INFO:teuthology.orchestra.run.smithi094.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-21T08:39:32.687 INFO:teuthology.orchestra.run.smithi094.stdout:mds.all 12/12 60s ago 2m smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-04-21T08:39:32.687 INFO:teuthology.orchestra.run.smithi094.stdout:mgr 3/3 60s ago 7m smithi094=x;smithi111=y;smithi173=z;count:3 2024-04-21T08:39:32.687 INFO:teuthology.orchestra.run.smithi094.stdout:mon 3/3 60s ago 8m smithi094:172.21.15.94=a;smithi111:172.21.15.111=b;smithi173:172.21.15.173=c;count:3 2024-04-21T08:39:32.687 INFO:teuthology.orchestra.run.smithi094.stdout:osd 12 60s ago - 2024-04-21T08:39:32.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.689+0000 7f8418c2f700 1 -- 172.21.15.94:0/1170131611 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83fc0a3c40 msgr2=0x7f83fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:32.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.689+0000 7f8418c2f700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83fc0a3c40 0x7f83fc0a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8408005950 tx=0x7f84080058e0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.689+0000 7f8418c2f700 1 -- 172.21.15.94:0/1170131611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84141019a0 msgr2=0x7f841406b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:32.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.689+0000 7f8418c2f700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84141019a0 0x7f841406b280 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f840400d980 tx=0x7f840400dd40 comp rx=0 tx=0).stop 2024-04-21T08:39:32.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.689+0000 7f8418c2f700 1 -- 172.21.15.94:0/1170131611 shutdown_connections 2024-04-21T08:39:32.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.689+0000 7f8418c2f700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f83fc0a3c40 0x7f83fc0a6100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.689+0000 7f8418c2f700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8414103540 0x7f841419ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.689+0000 7f8418c2f700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84141019a0 0x7f841406b280 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.689+0000 7f8418c2f700 1 --2- 172.21.15.94:0/1170131611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8414102ba0 0x7f841419d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:32.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.689+0000 7f8418c2f700 1 -- 172.21.15.94:0/1170131611 >> 172.21.15.94:0/1170131611 conn(0x7f84140fcf40 msgr2=0x7f8414105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:32.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.690+0000 7f8418c2f700 1 -- 172.21.15.94:0/1170131611 shutdown_connections 2024-04-21T08:39:32.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:32.690+0000 7f8418c2f700 1 -- 172.21.15.94:0/1170131611 wait complete. 2024-04-21T08:39:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:32 smithi111 ceph-mon[31493]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-21T08:39:32.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:32 smithi111 ceph-mon[31493]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:39:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:32 smithi094 ceph-mon[27988]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-21T08:39:33.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:32 smithi094 ceph-mon[27988]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:39:33.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:32 smithi173 ceph-mon[31428]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-21T08:39:33.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:32 smithi173 ceph-mon[31428]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:39:33.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-21T08:39:33.720 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:33.753 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:33 smithi094 ceph-mon[27988]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:33.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:33 smithi111 ceph-mon[31493]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:34.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:33 smithi173 ceph-mon[31428]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:34.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:34 smithi094 ceph-mon[27988]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-21T08:39:34.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:34 smithi111 ceph-mon[31493]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-21T08:39:35.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:34 smithi173 ceph-mon[31428]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-21T08:39:35.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.443+0000 7f6136080700 1 -- 172.21.15.94:0/3187829578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130102aa0 msgr2=0x7f6130102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:35.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.443+0000 7f6136080700 1 --2- 172.21.15.94:0/3187829578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130102aa0 0x7f6130102ec0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f61200099c0 tx=0x7f6120009cd0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.443+0000 7f6136080700 1 -- 172.21.15.94:0/3187829578 shutdown_connections 2024-04-21T08:39:35.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.443+0000 7f6136080700 1 --2- 172.21.15.94:0/3187829578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6130104640 0x7f613010aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.443+0000 7f6136080700 1 --2- 172.21.15.94:0/3187829578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130102aa0 0x7f6130102ec0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.443+0000 7f6136080700 1 --2- 172.21.15.94:0/3187829578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6130103ca0 0x7f6130104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.443+0000 7f6136080700 1 -- 172.21.15.94:0/3187829578 >> 172.21.15.94:0/3187829578 conn(0x7f61300fe040 msgr2=0x7f6130100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:35.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.443+0000 7f6136080700 1 -- 172.21.15.94:0/3187829578 shutdown_connections 2024-04-21T08:39:35.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.444+0000 7f6136080700 1 -- 172.21.15.94:0/3187829578 wait complete. 2024-04-21T08:39:35.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.444+0000 7f6136080700 1 Processor -- start 2024-04-21T08:39:35.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.444+0000 7f6136080700 1 -- start start 2024-04-21T08:39:35.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.444+0000 7f6136080700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6130102aa0 0x7f6130198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:35.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.444+0000 7f6136080700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130103ca0 0x7f6130199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:35.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.444+0000 7f6136080700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6130104640 0x7f61301a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:35.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.444+0000 7f6136080700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f613010e510 con 0x7f6130103ca0 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.444+0000 7f6136080700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f613010e390 con 0x7f6130104640 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.444+0000 7f6136080700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f613010e690 con 0x7f6130102aa0 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130103ca0 0x7f6130199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130103ca0 0x7f6130199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39534/0 (socket says 172.21.15.94:39534) 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612effd700 1 -- 172.21.15.94:0/211118166 learned_addr learned my addr 172.21.15.94:0/211118166 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612f7fe700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6130102aa0 0x7f6130198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612ffff700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6130104640 0x7f61301a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612effd700 1 -- 172.21.15.94:0/211118166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6130102aa0 msgr2=0x7f6130198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612effd700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6130102aa0 0x7f6130198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612effd700 1 -- 172.21.15.94:0/211118166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6130104640 msgr2=0x7f61301a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612effd700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6130104640 0x7f61301a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612effd700 1 -- 172.21.15.94:0/211118166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61301a0820 con 0x7f6130103ca0 2024-04-21T08:39:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612ffff700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6130104640 0x7f61301a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:35.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612effd700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130103ca0 0x7f6130199100 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f612400c950 tx=0x7f612400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:35.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612cff9700 1 -- 172.21.15.94:0/211118166 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6124007ac0 con 0x7f6130103ca0 2024-04-21T08:39:35.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f6136080700 1 -- 172.21.15.94:0/211118166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61301a0b10 con 0x7f6130103ca0 2024-04-21T08:39:35.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612cff9700 1 -- 172.21.15.94:0/211118166 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6124004ce0 con 0x7f6130103ca0 2024-04-21T08:39:35.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f612cff9700 1 -- 172.21.15.94:0/211118166 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6124005500 con 0x7f6130103ca0 2024-04-21T08:39:35.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.445+0000 7f6136080700 1 -- 172.21.15.94:0/211118166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61301a10a0 con 0x7f6130103ca0 2024-04-21T08:39:35.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.446+0000 7f6136080700 1 -- 172.21.15.94:0/211118166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6130069730 con 0x7f6130103ca0 2024-04-21T08:39:35.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.447+0000 7f612cff9700 1 -- 172.21.15.94:0/211118166 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6124007c60 con 0x7f6130103ca0 2024-04-21T08:39:35.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.448+0000 7f612cff9700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61180a3c40 0x7f61180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:35.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.448+0000 7f612f7fe700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61180a3c40 0x7f61180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:35.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.448+0000 7f612cff9700 1 -- 172.21.15.94:0/211118166 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f61240d0720 con 0x7f6130103ca0 2024-04-21T08:39:35.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.448+0000 7f612f7fe700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61180a3c40 0x7f61180a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6120006010 tx=0x7f6120000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:35.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.452+0000 7f612cff9700 1 -- 172.21.15.94:0/211118166 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6124014030 con 0x7f6130103ca0 2024-04-21T08:39:35.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.662+0000 7f6136080700 1 -- 172.21.15.94:0/211118166 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f6130062e80 con 0x7f61180a3c40 2024-04-21T08:39:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.663+0000 7f612cff9700 1 -- 172.21.15.94:0/211118166 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f6130062e80 con 0x7f61180a3c40 2024-04-21T08:39:35.663 INFO:teuthology.orchestra.run.smithi094.stdout:HOST ADDR LABELS STATUS 2024-04-21T08:39:35.663 INFO:teuthology.orchestra.run.smithi094.stdout:smithi094 172.21.15.94 2024-04-21T08:39:35.663 INFO:teuthology.orchestra.run.smithi094.stdout:smithi111 172.21.15.111 2024-04-21T08:39:35.663 INFO:teuthology.orchestra.run.smithi094.stdout:smithi173 172.21.15.173 2024-04-21T08:39:35.663 INFO:teuthology.orchestra.run.smithi094.stdout:3 hosts in cluster 2024-04-21T08:39:35.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 -- 172.21.15.94:0/211118166 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61180a3c40 msgr2=0x7f61180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:35.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61180a3c40 0x7f61180a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6120006010 tx=0x7f6120000bc0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 -- 172.21.15.94:0/211118166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130103ca0 msgr2=0x7f6130199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130103ca0 0x7f6130199100 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f612400c950 tx=0x7f612400cc60 comp rx=0 tx=0).stop 2024-04-21T08:39:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 -- 172.21.15.94:0/211118166 shutdown_connections 2024-04-21T08:39:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f61180a3c40 0x7f61180a6100 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6130104640 0x7f61301a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6130102aa0 0x7f6130198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 --2- 172.21.15.94:0/211118166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130103ca0 0x7f6130199100 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 -- 172.21.15.94:0/211118166 >> 172.21.15.94:0/211118166 conn(0x7f61300fe040 msgr2=0x7f61301070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 -- 172.21.15.94:0/211118166 shutdown_connections 2024-04-21T08:39:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:35.665+0000 7f6136080700 1 -- 172.21.15.94:0/211118166 wait complete. 2024-04-21T08:39:36.401 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-21T08:39:36.644 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:36 smithi111 ceph-mon[31493]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:39:36.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:36 smithi111 ceph-mon[31493]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:36 smithi094 ceph-mon[27988]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:39:37.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:36 smithi094 ceph-mon[27988]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:37.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:36 smithi173 ceph-mon[31428]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:39:37.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:36 smithi173 ceph-mon[31428]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:38.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.346+0000 7fb090bb7700 1 -- 172.21.15.94:0/2440770153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb08c104640 msgr2=0x7fb08c10aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:38.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.346+0000 7fb090bb7700 1 --2- 172.21.15.94:0/2440770153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb08c104640 0x7fb08c10aef0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb08400b0d0 tx=0x7fb08400b3e0 comp rx=0 tx=0).stop 2024-04-21T08:39:38.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.347+0000 7fb090bb7700 1 -- 172.21.15.94:0/2440770153 shutdown_connections 2024-04-21T08:39:38.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.347+0000 7fb090bb7700 1 --2- 172.21.15.94:0/2440770153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb08c102aa0 0x7fb08c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:38.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.347+0000 7fb090bb7700 1 --2- 172.21.15.94:0/2440770153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb08c104640 0x7fb08c10aef0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:38.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.347+0000 7fb090bb7700 1 --2- 172.21.15.94:0/2440770153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb08c103ca0 0x7fb08c104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:38.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.347+0000 7fb090bb7700 1 -- 172.21.15.94:0/2440770153 >> 172.21.15.94:0/2440770153 conn(0x7fb08c0fe040 msgr2=0x7fb08c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.347+0000 7fb090bb7700 1 -- 172.21.15.94:0/2440770153 shutdown_connections 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.347+0000 7fb090bb7700 1 -- 172.21.15.94:0/2440770153 wait complete. 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.347+0000 7fb090bb7700 1 Processor -- start 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.347+0000 7fb090bb7700 1 -- start start 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb090bb7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb08c102aa0 0x7fb08c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb090bb7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb08c103ca0 0x7fb08c199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb090bb7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb08c104640 0x7fb08c1a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb090bb7700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c10e590 con 0x7fb08c103ca0 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb090bb7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c10e410 con 0x7fb08c102aa0 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb090bb7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c10e710 con 0x7fb08c104640 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb089d9b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb08c103ca0 0x7fb08c199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb089d9b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb08c103ca0 0x7fb08c199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39546/0 (socket says 172.21.15.94:39546) 2024-04-21T08:39:38.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb089d9b700 1 -- 172.21.15.94:0/1087385046 learned_addr learned my addr 172.21.15.94:0/1087385046 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb08a59c700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb08c102aa0 0x7fb08c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb08ad9d700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb08c104640 0x7fb08c1a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb089d9b700 1 -- 172.21.15.94:0/1087385046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb08c104640 msgr2=0x7fb08c1a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb089d9b700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb08c104640 0x7fb08c1a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb089d9b700 1 -- 172.21.15.94:0/1087385046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb08c102aa0 msgr2=0x7fb08c198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb089d9b700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb08c102aa0 0x7fb08c198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb089d9b700 1 -- 172.21.15.94:0/1087385046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb08c1a07b0 con 0x7fb08c103ca0 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb08ad9d700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb08c104640 0x7fb08c1a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb08a59c700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb08c102aa0 0x7fb08c198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb089d9b700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb08c103ca0 0x7fb08c199090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb08000b560 tx=0x7fb08000b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb07b7fe700 1 -- 172.21.15.94:0/1087385046 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb080015020 con 0x7fb08c103ca0 2024-04-21T08:39:38.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb07b7fe700 1 -- 172.21.15.94:0/1087385046 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb080011ce0 con 0x7fb08c103ca0 2024-04-21T08:39:38.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb090bb7700 1 -- 172.21.15.94:0/1087385046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb08c1a0aa0 con 0x7fb08c103ca0 2024-04-21T08:39:38.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.348+0000 7fb07b7fe700 1 -- 172.21.15.94:0/1087385046 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb080010da0 con 0x7fb08c103ca0 2024-04-21T08:39:38.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.349+0000 7fb090bb7700 1 -- 172.21.15.94:0/1087385046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb08c1a0f90 con 0x7fb08c103ca0 2024-04-21T08:39:38.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.349+0000 7fb090bb7700 1 -- 172.21.15.94:0/1087385046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb08c069730 con 0x7fb08c103ca0 2024-04-21T08:39:38.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.351+0000 7fb07b7fe700 1 -- 172.21.15.94:0/1087385046 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb080018020 con 0x7fb08c103ca0 2024-04-21T08:39:38.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.352+0000 7fb07b7fe700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb0740a3c40 0x7fb0740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:38.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.352+0000 7fb08a59c700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb0740a3c40 0x7fb0740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:38.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.352+0000 7fb07b7fe700 1 -- 172.21.15.94:0/1087385046 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fb0800d0590 con 0x7fb08c103ca0 2024-04-21T08:39:38.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.352+0000 7fb08a59c700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb0740a3c40 0x7fb0740a6100 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb07c005fd0 tx=0x7fb07c005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:38.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.355+0000 7fb07b7fe700 1 -- 172.21.15.94:0/1087385046 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb08001f020 con 0x7fb08c103ca0 2024-04-21T08:39:38.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.567+0000 7fb090bb7700 1 -- 172.21.15.94:0/1087385046 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fb08c062e80 con 0x7fb0740a3c40 2024-04-21T08:39:38.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.569+0000 7fb07b7fe700 1 -- 172.21.15.94:0/1087385046 <== mgr.14152 v2:172.21.15.94:6800/3394521972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fb08c062e80 con 0x7fb0740a3c40 2024-04-21T08:39:38.568 INFO:teuthology.orchestra.run.smithi094.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-21T08:39:38.569 INFO:teuthology.orchestra.run.smithi094.stdout:smithi094 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513001L480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-21T08:39:38.569 INFO:teuthology.orchestra.run.smithi094.stdout:smithi111 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VZ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-21T08:39:38.569 INFO:teuthology.orchestra.run.smithi094.stdout:smithi173 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204019S400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-21T08:39:38.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 -- 172.21.15.94:0/1087385046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb0740a3c40 msgr2=0x7fb0740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:38.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb0740a3c40 0x7fb0740a6100 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb07c005fd0 tx=0x7fb07c005e40 comp rx=0 tx=0).stop 2024-04-21T08:39:38.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 -- 172.21.15.94:0/1087385046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb08c103ca0 msgr2=0x7fb08c199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:38.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb08c103ca0 0x7fb08c199090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb08000b560 tx=0x7fb08000b920 comp rx=0 tx=0).stop 2024-04-21T08:39:38.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 -- 172.21.15.94:0/1087385046 shutdown_connections 2024-04-21T08:39:38.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb0740a3c40 0x7fb0740a6100 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:38.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb08c102aa0 0x7fb08c198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:38.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb08c104640 0x7fb08c1a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:38.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 --2- 172.21.15.94:0/1087385046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb08c103ca0 0x7fb08c199090 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:38.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 -- 172.21.15.94:0/1087385046 >> 172.21.15.94:0/1087385046 conn(0x7fb08c0fe040 msgr2=0x7fb08c106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:38.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 -- 172.21.15.94:0/1087385046 shutdown_connections 2024-04-21T08:39:38.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:38.571+0000 7fb090bb7700 1 -- 172.21.15.94:0/1087385046 wait complete. 2024-04-21T08:39:39.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:38 smithi094 ceph-mon[27988]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:39:39.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:38 smithi173 ceph-mon[31428]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:39:39.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:38 smithi111 ceph-mon[31493]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:39:39.229 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T08:39:39.238 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:39:39.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-21T08:39:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:39.855 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:39 smithi094 ceph-mon[27988]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:40.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:39 smithi173 ceph-mon[31428]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:40.088 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:39 smithi111 ceph-mon[31493]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:40.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:40 smithi094 ceph-mon[27988]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T08:39:41.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:40 smithi173 ceph-mon[31428]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T08:39:41.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 -- 172.21.15.94:0/3488593603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb190101990 msgr2=0x7fb190101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 --2- 172.21.15.94:0/3488593603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb190101990 0x7fb190101db0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb1840099c0 tx=0x7fb184009cd0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 -- 172.21.15.94:0/3488593603 shutdown_connections 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 --2- 172.21.15.94:0/3488593603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb190103530 0x7fb190109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 --2- 172.21.15.94:0/3488593603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb190101990 0x7fb190101db0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 --2- 172.21.15.94:0/3488593603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb190102b90 0x7fb190102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 -- 172.21.15.94:0/3488593603 >> 172.21.15.94:0/3488593603 conn(0x7fb1900fcf30 msgr2=0x7fb1900ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 -- 172.21.15.94:0/3488593603 shutdown_connections 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 -- 172.21.15.94:0/3488593603 wait complete. 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 Processor -- start 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 -- start start 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb190101990 0x7fb19006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:41.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb190102b90 0x7fb19019d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb190103530 0x7fb19019ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb19010d400 con 0x7fb190101990 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb19010d280 con 0x7fb190102b90 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb196f30700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb19010d580 con 0x7fb190103530 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb194ccc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb190101990 0x7fb19006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.158+0000 7fb194ccc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb190101990 0x7fb19006bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39560/0 (socket says 172.21.15.94:39560) 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb194ccc700 1 -- 172.21.15.94:0/2294085932 learned_addr learned my addr 172.21.15.94:0/2294085932 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb1954cd700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb190103530 0x7fb19019ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb194ccc700 1 -- 172.21.15.94:0/2294085932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb190103530 msgr2=0x7fb19019ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb18ffff700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb190102b90 0x7fb19019d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb194ccc700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb190103530 0x7fb19019ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb194ccc700 1 -- 172.21.15.94:0/2294085932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb190102b90 msgr2=0x7fb19019d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb194ccc700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb190102b90 0x7fb19019d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb194ccc700 1 -- 172.21.15.94:0/2294085932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1901a0850 con 0x7fb190101990 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb1954cd700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb190103530 0x7fb19019ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb194ccc700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb190101990 0x7fb19006bb70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb1840038c0 tx=0x7fb1840047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb18dffb700 1 -- 172.21.15.94:0/2294085932 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb184027070 con 0x7fb190101990 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb18dffb700 1 -- 172.21.15.94:0/2294085932 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1840041a0 con 0x7fb190101990 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb196f30700 1 -- 172.21.15.94:0/2294085932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1901a0ae0 con 0x7fb190101990 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb18dffb700 1 -- 172.21.15.94:0/2294085932 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18400f460 con 0x7fb190101990 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.159+0000 7fb196f30700 1 -- 172.21.15.94:0/2294085932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1901a0fc0 con 0x7fb190101990 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.160+0000 7fb196f30700 1 -- 172.21.15.94:0/2294085932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb190069730 con 0x7fb190101990 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.161+0000 7fb18dffb700 1 -- 172.21.15.94:0/2294085932 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb18402a070 con 0x7fb190101990 2024-04-21T08:39:41.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.162+0000 7fb18dffb700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1800a3c40 0x7fb1800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:41.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.162+0000 7fb18ffff700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1800a3c40 0x7fb1800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:41.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.162+0000 7fb18dffb700 1 -- 172.21.15.94:0/2294085932 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fb1840d1bf0 con 0x7fb190101990 2024-04-21T08:39:41.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.162+0000 7fb18ffff700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1800a3c40 0x7fb1800a6100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb17c005950 tx=0x7fb17c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:41.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.165+0000 7fb18dffb700 1 -- 172.21.15.94:0/2294085932 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb184031070 con 0x7fb190101990 2024-04-21T08:39:41.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:40 smithi111 ceph-mon[31493]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T08:39:41.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.409+0000 7fb196f30700 1 -- 172.21.15.94:0/2294085932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fb190063650 con 0x7fb190101990 2024-04-21T08:39:41.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.409+0000 7fb18dffb700 1 -- 172.21.15.94:0/2294085932 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3060 (secure 0 0 0) 0x7fb184013070 con 0x7fb190101990 2024-04-21T08:39:41.409 INFO:teuthology.orchestra.run.smithi094.stdout:e23 2024-04-21T08:39:41.409 INFO:teuthology.orchestra.run.smithi094.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:legacy client fscid: 1 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:Filesystem 'cephfs' (1) 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:fs_name cephfs 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:epoch 23 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:created 2024-04-21T08:37:30.753966+0000 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:modified 2024-04-21T08:37:56.995828+0000 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:tableserver 0 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:root 0 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:session_timeout 300 2024-04-21T08:39:41.410 INFO:teuthology.orchestra.run.smithi094.stdout:session_autoclose 300 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:max_file_size 1099511627776 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:max_xattr_size 65536 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:required_client_features {} 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:last_failure 0 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:last_failure_osd_epoch 0 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:max_mds 3 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:in 0,1,2 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:up {0=24488,1=24454,2=24446} 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:failed 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:damaged 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:stopped 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:data_pools [3,4] 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:metadata_pool 2 2024-04-21T08:39:41.411 INFO:teuthology.orchestra.run.smithi094.stdout:inline_data disabled 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:balancer 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:bal_rank_mask -1 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:standby_count_wanted 0 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:qdb_cluster leader: 24488 members: 24454,24446,24488 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.b{0:24488} state up:active seq 3 addr [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.c{0:24433} state up:standby-replay seq 1 addr [v2:172.21.15.173:6836/1466520447,v1:172.21.15.173:6837/1466520447] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.i{1:24454} state up:active seq 11 addr [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.f{1:24421} state up:standby-replay seq 1 addr [v2:172.21.15.173:6834/3922099450,v1:172.21.15.173:6835/3922099450] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.h{2:24446} state up:active seq 12 addr [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.e{2:24416} state up:standby-replay seq 1 addr [v2:172.21.15.111:6834/3967081988,v1:172.21.15.111:6835/3967081988] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout:Standby daemons: 2024-04-21T08:39:41.412 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:41.413 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.j{-1:14547} state up:standby seq 1 addr [v2:172.21.15.94:6834/1080139605,v1:172.21.15.94:6835/1080139605] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.413 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.d{-1:14559} state up:standby seq 1 addr [v2:172.21.15.94:6836/4105566219,v1:172.21.15.94:6837/4105566219] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.413 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.g{-1:14616} state up:standby seq 1 addr [v2:172.21.15.94:6838/4167052698,v1:172.21.15.94:6839/4167052698] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.413 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.a{-1:14628} state up:standby seq 1 addr [v2:172.21.15.94:6840/2997402360,v1:172.21.15.94:6841/2997402360] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.413 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.l{-1:24379} state up:standby seq 1 addr [v2:172.21.15.173:6832/2495824461,v1:172.21.15.173:6833/2495824461] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.413 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.k{-1:24389} state up:standby seq 1 addr [v2:172.21.15.111:6832/4021460222,v1:172.21.15.111:6833/4021460222] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:39:41.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.411+0000 7fb196f30700 1 -- 172.21.15.94:0/2294085932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1800a3c40 msgr2=0x7fb1800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:41.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.411+0000 7fb196f30700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1800a3c40 0x7fb1800a6100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb17c005950 tx=0x7fb17c0058e0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.411+0000 7fb196f30700 1 -- 172.21.15.94:0/2294085932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb190101990 msgr2=0x7fb19006bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:41.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.411+0000 7fb196f30700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb190101990 0x7fb19006bb70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb1840038c0 tx=0x7fb1840047b0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.411+0000 7fb196f30700 1 -- 172.21.15.94:0/2294085932 shutdown_connections 2024-04-21T08:39:41.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.411+0000 7fb196f30700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1800a3c40 0x7fb1800a6100 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.411+0000 7fb196f30700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb190103530 0x7fb19019ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.411+0000 7fb196f30700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb190101990 0x7fb19006bb70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.411+0000 7fb196f30700 1 --2- 172.21.15.94:0/2294085932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb190102b90 0x7fb19019d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.411+0000 7fb196f30700 1 -- 172.21.15.94:0/2294085932 >> 172.21.15.94:0/2294085932 conn(0x7fb1900fcf30 msgr2=0x7fb190105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:41.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.412+0000 7fb196f30700 1 -- 172.21.15.94:0/2294085932 shutdown_connections 2024-04-21T08:39:41.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:41.412+0000 7fb196f30700 1 -- 172.21.15.94:0/2294085932 wait complete. 2024-04-21T08:39:41.415 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-04-21T08:39:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:41 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2294085932' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:39:42.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:41 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2294085932' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:39:42.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-21T08:39:42.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:41 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2294085932' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:39:42.347 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:42.726 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:42 smithi094 ceph-mon[27988]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-21T08:39:43.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:42 smithi173 ceph-mon[31428]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-21T08:39:43.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:42 smithi111 ceph-mon[31493]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-21T08:39:44.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 -- 172.21.15.94:0/2952412442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb908102d80 msgr2=0x7fb908109630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 --2- 172.21.15.94:0/2952412442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb908102d80 0x7fb908109630 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb90000b0d0 tx=0x7fb90000b3e0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 -- 172.21.15.94:0/2952412442 shutdown_connections 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 --2- 172.21.15.94:0/2952412442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9081011e0 0x7fb908101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 --2- 172.21.15.94:0/2952412442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb908102d80 0x7fb908109630 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 --2- 172.21.15.94:0/2952412442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9081023e0 0x7fb908102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 -- 172.21.15.94:0/2952412442 >> 172.21.15.94:0/2952412442 conn(0x7fb9080fc780 msgr2=0x7fb9080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 -- 172.21.15.94:0/2952412442 shutdown_connections 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 -- 172.21.15.94:0/2952412442 wait complete. 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 Processor -- start 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.139+0000 7fb90f8c3700 1 -- start start 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90f8c3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9081011e0 0x7fb908105170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90f8c3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9081023e0 0x7fb9081037c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90f8c3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb908102d80 0x7fb908103d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90f8c3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb90810cce0 con 0x7fb908102d80 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90f8c3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb90810cb60 con 0x7fb9081011e0 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90f8c3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb90810ce60 con 0x7fb9081023e0 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90de60700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb908102d80 0x7fb908103d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90de60700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb908102d80 0x7fb908103d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37218/0 (socket says 172.21.15.94:37218) 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90de60700 1 -- 172.21.15.94:0/69684651 learned_addr learned my addr 172.21.15.94:0/69684651 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90ce5e700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9081023e0 0x7fb9081037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90de60700 1 -- 172.21.15.94:0/69684651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9081023e0 msgr2=0x7fb9081037c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90de60700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9081023e0 0x7fb9081037c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90de60700 1 -- 172.21.15.94:0/69684651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9081011e0 msgr2=0x7fb908105170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90de60700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9081011e0 0x7fb908105170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90de60700 1 -- 172.21.15.94:0/69684651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9081045b0 con 0x7fb908102d80 2024-04-21T08:39:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90de60700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb908102d80 0x7fb908103d00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb900009fd0 tx=0x7fb900004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb8fe7fc700 1 -- 172.21.15.94:0/69684651 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90001f030 con 0x7fb908102d80 2024-04-21T08:39:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90f8c3700 1 -- 172.21.15.94:0/69684651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9081a2ce0 con 0x7fb908102d80 2024-04-21T08:39:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb8fe7fc700 1 -- 172.21.15.94:0/69684651 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb900007bb0 con 0x7fb908102d80 2024-04-21T08:39:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb8fe7fc700 1 -- 172.21.15.94:0/69684651 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb900004820 con 0x7fb908102d80 2024-04-21T08:39:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.140+0000 7fb90f8c3700 1 -- 172.21.15.94:0/69684651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9081a30e0 con 0x7fb908102d80 2024-04-21T08:39:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.141+0000 7fb90f8c3700 1 -- 172.21.15.94:0/69684651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9081a32b0 con 0x7fb908102d80 2024-04-21T08:39:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.142+0000 7fb8fe7fc700 1 -- 172.21.15.94:0/69684651 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb900003a00 con 0x7fb908102d80 2024-04-21T08:39:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.143+0000 7fb8fe7fc700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8f40a3c40 0x7fb8f40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.143+0000 7fb90d65f700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8f40a3c40 0x7fb8f40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.143+0000 7fb8fe7fc700 1 -- 172.21.15.94:0/69684651 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fb90000f020 con 0x7fb908102d80 2024-04-21T08:39:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.143+0000 7fb90d65f700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8f40a3c40 0x7fb8f40a6100 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb908102240 tx=0x7fb904005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.147+0000 7fb8fe7fc700 1 -- 172.21.15.94:0/69684651 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb900029020 con 0x7fb908102d80 2024-04-21T08:39:44.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.349+0000 7fb90f8c3700 1 -- 172.21.15.94:0/69684651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fb9081a3480 con 0x7fb908102d80 2024-04-21T08:39:44.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.350+0000 7fb8fe7fc700 1 -- 172.21.15.94:0/69684651 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5728 (secure 0 0 0) 0x7fb90009e690 con 0x7fb908102d80 2024-04-21T08:39:44.350 INFO:teuthology.orchestra.run.smithi094.stdout:epoch 93 2024-04-21T08:39:44.350 INFO:teuthology.orchestra.run.smithi094.stdout:fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:39:44.350 INFO:teuthology.orchestra.run.smithi094.stdout:created 2024-04-21T08:29:35.091752+0000 2024-04-21T08:39:44.350 INFO:teuthology.orchestra.run.smithi094.stdout:modified 2024-04-21T08:39:31.669420+0000 2024-04-21T08:39:44.350 INFO:teuthology.orchestra.run.smithi094.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-21T08:39:44.350 INFO:teuthology.orchestra.run.smithi094.stdout:crush_version 27 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:full_ratio 0.95 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:backfillfull_ratio 0.9 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:nearfull_ratio 0.85 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:require_min_compat_client luminous 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:min_compat_client luminous 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:require_osd_release squid 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:stretch_mode_enabled false 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:max_osd 12 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:osd.0 up in weight 1 up_from 8 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6802/676652749,v1:172.21.15.94:6803/676652749] [v2:172.21.15.94:6804/676652749,v1:172.21.15.94:6805/676652749] exists,up ec26f622-360a-48df-bc52-c85842709d88 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6800/2507021892,v1:172.21.15.111:6801/2507021892] [v2:172.21.15.111:6802/2507021892,v1:172.21.15.111:6803/2507021892] exists,up 188e7b8f-e006-4b64-8136-9e71097e107f 2024-04-21T08:39:44.351 INFO:teuthology.orchestra.run.smithi094.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/4262691909,v1:172.21.15.173:6801/4262691909] [v2:172.21.15.173:6802/4262691909,v1:172.21.15.173:6803/4262691909] exists,up 0f3dff21-6860-4c3b-9c15-3f120e0f3120 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6810/66052007,v1:172.21.15.94:6811/66052007] [v2:172.21.15.94:6812/66052007,v1:172.21.15.94:6813/66052007] exists,up b30afbd0-4fe9-432a-ac54-7db53d055e37 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6808/4084294051,v1:172.21.15.111:6809/4084294051] [v2:172.21.15.111:6810/4084294051,v1:172.21.15.111:6811/4084294051] exists,up 125dfc8c-d9ca-4ea4-a06f-605e4c76c919 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/2394839325,v1:172.21.15.173:6809/2394839325] [v2:172.21.15.173:6810/2394839325,v1:172.21.15.173:6811/2394839325] exists,up 01402189-deb7-49ad-bda8-0c35038a3348 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:osd.6 up in weight 1 up_from 46 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6818/234088342,v1:172.21.15.94:6819/234088342] [v2:172.21.15.94:6820/234088342,v1:172.21.15.94:6821/234088342] exists,up 7a09d9c1-e8fb-4ff8-a349-8d5aba865099 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6816/659250531,v1:172.21.15.111:6817/659250531] [v2:172.21.15.111:6818/659250531,v1:172.21.15.111:6819/659250531] exists,up ed806162-485b-4c84-9687-78fcc1f8cc59 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/1973310944,v1:172.21.15.173:6817/1973310944] [v2:172.21.15.173:6818/1973310944,v1:172.21.15.173:6819/1973310944] exists,up 0b61b68b-048f-4904-923a-df40dda39925 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6826/3583446749,v1:172.21.15.94:6827/3583446749] [v2:172.21.15.94:6828/3583446749,v1:172.21.15.94:6829/3583446749] exists,up 5f384696-fd22-4f83-b99c-e55623ce934e 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6824/176065870,v1:172.21.15.111:6825/176065870] [v2:172.21.15.111:6826/176065870,v1:172.21.15.111:6827/176065870] exists,up 7f6aaa6b-83b7-4f4e-8626-6bde0352caf9 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/1600242692,v1:172.21.15.173:6825/1600242692] [v2:172.21.15.173:6826/1600242692,v1:172.21.15.173:6827/1600242692] exists,up e7529c3d-ab5c-4963-9b8a-40aa40bf97dc 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 2.b [2,6] 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 2.e [2,4] 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 2.12 [5,11] 2024-04-21T08:39:44.352 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 2.1d [2,4,8,11] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 2.1f [8,1] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.1 [4,9] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.6 [3,7] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.14 [1,9] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.17 [4,8] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.1a [5,6] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.24 [4,9] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.39 [5,6] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 4.e [9,11] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 4.1c [9,8,5,10] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 4.1f [9,8] 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/441167573 expires 2024-04-22T08:30:29.483746+0000 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/2502723264 expires 2024-04-22T08:30:29.483746+0000 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/613544885 expires 2024-04-22T08:30:29.483746+0000 2024-04-21T08:39:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:6801/3877440809 expires 2024-04-22T08:30:29.483746+0000 2024-04-21T08:39:44.354 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/3977681577 expires 2024-04-22T08:30:03.511107+0000 2024-04-21T08:39:44.354 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/3965359185 expires 2024-04-22T08:30:03.511107+0000 2024-04-21T08:39:44.354 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/1761971189 expires 2024-04-22T08:30:03.511107+0000 2024-04-21T08:39:44.354 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:6800/2629131251 expires 2024-04-22T08:30:03.511107+0000 2024-04-21T08:39:44.354 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:6800/3877440809 expires 2024-04-22T08:30:29.483746+0000 2024-04-21T08:39:44.354 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:6801/2629131251 expires 2024-04-22T08:30:03.511107+0000 2024-04-21T08:39:44.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 -- 172.21.15.94:0/69684651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8f40a3c40 msgr2=0x7fb8f40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:44.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8f40a3c40 0x7fb8f40a6100 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb908102240 tx=0x7fb904005d20 comp rx=0 tx=0).stop 2024-04-21T08:39:44.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 -- 172.21.15.94:0/69684651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb908102d80 msgr2=0x7fb908103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:44.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb908102d80 0x7fb908103d00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb900009fd0 tx=0x7fb900004ef0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 -- 172.21.15.94:0/69684651 shutdown_connections 2024-04-21T08:39:44.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8f40a3c40 0x7fb8f40a6100 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9081011e0 0x7fb908105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb908102d80 0x7fb908103d00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 --2- 172.21.15.94:0/69684651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9081023e0 0x7fb9081037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:44.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 -- 172.21.15.94:0/69684651 >> 172.21.15.94:0/69684651 conn(0x7fb9080fc780 msgr2=0x7fb9080fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:44.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 -- 172.21.15.94:0/69684651 shutdown_connections 2024-04-21T08:39:44.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:44.353+0000 7fb90f8c3700 1 -- 172.21.15.94:0/69684651 wait complete. 2024-04-21T08:39:45.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:44 smithi094 ceph-mon[27988]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:45.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:44 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/69684651' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:39:45.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:44 smithi173 ceph-mon[31428]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:45.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:44 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/69684651' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:39:45.103 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-21T08:39:45.116 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:39:45.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:44 smithi111 ceph-mon[31493]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:45.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:44 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/69684651' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:39:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.216+0000 7fe5634a8640 1 -- 172.21.15.94:0/2790574271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe55c101820 msgr2=0x7fe55c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.216+0000 7fe5634a8640 1 --2- 172.21.15.94:0/2790574271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe55c101820 0x7fe55c101ca0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe550009900 tx=0x7fe55002bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.216+0000 7fe5634a8640 1 -- 172.21.15.94:0/2790574271 shutdown_connections 2024-04-21T08:39:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.216+0000 7fe5634a8640 1 --2- 172.21.15.94:0/2790574271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe55c1021e0 0x7fe55c108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.216+0000 7fe5634a8640 1 --2- 172.21.15.94:0/2790574271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe55c101820 0x7fe55c101ca0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.216+0000 7fe5634a8640 1 --2- 172.21.15.94:0/2790574271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c100630 0x7fe55c100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.216+0000 7fe5634a8640 1 -- 172.21.15.94:0/2790574271 >> 172.21.15.94:0/2790574271 conn(0x7fe55c0fbba0 msgr2=0x7fe55c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.216+0000 7fe5634a8640 1 -- 172.21.15.94:0/2790574271 shutdown_connections 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.216+0000 7fe5634a8640 1 -- 172.21.15.94:0/2790574271 wait complete. 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe5634a8640 1 Processor -- start 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe5634a8640 1 -- start start 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe5634a8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe55c100630 0x7fe55c198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe5634a8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe55c101820 0x7fe55c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe5634a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1021e0 0x7fe55c19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe5634a8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe55c10c9f0 con 0x7fe55c100630 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe5634a8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe55c10c870 con 0x7fe55c101820 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe5634a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe55c10cb70 con 0x7fe55c1021e0 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561244640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe55c100630 0x7fe55c198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561244640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe55c100630 0x7fe55c198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37242/0 (socket says 172.21.15.94:37242) 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561244640 1 -- 172.21.15.94:0/1223315046 learned_addr learned my addr 172.21.15.94:0/1223315046 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561a45640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1021e0 0x7fe55c19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe560a43640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe55c101820 0x7fe55c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561244640 1 -- 172.21.15.94:0/1223315046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1021e0 msgr2=0x7fe55c19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561244640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1021e0 0x7fe55c19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561244640 1 -- 172.21.15.94:0/1223315046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe55c101820 msgr2=0x7fe55c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561244640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe55c101820 0x7fe55c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561244640 1 -- 172.21.15.94:0/1223315046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55c19fda0 con 0x7fe55c100630 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561a45640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1021e0 0x7fe55c19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe560a43640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe55c101820 0x7fe55c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.217+0000 7fe561244640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe55c100630 0x7fe55c198180 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe55400ea80 tx=0x7fe55400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.218+0000 7fe55a7fc640 1 -- 172.21.15.94:0/1223315046 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55400ce40 con 0x7fe55c100630 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.218+0000 7fe5634a8640 1 -- 172.21.15.94:0/1223315046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe55c1a0090 con 0x7fe55c100630 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.218+0000 7fe55a7fc640 1 -- 172.21.15.94:0/1223315046 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe554011920 con 0x7fe55c100630 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.218+0000 7fe55a7fc640 1 -- 172.21.15.94:0/1223315046 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55401aaa0 con 0x7fe55c100630 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.218+0000 7fe5634a8640 1 -- 172.21.15.94:0/1223315046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe55c1a05e0 con 0x7fe55c100630 2024-04-21T08:39:45.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.219+0000 7fe5634a8640 1 -- 172.21.15.94:0/1223315046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe55c1a08a0 con 0x7fe55c100630 2024-04-21T08:39:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.220+0000 7fe55a7fc640 1 -- 172.21.15.94:0/1223315046 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe55401ac40 con 0x7fe55c100630 2024-04-21T08:39:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.221+0000 7fe55a7fc640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5440a3bb0 0x7fe5440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.221+0000 7fe55a7fc640 1 -- 172.21.15.94:0/1223315046 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fe5540d1090 con 0x7fe55c100630 2024-04-21T08:39:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.221+0000 7fe560a43640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5440a3bb0 0x7fe5440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.221+0000 7fe560a43640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5440a3bb0 0x7fe5440a6070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe55c199690 tx=0x7fe55002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:45.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.225+0000 7fe55a7fc640 1 -- 172.21.15.94:0/1223315046 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe554016030 con 0x7fe55c100630 2024-04-21T08:39:45.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.404+0000 7fe5634a8640 1 -- 172.21.15.94:0/1223315046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe55c1a0a70 con 0x7fe55c100630 2024-04-21T08:39:45.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.405+0000 7fe55a7fc640 1 -- 172.21.15.94:0/1223315046 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24890 (secure 0 0 0) 0x7fe55409d670 con 0x7fe55c100630 2024-04-21T08:39:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:45.406 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":93,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:31.669420+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:39:45.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.407+0000 7fe5634a8640 1 -- 172.21.15.94:0/1223315046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5440a3bb0 msgr2=0x7fe5440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.407+0000 7fe5634a8640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5440a3bb0 0x7fe5440a6070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe55c199690 tx=0x7fe55002c7e0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.407+0000 7fe5634a8640 1 -- 172.21.15.94:0/1223315046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe55c100630 msgr2=0x7fe55c198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.407+0000 7fe5634a8640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe55c100630 0x7fe55c198180 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe55400ea80 tx=0x7fe55400ef40 comp rx=0 tx=0).stop 2024-04-21T08:39:45.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.408+0000 7fe5634a8640 1 -- 172.21.15.94:0/1223315046 shutdown_connections 2024-04-21T08:39:45.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.408+0000 7fe5634a8640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fe5440a3bb0 0x7fe5440a6070 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.408+0000 7fe5634a8640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1021e0 0x7fe55c19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.408+0000 7fe5634a8640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe55c101820 0x7fe55c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.408+0000 7fe5634a8640 1 --2- 172.21.15.94:0/1223315046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe55c100630 0x7fe55c198180 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.408+0000 7fe5634a8640 1 -- 172.21.15.94:0/1223315046 >> 172.21.15.94:0/1223315046 conn(0x7fe55c0fbba0 msgr2=0x7fe55c0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:45.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.408+0000 7fe5634a8640 1 -- 172.21.15.94:0/1223315046 shutdown_connections 2024-04-21T08:39:45.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.408+0000 7fe5634a8640 1 -- 172.21.15.94:0/1223315046 wait complete. 2024-04-21T08:39:45.419 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:39:45.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:39:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.522+0000 7fc6d9df4640 1 -- 172.21.15.94:0/1740949063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d4101fa0 msgr2=0x7fc6d41023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.522+0000 7fc6d9df4640 1 --2- 172.21.15.94:0/1740949063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d4101fa0 0x7fc6d41023c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009900 tx=0x7fc6c402bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.522+0000 7fc6d9df4640 1 -- 172.21.15.94:0/1740949063 shutdown_connections 2024-04-21T08:39:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.522+0000 7fc6d9df4640 1 --2- 172.21.15.94:0/1740949063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6d4103190 0x7fc6d4103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.522+0000 7fc6d9df4640 1 --2- 172.21.15.94:0/1740949063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d4103b50 0x7fc6d410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.522+0000 7fc6d9df4640 1 --2- 172.21.15.94:0/1740949063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d4101fa0 0x7fc6d41023c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.522+0000 7fc6d9df4640 1 -- 172.21.15.94:0/1740949063 >> 172.21.15.94:0/1740949063 conn(0x7fc6d40fd530 msgr2=0x7fc6d40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.522+0000 7fc6d9df4640 1 -- 172.21.15.94:0/1740949063 shutdown_connections 2024-04-21T08:39:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.522+0000 7fc6d9df4640 1 -- 172.21.15.94:0/1740949063 wait complete. 2024-04-21T08:39:45.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.522+0000 7fc6d9df4640 1 Processor -- start 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d9df4640 1 -- start start 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d9df4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d4101fa0 0x7fc6d41981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d9df4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6d4103190 0x7fc6d41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d9df4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d4103b50 0x7fc6d419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d9df4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410dae0 con 0x7fc6d4101fa0 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d9df4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410d960 con 0x7fc6d4103b50 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d9df4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410dc60 con 0x7fc6d4103190 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d4101fa0 0x7fc6d41981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d4101fa0 0x7fc6d41981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37256/0 (socket says 172.21.15.94:37256) 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d37fe640 1 -- 172.21.15.94:0/2815508315 learned_addr learned my addr 172.21.15.94:0/2815508315 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d2ffd640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6d4103190 0x7fc6d41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d3fff640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d4103b50 0x7fc6d419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d37fe640 1 -- 172.21.15.94:0/2815508315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6d4103190 msgr2=0x7fc6d41986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d37fe640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6d4103190 0x7fc6d41986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d37fe640 1 -- 172.21.15.94:0/2815508315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d4103b50 msgr2=0x7fc6d419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d37fe640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d4103b50 0x7fc6d419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d37fe640 1 -- 172.21.15.94:0/2815508315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d419fdc0 con 0x7fc6d4101fa0 2024-04-21T08:39:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.523+0000 7fc6d37fe640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d4101fa0 0x7fc6d41981a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4036040 tx=0x7fc6c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.524+0000 7fc6d0ff9640 1 -- 172.21.15.94:0/2815508315 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4043020 con 0x7fc6d4101fa0 2024-04-21T08:39:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.524+0000 7fc6d0ff9640 1 -- 172.21.15.94:0/2815508315 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6c4004aa0 con 0x7fc6d4101fa0 2024-04-21T08:39:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.524+0000 7fc6d9df4640 1 -- 172.21.15.94:0/2815508315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d41a0050 con 0x7fc6d4101fa0 2024-04-21T08:39:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.524+0000 7fc6d0ff9640 1 -- 172.21.15.94:0/2815508315 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4003ce0 con 0x7fc6d4101fa0 2024-04-21T08:39:45.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.524+0000 7fc6d9df4640 1 -- 172.21.15.94:0/2815508315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d41a05a0 con 0x7fc6d4101fa0 2024-04-21T08:39:45.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.525+0000 7fc6d9df4640 1 -- 172.21.15.94:0/2815508315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d4109390 con 0x7fc6d4101fa0 2024-04-21T08:39:45.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.526+0000 7fc6d0ff9640 1 -- 172.21.15.94:0/2815508315 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc6c4046020 con 0x7fc6d4101fa0 2024-04-21T08:39:45.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.526+0000 7fc6d0ff9640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc6bc0a3bb0 0x7fc6bc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.526+0000 7fc6d2ffd640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc6bc0a3bb0 0x7fc6bc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.526+0000 7fc6d0ff9640 1 -- 172.21.15.94:0/2815508315 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fc6c40f2c10 con 0x7fc6d4101fa0 2024-04-21T08:39:45.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.527+0000 7fc6d2ffd640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc6bc0a3bb0 0x7fc6bc0a6070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc6d41996b0 tx=0x7fc6c800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.531+0000 7fc6d0ff9640 1 -- 172.21.15.94:0/2815508315 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc6c404d020 con 0x7fc6d4101fa0 2024-04-21T08:39:45.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.708+0000 7fc6d9df4640 1 -- 172.21.15.94:0/2815508315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc6d4062b50 con 0x7fc6d4101fa0 2024-04-21T08:39:45.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.711+0000 7fc6d0ff9640 1 -- 172.21.15.94:0/2815508315 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fc6c40bf1f0 con 0x7fc6d4101fa0 2024-04-21T08:39:45.711 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:39:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.712+0000 7fc6d9df4640 1 -- 172.21.15.94:0/2815508315 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc6bc0a3bb0 msgr2=0x7fc6bc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.712+0000 7fc6d9df4640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc6bc0a3bb0 0x7fc6bc0a6070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc6d41996b0 tx=0x7fc6c800a2d0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.712+0000 7fc6d9df4640 1 -- 172.21.15.94:0/2815508315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d4101fa0 msgr2=0x7fc6d41981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.712+0000 7fc6d9df4640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d4101fa0 0x7fc6d41981a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4036040 tx=0x7fc6c40042f0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.713+0000 7fc6d9df4640 1 -- 172.21.15.94:0/2815508315 shutdown_connections 2024-04-21T08:39:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.713+0000 7fc6d9df4640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc6bc0a3bb0 0x7fc6bc0a6070 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.713+0000 7fc6d9df4640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6d4103190 0x7fc6d41986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.713+0000 7fc6d9df4640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d4103b50 0x7fc6d419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.713+0000 7fc6d9df4640 1 --2- 172.21.15.94:0/2815508315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d4101fa0 0x7fc6d41981a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.713+0000 7fc6d9df4640 1 -- 172.21.15.94:0/2815508315 >> 172.21.15.94:0/2815508315 conn(0x7fc6d40fd530 msgr2=0x7fc6d4106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.713+0000 7fc6d9df4640 1 -- 172.21.15.94:0/2815508315 shutdown_connections 2024-04-21T08:39:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.713+0000 7fc6d9df4640 1 -- 172.21.15.94:0/2815508315 wait complete. 2024-04-21T08:39:45.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:39:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.825+0000 7f4d151ca640 1 -- 172.21.15.94:0/3555757774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d10101fa0 msgr2=0x7f4d101023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.825+0000 7f4d151ca640 1 --2- 172.21.15.94:0/3555757774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d10101fa0 0x7f4d101023c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009900 tx=0x7f4d0002bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.825+0000 7f4d151ca640 1 -- 172.21.15.94:0/3555757774 shutdown_connections 2024-04-21T08:39:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.825+0000 7f4d151ca640 1 --2- 172.21.15.94:0/3555757774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d10103b50 0x7f4d1010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.825+0000 7f4d151ca640 1 --2- 172.21.15.94:0/3555757774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d10103190 0x7f4d10103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.825+0000 7f4d151ca640 1 --2- 172.21.15.94:0/3555757774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d10101fa0 0x7f4d101023c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.825+0000 7f4d151ca640 1 -- 172.21.15.94:0/3555757774 >> 172.21.15.94:0/3555757774 conn(0x7f4d100fd530 msgr2=0x7f4d100ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.825+0000 7f4d151ca640 1 -- 172.21.15.94:0/3555757774 shutdown_connections 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.825+0000 7f4d151ca640 1 -- 172.21.15.94:0/3555757774 wait complete. 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d151ca640 1 Processor -- start 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d151ca640 1 -- start start 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d151ca640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d10101fa0 0x7f4d101981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d151ca640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d10103190 0x7f4d101986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d151ca640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d10103b50 0x7f4d1019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d151ca640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1010dae0 con 0x7f4d10103b50 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d151ca640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1010d960 con 0x7f4d10101fa0 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d151ca640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1010dc60 con 0x7f4d10103190 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d0f59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d10103b50 0x7f4d1019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d0f59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d10103b50 0x7f4d1019f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37264/0 (socket says 172.21.15.94:37264) 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d0f59e640 1 -- 172.21.15.94:0/341676671 learned_addr learned my addr 172.21.15.94:0/341676671 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:45.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d0ed9d640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d10101fa0 0x7f4d101981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d0f59e640 1 -- 172.21.15.94:0/341676671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d10103190 msgr2=0x7f4d101986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d0f59e640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d10103190 0x7f4d101986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d0f59e640 1 -- 172.21.15.94:0/341676671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d10101fa0 msgr2=0x7f4d101981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d0f59e640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d10101fa0 0x7f4d101981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d0f59e640 1 -- 172.21.15.94:0/341676671 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d1019fdc0 con 0x7f4d10103b50 2024-04-21T08:39:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.826+0000 7f4d0f59e640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d10103b50 0x7f4d1019f750 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800cad0 tx=0x7f4d0800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.827+0000 7f4d0cd99640 1 -- 172.21.15.94:0/341676671 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800ae20 con 0x7f4d10103b50 2024-04-21T08:39:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.827+0000 7f4d151ca640 1 -- 172.21.15.94:0/341676671 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d101a00b0 con 0x7f4d10103b50 2024-04-21T08:39:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.827+0000 7f4d0cd99640 1 -- 172.21.15.94:0/341676671 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d08005020 con 0x7f4d10103b50 2024-04-21T08:39:45.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.827+0000 7f4d0cd99640 1 -- 172.21.15.94:0/341676671 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800f6b0 con 0x7f4d10103b50 2024-04-21T08:39:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.827+0000 7f4d151ca640 1 -- 172.21.15.94:0/341676671 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d101a0600 con 0x7f4d10103b50 2024-04-21T08:39:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.827+0000 7f4d151ca640 1 -- 172.21.15.94:0/341676671 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d10109340 con 0x7f4d10103b50 2024-04-21T08:39:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.829+0000 7f4d0cd99640 1 -- 172.21.15.94:0/341676671 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4d080051d0 con 0x7f4d10103b50 2024-04-21T08:39:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.829+0000 7f4d0cd99640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4cfc0a3bb0 0x7f4cfc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.829+0000 7f4d0ed9d640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4cfc0a3bb0 0x7f4cfc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.829+0000 7f4d0cd99640 1 -- 172.21.15.94:0/341676671 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f4d080d0c70 con 0x7f4d10103b50 2024-04-21T08:39:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.830+0000 7f4d0ed9d640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4cfc0a3bb0 0x7f4cfc0a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4d00006010 tx=0x7f4d0002c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:45.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:45.833+0000 7f4d0cd99640 1 -- 172.21.15.94:0/341676671 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d08014030 con 0x7f4d10103b50 2024-04-21T08:39:46.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.009+0000 7f4d151ca640 1 -- 172.21.15.94:0/341676671 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4d1002d810 con 0x7f4d10103b50 2024-04-21T08:39:46.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.009+0000 7f4d0cd99640 1 -- 172.21.15.94:0/341676671 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f4d0809d250 con 0x7f4d10103b50 2024-04-21T08:39:46.009 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:39:46.009 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:45 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1223315046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:46.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:45 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1223315046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:46.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 -- 172.21.15.94:0/341676671 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4cfc0a3bb0 msgr2=0x7f4cfc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4cfc0a3bb0 0x7f4cfc0a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4d00006010 tx=0x7f4d0002c800 comp rx=0 tx=0).stop 2024-04-21T08:39:46.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 -- 172.21.15.94:0/341676671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d10103b50 msgr2=0x7f4d1019f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d10103b50 0x7f4d1019f750 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800cad0 tx=0x7f4d0800cf90 comp rx=0 tx=0).stop 2024-04-21T08:39:46.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 -- 172.21.15.94:0/341676671 shutdown_connections 2024-04-21T08:39:46.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f4cfc0a3bb0 0x7f4cfc0a6070 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d10103190 0x7f4d101986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d10103b50 0x7f4d1019f750 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 --2- 172.21.15.94:0/341676671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d10101fa0 0x7f4d101981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 -- 172.21.15.94:0/341676671 >> 172.21.15.94:0/341676671 conn(0x7f4d100fd530 msgr2=0x7f4d10106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:46.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 -- 172.21.15.94:0/341676671 shutdown_connections 2024-04-21T08:39:46.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.011+0000 7f4d151ca640 1 -- 172.21.15.94:0/341676671 wait complete. 2024-04-21T08:39:46.023 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:39:46.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.124+0000 7fd55026d640 1 -- 172.21.15.94:0/493944733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5481031a0 msgr2=0x7fd548103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.124+0000 7fd55026d640 1 --2- 172.21.15.94:0/493944733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5481031a0 0x7fd548103620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd538009900 tx=0x7fd53802bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:46.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.124+0000 7fd55026d640 1 -- 172.21.15.94:0/493944733 shutdown_connections 2024-04-21T08:39:46.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.124+0000 7fd55026d640 1 --2- 172.21.15.94:0/493944733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd548103b60 0x7fd54810a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.124+0000 7fd55026d640 1 --2- 172.21.15.94:0/493944733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5481031a0 0x7fd548103620 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.124+0000 7fd55026d640 1 --2- 172.21.15.94:0/493944733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd548101fb0 0x7fd5481023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.124+0000 7fd55026d640 1 -- 172.21.15.94:0/493944733 >> 172.21.15.94:0/493944733 conn(0x7fd5480fd520 msgr2=0x7fd5480ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:46.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.124+0000 7fd55026d640 1 -- 172.21.15.94:0/493944733 shutdown_connections 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.124+0000 7fd55026d640 1 -- 172.21.15.94:0/493944733 wait complete. 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd55026d640 1 Processor -- start 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd55026d640 1 -- start start 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd55026d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd548101fb0 0x7fd5481982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd55026d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5481031a0 0x7fd5481987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd55026d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd548103b60 0x7fd54819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd55026d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd54810db70 con 0x7fd548103b60 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd55026d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd54810d9f0 con 0x7fd548101fb0 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd55026d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd54810dcf0 con 0x7fd5481031a0 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54e80a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd548103b60 0x7fd54819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54e80a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd548103b60 0x7fd54819f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37290/0 (socket says 172.21.15.94:37290) 2024-04-21T08:39:46.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54e80a640 1 -- 172.21.15.94:0/3321148215 learned_addr learned my addr 172.21.15.94:0/3321148215 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54e009640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd548101fb0 0x7fd5481982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54d808640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5481031a0 0x7fd5481987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54e80a640 1 -- 172.21.15.94:0/3321148215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5481031a0 msgr2=0x7fd5481987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54e80a640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5481031a0 0x7fd5481987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54e80a640 1 -- 172.21.15.94:0/3321148215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd548101fb0 msgr2=0x7fd5481982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54e80a640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd548101fb0 0x7fd5481982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54e80a640 1 -- 172.21.15.94:0/3321148215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54819fec0 con 0x7fd548103b60 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54e009640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd548101fb0 0x7fd5481982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.125+0000 7fd54d808640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5481031a0 0x7fd5481987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.126+0000 7fd54e80a640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd548103b60 0x7fd54819f850 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd54000cad0 tx=0x7fd54000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.126+0000 7fd53f7fe640 1 -- 172.21.15.94:0/3321148215 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54000ae20 con 0x7fd548103b60 2024-04-21T08:39:46.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.126+0000 7fd53f7fe640 1 -- 172.21.15.94:0/3321148215 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd540005020 con 0x7fd548103b60 2024-04-21T08:39:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.126+0000 7fd55026d640 1 -- 172.21.15.94:0/3321148215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5481a01b0 con 0x7fd548103b60 2024-04-21T08:39:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.126+0000 7fd53f7fe640 1 -- 172.21.15.94:0/3321148215 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54000f6b0 con 0x7fd548103b60 2024-04-21T08:39:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.126+0000 7fd55026d640 1 -- 172.21.15.94:0/3321148215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5481a0780 con 0x7fd548103b60 2024-04-21T08:39:46.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.127+0000 7fd55026d640 1 -- 172.21.15.94:0/3321148215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd548109350 con 0x7fd548103b60 2024-04-21T08:39:46.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.128+0000 7fd53f7fe640 1 -- 172.21.15.94:0/3321148215 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd5400051d0 con 0x7fd548103b60 2024-04-21T08:39:46.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.129+0000 7fd53f7fe640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd5340a3bb0 0x7fd5340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.129+0000 7fd54e009640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd5340a3bb0 0x7fd5340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.129+0000 7fd53f7fe640 1 -- 172.21.15.94:0/3321148215 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fd5400d0c70 con 0x7fd548103b60 2024-04-21T08:39:46.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.130+0000 7fd54e009640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd5340a3bb0 0x7fd5340a6070 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd548103000 tx=0x7fd544008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:46.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.133+0000 7fd53f7fe640 1 -- 172.21.15.94:0/3321148215 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd540014030 con 0x7fd548103b60 2024-04-21T08:39:46.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:45 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1223315046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:46.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.311+0000 7fd55026d640 1 -- 172.21.15.94:0/3321148215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd54802d5c0 con 0x7fd548103b60 2024-04-21T08:39:46.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.311+0000 7fd53f7fe640 1 -- 172.21.15.94:0/3321148215 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fd54009d250 con 0x7fd548103b60 2024-04-21T08:39:46.311 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:39:46.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 -- 172.21.15.94:0/3321148215 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd5340a3bb0 msgr2=0x7fd5340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd5340a3bb0 0x7fd5340a6070 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd548103000 tx=0x7fd544008040 comp rx=0 tx=0).stop 2024-04-21T08:39:46.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 -- 172.21.15.94:0/3321148215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd548103b60 msgr2=0x7fd54819f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd548103b60 0x7fd54819f850 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd54000cad0 tx=0x7fd54000cf90 comp rx=0 tx=0).stop 2024-04-21T08:39:46.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 -- 172.21.15.94:0/3321148215 shutdown_connections 2024-04-21T08:39:46.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fd5340a3bb0 0x7fd5340a6070 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5481031a0 0x7fd5481987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd548103b60 0x7fd54819f850 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 --2- 172.21.15.94:0/3321148215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd548101fb0 0x7fd5481982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 -- 172.21.15.94:0/3321148215 >> 172.21.15.94:0/3321148215 conn(0x7fd5480fd520 msgr2=0x7fd548106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:46.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 -- 172.21.15.94:0/3321148215 shutdown_connections 2024-04-21T08:39:46.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.314+0000 7fd55026d640 1 -- 172.21.15.94:0/3321148215 wait complete. 2024-04-21T08:39:46.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:39:46.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.427+0000 7fb83e9b2640 1 -- 172.21.15.94:0/2678521475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb838105380 msgr2=0x7fb838107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.427+0000 7fb83e9b2640 1 --2- 172.21.15.94:0/2678521475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb838105380 0x7fb838107770 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009900 tx=0x7fb82c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:46.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.427+0000 7fb83e9b2640 1 -- 172.21.15.94:0/2678521475 shutdown_connections 2024-04-21T08:39:46.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.427+0000 7fb83e9b2640 1 --2- 172.21.15.94:0/2678521475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838107cb0 0x7fb83810a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.427+0000 7fb83e9b2640 1 --2- 172.21.15.94:0/2678521475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb838105380 0x7fb838107770 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.427+0000 7fb83e9b2640 1 --2- 172.21.15.94:0/2678521475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb838102a50 0x7fb838104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.427+0000 7fb83e9b2640 1 -- 172.21.15.94:0/2678521475 >> 172.21.15.94:0/2678521475 conn(0x7fb8380fc400 msgr2=0x7fb8380fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:46.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.427+0000 7fb83e9b2640 1 -- 172.21.15.94:0/2678521475 shutdown_connections 2024-04-21T08:39:46.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.427+0000 7fb83e9b2640 1 -- 172.21.15.94:0/2678521475 wait complete. 2024-04-21T08:39:46.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb83e9b2640 1 Processor -- start 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb83e9b2640 1 -- start start 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb83e9b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb838102a50 0x7fb8381982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb83e9b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb838105380 0x7fb8381987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb83e9b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838107cb0 0x7fb83819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb83e9b2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb83810dac0 con 0x7fb838102a50 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb83e9b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb83810d940 con 0x7fb838107cb0 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb83e9b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb83810dc40 con 0x7fb838105380 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb837fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb838102a50 0x7fb8381982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb837fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb838102a50 0x7fb8381982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37306/0 (socket says 172.21.15.94:37306) 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb837fff640 1 -- 172.21.15.94:0/3414625069 learned_addr learned my addr 172.21.15.94:0/3414625069 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb8377fe640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb838105380 0x7fb8381987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.428+0000 7fb83cf4f640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838107cb0 0x7fb83819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb837fff640 1 -- 172.21.15.94:0/3414625069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb838105380 msgr2=0x7fb8381987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb837fff640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb838105380 0x7fb8381987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb837fff640 1 -- 172.21.15.94:0/3414625069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838107cb0 msgr2=0x7fb83819f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb837fff640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838107cb0 0x7fb83819f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb837fff640 1 -- 172.21.15.94:0/3414625069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83819fed0 con 0x7fb838102a50 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb83cf4f640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838107cb0 0x7fb83819f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb837fff640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb838102a50 0x7fb8381982b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb82800e9f0 tx=0x7fb82800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb8357fa640 1 -- 172.21.15.94:0/3414625069 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82800cd80 con 0x7fb838102a50 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb8357fa640 1 -- 172.21.15.94:0/3414625069 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8280054d0 con 0x7fb838102a50 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb83e9b2640 1 -- 172.21.15.94:0/3414625069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8381a01c0 con 0x7fb838102a50 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb8357fa640 1 -- 172.21.15.94:0/3414625069 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb828010690 con 0x7fb838102a50 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.429+0000 7fb83e9b2640 1 -- 172.21.15.94:0/3414625069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8381a06e0 con 0x7fb838102a50 2024-04-21T08:39:46.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.430+0000 7fb83e9b2640 1 -- 172.21.15.94:0/3414625069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8380fdef0 con 0x7fb838102a50 2024-04-21T08:39:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.431+0000 7fb8357fa640 1 -- 172.21.15.94:0/3414625069 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb828004e90 con 0x7fb838102a50 2024-04-21T08:39:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.432+0000 7fb8357fa640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8200a3c30 0x7fb8200a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.432+0000 7fb8377fe640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8200a3c30 0x7fb8200a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.432+0000 7fb8357fa640 1 -- 172.21.15.94:0/3414625069 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fb8280d0c40 con 0x7fb838102a50 2024-04-21T08:39:46.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.432+0000 7fb8377fe640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8200a3c30 0x7fb8200a60f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb82c005ab0 tx=0x7fb82c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:46.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.436+0000 7fb8357fa640 1 -- 172.21.15.94:0/3414625069 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb82809d220 con 0x7fb838102a50 2024-04-21T08:39:46.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.614+0000 7fb83e9b2640 1 -- 172.21.15.94:0/3414625069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb838062b50 con 0x7fb838102a50 2024-04-21T08:39:46.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.615+0000 7fb8357fa640 1 -- 172.21.15.94:0/3414625069 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fb82809cb20 con 0x7fb838102a50 2024-04-21T08:39:46.615 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:39:46.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 -- 172.21.15.94:0/3414625069 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8200a3c30 msgr2=0x7fb8200a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8200a3c30 0x7fb8200a60f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb82c005ab0 tx=0x7fb82c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 -- 172.21.15.94:0/3414625069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb838102a50 msgr2=0x7fb8381982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb838102a50 0x7fb8381982b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb82800e9f0 tx=0x7fb82800edf0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 -- 172.21.15.94:0/3414625069 shutdown_connections 2024-04-21T08:39:46.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb8200a3c30 0x7fb8200a60f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb838105380 0x7fb8381987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838107cb0 0x7fb83819f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 --2- 172.21.15.94:0/3414625069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb838102a50 0x7fb8381982b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 -- 172.21.15.94:0/3414625069 >> 172.21.15.94:0/3414625069 conn(0x7fb8380fc400 msgr2=0x7fb838103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:46.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 -- 172.21.15.94:0/3414625069 shutdown_connections 2024-04-21T08:39:46.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.617+0000 7fb83e9b2640 1 -- 172.21.15.94:0/3414625069 wait complete. 2024-04-21T08:39:46.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:39:46.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.727+0000 7f21a4481640 1 -- 172.21.15.94:0/2684852391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f219c101fe0 msgr2=0x7f219c102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.727+0000 7f21a4481640 1 --2- 172.21.15.94:0/2684852391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f219c101fe0 0x7f219c102460 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f218c009900 tx=0x7f218c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:46.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.727+0000 7f21a4481640 1 -- 172.21.15.94:0/2684852391 shutdown_connections 2024-04-21T08:39:46.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.727+0000 7f21a4481640 1 --2- 172.21.15.94:0/2684852391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c1029a0 0x7f219c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.727+0000 7f21a4481640 1 --2- 172.21.15.94:0/2684852391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f219c101fe0 0x7f219c102460 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 --2- 172.21.15.94:0/2684852391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f219c100e90 0x7f219c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 -- 172.21.15.94:0/2684852391 >> 172.21.15.94:0/2684852391 conn(0x7f219c0fc400 msgr2=0x7f219c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:46.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 -- 172.21.15.94:0/2684852391 shutdown_connections 2024-04-21T08:39:46.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 -- 172.21.15.94:0/2684852391 wait complete. 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 Processor -- start 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 -- start start 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f219c100e90 0x7f219c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c101fe0 0x7f219c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f219c1029a0 0x7f219c19f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f219c10cac0 con 0x7f219c100e90 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f219c10c940 con 0x7f219c101fe0 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a4481640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f219c10cc40 con 0x7f219c1029a0 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a221d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f219c100e90 0x7f219c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a221d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f219c100e90 0x7f219c198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37332/0 (socket says 172.21.15.94:37332) 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.728+0000 7f21a221d640 1 -- 172.21.15.94:0/3084752218 learned_addr learned my addr 172.21.15.94:0/3084752218 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a2a1e640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f219c1029a0 0x7f219c19f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a1a1c640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c101fe0 0x7f219c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a221d640 1 -- 172.21.15.94:0/3084752218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f219c1029a0 msgr2=0x7f219c19f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a221d640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f219c1029a0 0x7f219c19f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a221d640 1 -- 172.21.15.94:0/3084752218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c101fe0 msgr2=0x7f219c1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a221d640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c101fe0 0x7f219c1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a221d640 1 -- 172.21.15.94:0/3084752218 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f219c19ffa0 con 0x7f219c100e90 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a1a1c640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c101fe0 0x7f219c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a221d640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f219c100e90 0x7f219c198380 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f219800b4a0 tx=0x7f219800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21937fe640 1 -- 172.21.15.94:0/3084752218 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198014070 con 0x7f219c100e90 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21937fe640 1 -- 172.21.15.94:0/3084752218 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2198010ac0 con 0x7f219c100e90 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a4481640 1 -- 172.21.15.94:0/3084752218 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f219c1a0290 con 0x7f219c100e90 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21937fe640 1 -- 172.21.15.94:0/3084752218 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198017c30 con 0x7f219c100e90 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.729+0000 7f21a4481640 1 -- 172.21.15.94:0/3084752218 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f219c1a0860 con 0x7f219c100e90 2024-04-21T08:39:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.730+0000 7f21a4481640 1 -- 172.21.15.94:0/3084752218 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f219c1a0b20 con 0x7f219c100e90 2024-04-21T08:39:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.731+0000 7f21937fe640 1 -- 172.21.15.94:0/3084752218 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2198017070 con 0x7f219c100e90 2024-04-21T08:39:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.732+0000 7f21937fe640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21880a3bb0 0x7f21880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.732+0000 7f21a1a1c640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21880a3bb0 0x7f21880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.732+0000 7f21937fe640 1 -- 172.21.15.94:0/3084752218 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f21980cffe0 con 0x7f219c100e90 2024-04-21T08:39:46.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.732+0000 7f21a1a1c640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21880a3bb0 0x7f21880a6070 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f218c036040 tx=0x7f218c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:46.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.736+0000 7f21937fe640 1 -- 172.21.15.94:0/3084752218 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f219809c5c0 con 0x7f219c100e90 2024-04-21T08:39:46.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.950+0000 7f21a4481640 1 -- 172.21.15.94:0/3084752218 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f219c1a0cf0 con 0x7f219c100e90 2024-04-21T08:39:46.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.951+0000 7f21937fe640 1 -- 172.21.15.94:0/3084752218 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11885 (secure 0 0 0) 0x7f219809bec0 con 0x7f219c100e90 2024-04-21T08:39:46.950 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:46.950 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:37:56.995828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24446,24488]},"id":1}]} 2024-04-21T08:39:46.951 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-04-21T08:39:46.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 -- 172.21.15.94:0/3084752218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21880a3bb0 msgr2=0x7f21880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21880a3bb0 0x7f21880a6070 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f218c036040 tx=0x7f218c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 -- 172.21.15.94:0/3084752218 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f219c100e90 msgr2=0x7f219c198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:46.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f219c100e90 0x7f219c198380 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f219800b4a0 tx=0x7f219800b960 comp rx=0 tx=0).stop 2024-04-21T08:39:46.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 -- 172.21.15.94:0/3084752218 shutdown_connections 2024-04-21T08:39:46.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f21880a3bb0 0x7f21880a6070 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f219c1029a0 0x7f219c19f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c101fe0 0x7f219c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 --2- 172.21.15.94:0/3084752218 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f219c100e90 0x7f219c198380 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:46.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 -- 172.21.15.94:0/3084752218 >> 172.21.15.94:0/3084752218 conn(0x7f219c0fc400 msgr2=0x7f219c06a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:46.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 -- 172.21.15.94:0/3084752218 shutdown_connections 2024-04-21T08:39:46.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:46.953+0000 7f21a4481640 1 -- 172.21.15.94:0/3084752218 wait complete. 2024-04-21T08:39:46.962 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:46 smithi094 ceph-mon[27988]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:46.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:46 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2815508315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:46.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:46 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/341676671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:46.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:46 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3321148215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:46.963 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:46 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3414625069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:46.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:39:47.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:46 smithi173 ceph-mon[31428]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:47.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:46 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2815508315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:47.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:46 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/341676671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:47.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:46 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3321148215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:47.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:46 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3414625069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 -- 172.21.15.94:0/3559998199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9758101170 msgr2=0x7f97581015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 --2- 172.21.15.94:0/3559998199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9758101170 0x7f97581015f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f974c009900 tx=0x7f974c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 -- 172.21.15.94:0/3559998199 shutdown_connections 2024-04-21T08:39:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 --2- 172.21.15.94:0/3559998199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9758101b30 0x7f97581093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 --2- 172.21.15.94:0/3559998199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9758101170 0x7f97581015f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 --2- 172.21.15.94:0/3559998199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9758102440 0x7f9758102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 -- 172.21.15.94:0/3559998199 >> 172.21.15.94:0/3559998199 conn(0x7f97580fbba0 msgr2=0x7f97580fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 -- 172.21.15.94:0/3559998199 shutdown_connections 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 -- 172.21.15.94:0/3559998199 wait complete. 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 Processor -- start 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.063+0000 7f975e54a640 1 -- start start 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f975e54a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9758101170 0x7f97581981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f975e54a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9758101b30 0x7f9758198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f975e54a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9758102440 0x7f975819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f975e54a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f975810dbb0 con 0x7f9758101170 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f975e54a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f975810da30 con 0x7f9758102440 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f975e54a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f975810dd30 con 0x7f9758101b30 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f9757fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9758101170 0x7f97581981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f9757fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9758101170 0x7f97581981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37346/0 (socket says 172.21.15.94:37346) 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f9757fff640 1 -- 172.21.15.94:0/1364728192 learned_addr learned my addr 172.21.15.94:0/1364728192 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f97577fe640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9758101b30 0x7f9758198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f9757fff640 1 -- 172.21.15.94:0/1364728192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9758101b30 msgr2=0x7f9758198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f9757fff640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9758101b30 0x7f9758198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f9757fff640 1 -- 172.21.15.94:0/1364728192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9758102440 msgr2=0x7f975819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f9757fff640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9758102440 0x7f975819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f9757fff640 1 -- 172.21.15.94:0/1364728192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975819fdf0 con 0x7f9758101170 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f9757fff640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9758101170 0x7f97581981d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f974800b450 tx=0x7f974800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.064+0000 7f97557fa640 1 -- 172.21.15.94:0/1364728192 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9748015020 con 0x7f9758101170 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.065+0000 7f97557fa640 1 -- 172.21.15.94:0/1364728192 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9748011d40 con 0x7f9758101170 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.065+0000 7f975e54a640 1 -- 172.21.15.94:0/1364728192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97581a00e0 con 0x7f9758101170 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.065+0000 7f97557fa640 1 -- 172.21.15.94:0/1364728192 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974800f340 con 0x7f9758101170 2024-04-21T08:39:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.065+0000 7f975e54a640 1 -- 172.21.15.94:0/1364728192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97581a0660 con 0x7f9758101170 2024-04-21T08:39:47.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.066+0000 7f975e54a640 1 -- 172.21.15.94:0/1364728192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97581082b0 con 0x7f9758101170 2024-04-21T08:39:47.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.067+0000 7f97557fa640 1 -- 172.21.15.94:0/1364728192 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9748018020 con 0x7f9758101170 2024-04-21T08:39:47.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.067+0000 7f97557fa640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97400a3bb0 0x7f97400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.067+0000 7f97557fa640 1 -- 172.21.15.94:0/1364728192 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f97480d0960 con 0x7f9758101170 2024-04-21T08:39:47.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.067+0000 7f97577fe640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97400a3bb0 0x7f97400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.068+0000 7f97577fe640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97400a3bb0 0x7f97400a6070 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f97581996e0 tx=0x7f974c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:47.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.071+0000 7f97557fa640 1 -- 172.21.15.94:0/1364728192 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f974801f020 con 0x7f9758101170 2024-04-21T08:39:47.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:46 smithi111 ceph-mon[31493]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:39:47.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:46 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2815508315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:47.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:46 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/341676671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:47.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:46 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3321148215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:47.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:46 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3414625069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:47.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.248+0000 7f975e54a640 1 -- 172.21.15.94:0/1364728192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9758062b50 con 0x7f9758101170 2024-04-21T08:39:47.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.250+0000 7f97557fa640 1 -- 172.21.15.94:0/1364728192 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24890 (secure 0 0 0) 0x7f974800d040 con 0x7f9758101170 2024-04-21T08:39:47.248 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:47.249 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":93,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:31.669420+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 -- 172.21.15.94:0/1364728192 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97400a3bb0 msgr2=0x7f97400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97400a3bb0 0x7f97400a6070 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f97581996e0 tx=0x7f974c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 -- 172.21.15.94:0/1364728192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9758101170 msgr2=0x7f97581981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9758101170 0x7f97581981d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f974800b450 tx=0x7f974800b910 comp rx=0 tx=0).stop 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 -- 172.21.15.94:0/1364728192 shutdown_connections 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f97400a3bb0 0x7f97400a6070 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9758101b30 0x7f9758198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9758102440 0x7f975819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 --2- 172.21.15.94:0/1364728192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9758101170 0x7f97581981d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 -- 172.21.15.94:0/1364728192 >> 172.21.15.94:0/1364728192 conn(0x7f97580fbba0 msgr2=0x7f9758105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 -- 172.21.15.94:0/1364728192 shutdown_connections 2024-04-21T08:39:47.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.252+0000 7f975e54a640 1 -- 172.21.15.94:0/1364728192 wait complete. 2024-04-21T08:39:47.262 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:39:47.263 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:39:47.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.366+0000 7f8e7210e640 1 -- 172.21.15.94:0/3275755528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e6c102040 msgr2=0x7f8e6c1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.366+0000 7f8e7210e640 1 --2- 172.21.15.94:0/3275755528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e6c102040 0x7f8e6c1024c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009900 tx=0x7f8e5402bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.366+0000 7f8e7210e640 1 -- 172.21.15.94:0/3275755528 shutdown_connections 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.366+0000 7f8e7210e640 1 --2- 172.21.15.94:0/3275755528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e6c102a00 0x7f8e6c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.366+0000 7f8e7210e640 1 --2- 172.21.15.94:0/3275755528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e6c102040 0x7f8e6c1024c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.366+0000 7f8e7210e640 1 --2- 172.21.15.94:0/3275755528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e6c100e60 0x7f8e6c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.366+0000 7f8e7210e640 1 -- 172.21.15.94:0/3275755528 >> 172.21.15.94:0/3275755528 conn(0x7f8e6c0fc420 msgr2=0x7f8e6c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.366+0000 7f8e7210e640 1 -- 172.21.15.94:0/3275755528 shutdown_connections 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.366+0000 7f8e7210e640 1 -- 172.21.15.94:0/3275755528 wait complete. 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.366+0000 7f8e7210e640 1 Processor -- start 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e7210e640 1 -- start start 2024-04-21T08:39:47.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e7210e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e6c100e60 0x7f8e6c198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e7210e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e6c102040 0x7f8e6c198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e7210e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e6c102a00 0x7f8e6c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e7210e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6c10cad0 con 0x7f8e6c100e60 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e7210e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6c10c950 con 0x7f8e6c102040 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e7210e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6c10cc50 con 0x7f8e6c102a00 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e6c100e60 0x7f8e6c198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e6c100e60 0x7f8e6c198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37362/0 (socket says 172.21.15.94:37362) 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6b7fe640 1 -- 172.21.15.94:0/1787825999 learned_addr learned my addr 172.21.15.94:0/1787825999 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e63fff640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e6c102040 0x7f8e6c198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6bfff640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e6c102a00 0x7f8e6c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6b7fe640 1 -- 172.21.15.94:0/1787825999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e6c102a00 msgr2=0x7f8e6c19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6b7fe640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e6c102a00 0x7f8e6c19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6b7fe640 1 -- 172.21.15.94:0/1787825999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e6c102040 msgr2=0x7f8e6c198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6b7fe640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e6c102040 0x7f8e6c198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6b7fe640 1 -- 172.21.15.94:0/1787825999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6c19fed0 con 0x7f8e6c100e60 2024-04-21T08:39:47.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6bfff640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e6c102a00 0x7f8e6c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:47.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e63fff640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e6c102040 0x7f8e6c198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:47.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e6b7fe640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e6c100e60 0x7f8e6c198320 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00c970 tx=0x7f8e5c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:47.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e697fa640 1 -- 172.21.15.94:0/1787825999 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c004220 con 0x7f8e6c100e60 2024-04-21T08:39:47.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e697fa640 1 -- 172.21.15.94:0/1787825999 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e5c0043c0 con 0x7f8e6c100e60 2024-04-21T08:39:47.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e697fa640 1 -- 172.21.15.94:0/1787825999 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c004a00 con 0x7f8e6c100e60 2024-04-21T08:39:47.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e7210e640 1 -- 172.21.15.94:0/1787825999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e6c1a01c0 con 0x7f8e6c100e60 2024-04-21T08:39:47.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.367+0000 7f8e7210e640 1 -- 172.21.15.94:0/1787825999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e6c1a0790 con 0x7f8e6c100e60 2024-04-21T08:39:47.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.369+0000 7f8e7210e640 1 -- 172.21.15.94:0/1787825999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e6c1a0a50 con 0x7f8e6c100e60 2024-04-21T08:39:47.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.369+0000 7f8e697fa640 1 -- 172.21.15.94:0/1787825999 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8e5c0079b0 con 0x7f8e6c100e60 2024-04-21T08:39:47.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.370+0000 7f8e697fa640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8e580a3c30 0x7f8e580a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.370+0000 7f8e63fff640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8e580a3c30 0x7f8e580a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.370+0000 7f8e697fa640 1 -- 172.21.15.94:0/1787825999 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f8e5c0d05c0 con 0x7f8e6c100e60 2024-04-21T08:39:47.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.371+0000 7f8e63fff640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8e580a3c30 0x7f8e580a60f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c1997c0 tx=0x7f8e54005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:47.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.375+0000 7f8e697fa640 1 -- 172.21.15.94:0/1787825999 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e5c014030 con 0x7f8e6c100e60 2024-04-21T08:39:47.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.556+0000 7f8e7210e640 1 -- 172.21.15.94:0/1787825999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8e6c1a0c20 con 0x7f8e6c100e60 2024-04-21T08:39:47.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.556+0000 7f8e697fa640 1 -- 172.21.15.94:0/1787825999 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f8e5c09cba0 con 0x7f8e6c100e60 2024-04-21T08:39:47.555 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:39:47.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 -- 172.21.15.94:0/1787825999 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8e580a3c30 msgr2=0x7f8e580a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8e580a3c30 0x7f8e580a60f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c1997c0 tx=0x7f8e54005fd0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 -- 172.21.15.94:0/1787825999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e6c100e60 msgr2=0x7f8e6c198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e6c100e60 0x7f8e6c198320 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00c970 tx=0x7f8e5c00ce30 comp rx=0 tx=0).stop 2024-04-21T08:39:47.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 -- 172.21.15.94:0/1787825999 shutdown_connections 2024-04-21T08:39:47.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f8e580a3c30 0x7f8e580a60f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e6c102a00 0x7f8e6c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e6c102040 0x7f8e6c198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 --2- 172.21.15.94:0/1787825999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e6c100e60 0x7f8e6c198320 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 -- 172.21.15.94:0/1787825999 >> 172.21.15.94:0/1787825999 conn(0x7f8e6c0fc420 msgr2=0x7f8e6c0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:47.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 -- 172.21.15.94:0/1787825999 shutdown_connections 2024-04-21T08:39:47.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.558+0000 7f8e7210e640 1 -- 172.21.15.94:0/1787825999 wait complete. 2024-04-21T08:39:47.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:39:47.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 -- 172.21.15.94:0/2706524253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b08103190 msgr2=0x7f3b08103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 --2- 172.21.15.94:0/2706524253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b08103190 0x7f3b08103610 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009900 tx=0x7f3af802bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:47.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 -- 172.21.15.94:0/2706524253 shutdown_connections 2024-04-21T08:39:47.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 --2- 172.21.15.94:0/2706524253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b08103b50 0x7f3b0810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 --2- 172.21.15.94:0/2706524253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b08103190 0x7f3b08103610 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 --2- 172.21.15.94:0/2706524253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b08101fa0 0x7f3b081023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 -- 172.21.15.94:0/2706524253 >> 172.21.15.94:0/2706524253 conn(0x7f3b080fd530 msgr2=0x7f3b080ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:47.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 -- 172.21.15.94:0/2706524253 shutdown_connections 2024-04-21T08:39:47.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 -- 172.21.15.94:0/2706524253 wait complete. 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 Processor -- start 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 -- start start 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b08101fa0 0x7f3b08198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b08103190 0x7f3b081986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b08103b50 0x7f3b0819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0810dae0 con 0x7f3b08103b50 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0810d960 con 0x7f3b08103190 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b1045c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0810dc60 con 0x7f3b08101fa0 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.672+0000 7f3b0e9f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b08103b50 0x7f3b0819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0e9f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b08103b50 0x7f3b0819f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37382/0 (socket says 172.21.15.94:37382) 2024-04-21T08:39:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0e9f9640 1 -- 172.21.15.94:0/4042974687 learned_addr learned my addr 172.21.15.94:0/4042974687 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0e1f8640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b08101fa0 0x7f3b08198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0d9f7640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b08103190 0x7f3b081986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0e9f9640 1 -- 172.21.15.94:0/4042974687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b08101fa0 msgr2=0x7f3b08198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0e9f9640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b08101fa0 0x7f3b08198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0e9f9640 1 -- 172.21.15.94:0/4042974687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b08103190 msgr2=0x7f3b081986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0e9f9640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b08103190 0x7f3b081986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0e9f9640 1 -- 172.21.15.94:0/4042974687 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0819fda0 con 0x7f3b08103b50 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0d9f7640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b08103190 0x7f3b081986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b0e9f9640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b08103b50 0x7f3b0819f730 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000c970 tx=0x7f3b0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3aff7fe640 1 -- 172.21.15.94:0/4042974687 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b00007bb0 con 0x7f3b08103b50 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3aff7fe640 1 -- 172.21.15.94:0/4042974687 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b00007d50 con 0x7f3b08103b50 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b1045c640 1 -- 172.21.15.94:0/4042974687 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b081a0090 con 0x7f3b08103b50 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3aff7fe640 1 -- 172.21.15.94:0/4042974687 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b00018770 con 0x7f3b08103b50 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.673+0000 7f3b1045c640 1 -- 172.21.15.94:0/4042974687 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b081a05e0 con 0x7f3b08103b50 2024-04-21T08:39:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.674+0000 7f3b1045c640 1 -- 172.21.15.94:0/4042974687 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b08109390 con 0x7f3b08103b50 2024-04-21T08:39:47.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.675+0000 7f3aff7fe640 1 -- 172.21.15.94:0/4042974687 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3b0000f990 con 0x7f3b08103b50 2024-04-21T08:39:47.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.676+0000 7f3aff7fe640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3af40a3c30 0x7f3af40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.676+0000 7f3b0e1f8640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3af40a3c30 0x7f3af40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.676+0000 7f3aff7fe640 1 -- 172.21.15.94:0/4042974687 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f3b000d0900 con 0x7f3b08103b50 2024-04-21T08:39:47.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.676+0000 7f3b0e1f8640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3af40a3c30 0x7f3af40a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3b08102ff0 tx=0x7f3b04009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:47.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.680+0000 7f3aff7fe640 1 -- 172.21.15.94:0/4042974687 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b00014030 con 0x7f3b08103b50 2024-04-21T08:39:47.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.857+0000 7f3b1045c640 1 -- 172.21.15.94:0/4042974687 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3b08062b50 con 0x7f3b08103b50 2024-04-21T08:39:47.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.857+0000 7f3aff7fe640 1 -- 172.21.15.94:0/4042974687 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f3b0009cff0 con 0x7f3b08103b50 2024-04-21T08:39:47.856 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:39:47.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 -- 172.21.15.94:0/4042974687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3af40a3c30 msgr2=0x7f3af40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3af40a3c30 0x7f3af40a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3b08102ff0 tx=0x7f3b04009680 comp rx=0 tx=0).stop 2024-04-21T08:39:47.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 -- 172.21.15.94:0/4042974687 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b08103b50 msgr2=0x7f3b0819f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b08103b50 0x7f3b0819f730 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000c970 tx=0x7f3b0000ce30 comp rx=0 tx=0).stop 2024-04-21T08:39:47.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 -- 172.21.15.94:0/4042974687 shutdown_connections 2024-04-21T08:39:47.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3af40a3c30 0x7f3af40a60f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b08103b50 0x7f3b0819f730 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b08103190 0x7f3b081986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 --2- 172.21.15.94:0/4042974687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b08101fa0 0x7f3b08198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 -- 172.21.15.94:0/4042974687 >> 172.21.15.94:0/4042974687 conn(0x7f3b080fd530 msgr2=0x7f3b08106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:47.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 -- 172.21.15.94:0/4042974687 shutdown_connections 2024-04-21T08:39:47.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.859+0000 7f3b1045c640 1 -- 172.21.15.94:0/4042974687 wait complete. 2024-04-21T08:39:47.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:47 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3084752218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:47.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:47 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1364728192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:47.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:47 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1787825999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:47.871 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:39:47.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.973+0000 7f029b95f640 1 -- 172.21.15.94:0/3501887996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0294101fa0 msgr2=0x7f02941023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.973+0000 7f029b95f640 1 --2- 172.21.15.94:0/3501887996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0294101fa0 0x7f02941023c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f028c009ca0 tx=0x7f028c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 -- 172.21.15.94:0/3501887996 shutdown_connections 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 --2- 172.21.15.94:0/3501887996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0294103190 0x7f0294103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 --2- 172.21.15.94:0/3501887996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0294103b50 0x7f029410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 --2- 172.21.15.94:0/3501887996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0294101fa0 0x7f02941023c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 -- 172.21.15.94:0/3501887996 >> 172.21.15.94:0/3501887996 conn(0x7f02940fd530 msgr2=0x7f02940ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 -- 172.21.15.94:0/3501887996 shutdown_connections 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 -- 172.21.15.94:0/3501887996 wait complete. 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 Processor -- start 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 -- start start 2024-04-21T08:39:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0294101fa0 0x7f0294198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0294103190 0x7f0294198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0294103b50 0x7f029419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f029410dba0 con 0x7f0294103b50 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f029410da20 con 0x7f0294103190 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f029b95f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f029410dd20 con 0x7f0294101fa0 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f0299efc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0294103b50 0x7f029419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f0299efc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0294103b50 0x7f029419f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37404/0 (socket says 172.21.15.94:37404) 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.974+0000 7f0299efc640 1 -- 172.21.15.94:0/422712153 learned_addr learned my addr 172.21.15.94:0/422712153 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f02996fb640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0294101fa0 0x7f0294198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f0298efa640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0294103190 0x7f0294198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f0299efc640 1 -- 172.21.15.94:0/422712153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0294101fa0 msgr2=0x7f0294198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f0299efc640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0294101fa0 0x7f0294198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f0299efc640 1 -- 172.21.15.94:0/422712153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0294103190 msgr2=0x7f0294198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f0299efc640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0294103190 0x7f0294198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f0299efc640 1 -- 172.21.15.94:0/422712153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f029419ff40 con 0x7f0294103b50 2024-04-21T08:39:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f0299efc640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0294103b50 0x7f029419f8d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0284009fd0 tx=0x7f028400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f02927fc640 1 -- 172.21.15.94:0/422712153 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284011830 con 0x7f0294103b50 2024-04-21T08:39:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f02927fc640 1 -- 172.21.15.94:0/422712153 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0284016070 con 0x7f0294103b50 2024-04-21T08:39:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f02927fc640 1 -- 172.21.15.94:0/422712153 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284010de0 con 0x7f0294103b50 2024-04-21T08:39:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f029b95f640 1 -- 172.21.15.94:0/422712153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02941a0230 con 0x7f0294103b50 2024-04-21T08:39:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.975+0000 7f029b95f640 1 -- 172.21.15.94:0/422712153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02941a07b0 con 0x7f0294103b50 2024-04-21T08:39:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.976+0000 7f029b95f640 1 -- 172.21.15.94:0/422712153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02941093b0 con 0x7f0294103b50 2024-04-21T08:39:47.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.977+0000 7f02927fc640 1 -- 172.21.15.94:0/422712153 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f02840119d0 con 0x7f0294103b50 2024-04-21T08:39:47.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.978+0000 7f02927fc640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f027c0a3bb0 0x7f027c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:47.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.978+0000 7f02996fb640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f027c0a3bb0 0x7f027c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:47.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.978+0000 7f02927fc640 1 -- 172.21.15.94:0/422712153 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f02840d05c0 con 0x7f0294103b50 2024-04-21T08:39:47.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.978+0000 7f02996fb640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f027c0a3bb0 0x7f027c0a6070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f028c032380 tx=0x7f028c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:47.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:47.981+0000 7f02927fc640 1 -- 172.21.15.94:0/422712153 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f028409cba0 con 0x7f0294103b50 2024-04-21T08:39:48.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:47 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3084752218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:48.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:47 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1364728192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:48.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:47 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1787825999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:48.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.158+0000 7f029b95f640 1 -- 172.21.15.94:0/422712153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0294062b50 con 0x7f0294103b50 2024-04-21T08:39:48.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.159+0000 7f02927fc640 1 -- 172.21.15.94:0/422712153 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f028409c4a0 con 0x7f0294103b50 2024-04-21T08:39:48.157 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:39:48.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.160+0000 7f029b95f640 1 -- 172.21.15.94:0/422712153 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f027c0a3bb0 msgr2=0x7f027c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.160+0000 7f029b95f640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f027c0a3bb0 0x7f027c0a6070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f028c032380 tx=0x7f028c000f40 comp rx=0 tx=0).stop 2024-04-21T08:39:48.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.160+0000 7f029b95f640 1 -- 172.21.15.94:0/422712153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0294103b50 msgr2=0x7f029419f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.160+0000 7f029b95f640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0294103b50 0x7f029419f8d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0284009fd0 tx=0x7f028400b8e0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.160+0000 7f029b95f640 1 -- 172.21.15.94:0/422712153 shutdown_connections 2024-04-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.160+0000 7f029b95f640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f027c0a3bb0 0x7f027c0a6070 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.160+0000 7f029b95f640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0294103b50 0x7f029419f8d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.161+0000 7f029b95f640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0294103190 0x7f0294198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.161+0000 7f029b95f640 1 --2- 172.21.15.94:0/422712153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0294101fa0 0x7f0294198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.161+0000 7f029b95f640 1 -- 172.21.15.94:0/422712153 >> 172.21.15.94:0/422712153 conn(0x7f02940fd530 msgr2=0x7f0294106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.161+0000 7f029b95f640 1 -- 172.21.15.94:0/422712153 shutdown_connections 2024-04-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.161+0000 7f029b95f640 1 -- 172.21.15.94:0/422712153 wait complete. 2024-04-21T08:39:48.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:39:48.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:47 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3084752218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:48.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:47 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1364728192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:48.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:47 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1787825999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:39:48.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.272+0000 7fb5b263f640 1 -- 172.21.15.94:0/697167004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5ac103240 msgr2=0x7fb5ac1036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.272+0000 7fb5b263f640 1 --2- 172.21.15.94:0/697167004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5ac103240 0x7fb5ac1036c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009930 tx=0x7fb5a002bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:48.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.272+0000 7fb5b263f640 1 -- 172.21.15.94:0/697167004 shutdown_connections 2024-04-21T08:39:48.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.272+0000 7fb5b263f640 1 --2- 172.21.15.94:0/697167004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5ac103c00 0x7fb5ac10a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.272+0000 7fb5b263f640 1 --2- 172.21.15.94:0/697167004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5ac103240 0x7fb5ac1036c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.272+0000 7fb5b263f640 1 --2- 172.21.15.94:0/697167004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5ac100270 0x7fb5ac100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.272+0000 7fb5b263f640 1 -- 172.21.15.94:0/697167004 >> 172.21.15.94:0/697167004 conn(0x7fb5ac0fbb80 msgr2=0x7fb5ac0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.272+0000 7fb5b263f640 1 -- 172.21.15.94:0/697167004 shutdown_connections 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.272+0000 7fb5b263f640 1 -- 172.21.15.94:0/697167004 wait complete. 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b263f640 1 Processor -- start 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b263f640 1 -- start start 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b263f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5ac100270 0x7fb5ac198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b263f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5ac103240 0x7fb5ac1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b263f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5ac103c00 0x7fb5ac19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b263f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac10db10 con 0x7fb5ac103c00 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b263f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac10d990 con 0x7fb5ac100270 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b263f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac10dc90 con 0x7fb5ac103240 2024-04-21T08:39:48.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b0bdc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5ac103c00 0x7fb5ac19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b0bdc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5ac103c00 0x7fb5ac19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37418/0 (socket says 172.21.15.94:37418) 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5b0bdc640 1 -- 172.21.15.94:0/268677687 learned_addr learned my addr 172.21.15.94:0/268677687 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.273+0000 7fb5abfff640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5ac100270 0x7fb5ac198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5ab7fe640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5ac103240 0x7fb5ac1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5b0bdc640 1 -- 172.21.15.94:0/268677687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5ac103240 msgr2=0x7fb5ac1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5b0bdc640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5ac103240 0x7fb5ac1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5b0bdc640 1 -- 172.21.15.94:0/268677687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5ac100270 msgr2=0x7fb5ac198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5b0bdc640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5ac100270 0x7fb5ac198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5b0bdc640 1 -- 172.21.15.94:0/268677687 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac19fdb0 con 0x7fb5ac103c00 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5abfff640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5ac100270 0x7fb5ac198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5b0bdc640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5ac103c00 0x7fb5ac19f740 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400cce0 tx=0x7fb5a400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5a97fa640 1 -- 172.21.15.94:0/268677687 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400f450 con 0x7fb5ac103c00 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5a97fa640 1 -- 172.21.15.94:0/268677687 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a400faa0 con 0x7fb5ac103c00 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5b263f640 1 -- 172.21.15.94:0/268677687 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac1a00a0 con 0x7fb5ac103c00 2024-04-21T08:39:48.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5a97fa640 1 -- 172.21.15.94:0/268677687 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4018b70 con 0x7fb5ac103c00 2024-04-21T08:39:48.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.274+0000 7fb5b263f640 1 -- 172.21.15.94:0/268677687 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac1a0570 con 0x7fb5ac103c00 2024-04-21T08:39:48.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.275+0000 7fb5b263f640 1 -- 172.21.15.94:0/268677687 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5ac109370 con 0x7fb5ac103c00 2024-04-21T08:39:48.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.276+0000 7fb5a97fa640 1 -- 172.21.15.94:0/268677687 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb5a4018d10 con 0x7fb5ac103c00 2024-04-21T08:39:48.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.276+0000 7fb5a97fa640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb5940a3bb0 0x7fb5940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.277+0000 7fb5abfff640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb5940a3bb0 0x7fb5940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.277+0000 7fb5a97fa640 1 -- 172.21.15.94:0/268677687 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fb5a40d0db0 con 0x7fb5ac103c00 2024-04-21T08:39:48.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.277+0000 7fb5abfff640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb5940a3bb0 0x7fb5940a6070 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0058c0 tx=0x7fb59c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:48.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.280+0000 7fb5a97fa640 1 -- 172.21.15.94:0/268677687 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5a4014030 con 0x7fb5ac103c00 2024-04-21T08:39:48.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.455+0000 7fb5b263f640 1 -- 172.21.15.94:0/268677687 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb5ac02d5c0 con 0x7fb5ac103c00 2024-04-21T08:39:48.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.455+0000 7fb5a97fa640 1 -- 172.21.15.94:0/268677687 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fb5a409d420 con 0x7fb5ac103c00 2024-04-21T08:39:48.454 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:39:48.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.457+0000 7fb5b263f640 1 -- 172.21.15.94:0/268677687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb5940a3bb0 msgr2=0x7fb5940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.457+0000 7fb5b263f640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb5940a3bb0 0x7fb5940a6070 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0058c0 tx=0x7fb59c005850 comp rx=0 tx=0).stop 2024-04-21T08:39:48.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.457+0000 7fb5b263f640 1 -- 172.21.15.94:0/268677687 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5ac103c00 msgr2=0x7fb5ac19f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.457+0000 7fb5b263f640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5ac103c00 0x7fb5ac19f740 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400cce0 tx=0x7fb5a400a620 comp rx=0 tx=0).stop 2024-04-21T08:39:48.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.458+0000 7fb5b263f640 1 -- 172.21.15.94:0/268677687 shutdown_connections 2024-04-21T08:39:48.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.458+0000 7fb5b263f640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb5940a3bb0 0x7fb5940a6070 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.458+0000 7fb5b263f640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5ac103240 0x7fb5ac1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.458+0000 7fb5b263f640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5ac103c00 0x7fb5ac19f740 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.458+0000 7fb5b263f640 1 --2- 172.21.15.94:0/268677687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5ac100270 0x7fb5ac198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.458+0000 7fb5b263f640 1 -- 172.21.15.94:0/268677687 >> 172.21.15.94:0/268677687 conn(0x7fb5ac0fbb80 msgr2=0x7fb5ac106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:48.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.458+0000 7fb5b263f640 1 -- 172.21.15.94:0/268677687 shutdown_connections 2024-04-21T08:39:48.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.458+0000 7fb5b263f640 1 -- 172.21.15.94:0/268677687 wait complete. 2024-04-21T08:39:48.468 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:39:48.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.566+0000 7fc81c5ef640 1 -- 172.21.15.94:0/3591319332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc814107c50 msgr2=0x7fc81410a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.566+0000 7fc81c5ef640 1 --2- 172.21.15.94:0/3591319332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc814107c50 0x7fc81410a0a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009900 tx=0x7fc80c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:48.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 -- 172.21.15.94:0/3591319332 shutdown_connections 2024-04-21T08:39:48.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 --2- 172.21.15.94:0/3591319332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc814107c50 0x7fc81410a0a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 --2- 172.21.15.94:0/3591319332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc814105320 0x7fc814107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 --2- 172.21.15.94:0/3591319332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8141029f0 0x7fc814104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 -- 172.21.15.94:0/3591319332 >> 172.21.15.94:0/3591319332 conn(0x7fc8140fc3e0 msgr2=0x7fc8140fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:48.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 -- 172.21.15.94:0/3591319332 shutdown_connections 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 -- 172.21.15.94:0/3591319332 wait complete. 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 Processor -- start 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 -- start start 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8141029f0 0x7fc814198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc814105320 0x7fc8141986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc814107c50 0x7fc81419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc81410dac0 con 0x7fc8141029f0 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc81410d940 con 0x7fc814105320 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81c5ef640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc81410dc40 con 0x7fc814107c50 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81a38b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8141029f0 0x7fc814198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81a38b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8141029f0 0x7fc814198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37442/0 (socket says 172.21.15.94:37442) 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.567+0000 7fc81a38b640 1 -- 172.21.15.94:0/1442838970 learned_addr learned my addr 172.21.15.94:0/1442838970 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc81a38b640 1 -- 172.21.15.94:0/1442838970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc814107c50 msgr2=0x7fc81419f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc81ab8c640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc814107c50 0x7fc81419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc81a38b640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc814107c50 0x7fc81419f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc81a38b640 1 -- 172.21.15.94:0/1442838970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc814105320 msgr2=0x7fc8141986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc81a38b640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc814105320 0x7fc8141986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc81a38b640 1 -- 172.21.15.94:0/1442838970 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81419fdb0 con 0x7fc8141029f0 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc81ab8c640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc814107c50 0x7fc81419f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc81a38b640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8141029f0 0x7fc814198190 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc81000ea80 tx=0x7fc81000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc80b7fe640 1 -- 172.21.15.94:0/1442838970 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81000ce40 con 0x7fc8141029f0 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc80b7fe640 1 -- 172.21.15.94:0/1442838970 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc810005020 con 0x7fc8141029f0 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc81c5ef640 1 -- 172.21.15.94:0/1442838970 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8141a00a0 con 0x7fc8141029f0 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc80b7fe640 1 -- 172.21.15.94:0/1442838970 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8100116d0 con 0x7fc8141029f0 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.568+0000 7fc81c5ef640 1 -- 172.21.15.94:0/1442838970 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8141a05f0 con 0x7fc8141029f0 2024-04-21T08:39:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.569+0000 7fc81c5ef640 1 -- 172.21.15.94:0/1442838970 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8140fded0 con 0x7fc8141029f0 2024-04-21T08:39:48.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.571+0000 7fc80b7fe640 1 -- 172.21.15.94:0/1442838970 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc8100054d0 con 0x7fc8141029f0 2024-04-21T08:39:48.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.571+0000 7fc80b7fe640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc8000a3c30 0x7fc8000a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.572+0000 7fc819b8a640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc8000a3c30 0x7fc8000a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.572+0000 7fc80b7fe640 1 -- 172.21.15.94:0/1442838970 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fc8100d0ee0 con 0x7fc8141029f0 2024-04-21T08:39:48.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.572+0000 7fc819b8a640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc8000a3c30 0x7fc8000a60f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc8141996a0 tx=0x7fc804005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:48.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.575+0000 7fc80b7fe640 1 -- 172.21.15.94:0/1442838970 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc810016030 con 0x7fc8141029f0 2024-04-21T08:39:48.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:48 smithi094 ceph-mon[27988]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:39:48.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:48 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4042974687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:48.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:48 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/422712153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:48.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:48 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/268677687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:48.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.790+0000 7fc81c5ef640 1 -- 172.21.15.94:0/1442838970 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc81402d830 con 0x7fc8141029f0 2024-04-21T08:39:48.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.791+0000 7fc80b7fe640 1 -- 172.21.15.94:0/1442838970 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11885 (secure 0 0 0) 0x7fc81009d540 con 0x7fc8141029f0 2024-04-21T08:39:48.789 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:48.790 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:37:56.995828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24446,24488]},"id":1}]} 2024-04-21T08:39:48.790 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 -- 172.21.15.94:0/1442838970 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc8000a3c30 msgr2=0x7fc8000a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc8000a3c30 0x7fc8000a60f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc8141996a0 tx=0x7fc804005bf0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 -- 172.21.15.94:0/1442838970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8141029f0 msgr2=0x7fc814198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8141029f0 0x7fc814198190 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc81000ea80 tx=0x7fc81000ef40 comp rx=0 tx=0).stop 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 -- 172.21.15.94:0/1442838970 shutdown_connections 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fc8000a3c30 0x7fc8000a60f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc814107c50 0x7fc81419f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc814105320 0x7fc8141986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 --2- 172.21.15.94:0/1442838970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8141029f0 0x7fc814198190 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 -- 172.21.15.94:0/1442838970 >> 172.21.15.94:0/1442838970 conn(0x7fc8140fc3e0 msgr2=0x7fc814103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 -- 172.21.15.94:0/1442838970 shutdown_connections 2024-04-21T08:39:48.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.793+0000 7fc81c5ef640 1 -- 172.21.15.94:0/1442838970 wait complete. 2024-04-21T08:39:48.803 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:39:48.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.902+0000 7fa40c2fd640 1 -- 172.21.15.94:0/4078845227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa404100650 msgr2=0x7fa404107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.902+0000 7fa40c2fd640 1 --2- 172.21.15.94:0/4078845227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa404100650 0x7fa404107650 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009900 tx=0x7fa3f402bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.902+0000 7fa40c2fd640 1 -- 172.21.15.94:0/4078845227 shutdown_connections 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.902+0000 7fa40c2fd640 1 --2- 172.21.15.94:0/4078845227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa404107be0 0x7fa40410a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.902+0000 7fa40c2fd640 1 --2- 172.21.15.94:0/4078845227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa404100650 0x7fa404107650 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.902+0000 7fa40c2fd640 1 --2- 172.21.15.94:0/4078845227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4040ffcf0 0x7fa404100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.902+0000 7fa40c2fd640 1 -- 172.21.15.94:0/4078845227 >> 172.21.15.94:0/4078845227 conn(0x7fa4040fb740 msgr2=0x7fa4040fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.902+0000 7fa40c2fd640 1 -- 172.21.15.94:0/4078845227 shutdown_connections 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.902+0000 7fa40c2fd640 1 -- 172.21.15.94:0/4078845227 wait complete. 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40c2fd640 1 Processor -- start 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40c2fd640 1 -- start start 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40c2fd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4040ffcf0 0x7fa4041981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40c2fd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa404100650 0x7fa404198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40c2fd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa404107be0 0x7fa40419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40c2fd640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa40410daa0 con 0x7fa404107be0 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40c2fd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa40410d920 con 0x7fa4040ffcf0 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40c2fd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa40410dc20 con 0x7fa404100650 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40a89a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa404107be0 0x7fa40419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40a89a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa404107be0 0x7fa40419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37472/0 (socket says 172.21.15.94:37472) 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40a89a640 1 -- 172.21.15.94:0/1484408549 learned_addr learned my addr 172.21.15.94:0/1484408549 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa409898640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa404100650 0x7fa404198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40a099640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4040ffcf0 0x7fa4041981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40a89a640 1 -- 172.21.15.94:0/1484408549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa404100650 msgr2=0x7fa404198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.903+0000 7fa40a89a640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa404100650 0x7fa404198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.904+0000 7fa40a89a640 1 -- 172.21.15.94:0/1484408549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4040ffcf0 msgr2=0x7fa4041981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.904+0000 7fa40a89a640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4040ffcf0 0x7fa4041981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.904+0000 7fa40a89a640 1 -- 172.21.15.94:0/1484408549 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40419fd50 con 0x7fa404107be0 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.904+0000 7fa40a89a640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa404107be0 0x7fa40419f6e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc00cce0 tx=0x7fa3fc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:48.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.904+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/1484408549 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc00f450 con 0x7fa404107be0 2024-04-21T08:39:48.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.904+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/1484408549 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3fc00faa0 con 0x7fa404107be0 2024-04-21T08:39:48.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.904+0000 7fa40c2fd640 1 -- 172.21.15.94:0/1484408549 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa40419ffe0 con 0x7fa404107be0 2024-04-21T08:39:48.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.904+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/1484408549 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc018b70 con 0x7fa404107be0 2024-04-21T08:39:48.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.904+0000 7fa40c2fd640 1 -- 172.21.15.94:0/1484408549 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4041a04b0 con 0x7fa404107be0 2024-04-21T08:39:48.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.905+0000 7fa40c2fd640 1 -- 172.21.15.94:0/1484408549 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa404106460 con 0x7fa404107be0 2024-04-21T08:39:48.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.906+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/1484408549 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa3fc018d10 con 0x7fa404107be0 2024-04-21T08:39:48.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.906+0000 7fa3fb7fe640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa3f00a3bb0 0x7fa3f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:48.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.906+0000 7fa40a099640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa3f00a3bb0 0x7fa3f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:48.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.907+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/1484408549 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fa3fc0d0c50 con 0x7fa404107be0 2024-04-21T08:39:48.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.907+0000 7fa40a099640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa3f00a3bb0 0x7fa3f00a6070 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa400005fd0 tx=0x7fa400005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:48.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:48.911+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/1484408549 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa3fc0d6020 con 0x7fa404107be0 2024-04-21T08:39:49.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:48 smithi173 ceph-mon[31428]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:39:49.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:48 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4042974687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:49.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:48 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/422712153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:49.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:48 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/268677687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:49.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.091+0000 7fa40c2fd640 1 -- 172.21.15.94:0/1484408549 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa404062b50 con 0x7fa404107be0 2024-04-21T08:39:49.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.092+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/1484408549 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24890 (secure 0 0 0) 0x7fa3fc014030 con 0x7fa404107be0 2024-04-21T08:39:49.091 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:49.094 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":93,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:31.669420+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:39:49.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.094+0000 7fa40c2fd640 1 -- 172.21.15.94:0/1484408549 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa3f00a3bb0 msgr2=0x7fa3f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.094+0000 7fa40c2fd640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa3f00a3bb0 0x7fa3f00a6070 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa400005fd0 tx=0x7fa400005850 comp rx=0 tx=0).stop 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.094+0000 7fa40c2fd640 1 -- 172.21.15.94:0/1484408549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa404107be0 msgr2=0x7fa40419f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.094+0000 7fa40c2fd640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa404107be0 0x7fa40419f6e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc00cce0 tx=0x7fa3fc00a620 comp rx=0 tx=0).stop 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.094+0000 7fa40c2fd640 1 -- 172.21.15.94:0/1484408549 shutdown_connections 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.094+0000 7fa40c2fd640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fa3f00a3bb0 0x7fa3f00a6070 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.094+0000 7fa40c2fd640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa404100650 0x7fa404198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.094+0000 7fa40c2fd640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa404107be0 0x7fa40419f6e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.094+0000 7fa40c2fd640 1 --2- 172.21.15.94:0/1484408549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4040ffcf0 0x7fa4041981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.094+0000 7fa40c2fd640 1 -- 172.21.15.94:0/1484408549 >> 172.21.15.94:0/1484408549 conn(0x7fa4040fb740 msgr2=0x7fa404103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.095+0000 7fa40c2fd640 1 -- 172.21.15.94:0/1484408549 shutdown_connections 2024-04-21T08:39:49.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.095+0000 7fa40c2fd640 1 -- 172.21.15.94:0/1484408549 wait complete. 2024-04-21T08:39:49.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:37:30.753966+0000', 'modified': '2024-04-21T08:37:56.995828+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24488, 'mds_1': 24454, 'mds_2': 24446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24416': {'gid': 24416, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6835/3967081988', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 3967081988}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 3967081988}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24421': {'gid': 24421, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/3922099450', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 3922099450}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 3922099450}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24433': {'gid': 24433, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6837/1466520447', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 1466520447}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 1466520447}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.111:6837/550365815', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6836', 'nonce': 550365815}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 550365815}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.173:6839/2987518473', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 2987518473}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 2987518473}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3878872423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3878872423}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3878872423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24488, 'qdb_cluster': [24454, 24446, 24488]} 2024-04-21T08:39:49.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-21T08:39:49.106 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-21T08:39:49.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:48 smithi111 ceph-mon[31493]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:39:49.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:48 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4042974687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:39:49.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:48 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/422712153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:39:49.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:48 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/268677687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:39:49.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 -- 172.21.15.94:0/2610485790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6258102040 msgr2=0x7f62581024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 --2- 172.21.15.94:0/2610485790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6258102040 0x7f62581024c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6244009900 tx=0x7f624402bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 -- 172.21.15.94:0/2610485790 shutdown_connections 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 --2- 172.21.15.94:0/2610485790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6258102a00 0x7f6258109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 --2- 172.21.15.94:0/2610485790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6258102040 0x7f62581024c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 --2- 172.21.15.94:0/2610485790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6258100e60 0x7f6258101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 -- 172.21.15.94:0/2610485790 >> 172.21.15.94:0/2610485790 conn(0x7f62580fc420 msgr2=0x7f62580fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 -- 172.21.15.94:0/2610485790 shutdown_connections 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 -- 172.21.15.94:0/2610485790 wait complete. 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 Processor -- start 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 -- start start 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6258100e60 0x7f625810b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6258102040 0x7f6258109ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6258102a00 0x7f625810a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f625810db30 con 0x7f6258100e60 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f625810d9b0 con 0x7f6258102040 2024-04-21T08:39:49.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.206+0000 7f625ee74640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f625810dcb0 con 0x7f6258102a00 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625cc10640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6258100e60 0x7f625810b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625cc10640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6258100e60 0x7f625810b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37498/0 (socket says 172.21.15.94:37498) 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625cc10640 1 -- 172.21.15.94:0/2468035523 learned_addr learned my addr 172.21.15.94:0/2468035523 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625d411640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6258102a00 0x7f625810a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625cc10640 1 -- 172.21.15.94:0/2468035523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6258102a00 msgr2=0x7f625810a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625cc10640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6258102a00 0x7f625810a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625cc10640 1 -- 172.21.15.94:0/2468035523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6258102040 msgr2=0x7f6258109ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625cc10640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6258102040 0x7f6258109ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625cc10640 1 -- 172.21.15.94:0/2468035523 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f625810ac70 con 0x7f6258100e60 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625d411640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6258102a00 0x7f625810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625cc10640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6258100e60 0x7f625810b860 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f624c00c970 tx=0x7f624c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f6255ffb640 1 -- 172.21.15.94:0/2468035523 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c007c10 con 0x7f6258100e60 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f6255ffb640 1 -- 172.21.15.94:0/2468035523 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f624c007db0 con 0x7f6258100e60 2024-04-21T08:39:49.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f6255ffb640 1 -- 172.21.15.94:0/2468035523 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c00f870 con 0x7f6258100e60 2024-04-21T08:39:49.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625ee74640 1 -- 172.21.15.94:0/2468035523 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62581a22e0 con 0x7f6258100e60 2024-04-21T08:39:49.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.207+0000 7f625ee74640 1 -- 172.21.15.94:0/2468035523 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62581a2830 con 0x7f6258100e60 2024-04-21T08:39:49.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.209+0000 7f6255ffb640 1 -- 172.21.15.94:0/2468035523 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f624c00fa10 con 0x7f6258100e60 2024-04-21T08:39:49.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6261a6e740 1 Processor -- start 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6261a6e740 1 -- start start 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6255ffb640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62480a3bb0 0x7f62480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7de40 0x5637abf7e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf7ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 0x5637abf85eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6261a6e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5637abee2b10 con 0x5637abf7e830 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6261a6e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5637abe80240 con 0x5637abf7de40 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6261a6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5637abedead0 con 0x5637abf7f2d0 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6257fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf7ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f625d411640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62480a3bb0 0x7f62480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6257fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf7ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37514/0 (socket says 172.21.15.94:37514) 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6257fff640 1 -- 172.21.15.94:0/4034651251 learned_addr learned my addr 172.21.15.94:0/4034651251 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6255ffb640 1 -- 172.21.15.94:0/2468035523 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f624c0d0c80 con 0x7f6258100e60 2024-04-21T08:39:49.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f625cc10640 1 --2- 172.21.15.94:0/4034651251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7de40 0x5637abf7e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6257fff640 1 -- 172.21.15.94:0/4034651251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 msgr2=0x5637abf85eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6257fff640 1 --2- 172.21.15.94:0/4034651251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 0x5637abf85eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6257fff640 1 -- 172.21.15.94:0/4034651251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7de40 msgr2=0x5637abf7e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6257fff640 1 --2- 172.21.15.94:0/4034651251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7de40 0x5637abf7e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f6257fff640 1 -- 172.21.15.94:0/4034651251 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637abbf52d0 con 0x5637abf7e830 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.210+0000 7f625d411640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62480a3bb0 0x7f62480a6070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6250005fd0 tx=0x7f6250009980 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6257fff640 1 --2- 172.21.15.94:0/4034651251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf7ecb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6244005ab0 tx=0x7f62440095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d2b7964adb7125 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6242ffd640 1 -- 172.21.15.94:0/4034651251 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244041020 con 0x5637abf7e830 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6242ffd640 1 -- 172.21.15.94:0/4034651251 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f624402cae0 con 0x5637abf7e830 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6242ffd640 1 -- 172.21.15.94:0/4034651251 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244004920 con 0x5637abf7e830 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6261a6e740 1 -- 172.21.15.94:0/4034651251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 msgr2=0x5637abf7ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6261a6e740 1 --2- 172.21.15.94:0/4034651251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf7ecb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6244005ab0 tx=0x7f62440095e0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6261a6e740 1 -- 172.21.15.94:0/4034651251 shutdown_connections 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6261a6e740 1 --2- 172.21.15.94:0/4034651251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 0x5637abf85eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6261a6e740 1 --2- 172.21.15.94:0/4034651251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf7ecb0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6261a6e740 1 --2- 172.21.15.94:0/4034651251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7de40 0x5637abf7e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6261a6e740 1 -- 172.21.15.94:0/4034651251 >> 172.21.15.94:0/4034651251 conn(0x5637abf79f40 msgr2=0x5637abf78490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6261a6e740 1 -- 172.21.15.94:0/4034651251 shutdown_connections 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.211+0000 7f6261a6e740 1 -- 172.21.15.94:0/4034651251 wait complete. 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6261a6e740 1 -- start start 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6261a6e740 1 Processor -- start 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7de40 0x5637abf87e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf883a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7f2d0 0x5637abf8f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6261a6e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5637abf86450 con 0x5637abf7e830 2024-04-21T08:39:49.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6261a6e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5637abe6b190 con 0x5637abf7f2d0 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6261a6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5637abf865d0 con 0x5637abf7de40 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6261a6e740 10 client.0 _ll_register_callbacks cb 0x5637abe668b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6257fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf883a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6257fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf883a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37526/0 (socket says 172.21.15.94:37526) 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f6257fff640 1 -- 172.21.15.94:0/787619108 learned_addr learned my addr 172.21.15.94:0/787619108 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f625d411640 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7f2d0 0x5637abf8f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.212+0000 7f625cc10640 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7de40 0x5637abf87e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6257fff640 1 -- 172.21.15.94:0/787619108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7de40 msgr2=0x5637abf87e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6257fff640 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7de40 0x5637abf87e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6257fff640 1 -- 172.21.15.94:0/787619108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7f2d0 msgr2=0x5637abf8f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6257fff640 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7f2d0 0x5637abf8f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6257fff640 1 -- 172.21.15.94:0/787619108 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637abcbf6b0 con 0x5637abf7e830 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6257fff640 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf883a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f624402c6a0 tx=0x7f6244004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d213f2b4eefd432 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6237fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6237fff640 1 -- 172.21.15.94:0/787619108 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244041030 con 0x5637abf7e830 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6237fff640 1 -- 172.21.15.94:0/787619108 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f62440055a0 con 0x5637abf7e830 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6261a6e740 1 -- 172.21.15.94:0/787619108 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f624402cae0 con 0x5637abf7e830 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6237fff640 1 -- 172.21.15.94:0/787619108 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6244056050 con 0x5637abf7e830 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6261a6e740 10 client.15387 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6261a6e740 1 -- 172.21.15.94:0/787619108 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5637abf92750 con 0x5637abf7e830 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6237fff640 1 -- 172.21.15.94:0/787619108 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f624402cae0 con 0x5637abf7e830 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6237fff640 1 -- 172.21.15.94:0/787619108 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f624404f450 con 0x5637abf7e830 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6261a6e740 10 client.15387 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6261a6e740 10 client.15387 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-21T08:39:49.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6261a6e740 20 client.15387 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6261a6e740 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5637abf94870 0x5637abf95ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6261a6e740 4 client.15387 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.213+0000 7f6261a6e740 1 -- 172.21.15.94:0/787619108 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637abef0530 con 0x5637abf94870 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.214+0000 7f625cc10640 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5637abf94870 0x5637abf95ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.214+0000 7f6237fff640 10 client.15387 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.214+0000 7f625cc10640 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5637abf94870 0x5637abf95ce0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=75c8b31ebfae6506 server_cookie=586963867c65f9aa in_seq=0 out_seq=0 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6237fff640 1 -- 172.21.15.94:0/787619108 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5637abef0530 con 0x5637abf94870 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6237fff640 10 client.15387 handle_command_reply: tid=0 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 client.15387 shutdown 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 -- 172.21.15.94:0/787619108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 msgr2=0x5637abf883a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf883a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f624402c6a0 tx=0x7f6244004270 comp rx=0 tx=0).stop 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 -- 172.21.15.94:0/787619108 shutdown_connections 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5637abf94870 0x5637abf95ce0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7f2d0 0x5637abf8f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6237fff640 0 client.15387 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:39:49.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf883a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 --2- 172.21.15.94:0/787619108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7de40 0x5637abf87e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 -- 172.21.15.94:0/787619108 >> 172.21.15.94:0/787619108 conn(0x5637abf79f40 msgr2=0x5637abf77a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 -- 172.21.15.94:0/787619108 shutdown_connections 2024-04-21T08:39:49.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 1 -- 172.21.15.94:0/787619108 wait complete. 2024-04-21T08:39:49.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.215+0000 7f6261a6e740 20 client.15387 trim_cache size 0 max 16384 2024-04-21T08:39:49.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f6261a6e740 1 Processor -- start 2024-04-21T08:39:49.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f6261a6e740 1 -- start start 2024-04-21T08:39:49.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7de40 0x5637abf82840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7e830 0x5637abf82d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 0x5637abf8f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f6261a6e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5637abf94020 con 0x5637abf7de40 2024-04-21T08:39:49.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f6261a6e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5637abf946f0 con 0x5637abf7e830 2024-04-21T08:39:49.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f6261a6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5637abf94dc0 con 0x5637abf7f2d0 2024-04-21T08:39:49.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f625cc10640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7de40 0x5637abf82840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f625cc10640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7de40 0x5637abf82840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37528/0 (socket says 172.21.15.94:37528) 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f625cc10640 1 -- 172.21.15.94:0/1145091638 learned_addr learned my addr 172.21.15.94:0/1145091638 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f6257fff640 1 --2- 172.21.15.94:0/1145091638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7e830 0x5637abf82d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f625cc10640 1 -- 172.21.15.94:0/1145091638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 msgr2=0x5637abf8f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f625cc10640 1 --2- 172.21.15.94:0/1145091638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 0x5637abf8f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f625cc10640 1 -- 172.21.15.94:0/1145091638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7e830 msgr2=0x5637abf82d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f625cc10640 1 --2- 172.21.15.94:0/1145091638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7e830 0x5637abf82d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.233+0000 7f625cc10640 1 -- 172.21.15.94:0/1145091638 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637abf88fe0 con 0x5637abf7de40 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f625cc10640 1 --2- 172.21.15.94:0/1145091638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7de40 0x5637abf82840 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f624c0ce6f0 tx=0x7f624c0bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395b0358a70640dc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f62427fc640 1 -- 172.21.15.94:0/1145091638 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c09ac90 con 0x5637abf7de40 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f62427fc640 1 -- 172.21.15.94:0/1145091638 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f624c087040 con 0x5637abf7de40 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f62427fc640 1 -- 172.21.15.94:0/1145091638 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c099260 con 0x5637abf7de40 2024-04-21T08:39:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f6261a6e740 1 -- 172.21.15.94:0/1145091638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7de40 msgr2=0x5637abf82840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f6261a6e740 1 --2- 172.21.15.94:0/1145091638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7de40 0x5637abf82840 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f624c0ce6f0 tx=0x7f624c0bd040 comp rx=0 tx=0).stop 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f6261a6e740 1 -- 172.21.15.94:0/1145091638 shutdown_connections 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f6261a6e740 1 --2- 172.21.15.94:0/1145091638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 0x5637abf8f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f6261a6e740 1 --2- 172.21.15.94:0/1145091638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7e830 0x5637abf82d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f6261a6e740 1 --2- 172.21.15.94:0/1145091638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7de40 0x5637abf82840 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f6261a6e740 1 -- 172.21.15.94:0/1145091638 >> 172.21.15.94:0/1145091638 conn(0x5637abf79f40 msgr2=0x5637abf85e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f6261a6e740 1 -- 172.21.15.94:0/1145091638 shutdown_connections 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.234+0000 7f6261a6e740 1 -- 172.21.15.94:0/1145091638 wait complete. 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6261a6e740 1 -- start start 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6261a6e740 1 Processor -- start 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7de40 0x5637abf895c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf89b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6261a6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 0x5637abf8e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6261a6e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5637abf95b60 con 0x5637abf7e830 2024-04-21T08:39:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6261a6e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5637abf95490 con 0x5637abf7de40 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6261a6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5637abf96230 con 0x5637abf7f2d0 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6261a6e740 10 client.0 _ll_register_callbacks cb 0x7f6244005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6257fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf89b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6257fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf89b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37544/0 (socket says 172.21.15.94:37544) 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6257fff640 1 -- 172.21.15.94:0/3736081506 learned_addr learned my addr 172.21.15.94:0/3736081506 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f625cc10640 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7de40 0x5637abf895c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f625d411640 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 0x5637abf8e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6257fff640 1 -- 172.21.15.94:0/3736081506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 msgr2=0x5637abf8e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6257fff640 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 0x5637abf8e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6257fff640 1 -- 172.21.15.94:0/3736081506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7de40 msgr2=0x5637abf895c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6257fff640 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7de40 0x5637abf895c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.235+0000 7f6257fff640 1 -- 172.21.15.94:0/3736081506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637abf8ebb0 con 0x5637abf7e830 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6257fff640 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf89b00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x5637abf73f30 tx=0x7f624404f760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b0e9a8e0b8f6f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6237fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:49.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6237fff640 1 -- 172.21.15.94:0/3736081506 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244041040 con 0x5637abf7e830 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6237fff640 1 -- 172.21.15.94:0/3736081506 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f6244005560 con 0x5637abf7e830 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6261a6e740 1 -- 172.21.15.94:0/3736081506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f624c087040 con 0x5637abf7e830 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6237fff640 1 -- 172.21.15.94:0/3736081506 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6244062050 con 0x5637abf7e830 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6261a6e740 10 client.15393 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6261a6e740 1 -- 172.21.15.94:0/3736081506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5637abffacc0 con 0x5637abf7e830 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6237fff640 1 -- 172.21.15.94:0/3736081506 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f624c087040 con 0x5637abf7e830 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6237fff640 1 -- 172.21.15.94:0/3736081506 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f6244004c00 con 0x5637abf7e830 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6261a6e740 10 client.15393 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6261a6e740 10 client.15393 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6261a6e740 20 client.15393 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6261a6e740 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5637abfff830 0x5637abffc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6261a6e740 4 client.15393 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f6261a6e740 1 -- 172.21.15.94:0/3736081506 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "status"}) -- 0x5637abeea3a0 con 0x5637abfff830 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.236+0000 7f625cc10640 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5637abfff830 0x5637abffc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6237fff640 10 client.15393 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:39:49.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f625cc10640 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5637abfff830 0x5637abffc340 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88a03d53fdb60df8 server_cookie=99967d6667ae1c37 in_seq=0 out_seq=0 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6237fff640 1 -- 172.21.15.94:0/3736081506 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5637abeea3a0 con 0x5637abfff830 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6237fff640 10 client.15393 handle_command_reply: tid=0 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 client.15393 shutdown 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 -- 172.21.15.94:0/3736081506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 msgr2=0x5637abf89b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf89b00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x5637abf73f30 tx=0x7f624404f760 comp rx=0 tx=0).stop 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 -- 172.21.15.94:0/3736081506 shutdown_connections 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5637abfff830 0x5637abffc340 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5637abf7f2d0 0x5637abf8e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5637abf7e830 0x5637abf89b00 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 --2- 172.21.15.94:0/3736081506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637abf7de40 0x5637abf895c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6237fff640 0 client.15393 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 -- 172.21.15.94:0/3736081506 >> 172.21.15.94:0/3736081506 conn(0x5637abf79f40 msgr2=0x5637abf77170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 -- 172.21.15.94:0/3736081506 shutdown_connections 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 1 -- 172.21.15.94:0/3736081506 wait complete. 2024-04-21T08:39:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.237+0000 7f6261a6e740 20 client.15393 trim_cache size 0 max 16384 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 2, 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24446, 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 112.580383764, 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 23, 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 93, 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:39:49.243 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 154.27788830899999 2024-04-21T08:39:49.244 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:39:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 -- 172.21.15.94:0/2468035523 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62480a3bb0 msgr2=0x7f62480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62480a3bb0 0x7f62480a6070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6250005fd0 tx=0x7f6250009980 comp rx=0 tx=0).stop 2024-04-21T08:39:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 -- 172.21.15.94:0/2468035523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6258100e60 msgr2=0x7f625810b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6258100e60 0x7f625810b860 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f624c00c970 tx=0x7f624c00ce30 comp rx=0 tx=0).stop 2024-04-21T08:39:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 -- 172.21.15.94:0/2468035523 shutdown_connections 2024-04-21T08:39:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f62480a3bb0 0x7f62480a6070 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6258102a00 0x7f625810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6258102040 0x7f6258109ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 --2- 172.21.15.94:0/2468035523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6258100e60 0x7f625810b860 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 -- 172.21.15.94:0/2468035523 >> 172.21.15.94:0/2468035523 conn(0x7f62580fc420 msgr2=0x7f62581053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 -- 172.21.15.94:0/2468035523 shutdown_connections 2024-04-21T08:39:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.238+0000 7f625ee74640 1 -- 172.21.15.94:0/2468035523 wait complete. 2024-04-21T08:39:49.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-21T08:39:49.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.348+0000 7fb3d9a25640 1 -- 172.21.15.94:0/3461124774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d4102040 msgr2=0x7fb3d41024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.348+0000 7fb3d9a25640 1 --2- 172.21.15.94:0/3461124774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d4102040 0x7fb3d41024c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009900 tx=0x7fb3c802bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.348+0000 7fb3d9a25640 1 -- 172.21.15.94:0/3461124774 shutdown_connections 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.348+0000 7fb3d9a25640 1 --2- 172.21.15.94:0/3461124774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3d4102a00 0x7fb3d4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.348+0000 7fb3d9a25640 1 --2- 172.21.15.94:0/3461124774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d4102040 0x7fb3d41024c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.348+0000 7fb3d9a25640 1 --2- 172.21.15.94:0/3461124774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4100e60 0x7fb3d4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.348+0000 7fb3d9a25640 1 -- 172.21.15.94:0/3461124774 >> 172.21.15.94:0/3461124774 conn(0x7fb3d40fc440 msgr2=0x7fb3d40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.348+0000 7fb3d9a25640 1 -- 172.21.15.94:0/3461124774 shutdown_connections 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.348+0000 7fb3d9a25640 1 -- 172.21.15.94:0/3461124774 wait complete. 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.348+0000 7fb3d9a25640 1 Processor -- start 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d9a25640 1 -- start start 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d9a25640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4100e60 0x7fb3d419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d9a25640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d4102040 0x7fb3d419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d9a25640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3d4102a00 0x7fb3d41a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d9a25640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d410ca60 con 0x7fb3d4102040 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d9a25640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d410c8e0 con 0x7fb3d4102a00 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d9a25640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d410cbe0 con 0x7fb3d4100e60 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d4102040 0x7fb3d419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d4102040 0x7fb3d419a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37572/0 (socket says 172.21.15.94:37572) 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d27fc640 1 -- 172.21.15.94:0/4183765658 learned_addr learned my addr 172.21.15.94:0/4183765658 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d37fe640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3d4102a00 0x7fb3d41a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4100e60 0x7fb3d419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d27fc640 1 -- 172.21.15.94:0/4183765658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4100e60 msgr2=0x7fb3d419a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4100e60 0x7fb3d419a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d27fc640 1 -- 172.21.15.94:0/4183765658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3d4102a00 msgr2=0x7fb3d41a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3d4102a00 0x7fb3d41a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d27fc640 1 -- 172.21.15.94:0/4183765658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d41a2050 con 0x7fb3d4102040 2024-04-21T08:39:49.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d37fe640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3d4102a00 0x7fb3d41a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:49.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4100e60 0x7fb3d419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:49.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d4102040 0x7fb3d419a970 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8036040 tx=0x7fb3c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d8a23640 1 -- 172.21.15.94:0/4183765658 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8043020 con 0x7fb3d4102040 2024-04-21T08:39:49.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d8a23640 1 -- 172.21.15.94:0/4183765658 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c80054c0 con 0x7fb3d4102040 2024-04-21T08:39:49.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d9a25640 1 -- 172.21.15.94:0/4183765658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d41a22e0 con 0x7fb3d4102040 2024-04-21T08:39:49.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d8a23640 1 -- 172.21.15.94:0/4183765658 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8003db0 con 0x7fb3d4102040 2024-04-21T08:39:49.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.349+0000 7fb3d9a25640 1 -- 172.21.15.94:0/4183765658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d41a2900 con 0x7fb3d4102040 2024-04-21T08:39:49.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.351+0000 7fb3d8a23640 1 -- 172.21.15.94:0/4183765658 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb3c8046020 con 0x7fb3d4102040 2024-04-21T08:39:49.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3dc612740 1 Processor -- start 2024-04-21T08:39:49.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3dc612740 1 -- start start 2024-04-21T08:39:49.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e99260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e99830 0x5605c1e99cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e9a2d0 0x5605c1ea0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3dc612740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5605c1dfdb10 con 0x5605c1e98e40 2024-04-21T08:39:49.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3dc612740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5605c1d9b240 con 0x5605c1e9a2d0 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3dc612740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5605c1df9ad0 con 0x5605c1e99830 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3d8a23640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb3bc0a3c30 0x7fb3bc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3d37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e99260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3d37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e99260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37582/0 (socket says 172.21.15.94:37582) 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3d37fe640 1 -- 172.21.15.94:0/1988549832 learned_addr learned my addr 172.21.15.94:0/1988549832 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/1988549832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e9a2d0 0x5605c1ea0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3d37fe640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb3bc0a3c30 0x7fb3bc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3d8a23640 1 -- 172.21.15.94:0/4183765658 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fb3c80f2cd0 con 0x7fb3d4102040 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.352+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/1988549832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e99830 0x5605c1e99cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.353+0000 7fb3d37fe640 1 -- 172.21.15.94:0/1988549832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e99830 msgr2=0x5605c1e99cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.353+0000 7fb3d37fe640 1 --2- 172.21.15.94:0/1988549832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e99830 0x5605c1e99cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.353+0000 7fb3d37fe640 1 -- 172.21.15.94:0/1988549832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e9a2d0 msgr2=0x5605c1ea0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.353+0000 7fb3d37fe640 1 --2- 172.21.15.94:0/1988549832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e9a2d0 0x5605c1ea0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.353+0000 7fb3d37fe640 1 -- 172.21.15.94:0/1988549832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605c1b102d0 con 0x5605c1e98e40 2024-04-21T08:39:49.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.353+0000 7fb3d37fe640 1 --2- 172.21.15.94:0/1988549832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e99260 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400ed70 tx=0x7fb3c400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584863310fb02051 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.353+0000 7fb3d37fe640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb3bc0a3c30 0x7fb3bc0a60f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400cc40 tx=0x7fb3c4019000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.353+0000 7fb3b9ffb640 1 -- 172.21.15.94:0/1988549832 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c400ee60 con 0x5605c1e98e40 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.353+0000 7fb3b9ffb640 1 -- 172.21.15.94:0/1988549832 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c4018b70 con 0x5605c1e98e40 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.353+0000 7fb3b9ffb640 1 -- 172.21.15.94:0/1988549832 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40233a0 con 0x5605c1e98e40 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 -- 172.21.15.94:0/1988549832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 msgr2=0x5605c1e99260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 --2- 172.21.15.94:0/1988549832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e99260 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400ed70 tx=0x7fb3c400c740 comp rx=0 tx=0).stop 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 -- 172.21.15.94:0/1988549832 shutdown_connections 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 --2- 172.21.15.94:0/1988549832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e99830 0x5605c1e99cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 --2- 172.21.15.94:0/1988549832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e9a2d0 0x5605c1ea0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 --2- 172.21.15.94:0/1988549832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e99260 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 -- 172.21.15.94:0/1988549832 >> 172.21.15.94:0/1988549832 conn(0x5605c1e94f40 msgr2=0x5605c1e93490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 -- 172.21.15.94:0/1988549832 shutdown_connections 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 -- 172.21.15.94:0/1988549832 wait complete. 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 -- start start 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 Processor -- start 2024-04-21T08:39:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1ea2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 0x5605c1ea33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e9a2d0 0x5605c1eaa410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5605c1ea1450 con 0x5605c1e98e40 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5605c1d86190 con 0x5605c1e99830 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5605c1ea15d0 con 0x5605c1e9a2d0 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3dc612740 10 client.0 _ll_register_callbacks cb 0x5605c1d818b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.354+0000 7fb3d2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1ea2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1ea2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37584/0 (socket says 172.21.15.94:37584) 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/2668083580 learned_addr learned my addr 172.21.15.94:0/2668083580 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d37fe640 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e9a2d0 0x5605c1eaa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 0x5605c1ea33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/2668083580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e9a2d0 msgr2=0x5605c1eaa410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e9a2d0 0x5605c1eaa410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/2668083580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 msgr2=0x5605c1ea33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 0x5605c1ea33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/2668083580 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605c1bda6b0 con 0x5605c1e98e40 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d37fe640 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e9a2d0 0x5605c1eaa410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 0x5605c1ea33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1ea2e90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0107f0 tx=0x7fb3cc010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0b87bc01cde28b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/2668083580 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc00ed40 con 0x5605c1e98e40 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/2668083580 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fb3cc005020 con 0x5605c1e98e40 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3dc612740 1 -- 172.21.15.94:0/2668083580 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3c4018b70 con 0x5605c1e98e40 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/2668083580 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb3cc023050 con 0x5605c1e98e40 2024-04-21T08:39:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3dc612740 10 client.15405 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3dc612740 1 -- 172.21.15.94:0/2668083580 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5605c1ead8a0 con 0x5605c1e98e40 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.355+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/2668083580 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c4018b70 con 0x5605c1e98e40 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.356+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/2668083580 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fb3cc018430 con 0x5605c1e98e40 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.356+0000 7fb3dc612740 10 client.15405 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.356+0000 7fb3dc612740 10 client.15405 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.356+0000 7fb3dc612740 20 client.15405 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.356+0000 7fb3dc612740 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5605c1eb0000 0x5605c1eb1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.356+0000 7fb3dc612740 4 client.15405 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.356+0000 7fb3dc612740 1 -- 172.21.15.94:0/2668083580 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605c1e0b530 con 0x5605c1eb0000 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.356+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5605c1eb0000 0x5605c1eb1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.356+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5605c1eb0000 0x5605c1eb1470 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=31b9f49087e75993 server_cookie=e0a9bb7e4c056501 in_seq=0 out_seq=0 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.356+0000 7fb3b2ffd640 10 client.15405 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/2668083580 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5605c1e0b530 con 0x5605c1eb0000 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3b2ffd640 10 client.15405 handle_command_reply: tid=0 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3dc612740 1 client.15405 shutdown 2024-04-21T08:39:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3dc612740 1 -- 172.21.15.94:0/2668083580 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 msgr2=0x5605c1ea2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3dc612740 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1ea2e90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0107f0 tx=0x7fb3cc010cb0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3dc612740 1 -- 172.21.15.94:0/2668083580 shutdown_connections 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3dc612740 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e9a2d0 0x5605c1eaa410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3dc612740 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5605c1eb0000 0x5605c1eb1470 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3dc612740 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 0x5605c1ea33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3dc612740 1 --2- 172.21.15.94:0/2668083580 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1ea2e90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3dc612740 1 -- 172.21.15.94:0/2668083580 >> 172.21.15.94:0/2668083580 conn(0x5605c1e94f40 msgr2=0x5605c1e92a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.357+0000 7fb3b2ffd640 0 client.15405 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.358+0000 7fb3dc612740 1 -- 172.21.15.94:0/2668083580 shutdown_connections 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.358+0000 7fb3dc612740 1 -- 172.21.15.94:0/2668083580 wait complete. 2024-04-21T08:39:49.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.358+0000 7fb3dc612740 20 client.15405 trim_cache size 0 max 16384 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.375+0000 7fb3dc612740 1 Processor -- start 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.375+0000 7fb3dc612740 1 -- start start 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.375+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.375+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e99830 0x5605c1e9dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.375+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e9a2d0 0x5605c1eaa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.375+0000 7fb3dc612740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5605c1eaf7b0 con 0x5605c1e98e40 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.375+0000 7fb3dc612740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5605c1eafe80 con 0x5605c1e9a2d0 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.375+0000 7fb3dc612740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5605c1eb0550 con 0x5605c1e99830 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.375+0000 7fb3d2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3d2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37592/0 (socket says 172.21.15.94:37592) 2024-04-21T08:39:49.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/3287160098 learned_addr learned my addr 172.21.15.94:0/3287160098 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/3287160098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e99830 0x5605c1e9dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3d37fe640 1 --2- 172.21.15.94:0/3287160098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e9a2d0 0x5605c1eaa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/3287160098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e99830 msgr2=0x5605c1e9dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/3287160098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e99830 0x5605c1e9dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/3287160098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e9a2d0 msgr2=0x5605c1eaa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/3287160098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e9a2d0 0x5605c1eaa3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/3287160098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605c1e94830 con 0x5605c1e98e40 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/3287160098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9d840 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0188a0 tx=0x7fb3cc019ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f550b718b70a85ff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3b97fa640 1 -- 172.21.15.94:0/3287160098 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc018630 con 0x5605c1e98e40 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3b97fa640 1 -- 172.21.15.94:0/3287160098 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3cc018b10 con 0x5605c1e98e40 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.376+0000 7fb3b97fa640 1 -- 172.21.15.94:0/3287160098 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc0052d0 con 0x5605c1e98e40 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 -- 172.21.15.94:0/3287160098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 msgr2=0x5605c1e9d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 --2- 172.21.15.94:0/3287160098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9d840 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0188a0 tx=0x7fb3cc019ef0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 -- 172.21.15.94:0/3287160098 shutdown_connections 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 --2- 172.21.15.94:0/3287160098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e99830 0x5605c1e9dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 --2- 172.21.15.94:0/3287160098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e9a2d0 0x5605c1eaa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 --2- 172.21.15.94:0/3287160098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9d840 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 -- 172.21.15.94:0/3287160098 >> 172.21.15.94:0/3287160098 conn(0x5605c1e94f40 msgr2=0x5605c1e93400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 -- 172.21.15.94:0/3287160098 shutdown_connections 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 -- 172.21.15.94:0/3287160098 wait complete. 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 -- start start 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 Processor -- start 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 0x5605c1e9c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e9a2d0 0x5605c1ea8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5605c1eb12f0 con 0x5605c1e98e40 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5605c1eb0c20 con 0x5605c1e99830 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5605c1eb19c0 con 0x5605c1e9a2d0 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3dc612740 10 client.0 _ll_register_callbacks cb 0x7fb3cc00b6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3d2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3d2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37606/0 (socket says 172.21.15.94:37606) 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.377+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/3579231921 learned_addr learned my addr 172.21.15.94:0/3579231921 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 0x5605c1e9c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/3579231921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e9a2d0 msgr2=0x5605c1ea8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e9a2d0 0x5605c1ea8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/3579231921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 msgr2=0x5605c1e9c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 0x5605c1e9c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3d2ffd640 1 -- 172.21.15.94:0/3579231921 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605c1ea9000 con 0x5605c1e98e40 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 0x5605c1e9c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3d2ffd640 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9bdd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc000c00 tx=0x7fb3cc010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96cd3f0c9ed68a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/3579231921 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc03d060 con 0x5605c1e98e40 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/3579231921 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fb3cc005040 con 0x5605c1e98e40 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3dc612740 1 -- 172.21.15.94:0/3579231921 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3cc018b10 con 0x5605c1e98e40 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/3579231921 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb3cc044050 con 0x5605c1e98e40 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3dc612740 10 client.15411 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3dc612740 1 -- 172.21.15.94:0/3579231921 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5605c1eaa300 con 0x5605c1e98e40 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/3579231921 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3cc018b10 con 0x5605c1e98e40 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/3579231921 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fb3cc0049b0 con 0x5605c1e98e40 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.378+0000 7fb3dc612740 10 client.15411 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:49.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3dc612740 10 client.15411 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3dc612740 20 client.15411 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3dc612740 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5605c1f1a1e0 0x5605c1f16620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3dc612740 4 client.15411 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3dc612740 1 -- 172.21.15.94:0/3579231921 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "status"}) -- 0x5605c1e053a0 con 0x5605c1f1a1e0 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5605c1f1a1e0 0x5605c1f16620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3d27fc640 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5605c1f1a1e0 0x5605c1f16620 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=208763c3af3085ad server_cookie=3e9e7b24b2d44b3a in_seq=0 out_seq=0 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3b2ffd640 10 client.15411 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3b2ffd640 1 -- 172.21.15.94:0/3579231921 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5605c1e053a0 con 0x5605c1f1a1e0 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3b2ffd640 10 client.15411 handle_command_reply: tid=0 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.379+0000 7fb3dc612740 1 client.15411 shutdown 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 1 -- 172.21.15.94:0/3579231921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 msgr2=0x5605c1e9bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9bdd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc000c00 tx=0x7fb3cc010430 comp rx=0 tx=0).stop 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 1 -- 172.21.15.94:0/3579231921 shutdown_connections 2024-04-21T08:39:49.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5605c1e9a2d0 0x5605c1ea8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5605c1f1a1e0 0x5605c1f16620 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605c1e99830 0x5605c1e9c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 1 --2- 172.21.15.94:0/3579231921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5605c1e98e40 0x5605c1e9bdd0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3b2ffd640 0 client.15411 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:39:49.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 1 -- 172.21.15.94:0/3579231921 >> 172.21.15.94:0/3579231921 conn(0x5605c1e94f40 msgr2=0x5605c1ea0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 1 -- 172.21.15.94:0/3579231921 shutdown_connections 2024-04-21T08:39:49.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 1 -- 172.21.15.94:0/3579231921 wait complete. 2024-04-21T08:39:49.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3dc612740 20 client.15411 trim_cache size 0 max 16384 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 1, 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24454, 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 113.728159835, 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 23, 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 93, 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:39:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 151.61961895799999 2024-04-21T08:39:49.388 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:39:49.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 -- 172.21.15.94:0/4183765658 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb3bc0a3c30 msgr2=0x7fb3bc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb3bc0a3c30 0x7fb3bc0a60f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400cc40 tx=0x7fb3c4019000 comp rx=0 tx=0).stop 2024-04-21T08:39:49.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 -- 172.21.15.94:0/4183765658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d4102040 msgr2=0x7fb3d419a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d4102040 0x7fb3d419a970 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8036040 tx=0x7fb3c80042f0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 -- 172.21.15.94:0/4183765658 shutdown_connections 2024-04-21T08:39:49.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb3bc0a3c30 0x7fb3bc0a60f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3d4102a00 0x7fb3d41a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d4102040 0x7fb3d419a970 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 --2- 172.21.15.94:0/4183765658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d4100e60 0x7fb3d419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 -- 172.21.15.94:0/4183765658 >> 172.21.15.94:0/4183765658 conn(0x7fb3d40fc440 msgr2=0x7fb3d40fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 -- 172.21.15.94:0/4183765658 shutdown_connections 2024-04-21T08:39:49.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.380+0000 7fb3d9a25640 1 -- 172.21.15.94:0/4183765658 wait complete. 2024-04-21T08:39:49.390 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:39:49.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.490+0000 7f26ac7c7640 1 -- 172.21.15.94:0/3234809648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26a41029a0 msgr2=0x7f26a41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.490+0000 7f26ac7c7640 1 --2- 172.21.15.94:0/3234809648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26a41029a0 0x7f26a41092c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f269c009900 tx=0x7f269c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.490+0000 7f26ac7c7640 1 -- 172.21.15.94:0/3234809648 shutdown_connections 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.490+0000 7f26ac7c7640 1 --2- 172.21.15.94:0/3234809648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26a4101fe0 0x7f26a4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.490+0000 7f26ac7c7640 1 --2- 172.21.15.94:0/3234809648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26a41029a0 0x7f26a41092c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.490+0000 7f26ac7c7640 1 --2- 172.21.15.94:0/3234809648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a4100e90 0x7f26a41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.490+0000 7f26ac7c7640 1 -- 172.21.15.94:0/3234809648 >> 172.21.15.94:0/3234809648 conn(0x7f26a40fc400 msgr2=0x7f26a40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.490+0000 7f26ac7c7640 1 -- 172.21.15.94:0/3234809648 shutdown_connections 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.490+0000 7f26ac7c7640 1 -- 172.21.15.94:0/3234809648 wait complete. 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26ac7c7640 1 Processor -- start 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26ac7c7640 1 -- start start 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26ac7c7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a4100e90 0x7f26a41981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26ac7c7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26a4101fe0 0x7f26a4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26ac7c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26a41029a0 0x7f26a419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26ac7c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f26a410db30 con 0x7f26a41029a0 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26ac7c7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f26a410d9b0 con 0x7f26a4100e90 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26ac7c7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f26a410dcb0 con 0x7f26a4101fe0 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26aad64640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26a41029a0 0x7f26a419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26aad64640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26a41029a0 0x7f26a419f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37630/0 (socket says 172.21.15.94:37630) 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26aad64640 1 -- 172.21.15.94:0/2158185588 learned_addr learned my addr 172.21.15.94:0/2158185588 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.491+0000 7f26a9d62640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26a4101fe0 0x7f26a4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f26aad64640 1 -- 172.21.15.94:0/2158185588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26a4101fe0 msgr2=0x7f26a4198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f26aad64640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26a4101fe0 0x7f26a4198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f26aad64640 1 -- 172.21.15.94:0/2158185588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a4100e90 msgr2=0x7f26a41981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f26aa563640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a4100e90 0x7f26a41981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f26aad64640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a4100e90 0x7f26a41981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f26aad64640 1 -- 172.21.15.94:0/2158185588 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a419fde0 con 0x7f26a41029a0 2024-04-21T08:39:49.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f26aa563640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a4100e90 0x7f26a41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:49.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f26aad64640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26a41029a0 0x7f26a419f770 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f269c009230 tx=0x7f269c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f269b7fe640 1 -- 172.21.15.94:0/2158185588 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269c043020 con 0x7f26a41029a0 2024-04-21T08:39:49.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f26ac7c7640 1 -- 172.21.15.94:0/2158185588 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a41a0070 con 0x7f26a41029a0 2024-04-21T08:39:49.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f269b7fe640 1 -- 172.21.15.94:0/2158185588 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f269c004aa0 con 0x7f26a41029a0 2024-04-21T08:39:49.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f269b7fe640 1 -- 172.21.15.94:0/2158185588 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269c003c70 con 0x7f26a41029a0 2024-04-21T08:39:49.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.492+0000 7f26ac7c7640 1 -- 172.21.15.94:0/2158185588 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a41a05c0 con 0x7f26a41029a0 2024-04-21T08:39:49.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.494+0000 7f269b7fe640 1 -- 172.21.15.94:0/2158185588 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f269c046020 con 0x7f26a41029a0 2024-04-21T08:39:49.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.494+0000 7f26af3ba740 1 Processor -- start 2024-04-21T08:39:49.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.494+0000 7f26af3ba740 1 -- start start 2024-04-21T08:39:49.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d0a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0a830 0x55dd21d0acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0b2d0 0x55dd21d11eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26af3ba740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55dd21c6eb10 con 0x55dd21d09e40 2024-04-21T08:39:49.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26af3ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd21c0c240 con 0x55dd21d0b2d0 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26af3ba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dd21c6aad0 con 0x55dd21d0a830 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26a9d62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d0a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26a9d62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d0a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37634/0 (socket says 172.21.15.94:37634) 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26a9d62640 1 -- 172.21.15.94:0/2563318306 learned_addr learned my addr 172.21.15.94:0/2563318306 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f269b7fe640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f26900a3bb0 0x7f26900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26aa563640 1 --2- 172.21.15.94:0/2563318306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0a830 0x55dd21d0acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26aad64640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f26900a3bb0 0x7f26900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26a9d62640 1 -- 172.21.15.94:0/2563318306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0a830 msgr2=0x55dd21d0acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26a9d62640 1 --2- 172.21.15.94:0/2563318306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0a830 0x55dd21d0acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26a9d62640 1 -- 172.21.15.94:0/2563318306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0b2d0 msgr2=0x55dd21d11eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26a9d62640 1 --2- 172.21.15.94:0/2563318306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0b2d0 0x55dd21d11eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26a9d62640 1 -- 172.21.15.94:0/2563318306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd219812d0 con 0x55dd21d09e40 2024-04-21T08:39:49.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f269b7fe640 1 -- 172.21.15.94:0/2158185588 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f269c0f2d10 con 0x7f26a41029a0 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26aa563640 1 --2- 172.21.15.94:0/2563318306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0a830 0x55dd21d0acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26a9d62640 1 --2- 172.21.15.94:0/2563318306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d0a260 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f26940058c0 tx=0x7f26940106e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e8d2356ededd89 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f26aad64640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f26900a3bb0 0x7f26900a6070 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f269c0c72d0 tx=0x7f269c089680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f2698ff9640 1 -- 172.21.15.94:0/2563318306 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2694016070 con 0x55dd21d09e40 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f2698ff9640 1 -- 172.21.15.94:0/2563318306 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2694009e90 con 0x55dd21d09e40 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.495+0000 7f2698ff9640 1 -- 172.21.15.94:0/2563318306 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2694011670 con 0x55dd21d09e40 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.496+0000 7f26af3ba740 1 -- 172.21.15.94:0/2563318306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 msgr2=0x55dd21d0a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.496+0000 7f26af3ba740 1 --2- 172.21.15.94:0/2563318306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d0a260 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f26940058c0 tx=0x7f26940106e0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.496+0000 7f26af3ba740 1 -- 172.21.15.94:0/2563318306 shutdown_connections 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.496+0000 7f26af3ba740 1 --2- 172.21.15.94:0/2563318306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0a830 0x55dd21d0acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.496+0000 7f26af3ba740 1 --2- 172.21.15.94:0/2563318306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0b2d0 0x55dd21d11eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.496+0000 7f26af3ba740 1 --2- 172.21.15.94:0/2563318306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d0a260 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.496+0000 7f26af3ba740 1 -- 172.21.15.94:0/2563318306 >> 172.21.15.94:0/2563318306 conn(0x55dd21d05f40 msgr2=0x55dd21d04490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.496+0000 7f26af3ba740 1 -- 172.21.15.94:0/2563318306 shutdown_connections 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.496+0000 7f26af3ba740 1 -- 172.21.15.94:0/2563318306 wait complete. 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.496+0000 7f26af3ba740 1 -- start start 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26af3ba740 1 Processor -- start 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d09e40 0x55dd21d13e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0a830 0x55dd21d143d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0b2d0 0x55dd21d1b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26af3ba740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55dd21d124b0 con 0x55dd21d0a830 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26af3ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd21bf7190 con 0x55dd21d0b2d0 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26af3ba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dd21d12630 con 0x55dd21d09e40 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26af3ba740 10 client.0 _ll_register_callbacks cb 0x55dd21bf28b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26aa563640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0a830 0x55dd21d143d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26aa563640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0a830 0x55dd21d143d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37650/0 (socket says 172.21.15.94:37650) 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26aa563640 1 -- 172.21.15.94:0/3159239408 learned_addr learned my addr 172.21.15.94:0/3159239408 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d09e40 0x55dd21d13e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0b2d0 0x55dd21d1b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26aa563640 1 -- 172.21.15.94:0/3159239408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d09e40 msgr2=0x55dd21d13e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26aa563640 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d09e40 0x55dd21d13e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26aa563640 1 -- 172.21.15.94:0/3159239408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0b2d0 msgr2=0x55dd21d1b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26aa563640 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0b2d0 0x55dd21d1b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26aa563640 1 -- 172.21.15.94:0/3159239408 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd21a4b6b0 con 0x55dd21d0a830 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0b2d0 0x55dd21d1b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f26aa563640 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0a830 0x55dd21d143d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f26a000fad0 tx=0x7f26a000ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2ffcdbd4477000 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f268dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.497+0000 7f268dffb640 1 -- 172.21.15.94:0/3159239408 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a000de50 con 0x55dd21d0a830 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f268dffb640 1 -- 172.21.15.94:0/3159239408 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f26a001b480 con 0x55dd21d0a830 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f26af3ba740 1 -- 172.21.15.94:0/3159239408 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2694009e90 con 0x55dd21d0a830 2024-04-21T08:39:49.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f268dffb640 1 -- 172.21.15.94:0/3159239408 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f26a001b680 con 0x55dd21d0a830 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f26af3ba740 10 client.15423 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f26af3ba740 1 -- 172.21.15.94:0/3159239408 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55dd21d1e850 con 0x55dd21d0a830 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f268dffb640 1 -- 172.21.15.94:0/3159239408 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2694009e90 con 0x55dd21d0a830 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f268dffb640 1 -- 172.21.15.94:0/3159239408 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f26a00109d0 con 0x55dd21d0a830 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f26af3ba740 10 client.15423 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f26af3ba740 10 client.15423 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f26af3ba740 20 client.15423 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd21d20ed0 0x55dd21d22340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f26af3ba740 4 client.15423 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f26af3ba740 1 -- 172.21.15.94:0/3159239408 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd21c7c530 con 0x55dd21d20ed0 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.498+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd21d20ed0 0x55dd21d22340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.499+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd21d20ed0 0x55dd21d22340 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec101671f826973b server_cookie=a46a38d261ca988a in_seq=0 out_seq=0 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.499+0000 7f268dffb640 10 client.15423 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.499+0000 7f268dffb640 1 -- 172.21.15.94:0/3159239408 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd21c7c530 con 0x55dd21d20ed0 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.499+0000 7f268dffb640 10 client.15423 handle_command_reply: tid=0 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.499+0000 7f26af3ba740 1 client.15423 shutdown 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 1 -- 172.21.15.94:0/3159239408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0a830 msgr2=0x55dd21d143d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0a830 0x55dd21d143d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f26a000fad0 tx=0x7f26a000ff90 comp rx=0 tx=0).stop 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 1 -- 172.21.15.94:0/3159239408 shutdown_connections 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd21d20ed0 0x55dd21d22340 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0b2d0 0x55dd21d1b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f268dffb640 0 client.15423 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0a830 0x55dd21d143d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3159239408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d09e40 0x55dd21d13e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 1 -- 172.21.15.94:0/3159239408 >> 172.21.15.94:0/3159239408 conn(0x55dd21d05f40 msgr2=0x55dd21d03a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 1 -- 172.21.15.94:0/3159239408 shutdown_connections 2024-04-21T08:39:49.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 1 -- 172.21.15.94:0/3159239408 wait complete. 2024-04-21T08:39:49.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.500+0000 7f26af3ba740 20 client.15423 trim_cache size 0 max 16384 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.517+0000 7f26af3ba740 1 Processor -- start 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26af3ba740 1 -- start start 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d09e40 0x55dd21d11e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0a830 0x55dd21d1b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0b2d0 0x55dd21d123a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26af3ba740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55dd21d20680 con 0x55dd21d0b2d0 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26af3ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd21d20d50 con 0x55dd21d0a830 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26af3ba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dd21d21420 con 0x55dd21d09e40 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26a9d62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0b2d0 0x55dd21d123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26a9d62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0b2d0 0x55dd21d123a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37656/0 (socket says 172.21.15.94:37656) 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26a9d62640 1 -- 172.21.15.94:0/3663225219 learned_addr learned my addr 172.21.15.94:0/3663225219 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26a9d62640 1 -- 172.21.15.94:0/3663225219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d09e40 msgr2=0x55dd21d11e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3663225219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d09e40 0x55dd21d11e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26a9d62640 1 -- 172.21.15.94:0/3663225219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0a830 msgr2=0x55dd21d1b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3663225219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0a830 0x55dd21d1b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26a9d62640 1 -- 172.21.15.94:0/3663225219 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd21d11ae0 con 0x55dd21d0b2d0 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3663225219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0b2d0 0x55dd21d123a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2694005c90 tx=0x7f26940094b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e7345b0ea26bb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f268ffff640 1 -- 172.21.15.94:0/3663225219 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2694016040 con 0x55dd21d0b2d0 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f268ffff640 1 -- 172.21.15.94:0/3663225219 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2694009e10 con 0x55dd21d0b2d0 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.518+0000 7f268ffff640 1 -- 172.21.15.94:0/3663225219 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2694004620 con 0x55dd21d0b2d0 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 -- 172.21.15.94:0/3663225219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0b2d0 msgr2=0x55dd21d123a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3663225219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0b2d0 0x55dd21d123a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2694005c90 tx=0x7f26940094b0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 -- 172.21.15.94:0/3663225219 shutdown_connections 2024-04-21T08:39:49.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3663225219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d0b2d0 0x55dd21d123a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3663225219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0a830 0x55dd21d1b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3663225219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d09e40 0x55dd21d11e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 -- 172.21.15.94:0/3663225219 >> 172.21.15.94:0/3663225219 conn(0x55dd21d05f40 msgr2=0x55dd21d0e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 -- 172.21.15.94:0/3663225219 shutdown_connections 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 -- 172.21.15.94:0/3663225219 wait complete. 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 -- start start 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 Processor -- start 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d17600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0a830 0x55dd21d19b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0b2d0 0x55dd21d1a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55dd21d221c0 con 0x55dd21d09e40 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd21d21af0 con 0x55dd21d0a830 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.519+0000 7f26af3ba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dd21d22890 con 0x55dd21d0b2d0 2024-04-21T08:39:49.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26af3ba740 10 client.0 _ll_register_callbacks cb 0x7f26a0008690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26a9d62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d17600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26a9d62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d17600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37662/0 (socket says 172.21.15.94:37662) 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26a9d62640 1 -- 172.21.15.94:0/3358251273 learned_addr learned my addr 172.21.15.94:0/3358251273 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0b2d0 0x55dd21d1a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26aa563640 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0a830 0x55dd21d19b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26a9d62640 1 -- 172.21.15.94:0/3358251273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0b2d0 msgr2=0x55dd21d1a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0b2d0 0x55dd21d1a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26a9d62640 1 -- 172.21.15.94:0/3358251273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0a830 msgr2=0x55dd21d19b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0a830 0x55dd21d19b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26a9d62640 1 -- 172.21.15.94:0/3358251273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd21d1a8a0 con 0x55dd21d09e40 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26aa563640 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0a830 0x55dd21d19b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:39:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26a9d62640 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d17600 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2694006010 tx=0x7f2694010c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ff53c58baea0af server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f268dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f268dffb640 1 -- 172.21.15.94:0/3358251273 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2694016070 con 0x55dd21d09e40 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f268dffb640 1 -- 172.21.15.94:0/3358251273 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f269402f040 con 0x55dd21d09e40 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26af3ba740 1 -- 172.21.15.94:0/3358251273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2694009e10 con 0x55dd21d09e40 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f268dffb640 1 -- 172.21.15.94:0/3358251273 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f269403e050 con 0x55dd21d09e40 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26af3ba740 10 client.15429 fetch_fsmap learned FSMap version 23 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f26af3ba740 1 -- 172.21.15.94:0/3358251273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55dd21d87100 con 0x55dd21d09e40 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.520+0000 7f268dffb640 1 -- 172.21.15.94:0/3358251273 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2694009e10 con 0x55dd21d09e40 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.521+0000 7f268dffb640 1 -- 172.21.15.94:0/3358251273 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f2694032070 con 0x55dd21d09e40 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.521+0000 7f26af3ba740 10 client.15429 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.521+0000 7f26af3ba740 10 client.15429 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.521+0000 7f26af3ba740 20 client.15429 populate_metadata read hostname 'smithi094' 2024-04-21T08:39:49.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.521+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd21d8b5f0 0x55dd21d8a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.521+0000 7f26af3ba740 4 client.15429 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.521+0000 7f26af3ba740 1 -- 172.21.15.94:0/3358251273 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "status"}) -- 0x55dd21c763a0 con 0x55dd21d8b5f0 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.521+0000 7f26aa563640 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd21d8b5f0 0x55dd21d8a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.521+0000 7f268dffb640 10 client.15429 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.521+0000 7f26aa563640 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd21d8b5f0 0x55dd21d8a300 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a3e4658b401a65d server_cookie=596f8de76fddddc1 in_seq=0 out_seq=0 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f268dffb640 1 -- 172.21.15.94:0/3358251273 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55dd21c763a0 con 0x55dd21d8b5f0 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f268dffb640 10 client.15429 handle_command_reply: tid=0 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 client.15429 shutdown 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 -- 172.21.15.94:0/3358251273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 msgr2=0x55dd21d17600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d17600 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2694006010 tx=0x7f2694010c90 comp rx=0 tx=0).stop 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 -- 172.21.15.94:0/3358251273 shutdown_connections 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd21d8b5f0 0x55dd21d8a300 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd21d0b2d0 0x55dd21d1a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd21d0a830 0x55dd21d19b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 --2- 172.21.15.94:0/3358251273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd21d09e40 0x55dd21d17600 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f268dffb640 0 client.15429 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:39:49.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 -- 172.21.15.94:0/3358251273 >> 172.21.15.94:0/3358251273 conn(0x55dd21d05f40 msgr2=0x55dd21d11d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 -- 172.21.15.94:0/3358251273 shutdown_connections 2024-04-21T08:39:49.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 1 -- 172.21.15.94:0/3358251273 wait complete. 2024-04-21T08:39:49.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26af3ba740 20 client.15429 trim_cache size 0 max 16384 2024-04-21T08:39:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:39:49.528 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:39:49.528 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-04-21T08:39:49.529 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24488, 2024-04-21T08:39:49.529 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-21T08:39:49.529 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-21T08:39:49.529 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-21T08:39:49.529 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 137.95191736199999, 2024-04-21T08:39:49.529 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 23, 2024-04-21T08:39:49.529 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 93, 2024-04-21T08:39:49.529 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:39:49.529 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 143.223741273 2024-04-21T08:39:49.529 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:39:49.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 -- 172.21.15.94:0/2158185588 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f26900a3bb0 msgr2=0x7f26900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f26900a3bb0 0x7f26900a6070 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f269c0c72d0 tx=0x7f269c089680 comp rx=0 tx=0).stop 2024-04-21T08:39:49.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 -- 172.21.15.94:0/2158185588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26a41029a0 msgr2=0x7f26a419f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26a41029a0 0x7f26a419f770 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f269c009230 tx=0x7f269c0042f0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 -- 172.21.15.94:0/2158185588 shutdown_connections 2024-04-21T08:39:49.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f26900a3bb0 0x7f26900a6070 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26a4101fe0 0x7f26a4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26a41029a0 0x7f26a419f770 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 --2- 172.21.15.94:0/2158185588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a4100e90 0x7f26a41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 -- 172.21.15.94:0/2158185588 >> 172.21.15.94:0/2158185588 conn(0x7f26a40fc400 msgr2=0x7f26a41053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 -- 172.21.15.94:0/2158185588 shutdown_connections 2024-04-21T08:39:49.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.522+0000 7f26ac7c7640 1 -- 172.21.15.94:0/2158185588 wait complete. 2024-04-21T08:39:49.532 INFO:teuthology.run_tasks:Running task sequential... 2024-04-21T08:39:49.544 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-21T08:39:49.544 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-21T08:39:49.544 INFO:teuthology.task.print:Enabling mgr modules 2024-04-21T08:39:49.544 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-21T08:39:49.545 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:39:49.545 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:39:49.545 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-21T08:39:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.645+0000 7fb1c325c640 1 -- 172.21.15.94:0/338660846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1bc1031a0 msgr2=0x7fb1bc103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.645+0000 7fb1c325c640 1 --2- 172.21.15.94:0/338660846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1bc1031a0 0x7fb1bc103620 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009900 tx=0x7fb1a802bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.645+0000 7fb1c325c640 1 -- 172.21.15.94:0/338660846 shutdown_connections 2024-04-21T08:39:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.645+0000 7fb1c325c640 1 --2- 172.21.15.94:0/338660846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1bc103b60 0x7fb1bc10a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.645+0000 7fb1c325c640 1 --2- 172.21.15.94:0/338660846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1bc1031a0 0x7fb1bc103620 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.645+0000 7fb1c325c640 1 --2- 172.21.15.94:0/338660846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc101fb0 0x7fb1bc1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.645+0000 7fb1c325c640 1 -- 172.21.15.94:0/338660846 >> 172.21.15.94:0/338660846 conn(0x7fb1bc0fd540 msgr2=0x7fb1bc0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.645+0000 7fb1c325c640 1 -- 172.21.15.94:0/338660846 shutdown_connections 2024-04-21T08:39:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.645+0000 7fb1c325c640 1 -- 172.21.15.94:0/338660846 wait complete. 2024-04-21T08:39:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c325c640 1 Processor -- start 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c325c640 1 -- start start 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c325c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1bc101fb0 0x7fb1bc1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c325c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1bc1031a0 0x7fb1bc198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c325c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc103b60 0x7fb1bc19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c325c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc10daf0 con 0x7fb1bc1031a0 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c325c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc10d970 con 0x7fb1bc103b60 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c325c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc10dc70 con 0x7fb1bc101fb0 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1bbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1bc1031a0 0x7fb1bc198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1bbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1bc1031a0 0x7fb1bc198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37690/0 (socket says 172.21.15.94:37690) 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1bbfff640 1 -- 172.21.15.94:0/3986212046 learned_addr learned my addr 172.21.15.94:0/3986212046 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c17f9640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc103b60 0x7fb1bc19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1bbfff640 1 -- 172.21.15.94:0/3986212046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1bc101fb0 msgr2=0x7fb1bc1980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c0ff8640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1bc101fb0 0x7fb1bc1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1bbfff640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1bc101fb0 0x7fb1bc1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1bbfff640 1 -- 172.21.15.94:0/3986212046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc103b60 msgr2=0x7fb1bc19f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1bbfff640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc103b60 0x7fb1bc19f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1bbfff640 1 -- 172.21.15.94:0/3986212046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1bc19fd00 con 0x7fb1bc1031a0 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.646+0000 7fb1c17f9640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc103b60 0x7fb1bc19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.647+0000 7fb1bbfff640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1bc1031a0 0x7fb1bc198620 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8006010 tx=0x7fb1a80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.647+0000 7fb1b9ffb640 1 -- 172.21.15.94:0/3986212046 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8043020 con 0x7fb1bc1031a0 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.647+0000 7fb1b9ffb640 1 -- 172.21.15.94:0/3986212046 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1a8004aa0 con 0x7fb1bc1031a0 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.647+0000 7fb1c325c640 1 -- 172.21.15.94:0/3986212046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1bc19ff90 con 0x7fb1bc1031a0 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.647+0000 7fb1b9ffb640 1 -- 172.21.15.94:0/3986212046 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8003c70 con 0x7fb1bc1031a0 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.647+0000 7fb1c325c640 1 -- 172.21.15.94:0/3986212046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1bc1a04a0 con 0x7fb1bc1031a0 2024-04-21T08:39:49.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.648+0000 7fb1c325c640 1 -- 172.21.15.94:0/3986212046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1bc1092d0 con 0x7fb1bc1031a0 2024-04-21T08:39:49.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.649+0000 7fb1b9ffb640 1 -- 172.21.15.94:0/3986212046 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb1a8046020 con 0x7fb1bc1031a0 2024-04-21T08:39:49.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.650+0000 7fb1b9ffb640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1ac0a3cd0 0x7fb1ac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:49.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.650+0000 7fb1c0ff8640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1ac0a3cd0 0x7fb1ac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:49.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.650+0000 7fb1b9ffb640 1 -- 172.21.15.94:0/3986212046 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fb1a80f31b0 con 0x7fb1bc1031a0 2024-04-21T08:39:49.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.650+0000 7fb1c0ff8640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1ac0a3cd0 0x7fb1ac0a6190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0005fd0 tx=0x7fb1b0005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:49.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.653+0000 7fb1b9ffb640 1 -- 172.21.15.94:0/3986212046 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb1a804d020 con 0x7fb1bc1031a0 2024-04-21T08:39:49.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:49.879+0000 7fb1c325c640 1 -- 172.21.15.94:0/3986212046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fb1bc02d7e0 con 0x7fb1bc1031a0 2024-04-21T08:39:50.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:49 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1442838970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:50.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:49 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1484408549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:50.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:49 smithi094 ceph-mon[27988]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:50.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:49 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1442838970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:50.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:49 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1484408549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:50.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:49 smithi173 ceph-mon[31428]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:50.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:49 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1442838970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:39:50.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:49 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1484408549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:39:50.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:49 smithi111 ceph-mon[31493]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:39:50.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.727+0000 7fb1b9ffb640 1 -- 172.21.15.94:0/3986212046 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fb1a80bf790 con 0x7fb1bc1031a0 2024-04-21T08:39:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 -- 172.21.15.94:0/3986212046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1ac0a3cd0 msgr2=0x7fb1ac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1ac0a3cd0 0x7fb1ac0a6190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0005fd0 tx=0x7fb1b0005c70 comp rx=0 tx=0).stop 2024-04-21T08:39:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 -- 172.21.15.94:0/3986212046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1bc1031a0 msgr2=0x7fb1bc198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1bc1031a0 0x7fb1bc198620 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8006010 tx=0x7fb1a80042f0 comp rx=0 tx=0).stop 2024-04-21T08:39:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 -- 172.21.15.94:0/3986212046 shutdown_connections 2024-04-21T08:39:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7fb1ac0a3cd0 0x7fb1ac0a6190 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc103b60 0x7fb1bc19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb1bc1031a0 0x7fb1bc198620 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 --2- 172.21.15.94:0/3986212046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1bc101fb0 0x7fb1bc1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 -- 172.21.15.94:0/3986212046 >> 172.21.15.94:0/3986212046 conn(0x7fb1bc0fd540 msgr2=0x7fb1bc1064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 -- 172.21.15.94:0/3986212046 shutdown_connections 2024-04-21T08:39:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.730+0000 7fb1c325c640 1 -- 172.21.15.94:0/3986212046 wait complete. 2024-04-21T08:39:50.751 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-21T08:39:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.859+0000 7f3aa85f4640 1 -- 172.21.15.94:0/2532251993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3aa0102060 msgr2=0x7f3aa01024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.859+0000 7f3aa85f4640 1 --2- 172.21.15.94:0/2532251993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3aa0102060 0x7f3aa01024e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009900 tx=0x7f3a9002bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.859+0000 7f3aa85f4640 1 -- 172.21.15.94:0/2532251993 shutdown_connections 2024-04-21T08:39:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.859+0000 7f3aa85f4640 1 --2- 172.21.15.94:0/2532251993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3aa0102a20 0x7f3aa0109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.859+0000 7f3aa85f4640 1 --2- 172.21.15.94:0/2532251993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3aa0102060 0x7f3aa01024e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.859+0000 7f3aa85f4640 1 --2- 172.21.15.94:0/2532251993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa0100e80 0x7f3aa01012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.859+0000 7f3aa85f4640 1 -- 172.21.15.94:0/2532251993 >> 172.21.15.94:0/2532251993 conn(0x7f3aa00fc440 msgr2=0x7f3aa00fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.859+0000 7f3aa85f4640 1 -- 172.21.15.94:0/2532251993 shutdown_connections 2024-04-21T08:39:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.859+0000 7f3aa85f4640 1 -- 172.21.15.94:0/2532251993 wait complete. 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa85f4640 1 Processor -- start 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa85f4640 1 -- start start 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa85f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3aa0100e80 0x7f3aa0104e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa85f4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa0102060 0x7f3aa01034d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa85f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3aa0102a20 0x7f3aa0103a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa85f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa010ca00 con 0x7f3aa0102a20 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa85f4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa010c880 con 0x7f3aa0102060 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa85f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa010cb80 con 0x7f3aa0100e80 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa6b91640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3aa0102a20 0x7f3aa0103a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa6b91640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3aa0102a20 0x7f3aa0103a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37722/0 (socket says 172.21.15.94:37722) 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa6b91640 1 -- 172.21.15.94:0/1893286139 learned_addr learned my addr 172.21.15.94:0/1893286139 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa5b8f640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa0102060 0x7f3aa01034d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa6390640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3aa0100e80 0x7f3aa0104e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa6b91640 1 -- 172.21.15.94:0/1893286139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3aa0100e80 msgr2=0x7f3aa0104e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa6b91640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3aa0100e80 0x7f3aa0104e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:50.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa6b91640 1 -- 172.21.15.94:0/1893286139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa0102060 msgr2=0x7f3aa01034d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:50.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa6b91640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa0102060 0x7f3aa01034d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:50.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.860+0000 7f3aa6b91640 1 -- 172.21.15.94:0/1893286139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa01042a0 con 0x7f3aa0102a20 2024-04-21T08:39:50.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.861+0000 7f3aa6b91640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3aa0102a20 0x7f3aa0103a10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3a98009fd0 tx=0x7f3a9800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:50.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.861+0000 7f3a977fe640 1 -- 172.21.15.94:0/1893286139 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98017070 con 0x7f3aa0102a20 2024-04-21T08:39:50.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.861+0000 7f3a977fe640 1 -- 172.21.15.94:0/1893286139 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a98003e20 con 0x7f3aa0102a20 2024-04-21T08:39:50.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.861+0000 7f3aa85f4640 1 -- 172.21.15.94:0/1893286139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa01a22b0 con 0x7f3aa0102a20 2024-04-21T08:39:50.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.861+0000 7f3a977fe640 1 -- 172.21.15.94:0/1893286139 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a980106c0 con 0x7f3aa0102a20 2024-04-21T08:39:50.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.861+0000 7f3aa85f4640 1 -- 172.21.15.94:0/1893286139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa01a2780 con 0x7f3aa0102a20 2024-04-21T08:39:50.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.862+0000 7f3aa85f4640 1 -- 172.21.15.94:0/1893286139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa01a2a40 con 0x7f3aa0102a20 2024-04-21T08:39:50.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.863+0000 7f3a977fe640 1 -- 172.21.15.94:0/1893286139 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f3a98004140 con 0x7f3aa0102a20 2024-04-21T08:39:50.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.864+0000 7f3a977fe640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a8c0a3ca0 0x7f3a8c0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:50.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.864+0000 7f3aa6390640 1 -- 172.21.15.94:0/1893286139 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a8c0a3ca0 msgr2=0x7f3a8c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3394521972 2024-04-21T08:39:50.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.864+0000 7f3a977fe640 1 -- 172.21.15.94:0/1893286139 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f3a980d03f0 con 0x7f3aa0102a20 2024-04-21T08:39:50.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.864+0000 7f3aa6390640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a8c0a3ca0 0x7f3a8c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:39:50.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:50.868+0000 7f3a977fe640 1 -- 172.21.15.94:0/1893286139 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a980047e0 con 0x7f3aa0102a20 2024-04-21T08:39:51.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:50 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3986212046' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:39:51.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:50 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: ignoring --setuser ceph since I am not root 2024-04-21T08:39:51.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:50 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: ignoring --setgroup ceph since I am not root 2024-04-21T08:39:51.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:50 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:50.920+0000 7f5d310f0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:39:51.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:50 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3986212046' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:39:51.011 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:50 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: ignoring --setuser ceph since I am not root 2024-04-21T08:39:51.011 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:50 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: ignoring --setgroup ceph since I am not root 2024-04-21T08:39:51.011 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:50 smithi173 ceph-mgr[33243]: -- 172.21.15.173:0/4220270388 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560e846385a0 con 0x560e8460d000 2024-04-21T08:39:51.011 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:50 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:50.909+0000 7f068bbc7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:39:51.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.054+0000 7f3aa85f4640 1 -- 172.21.15.94:0/1893286139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f3aa01a3080 con 0x7f3aa0102a20 2024-04-21T08:39:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.058+0000 7f3a977fe640 1 -- 172.21.15.94:0/1893286139 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f3a9809c380 con 0x7f3aa0102a20 2024-04-21T08:39:51.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 -- 172.21.15.94:0/1893286139 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a8c0a3ca0 msgr2=0x7f3a8c0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a8c0a3ca0 0x7f3a8c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 -- 172.21.15.94:0/1893286139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3aa0102a20 msgr2=0x7f3aa0103a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3aa0102a20 0x7f3aa0103a10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3a98009fd0 tx=0x7f3a9800b260 comp rx=0 tx=0).stop 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 -- 172.21.15.94:0/1893286139 shutdown_connections 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f3a8c0a3ca0 0x7f3a8c0a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3aa0102a20 0x7f3aa0103a10 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa0102060 0x7f3aa01034d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 --2- 172.21.15.94:0/1893286139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3aa0100e80 0x7f3aa0104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 -- 172.21.15.94:0/1893286139 >> 172.21.15.94:0/1893286139 conn(0x7f3aa00fc440 msgr2=0x7f3aa00fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 -- 172.21.15.94:0/1893286139 shutdown_connections 2024-04-21T08:39:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.061+0000 7f3aa85f4640 1 -- 172.21.15.94:0/1893286139 wait complete. 2024-04-21T08:39:51.071 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-21T08:39:51.087 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:50 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3986212046' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:39:51.087 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:50 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: ignoring --setuser ceph since I am not root 2024-04-21T08:39:51.088 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:50 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: ignoring --setgroup ceph since I am not root 2024-04-21T08:39:51.088 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:50 smithi111 ceph-mgr[32920]: -- 172.21.15.111:0/2262707804 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563178e065a0 con 0x563178ddac00 2024-04-21T08:39:51.088 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:50 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:50.905+0000 7f5e25acf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:39:51.088 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:51 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:51.088+0000 7f5e25acf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:39:51.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 -- 172.21.15.94:0/2498371518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6c103b70 msgr2=0x7f9d6c10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:51.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 --2- 172.21.15.94:0/2498371518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6c103b70 0x7f9d6c10a400 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009ca0 tx=0x7f9d6402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 -- 172.21.15.94:0/2498371518 shutdown_connections 2024-04-21T08:39:51.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 --2- 172.21.15.94:0/2498371518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6c103b70 0x7f9d6c10a400 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 --2- 172.21.15.94:0/2498371518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d6c1031b0 0x7f9d6c103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 --2- 172.21.15.94:0/2498371518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c101fc0 0x7f9d6c1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 -- 172.21.15.94:0/2498371518 >> 172.21.15.94:0/2498371518 conn(0x7f9d6c0fd530 msgr2=0x7f9d6c0ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:51.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 -- 172.21.15.94:0/2498371518 shutdown_connections 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 -- 172.21.15.94:0/2498371518 wait complete. 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 Processor -- start 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.177+0000 7f9d725f4640 1 -- start start 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d725f4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d6c101fc0 0x7f9d6c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d725f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6c1031b0 0x7f9d6c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d725f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c103b70 0x7f9d6c19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d725f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c10db00 con 0x7f9d6c1031b0 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d725f4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c10d980 con 0x7f9d6c101fc0 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d725f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c10dc80 con 0x7f9d6c103b70 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d6b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6c1031b0 0x7f9d6c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d6b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6c1031b0 0x7f9d6c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37744/0 (socket says 172.21.15.94:37744) 2024-04-21T08:39:51.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d6b7fe640 1 -- 172.21.15.94:0/3338195246 learned_addr learned my addr 172.21.15.94:0/3338195246 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d6bfff640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d6c101fc0 0x7f9d6c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d70b91640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c103b70 0x7f9d6c19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d6b7fe640 1 -- 172.21.15.94:0/3338195246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c103b70 msgr2=0x7f9d6c19f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d6b7fe640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c103b70 0x7f9d6c19f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d6b7fe640 1 -- 172.21.15.94:0/3338195246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d6c101fc0 msgr2=0x7f9d6c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d6b7fe640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d6c101fc0 0x7f9d6c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d6b7fe640 1 -- 172.21.15.94:0/3338195246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c19fcd0 con 0x7f9d6c1031b0 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d70b91640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c103b70 0x7f9d6c19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.178+0000 7f9d6b7fe640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6c1031b0 0x7f9d6c198730 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000e9f0 tx=0x7f9d6000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.179+0000 7f9d697fa640 1 -- 172.21.15.94:0/3338195246 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6000cd80 con 0x7f9d6c1031b0 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.179+0000 7f9d725f4640 1 -- 172.21.15.94:0/3338195246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6c19ffc0 con 0x7f9d6c1031b0 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.179+0000 7f9d697fa640 1 -- 172.21.15.94:0/3338195246 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d600108e0 con 0x7f9d6c1031b0 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.179+0000 7f9d697fa640 1 -- 172.21.15.94:0/3338195246 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60018980 con 0x7f9d6c1031b0 2024-04-21T08:39:51.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.179+0000 7f9d725f4640 1 -- 172.21.15.94:0/3338195246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d6c1a0480 con 0x7f9d6c1031b0 2024-04-21T08:39:51.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.180+0000 7f9d725f4640 1 -- 172.21.15.94:0/3338195246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d6c109280 con 0x7f9d6c1031b0 2024-04-21T08:39:51.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.182+0000 7f9d697fa640 1 -- 172.21.15.94:0/3338195246 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f9d60010430 con 0x7f9d6c1031b0 2024-04-21T08:39:51.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.183+0000 7f9d697fa640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9d540a3ca0 0x7f9d540a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:51.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.183+0000 7f9d6bfff640 1 -- 172.21.15.94:0/3338195246 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9d540a3ca0 msgr2=0x7f9d540a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3394521972 2024-04-21T08:39:51.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.183+0000 7f9d697fa640 1 -- 172.21.15.94:0/3338195246 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f9d600d0f50 con 0x7f9d6c1031b0 2024-04-21T08:39:51.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.183+0000 7f9d6bfff640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9d540a3ca0 0x7f9d540a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:39:51.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.185+0000 7f9d697fa640 1 -- 172.21.15.94:0/3338195246 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d6009cee0 con 0x7f9d6c1031b0 2024-04-21T08:39:51.341 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:51 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:51.114+0000 7f5d310f0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:39:51.341 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:51 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:51.226+0000 7f5d310f0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:39:51.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.370+0000 7f9d725f4640 1 -- 172.21.15.94:0/3338195246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f9d6c062bc0 con 0x7f9d6c1031b0 2024-04-21T08:39:51.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.374+0000 7f9d697fa640 1 -- 172.21.15.94:0/3338195246 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f9d600a11e0 con 0x7f9d6c1031b0 2024-04-21T08:39:51.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.377+0000 7f9d725f4640 1 -- 172.21.15.94:0/3338195246 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9d540a3ca0 msgr2=0x7f9d540a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:39:51.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.377+0000 7f9d725f4640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9d540a3ca0 0x7f9d540a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.377+0000 7f9d725f4640 1 -- 172.21.15.94:0/3338195246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6c1031b0 msgr2=0x7f9d6c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:51.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.377+0000 7f9d725f4640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6c1031b0 0x7f9d6c198730 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000e9f0 tx=0x7f9d6000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.377+0000 7f9d725f4640 1 -- 172.21.15.94:0/3338195246 shutdown_connections 2024-04-21T08:39:51.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.377+0000 7f9d725f4640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9d540a3ca0 0x7f9d540a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.378+0000 7f9d725f4640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c103b70 0x7f9d6c19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.378+0000 7f9d725f4640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6c1031b0 0x7f9d6c198730 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.378+0000 7f9d725f4640 1 --2- 172.21.15.94:0/3338195246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d6c101fc0 0x7f9d6c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:51.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.378+0000 7f9d725f4640 1 -- 172.21.15.94:0/3338195246 >> 172.21.15.94:0/3338195246 conn(0x7f9d6c0fd530 msgr2=0x7f9d6c106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:51.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.378+0000 7f9d725f4640 1 -- 172.21.15.94:0/3338195246 shutdown_connections 2024-04-21T08:39:51.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:51.378+0000 7f9d725f4640 1 -- 172.21.15.94:0/3338195246 wait complete. 2024-04-21T08:39:51.392 INFO:teuthology.run_tasks:Running task kclient... 2024-04-21T08:39:51.405 INFO:tasks.kclient:Mounting kernel clients... 2024-04-21T08:39:51.405 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-21T08:39:51.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:39:51.405 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-04-21T08:39:51.421 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-21T08:39:51.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:39:51.422 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:39:51.437 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:51 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:51.192+0000 7f5e25acf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:39:51.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:39:51.487 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2024-04-21T08:39:51.507 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:51.095+0000 7f068bbc7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:39:51.507 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:51.198+0000 7f068bbc7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:39:51.596 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:51 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:51.568+0000 7f5d310f0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:39:51.597 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-21T08:39:51.597 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T08:39:51.597 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T08:39:51.597 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph fs ls 2024-04-21T08:39:51.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:51 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3986212046' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:39:51.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:51 smithi173 ceph-mon[31428]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:39:51.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:51 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1893286139' entity='client.admin' 2024-04-21T08:39:51.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:51 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3338195246' entity='client.admin' 2024-04-21T08:39:51.761 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:51.507+0000 7f068bbc7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:39:51.761 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:51.592+0000 7f068bbc7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:39:51.854 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:51.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:51 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3986212046' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:39:51.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:51 smithi111 ceph-mon[31493]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:39:51.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:51 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1893286139' entity='client.admin' 2024-04-21T08:39:51.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:51 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3338195246' entity='client.admin' 2024-04-21T08:39:51.937 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:51 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:51.498+0000 7f5e25acf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:39:51.937 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:51 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:51.582+0000 7f5e25acf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:39:51.984 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:51 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:51.664+0000 7f5d310f0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:39:51.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:51 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3986212046' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:39:51.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:51 smithi094 ceph-mon[27988]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:39:51.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:51 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1893286139' entity='client.admin' 2024-04-21T08:39:51.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:51 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3338195246' entity='client.admin' 2024-04-21T08:39:53.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:52 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:52.914+0000 7f5d310f0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:39:53.092 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:52 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:52.757+0000 7f5e25acf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:39:53.092 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:52 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:52.926+0000 7f5e25acf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:39:53.093 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:53 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:53.093+0000 7f5e25acf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:39:53.120 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:52 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:52.779+0000 7f068bbc7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:39:53.120 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:52 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:52.948+0000 7f068bbc7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:39:53.120 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:53 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:53.121+0000 7f068bbc7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:39:53.330 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:53.094+0000 7f5d310f0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:39:53.330 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:53.267+0000 7f5d310f0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:39:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.565+0000 7f6f10476700 1 -- 172.21.15.94:0/353211092 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f08107980 msgr2=0x7f6f08107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.565+0000 7f6f10476700 1 --2- 172.21.15.94:0/353211092 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f08107980 0x7f6f08107d60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6f04009990 tx=0x7f6f04009ca0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.566+0000 7f6f10476700 1 -- 172.21.15.94:0/353211092 shutdown_connections 2024-04-21T08:39:53.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.566+0000 7f6f10476700 1 --2- 172.21.15.94:0/353211092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f081022f0 0x7f6f0810e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.566+0000 7f6f10476700 1 --2- 172.21.15.94:0/353211092 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f08107980 0x7f6f08107d60 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.566+0000 7f6f10476700 1 --2- 172.21.15.94:0/353211092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f08101930 0x7f6f08101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.566+0000 7f6f10476700 1 -- 172.21.15.94:0/353211092 >> 172.21.15.94:0/353211092 conn(0x7f6f080788b0 msgr2=0x7f6f08078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:53.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.566+0000 7f6f10476700 1 -- 172.21.15.94:0/353211092 shutdown_connections 2024-04-21T08:39:53.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.566+0000 7f6f10476700 1 -- 172.21.15.94:0/353211092 wait complete. 2024-04-21T08:39:53.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.566+0000 7f6f10476700 1 Processor -- start 2024-04-21T08:39:53.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f10476700 1 -- start start 2024-04-21T08:39:53.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f10476700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f08101930 0x7f6f08199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:53.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f10476700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f081022f0 0x7f6f0819a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:53.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f10476700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f08107980 0x7f6f081a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:53.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f10476700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6f08111ad0 con 0x7f6f08107980 2024-04-21T08:39:53.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f10476700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6f08111950 con 0x7f6f081022f0 2024-04-21T08:39:53.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f10476700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f08111c50 con 0x7f6f08101930 2024-04-21T08:39:53.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0ea13700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f08107980 0x7f6f081a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:53.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0e212700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f08101930 0x7f6f08199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:53.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0ea13700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f08107980 0x7f6f081a5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58240/0 (socket says 172.21.15.94:58240) 2024-04-21T08:39:53.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0ea13700 1 -- 172.21.15.94:0/848352717 learned_addr learned my addr 172.21.15.94:0/848352717 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:53.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0da11700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f081022f0 0x7f6f0819a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:53.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0ea13700 1 -- 172.21.15.94:0/848352717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f08101930 msgr2=0x7f6f08199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:53.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0ea13700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f08101930 0x7f6f08199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0ea13700 1 -- 172.21.15.94:0/848352717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f081022f0 msgr2=0x7f6f0819a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:53.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0ea13700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f081022f0 0x7f6f0819a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0ea13700 1 -- 172.21.15.94:0/848352717 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f081a5590 con 0x7f6f08107980 2024-04-21T08:39:53.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.567+0000 7f6f0e212700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f08101930 0x7f6f08199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:39:53.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.568+0000 7f6f0ea13700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f08107980 0x7f6f081a5050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6f00007ee0 tx=0x7f6f0000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:53.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.568+0000 7f6eff7fe700 1 -- 172.21.15.94:0/848352717 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00004bb0 con 0x7f6f08107980 2024-04-21T08:39:53.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.568+0000 7f6eff7fe700 1 -- 172.21.15.94:0/848352717 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f00004d50 con 0x7f6f08107980 2024-04-21T08:39:53.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.568+0000 7f6eff7fe700 1 -- 172.21.15.94:0/848352717 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00005920 con 0x7f6f08107980 2024-04-21T08:39:53.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.568+0000 7f6f10476700 1 -- 172.21.15.94:0/848352717 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f081a57c0 con 0x7f6f08107980 2024-04-21T08:39:53.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.568+0000 7f6f10476700 1 -- 172.21.15.94:0/848352717 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f081a5cb0 con 0x7f6f08107980 2024-04-21T08:39:53.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.569+0000 7f6f10476700 1 -- 172.21.15.94:0/848352717 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f08109ea0 con 0x7f6f08107980 2024-04-21T08:39:53.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.570+0000 7f6eff7fe700 1 -- 172.21.15.94:0/848352717 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f6f0001a020 con 0x7f6f08107980 2024-04-21T08:39:53.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.571+0000 7f6eff7fe700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6ef40a3d10 0x7f6ef40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:53.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.571+0000 7f6f0e212700 1 -- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6ef40a3d10 msgr2=0x7f6ef40a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3394521972 2024-04-21T08:39:53.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.571+0000 7f6eff7fe700 1 -- 172.21.15.94:0/848352717 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7f6f000d03b0 con 0x7f6f08107980 2024-04-21T08:39:53.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.571+0000 7f6f0e212700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6ef40a3d10 0x7f6ef40a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:39:53.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.574+0000 7f6eff7fe700 1 -- 172.21.15.94:0/848352717 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6f00010070 con 0x7f6f08107980 2024-04-21T08:39:53.667 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:53.572+0000 7f5d310f0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:39:53.667 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:53.668+0000 7f5d310f0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:39:53.668 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:53 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:53.378+0000 7f5e25acf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:39:53.668 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:53 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:53.469+0000 7f5e25acf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:39:53.668 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:53 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:53.669+0000 7f5e25acf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:39:53.705 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:53 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:53.409+0000 7f068bbc7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:39:53.705 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:53 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:53.501+0000 7f068bbc7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:39:53.705 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:53 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:53.705+0000 7f068bbc7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:39:53.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.772+0000 7f6f0e212700 1 -- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6ef40a3d10 msgr2=0x7f6ef40a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3394521972 2024-04-21T08:39:53.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.772+0000 7f6f0e212700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6ef40a3d10 0x7f6ef40a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:39:53.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.827+0000 7f6f10476700 1 -- 172.21.15.94:0/848352717 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6f08069730 con 0x7f6f08107980 2024-04-21T08:39:53.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.828+0000 7f6eff7fe700 1 -- 172.21.15.94:0/848352717 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+88 (secure 0 0 0) 0x7f6f0009ca30 con 0x7f6f08107980 2024-04-21T08:39:53.826 INFO:teuthology.orchestra.run.smithi094.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T08:39:53.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 -- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6ef40a3d10 msgr2=0x7f6ef40a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:39:53.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6ef40a3d10 0x7f6ef40a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 -- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f08107980 msgr2=0x7f6f081a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:53.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f08107980 0x7f6f081a5050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6f00007ee0 tx=0x7f6f0000d3b0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 -- 172.21.15.94:0/848352717 shutdown_connections 2024-04-21T08:39:53.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f6ef40a3d10 0x7f6ef40a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f08107980 0x7f6f081a5050 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f08101930 0x7f6f08199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 --2- 172.21.15.94:0/848352717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f081022f0 0x7f6f0819a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 -- 172.21.15.94:0/848352717 >> 172.21.15.94:0/848352717 conn(0x7f6f080788b0 msgr2=0x7f6f08100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 -- 172.21.15.94:0/848352717 shutdown_connections 2024-04-21T08:39:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:53.830+0000 7f6f10476700 1 -- 172.21.15.94:0/848352717 wait complete. 2024-04-21T08:39:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:53 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/848352717' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:39:53.937 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:53 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:53.862+0000 7f5e25acf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:39:54.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:53 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/848352717' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:39:54.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:53.881+0000 7f5d310f0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:39:54.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:53 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/848352717' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:39:54.011 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:53 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:53.898+0000 7f068bbc7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:39:54.510 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:54 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:54.077+0000 7f5d310f0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:39:54.659 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-21T08:39:54.659 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T08:39:54.659 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi094.front.sepia.ceph.com 2024-04-21T08:39:54.659 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-21T08:39:54.659 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T08:39:54.660 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-21T08:39:54.660 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T08:39:54.660 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T08:39:54.660 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-21T08:39:54.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:39:54.660 DEBUG:teuthology.orchestra.run.smithi094:> ip addr 2024-04-21T08:39:54.676 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T08:39:54.676 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T08:39:54.676 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T08:39:54.676 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4c brd ff:ff:ff:ff:ff:ff 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f0 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4d brd ff:ff:ff:ff:ff:ff 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f1 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:62 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:63 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout: inet 172.21.15.94/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-21T08:39:54.677 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft 42061sec preferred_lft 42061sec 2024-04-21T08:39:54.678 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::cabc:3417:31e5:23d5/64 scope link noprefixroute 2024-04-21T08:39:54.678 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:39:54.678 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T08:39:54.678 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:39:54.678 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-21T08:39:54.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add name ceph-brx type bridge 2024-04-21T08:39:54.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr flush dev ceph-brx 2024-04-21T08:39:54.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set ceph-brx up 2024-04-21T08:39:54.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T08:39:54.678 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-21T08:39:54.760 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:54 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:54.498+0000 7f068bbc7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:39:54.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:39:54.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:39:54.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:39:54.818 DEBUG:teuthology.orchestra.run.smithi094:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T08:39:54.841 INFO:teuthology.orchestra.run.smithi094.stdout:1 2024-04-21T08:39:54.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:39:54.842 DEBUG:teuthology.orchestra.run.smithi094:> route 2024-04-21T08:39:54.874 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:54 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:54.455+0000 7f5e25acf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:39:54.898 INFO:teuthology.orchestra.run.smithi094.stdout:Kernel IP routing table 2024-04-21T08:39:54.898 INFO:teuthology.orchestra.run.smithi094.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T08:39:54.898 INFO:teuthology.orchestra.run.smithi094.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-21T08:39:54.898 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-21T08:39:54.898 INFO:teuthology.orchestra.run.smithi094.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T08:39:54.899 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:39:54.899 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-21T08:39:54.899 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-21T08:39:54.899 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-21T08:39:54.899 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-21T08:39:54.899 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-21T08:39:54.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:39:54.994 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:54 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:54.685+0000 7f5d310f0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:39:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:39:55.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:39:55.024 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-04-21T08:39:55.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:39:55.039 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list-id 2024-04-21T08:39:55.094 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:39:55.094 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-21T08:39:55.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T08:39:55.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-21T08:39:55.095 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-21T08:39:55.170 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:54 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:54.873+0000 7f5e25acf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:39:55.170 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:54 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:54.969+0000 7f5e25acf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:39:55.170 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:55 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:55.169+0000 7f5e25acf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:39:55.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:39:55.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:39:55.210 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-21T08:39:55.210 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:39:55.211 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-21T08:39:55.211 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-21T08:39:55.211 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T08:39:55.211 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-21T08:39:55.211 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-21T08:39:55.211 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-21T08:39:55.211 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-21T08:39:55.213 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:54 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:54.916+0000 7f068bbc7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:39:55.213 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:55.012+0000 7f068bbc7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:39:55.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:39:55.259 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:55 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:55.138+0000 7f5d310f0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:39:55.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:55 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:55.242+0000 7f5d310f0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:39:55.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:39:55.321 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:39:55.321 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-21T08:39:55.321 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set brx.0 up 2024-04-21T08:39:55.321 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T08:39:55.321 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-21T08:39:55.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:39:55.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:39:55.396 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:39:55.397 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-21T08:39:55.411 INFO:teuthology.orchestra.run.smithi094.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-21T08:39:55.411 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:39:55.411 DEBUG:teuthology.orchestra.run.smithi094:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-21T08:39:55.467 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-21T08:39:55.467 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-21T08:39:55.467 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/2736019d-b614-4eb2-b627-65349b1b5fba 2024-04-21T08:39:55.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:39:55.467 DEBUG:teuthology.orchestra.run.smithi094:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/2736019d-b614-4eb2-b627-65349b1b5fba /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-21T08:39:55.510 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:55.213+0000 7f068bbc7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:39:55.528 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:55 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:55.477+0000 7f5d310f0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:39:55.550 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-21T08:39:55.550 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-21T08:39:55.550 INFO:teuthology.orchestra.run.smithi094.stdout:invalid new device string format 2024-04-21T08:39:55.678 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-21T08:39:55.678 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-21T08:39:55.678 INFO:teuthology.orchestra.run.smithi094.stdout:invalid new device string format 2024-04-21T08:39:55.678 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.94:3300,172.21.15.111:3300,172.21.15.173:3300" 2024-04-21T08:39:55.678 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.94:3300,172.21.15.111:3300,172.21.15.173:3300:/volumes/qa/sv_1/2736019d-b614-4eb2-b627-65349b1b5fba 2024-04-21T08:39:55.678 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=2518ed94-ffb9-11ee-bc93-c7b262605968" will pass to kernel 2024-04-21T08:39:55.680 INFO:teuthology.orchestra.run.smithi094.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-21T08:39:55.680 INFO:teuthology.orchestra.run.smithi094.stdout: You just mounted a file system that supports labels which does not 2024-04-21T08:39:55.680 INFO:teuthology.orchestra.run.smithi094.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-21T08:39:55.680 INFO:teuthology.orchestra.run.smithi094.stdout: applications will generate AVC messages and not be allowed access to 2024-04-21T08:39:55.680 INFO:teuthology.orchestra.run.smithi094.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-21T08:39:55.681 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-21T08:39:55.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:39:55.682 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-21T08:39:55.747 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:39:55.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:39:55.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:39:55.843 INFO:teuthology.orchestra.run.smithi094.stdout:client15474 2024-04-21T08:39:55.843 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:39:55.843 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:39:55.860 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:39:55.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:39:55.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:39:55.956 INFO:teuthology.orchestra.run.smithi094.stdout:client15474 2024-04-21T08:39:55.956 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:39:56.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:39:56.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:39:56.010 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:55.741+0000 7f068bbc7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:39:56.010 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:55.832+0000 7f068bbc7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:39:56.012 INFO:teuthology.orchestra.run.smithi094.stdout:client15474 2024-04-21T08:39:56.013 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968.client15474/status') 2024-04-21T08:39:56.029 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:55 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:55.693+0000 7f5e25acf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:39:56.029 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:55 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:55.785+0000 7f5e25acf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:39:56.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:39:56.070 INFO:teuthology.orchestra.run.smithi094.stdout:instance: client.15474 (3)192.168.0.1:0/2256865590 2024-04-21T08:39:56.070 INFO:teuthology.orchestra.run.smithi094.stdout:blocklisted: false 2024-04-21T08:39:56.070 INFO:teuthology.orchestra.run.smithi094.stderr:0+1 records in 2024-04-21T08:39:56.070 INFO:teuthology.orchestra.run.smithi094.stderr:0+1 records out 2024-04-21T08:39:56.070 INFO:teuthology.orchestra.run.smithi094.stderr:70 bytes copied, 2.2615e-05 s, 3.1 MB/s 2024-04-21T08:39:56.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:39:56.077 INFO:teuthology.run_tasks:Running task exec... 2024-04-21T08:39:56.087 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:39:56.087 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:39:56.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-21T08:39:56.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.190+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2610212796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbd8100ea0 msgr2=0x7ffbd81012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:56.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.190+0000 7ffbdd31e640 1 --2- 172.21.15.94:0/2610212796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbd8100ea0 0x7ffbd81012c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009900 tx=0x7ffbc802bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:56.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.190+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2610212796 shutdown_connections 2024-04-21T08:39:56.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.190+0000 7ffbdd31e640 1 --2- 172.21.15.94:0/2610212796 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd8102a40 0x7ffbd8109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.190+0000 7ffbdd31e640 1 --2- 172.21.15.94:0/2610212796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd8102080 0x7ffbd8102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.190+0000 7ffbdd31e640 1 --2- 172.21.15.94:0/2610212796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbd8100ea0 0x7ffbd81012c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.190+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2610212796 >> 172.21.15.94:0/2610212796 conn(0x7ffbd80fc460 msgr2=0x7ffbd80fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:56.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.190+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2610212796 shutdown_connections 2024-04-21T08:39:56.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.190+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2610212796 wait complete. 2024-04-21T08:39:56.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbdd31e640 1 Processor -- start 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbdd31e640 1 -- start start 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbdd31e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd8100ea0 0x7ffbd806b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbdd31e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbd8102080 0x7ffbd819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbdd31e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd8102a40 0x7ffbd819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbdd31e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd810ca20 con 0x7ffbd8102080 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbdd31e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd810c8a0 con 0x7ffbd8100ea0 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbdd31e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd810cba0 con 0x7ffbd8102a40 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbd67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbd8102080 0x7ffbd819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbd6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd8100ea0 0x7ffbd806b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbd67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbd8102080 0x7ffbd819ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58256/0 (socket says 172.21.15.94:58256) 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbd67fc640 1 -- 172.21.15.94:0/2497792609 learned_addr learned my addr 172.21.15.94:0/2497792609 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.191+0000 7ffbd77fe640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd8102a40 0x7ffbd819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbd67fc640 1 -- 172.21.15.94:0/2497792609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd8102a40 msgr2=0x7ffbd819f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:56.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbd67fc640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd8102a40 0x7ffbd819f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbd67fc640 1 -- 172.21.15.94:0/2497792609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd8100ea0 msgr2=0x7ffbd806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:56.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbd67fc640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd8100ea0 0x7ffbd806b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbd67fc640 1 -- 172.21.15.94:0/2497792609 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd819fdd0 con 0x7ffbd8102080 2024-04-21T08:39:56.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbd67fc640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbd8102080 0x7ffbd819ced0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc00d950 tx=0x7ffbcc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:56.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbbffff640 1 -- 172.21.15.94:0/2497792609 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc017030 con 0x7ffbd8102080 2024-04-21T08:39:56.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2497792609 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd81a00c0 con 0x7ffbd8102080 2024-04-21T08:39:56.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbbffff640 1 -- 172.21.15.94:0/2497792609 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbcc004020 con 0x7ffbd8102080 2024-04-21T08:39:56.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbbffff640 1 -- 172.21.15.94:0/2497792609 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc0047a0 con 0x7ffbd8102080 2024-04-21T08:39:56.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.192+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2497792609 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd81a0610 con 0x7ffbd8102080 2024-04-21T08:39:56.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.193+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2497792609 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd8108240 con 0x7ffbd8102080 2024-04-21T08:39:56.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.195+0000 7ffbbffff640 1 -- 172.21.15.94:0/2497792609 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7ffbcc009a30 con 0x7ffbd8102080 2024-04-21T08:39:56.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.195+0000 7ffbbffff640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffbc00a3ca0 0x7ffbc00a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:56.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.195+0000 7ffbd6ffd640 1 -- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffbc00a3ca0 msgr2=0x7ffbc00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3394521972 2024-04-21T08:39:56.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.195+0000 7ffbbffff640 1 -- 172.21.15.94:0/2497792609 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7ffbcc0d0270 con 0x7ffbd8102080 2024-04-21T08:39:56.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.195+0000 7ffbd6ffd640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffbc00a3ca0 0x7ffbc00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:39:56.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.200+0000 7ffbbffff640 1 -- 172.21.15.94:0/2497792609 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffbcc09c850 con 0x7ffbd8102080 2024-04-21T08:39:56.370 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:56.029+0000 7f5e25acf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:39:56.371 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:56.161+0000 7f5e25acf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:39:56.371 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:56.256+0000 7f5e25acf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:39:56.371 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:56.370+0000 7f5e25acf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:39:56.390 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:56.035+0000 7f5d310f0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:39:56.390 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:56.137+0000 7f5d310f0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:39:56.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.396+0000 7ffbd6ffd640 1 -- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffbc00a3ca0 msgr2=0x7ffbc00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3394521972 2024-04-21T08:39:56.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.396+0000 7ffbd6ffd640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffbc00a3ca0 0x7ffbc00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:39:56.416 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:56 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:56.074+0000 7f068bbc7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:39:56.416 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:56 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:56.207+0000 7f068bbc7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:39:56.416 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:56 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:56.301+0000 7f068bbc7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:39:56.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.418+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2497792609 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7ffbd8062b50 con 0x7ffbd8102080 2024-04-21T08:39:56.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.470+0000 7ffbbffff640 1 -- 172.21.15.94:0/2497792609 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7ffbcc09c150 con 0x7ffbd8102080 2024-04-21T08:39:56.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.472+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffbc00a3ca0 msgr2=0x7ffbc00a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:39:56.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.472+0000 7ffbdd31e640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffbc00a3ca0 0x7ffbc00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.472+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbd8102080 msgr2=0x7ffbd819ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:56.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.472+0000 7ffbdd31e640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbd8102080 0x7ffbd819ced0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc00d950 tx=0x7ffbcc00de10 comp rx=0 tx=0).stop 2024-04-21T08:39:56.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.472+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2497792609 shutdown_connections 2024-04-21T08:39:56.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.472+0000 7ffbdd31e640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7ffbc00a3ca0 0x7ffbc00a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.472+0000 7ffbdd31e640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbd8102a40 0x7ffbd819f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.472+0000 7ffbdd31e640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbd8102080 0x7ffbd819ced0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.472+0000 7ffbdd31e640 1 --2- 172.21.15.94:0/2497792609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd8100ea0 0x7ffbd806b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.472+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2497792609 >> 172.21.15.94:0/2497792609 conn(0x7ffbd80fc460 msgr2=0x7ffbd8105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:56.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.473+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2497792609 shutdown_connections 2024-04-21T08:39:56.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.473+0000 7ffbdd31e640 1 -- 172.21.15.94:0/2497792609 wait complete. 2024-04-21T08:39:56.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-21T08:39:56.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.587+0000 7fa01a85c640 1 -- 172.21.15.94:0/2406335519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa014103b90 msgr2=0x7fa01410a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:56.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.587+0000 7fa01a85c640 1 --2- 172.21.15.94:0/2406335519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa014103b90 0x7fa01410a420 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009ca0 tx=0x7fa00c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.587+0000 7fa01a85c640 1 -- 172.21.15.94:0/2406335519 shutdown_connections 2024-04-21T08:39:56.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.587+0000 7fa01a85c640 1 --2- 172.21.15.94:0/2406335519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0141031d0 0x7fa014103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.587+0000 7fa01a85c640 1 --2- 172.21.15.94:0/2406335519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa014103b90 0x7fa01410a420 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.587+0000 7fa01a85c640 1 --2- 172.21.15.94:0/2406335519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa014101fe0 0x7fa014102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.587+0000 7fa01a85c640 1 -- 172.21.15.94:0/2406335519 >> 172.21.15.94:0/2406335519 conn(0x7fa0140fd570 msgr2=0x7fa0140ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:56.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.587+0000 7fa01a85c640 1 -- 172.21.15.94:0/2406335519 shutdown_connections 2024-04-21T08:39:56.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.587+0000 7fa01a85c640 1 -- 172.21.15.94:0/2406335519 wait complete. 2024-04-21T08:39:56.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.588+0000 7fa01a85c640 1 Processor -- start 2024-04-21T08:39:56.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.588+0000 7fa01a85c640 1 -- start start 2024-04-21T08:39:56.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.588+0000 7fa01a85c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa014101fe0 0x7fa014198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:56.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.588+0000 7fa01a85c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0141031d0 0x7fa014198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:56.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa01a85c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa014103b90 0x7fa01419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:56.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa01a85c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa01410db20 con 0x7fa0141031d0 2024-04-21T08:39:56.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa01a85c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa01410d9a0 con 0x7fa014101fe0 2024-04-21T08:39:56.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa01a85c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa01410dca0 con 0x7fa014103b90 2024-04-21T08:39:56.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0141031d0 0x7fa014198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa013fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa014101fe0 0x7fa014198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0141031d0 0x7fa014198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58264/0 (socket says 172.21.15.94:58264) 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0137fe640 1 -- 172.21.15.94:0/3504987218 learned_addr learned my addr 172.21.15.94:0/3504987218 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa018df9640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa014103b90 0x7fa01419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0137fe640 1 -- 172.21.15.94:0/3504987218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa014103b90 msgr2=0x7fa01419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0137fe640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa014103b90 0x7fa01419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0137fe640 1 -- 172.21.15.94:0/3504987218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa014101fe0 msgr2=0x7fa014198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0137fe640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa014101fe0 0x7fa014198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0137fe640 1 -- 172.21.15.94:0/3504987218 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01419fd10 con 0x7fa0141031d0 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0137fe640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0141031d0 0x7fa014198680 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa00800d770 tx=0x7fa00800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa008017070 con 0x7fa0141031d0 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa01a85c640 1 -- 172.21.15.94:0/3504987218 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0140741e0 con 0x7fa0141031d0 2024-04-21T08:39:56.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0080054d0 con 0x7fa0141031d0 2024-04-21T08:39:56.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa008004c30 con 0x7fa0141031d0 2024-04-21T08:39:56.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.589+0000 7fa01a85c640 1 -- 172.21.15.94:0/3504987218 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa014074730 con 0x7fa0141031d0 2024-04-21T08:39:56.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.591+0000 7fa01a85c640 1 -- 172.21.15.94:0/3504987218 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa014109300 con 0x7fa0141031d0 2024-04-21T08:39:56.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.592+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fa008004e10 con 0x7fa0141031d0 2024-04-21T08:39:56.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.592+0000 7fa0117fa640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9ffc0a3ca0 0x7f9ffc0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:56.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.592+0000 7fa013fff640 1 -- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9ffc0a3ca0 msgr2=0x7f9ffc0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3394521972 2024-04-21T08:39:56.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.592+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9253+0+0 (secure 0 0 0) 0x7fa0080d0270 con 0x7fa0141031d0 2024-04-21T08:39:56.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.592+0000 7fa013fff640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9ffc0a3ca0 0x7f9ffc0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:39:56.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.597+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa00809c200 con 0x7fa0141031d0 2024-04-21T08:39:56.671 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:56 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2497792609' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T08:39:56.723 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:56 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2497792609' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T08:39:56.723 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:56 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:56.417+0000 7f068bbc7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:39:56.756 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:56 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2497792609' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T08:39:56.756 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:56.391+0000 7f5d310f0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:39:56.756 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:56.534+0000 7f5d310f0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:39:56.756 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:56.637+0000 7f5d310f0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:39:56.756 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:56.757+0000 7f5d310f0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:39:56.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.793+0000 7fa013fff640 1 -- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9ffc0a3ca0 msgr2=0x7f9ffc0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3394521972 2024-04-21T08:39:56.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.793+0000 7fa013fff640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9ffc0a3ca0 0x7f9ffc0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:39:56.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:56.811+0000 7fa01a85c640 1 -- 172.21.15.94:0/3504987218 --> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fa014075ce0 con 0x7f9ffc0a3ca0 2024-04-21T08:39:56.937 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:56.670+0000 7f5e25acf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:39:57.010 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:56 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:56.723+0000 7f068bbc7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:39:57.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:57.194+0000 7fa013fff640 1 -- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9ffc0a3ca0 msgr2=0x7f9ffc0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3394521972 2024-04-21T08:39:57.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:57.194+0000 7fa013fff640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9ffc0a3ca0 0x7f9ffc0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T08:39:57.475 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:57.063+0000 7f5d310f0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:39:57.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:57 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2497792609' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-21T08:39:57.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:57 smithi111 ceph-mon[31493]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:39:57.687 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:57 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:57.273+0000 7f5e25acf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:39:57.687 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:39:57 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y[32916]: 2024-04-21T08:39:57.490+0000 7f5e25acf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:39:57.760 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:57.673+0000 7f5d310f0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:39:57.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:57 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2497792609' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-21T08:39:57.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:57 smithi094 ceph-mon[27988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:39:57.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:57 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2497792609' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-21T08:39:57.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:57 smithi173 ceph-mon[31428]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:39:57.761 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:57 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:57.324+0000 7f068bbc7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:39:57.761 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:39:57 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z[33203]: 2024-04-21T08:39:57.537+0000 7f068bbc7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:39:57.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:57.909+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mgrmap(e 20) ==== 138331+0+0 (secure 0 0 0) 0x7fa008098870 con 0x7fa0141031d0 2024-04-21T08:39:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:57.910+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9ffc0a3ca0 msgr2=0x7f9ffc0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:39:57.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:57.910+0000 7fa0117fa640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/3394521972,v1:172.21.15.94:6801/3394521972] conn(0x7f9ffc0a3ca0 0x7f9ffc0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:58.146 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:57.897+0000 7f5d310f0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:39:58.147 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.145+0000 7f5c90a4e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:58.147 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.145+0000 7f5c90a4e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:58.147 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.145+0000 7f5c90a4e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:58.147 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.145+0000 7f5c90a4e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:58.147 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.145+0000 7f5c90a4e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:58.477 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.147+0000 7f5c93a94700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:58.478 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.147+0000 7f5c93a94700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:58.478 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.147+0000 7f5c93a94700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:58.478 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.147+0000 7f5c93a94700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:58.478 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.147+0000 7f5c93a94700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:58.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: Standby manager daemon y restarted 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: Standby manager daemon y started 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: Standby manager daemon z restarted 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: Standby manager daemon z started 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:39:58.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: Active manager daemon x restarted 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: Activating manager daemon x 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: mgrmap e20: x(active, starting, since 0.0117936s), standbys: y, z 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:39:58.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:39:58.763 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: Manager daemon x is now available 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:39:58.764 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:39:58.765 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: Standby manager daemon y restarted 2024-04-21T08:39:58.765 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: Standby manager daemon y started 2024-04-21T08:39:58.765 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:39:58.765 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:39:58.765 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:39:58.765 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:39:58.765 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: Standby manager daemon z restarted 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: Standby manager daemon z started 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: Active manager daemon x restarted 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: Activating manager daemon x 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: mgrmap e20: x(active, starting, since 0.0117936s), standbys: y, z 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:39:58.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:39:58.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:39:58.768 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: Manager daemon x is now available 2024-04-21T08:39:58.769 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:58.769 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:39:58.769 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:58 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:39:58.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.917+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 <== mon.0 v2:172.21.15.94:3300/0 8 ==== mgrmap(e 21) ==== 138458+0+0 (secure 0 0 0) 0x7fa008005100 con 0x7fa0141031d0 2024-04-21T08:39:58.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.918+0000 7fa0117fa640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9ffc0528d0 0x7f9ffc052cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:58.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.918+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fa00801dec0 con 0x7f9ffc0528d0 2024-04-21T08:39:58.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.923+0000 7fa013fff640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9ffc0528d0 0x7f9ffc052cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:58.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.925+0000 7fa013fff640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9ffc0528d0 0x7f9ffc052cb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa014103030 tx=0x7fa00400b390 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: Standby manager daemon y restarted 2024-04-21T08:39:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: Standby manager daemon y started 2024-04-21T08:39:58.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.111:0/4215462202' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: Standby manager daemon z restarted 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: Standby manager daemon z started 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.? 172.21.15.173:0/17355802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: Active manager daemon x restarted 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: Activating manager daemon x 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: mgrmap e20: x(active, starting, since 0.0117936s), standbys: y, z 2024-04-21T08:39:58.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:39:58.939 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:39:58.940 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:39:58.941 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:39:58.941 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:39:58.941 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:39:58.941 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: Manager daemon x is now available 2024-04-21T08:39:58.941 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:58.941 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:39:58.941 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:58 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:39:58.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.945+0000 7fa0117fa640 1 -- 172.21.15.94:0/3504987218 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fa00801dec0 con 0x7f9ffc0528d0 2024-04-21T08:39:58.943 INFO:teuthology.orchestra.run.smithi094.stdout:{} 2024-04-21T08:39:58.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.947+0000 7fa01a85c640 1 -- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9ffc0528d0 msgr2=0x7f9ffc052cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:58.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.947+0000 7fa01a85c640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9ffc0528d0 0x7f9ffc052cb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa014103030 tx=0x7fa00400b390 comp rx=0 tx=0).stop 2024-04-21T08:39:58.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.947+0000 7fa01a85c640 1 -- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0141031d0 msgr2=0x7fa014198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:58.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.947+0000 7fa01a85c640 1 --2- 172.21.15.94:0/3504987218 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0141031d0 0x7fa014198680 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa00800d770 tx=0x7fa00800dc30 comp rx=0 tx=0).stop 2024-04-21T08:39:58.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.947+0000 7fa018df9640 1 -- 172.21.15.94:0/3504987218 reap_dead start 2024-04-21T08:39:58.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.948+0000 7fa01a85c640 1 -- 172.21.15.94:0/3504987218 shutdown_connections 2024-04-21T08:39:58.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.948+0000 7fa01a85c640 1 -- 172.21.15.94:0/3504987218 >> 172.21.15.94:0/3504987218 conn(0x7fa0140fd570 msgr2=0x7fa0141064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:58.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.948+0000 7fa01a85c640 1 -- 172.21.15.94:0/3504987218 shutdown_connections 2024-04-21T08:39:58.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:58.948+0000 7fa01a85c640 1 -- 172.21.15.94:0/3504987218 wait complete. 2024-04-21T08:39:58.959 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-21T08:39:58.970 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-21T08:39:59.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:39:59.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.112+0000 7f32f34c1640 1 -- 172.21.15.94:0/2188364688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec1029b0 msgr2=0x7f32ec1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.112+0000 7f32f34c1640 1 --2- 172.21.15.94:0/2188364688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec1029b0 0x7f32ec1092d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f32e4009ca0 tx=0x7f32e402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.112+0000 7f32f34c1640 1 -- 172.21.15.94:0/2188364688 shutdown_connections 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.112+0000 7f32f34c1640 1 --2- 172.21.15.94:0/2188364688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32ec101ff0 0x7f32ec102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.112+0000 7f32f34c1640 1 --2- 172.21.15.94:0/2188364688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec1029b0 0x7f32ec1092d0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.112+0000 7f32f34c1640 1 --2- 172.21.15.94:0/2188364688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32ec100ea0 0x7f32ec1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.112+0000 7f32f34c1640 1 -- 172.21.15.94:0/2188364688 >> 172.21.15.94:0/2188364688 conn(0x7f32ec0fc410 msgr2=0x7f32ec0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.112+0000 7f32f34c1640 1 -- 172.21.15.94:0/2188364688 shutdown_connections 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.112+0000 7f32f34c1640 1 -- 172.21.15.94:0/2188364688 wait complete. 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f34c1640 1 Processor -- start 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f34c1640 1 -- start start 2024-04-21T08:39:59.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f34c1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32ec100ea0 0x7f32ec198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f34c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec101ff0 0x7f32ec198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f34c1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32ec1029b0 0x7f32ec19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f34c1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec10c9f0 con 0x7f32ec101ff0 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f34c1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec10c870 con 0x7f32ec100ea0 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f34c1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec10cb70 con 0x7f32ec1029b0 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f0a5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec101ff0 0x7f32ec198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f0a5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec101ff0 0x7f32ec198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58450/0 (socket says 172.21.15.94:58450) 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f0a5c640 1 -- 172.21.15.94:0/1241243095 learned_addr learned my addr 172.21.15.94:0/1241243095 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f125d640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32ec100ea0 0x7f32ec198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f1a5e640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32ec1029b0 0x7f32ec19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f0a5c640 1 -- 172.21.15.94:0/1241243095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32ec1029b0 msgr2=0x7f32ec19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f0a5c640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32ec1029b0 0x7f32ec19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f0a5c640 1 -- 172.21.15.94:0/1241243095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32ec100ea0 msgr2=0x7f32ec198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f0a5c640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32ec100ea0 0x7f32ec198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f0a5c640 1 -- 172.21.15.94:0/1241243095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ec19fd60 con 0x7f32ec101ff0 2024-04-21T08:39:59.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32f0a5c640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec101ff0 0x7f32ec198680 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00d950 tx=0x7f32dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:59.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.113+0000 7f32e27fc640 1 -- 172.21.15.94:0/1241243095 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32dc017070 con 0x7f32ec101ff0 2024-04-21T08:39:59.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.114+0000 7f32f34c1640 1 -- 172.21.15.94:0/1241243095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32ec19fff0 con 0x7f32ec101ff0 2024-04-21T08:39:59.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.114+0000 7f32e27fc640 1 -- 172.21.15.94:0/1241243095 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32dc009c30 con 0x7f32ec101ff0 2024-04-21T08:39:59.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.114+0000 7f32e27fc640 1 -- 172.21.15.94:0/1241243095 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32dc00f620 con 0x7f32ec101ff0 2024-04-21T08:39:59.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.114+0000 7f32f34c1640 1 -- 172.21.15.94:0/1241243095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32ec1a04c0 con 0x7f32ec101ff0 2024-04-21T08:39:59.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.115+0000 7f32f34c1640 1 -- 172.21.15.94:0/1241243095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32ec1a0750 con 0x7f32ec101ff0 2024-04-21T08:39:59.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.116+0000 7f32e27fc640 1 -- 172.21.15.94:0/1241243095 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 21) ==== 138458+0+0 (secure 0 0 0) 0x7f32dc01a070 con 0x7f32ec101ff0 2024-04-21T08:39:59.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.116+0000 7f32e27fc640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f32d80a39b0 0x7f32d80a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.116+0000 7f32f125d640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f32d80a39b0 0x7f32d80a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.116+0000 7f32e27fc640 1 -- 172.21.15.94:0/1241243095 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f32dc0cfe10 con 0x7f32ec101ff0 2024-04-21T08:39:59.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.117+0000 7f32f125d640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f32d80a39b0 0x7f32d80a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f32e80058c0 tx=0x7f32e800a2d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:59.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.120+0000 7f32e27fc640 1 -- 172.21.15.94:0/1241243095 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f32dc009de0 con 0x7f32ec101ff0 2024-04-21T08:39:59.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.932+0000 7f5c8da48700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:59.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.932+0000 7f5c8da48700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:59.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.932+0000 7f5c8da48700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:59.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.932+0000 7f5c8da48700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:59.260 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:39:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:39:58.932+0000 7f5c8da48700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:39:59.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.314+0000 7f32f34c1640 1 -- 172.21.15.94:0/1241243095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32ec1a0a40 con 0x7f32ec101ff0 2024-04-21T08:39:59.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.316+0000 7f32e27fc640 1 -- 172.21.15.94:0/1241243095 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25266 (secure 0 0 0) 0x7f32dc09bf90 con 0x7f32ec101ff0 2024-04-21T08:39:59.315 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:39:59.315 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:57.898777+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:39:59.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 -- 172.21.15.94:0/1241243095 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f32d80a39b0 msgr2=0x7f32d80a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f32d80a39b0 0x7f32d80a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f32e80058c0 tx=0x7f32e800a2d0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 -- 172.21.15.94:0/1241243095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec101ff0 msgr2=0x7f32ec198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec101ff0 0x7f32ec198680 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00d950 tx=0x7f32dc00de10 comp rx=0 tx=0).stop 2024-04-21T08:39:59.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 -- 172.21.15.94:0/1241243095 shutdown_connections 2024-04-21T08:39:59.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f32d80a39b0 0x7f32d80a5e70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32ec1029b0 0x7f32ec19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec101ff0 0x7f32ec198680 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 --2- 172.21.15.94:0/1241243095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32ec100ea0 0x7f32ec198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 -- 172.21.15.94:0/1241243095 >> 172.21.15.94:0/1241243095 conn(0x7f32ec0fc410 msgr2=0x7f32ec06a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:59.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 -- 172.21.15.94:0/1241243095 shutdown_connections 2024-04-21T08:39:59.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.318+0000 7f32f34c1640 1 -- 172.21.15.94:0/1241243095 wait complete. 2024-04-21T08:39:59.329 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:39:59.329 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.434+0000 7fdd9acc1640 1 -- 172.21.15.94:0/2502746620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd94101fe0 msgr2=0x7fdd94102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.434+0000 7fdd9acc1640 1 --2- 172.21.15.94:0/2502746620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd94101fe0 0x7fdd94102460 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c009900 tx=0x7fdd7c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.434+0000 7fdd9acc1640 1 -- 172.21.15.94:0/2502746620 shutdown_connections 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.434+0000 7fdd9acc1640 1 --2- 172.21.15.94:0/2502746620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd941029a0 0x7fdd941092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.434+0000 7fdd9acc1640 1 --2- 172.21.15.94:0/2502746620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd94101fe0 0x7fdd94102460 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.434+0000 7fdd9acc1640 1 --2- 172.21.15.94:0/2502746620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd94100e90 0x7fdd941012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.434+0000 7fdd9acc1640 1 -- 172.21.15.94:0/2502746620 >> 172.21.15.94:0/2502746620 conn(0x7fdd940fc400 msgr2=0x7fdd940fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.434+0000 7fdd9acc1640 1 -- 172.21.15.94:0/2502746620 shutdown_connections 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.434+0000 7fdd9acc1640 1 -- 172.21.15.94:0/2502746620 wait complete. 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9acc1640 1 Processor -- start 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9acc1640 1 -- start start 2024-04-21T08:39:59.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9acc1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd94100e90 0x7fdd941981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9acc1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd94101fe0 0x7fdd94198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9acc1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd941029a0 0x7fdd9419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9acc1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9410c9d0 con 0x7fdd941029a0 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9acc1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9410c850 con 0x7fdd94100e90 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9acc1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9410cb50 con 0x7fdd94101fe0 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9925e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd941029a0 0x7fdd9419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9925e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd941029a0 0x7fdd9419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58460/0 (socket says 172.21.15.94:58460) 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9925e640 1 -- 172.21.15.94:0/3434161103 learned_addr learned my addr 172.21.15.94:0/3434161103 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd98a5d640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd94100e90 0x7fdd941981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd93fff640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd94101fe0 0x7fdd94198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9925e640 1 -- 172.21.15.94:0/3434161103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd94101fe0 msgr2=0x7fdd94198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9925e640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd94101fe0 0x7fdd94198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9925e640 1 -- 172.21.15.94:0/3434161103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd94100e90 msgr2=0x7fdd941981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9925e640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd94100e90 0x7fdd941981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd9925e640 1 -- 172.21.15.94:0/3434161103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9419fe10 con 0x7fdd941029a0 2024-04-21T08:39:59.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.435+0000 7fdd93fff640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd94101fe0 0x7fdd94198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:59.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.436+0000 7fdd9925e640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd941029a0 0x7fdd9419f7a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400d750 tx=0x7fdd8400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:59.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.436+0000 7fdd91ffb640 1 -- 172.21.15.94:0/3434161103 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84017070 con 0x7fdd941029a0 2024-04-21T08:39:59.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.436+0000 7fdd9acc1640 1 -- 172.21.15.94:0/3434161103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd941a0100 con 0x7fdd941029a0 2024-04-21T08:39:59.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.436+0000 7fdd91ffb640 1 -- 172.21.15.94:0/3434161103 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd840054d0 con 0x7fdd941029a0 2024-04-21T08:39:59.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.436+0000 7fdd91ffb640 1 -- 172.21.15.94:0/3434161103 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84004c60 con 0x7fdd941029a0 2024-04-21T08:39:59.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.436+0000 7fdd9acc1640 1 -- 172.21.15.94:0/3434161103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd941a0620 con 0x7fdd941029a0 2024-04-21T08:39:59.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.437+0000 7fdd9acc1640 1 -- 172.21.15.94:0/3434161103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd941a0880 con 0x7fdd941029a0 2024-04-21T08:39:59.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.438+0000 7fdd91ffb640 1 -- 172.21.15.94:0/3434161103 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 21) ==== 138458+0+0 (secure 0 0 0) 0x7fdd84009bd0 con 0x7fdd941029a0 2024-04-21T08:39:59.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.438+0000 7fdd91ffb640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd800a39b0 0x7fdd800a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.439+0000 7fdd98a5d640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd800a39b0 0x7fdd800a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.439+0000 7fdd91ffb640 1 -- 172.21.15.94:0/3434161103 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fdd840cfe50 con 0x7fdd941029a0 2024-04-21T08:39:59.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.439+0000 7fdd98a5d640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd800a39b0 0x7fdd800a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c005fd0 tx=0x7fdd8c009760 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:59.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.443+0000 7fdd91ffb640 1 -- 172.21.15.94:0/3434161103 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd8409c620 con 0x7fdd941029a0 2024-04-21T08:39:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.626+0000 7fdd9acc1640 1 -- 172.21.15.94:0/3434161103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdd941a0a50 con 0x7fdd941029a0 2024-04-21T08:39:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.627+0000 7fdd91ffb640 1 -- 172.21.15.94:0/3434161103 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fdd840d1020 con 0x7fdd941029a0 2024-04-21T08:39:59.626 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:39:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 -- 172.21.15.94:0/3434161103 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd800a39b0 msgr2=0x7fdd800a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd800a39b0 0x7fdd800a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c005fd0 tx=0x7fdd8c009760 comp rx=0 tx=0).stop 2024-04-21T08:39:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 -- 172.21.15.94:0/3434161103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd941029a0 msgr2=0x7fdd9419f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd941029a0 0x7fdd9419f7a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400d750 tx=0x7fdd8400dc10 comp rx=0 tx=0).stop 2024-04-21T08:39:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 -- 172.21.15.94:0/3434161103 shutdown_connections 2024-04-21T08:39:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd800a39b0 0x7fdd800a5e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd94101fe0 0x7fdd94198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd941029a0 0x7fdd9419f7a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 --2- 172.21.15.94:0/3434161103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd94100e90 0x7fdd941981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 -- 172.21.15.94:0/3434161103 >> 172.21.15.94:0/3434161103 conn(0x7fdd940fc400 msgr2=0x7fdd940690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 -- 172.21.15.94:0/3434161103 shutdown_connections 2024-04-21T08:39:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.629+0000 7fdd9acc1640 1 -- 172.21.15.94:0/3434161103 wait complete. 2024-04-21T08:39:59.639 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:39:59.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.752+0000 7f50d496d640 1 -- 172.21.15.94:0/2277917308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50d0103b40 msgr2=0x7f50d010a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.752+0000 7f50d496d640 1 --2- 172.21.15.94:0/2277917308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50d0103b40 0x7f50d010a3d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f50c8009ca0 tx=0x7f50c802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.752+0000 7f50d496d640 1 -- 172.21.15.94:0/2277917308 shutdown_connections 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.752+0000 7f50d496d640 1 --2- 172.21.15.94:0/2277917308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50d0103b40 0x7f50d010a3d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.752+0000 7f50d496d640 1 --2- 172.21.15.94:0/2277917308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d0103180 0x7f50d0103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.752+0000 7f50d496d640 1 --2- 172.21.15.94:0/2277917308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d0101f90 0x7f50d01023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.752+0000 7f50d496d640 1 -- 172.21.15.94:0/2277917308 >> 172.21.15.94:0/2277917308 conn(0x7f50d00fd500 msgr2=0x7f50d00ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.752+0000 7f50d496d640 1 -- 172.21.15.94:0/2277917308 shutdown_connections 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.752+0000 7f50d496d640 1 -- 172.21.15.94:0/2277917308 wait complete. 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.752+0000 7f50d496d640 1 Processor -- start 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50d496d640 1 -- start start 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50d496d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50d0101f90 0x7f50d0198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50d496d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d0103180 0x7f50d01986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50d496d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d0103b40 0x7f50d019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50d496d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f50d010dad0 con 0x7f50d0101f90 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50d496d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f50d010d950 con 0x7f50d0103b40 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50d496d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f50d010dc50 con 0x7f50d0103180 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ce59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50d0101f90 0x7f50d0198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ce59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50d0101f90 0x7f50d0198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58478/0 (socket says 172.21.15.94:58478) 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ce59c640 1 -- 172.21.15.94:0/2385645836 learned_addr learned my addr 172.21.15.94:0/2385645836 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ced9d640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d0103b40 0x7f50d019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50c7fff640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d0103180 0x7f50d01986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ced9d640 1 -- 172.21.15.94:0/2385645836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d0103180 msgr2=0x7f50d01986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ced9d640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d0103180 0x7f50d01986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ced9d640 1 -- 172.21.15.94:0/2385645836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50d0101f90 msgr2=0x7f50d0198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ced9d640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50d0101f90 0x7f50d0198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ced9d640 1 -- 172.21.15.94:0/2385645836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d019fd80 con 0x7f50d0103b40 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ce59c640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50d0101f90 0x7f50d0198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:39:59.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.753+0000 7f50ced9d640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d0103b40 0x7f50d019f710 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f50c802c630 tx=0x7f50c80346d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:59.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.754+0000 7f50c77fe640 1 -- 172.21.15.94:0/2385645836 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8041020 con 0x7f50d0103b40 2024-04-21T08:39:59.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.754+0000 7f50c77fe640 1 -- 172.21.15.94:0/2385645836 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50c8034ca0 con 0x7f50d0103b40 2024-04-21T08:39:59.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.754+0000 7f50d496d640 1 -- 172.21.15.94:0/2385645836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d01a0010 con 0x7f50d0103b40 2024-04-21T08:39:59.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.754+0000 7f50c77fe640 1 -- 172.21.15.94:0/2385645836 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8039040 con 0x7f50d0103b40 2024-04-21T08:39:59.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.754+0000 7f50d496d640 1 -- 172.21.15.94:0/2385645836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d01a0500 con 0x7f50d0103b40 2024-04-21T08:39:59.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.755+0000 7f50d496d640 1 -- 172.21.15.94:0/2385645836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d01093c0 con 0x7f50d0103b40 2024-04-21T08:39:59.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.756+0000 7f50c77fe640 1 -- 172.21.15.94:0/2385645836 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 21) ==== 138458+0+0 (secure 0 0 0) 0x7f50c8044020 con 0x7f50d0103b40 2024-04-21T08:39:59.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.756+0000 7f50c77fe640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f50bc0a3a30 0x7f50bc0a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:59.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.757+0000 7f50c77fe640 1 -- 172.21.15.94:0/2385645836 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f50c80f2520 con 0x7f50d0103b40 2024-04-21T08:39:59.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.757+0000 7f50ce59c640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f50bc0a3a30 0x7f50bc0a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:59.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.757+0000 7f50ce59c640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f50bc0a3a30 0x7f50bc0a5ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f50c00058c0 tx=0x7f50c0005f90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:59.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.761+0000 7f50c77fe640 1 -- 172.21.15.94:0/2385645836 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f50c804b020 con 0x7f50d0103b40 2024-04-21T08:39:59.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.922+0000 7f50c77fe640 1 -- 172.21.15.94:0/2385645836 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 22) ==== 138817+0+0 (secure 0 0 0) 0x7f50c80bac20 con 0x7f50d0103b40 2024-04-21T08:39:59.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.960+0000 7f50d496d640 1 -- 172.21.15.94:0/2385645836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f50d002d5c0 con 0x7f50d0103b40 2024-04-21T08:39:59.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.961+0000 7f50c77fe640 1 -- 172.21.15.94:0/2385645836 <== mon.2 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f50c80bbc00 con 0x7f50d0103b40 2024-04-21T08:39:59.960 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:39:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 -- 172.21.15.94:0/2385645836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f50bc0a3a30 msgr2=0x7f50bc0a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f50bc0a3a30 0x7f50bc0a5ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f50c00058c0 tx=0x7f50c0005f90 comp rx=0 tx=0).stop 2024-04-21T08:39:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 -- 172.21.15.94:0/2385645836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d0103b40 msgr2=0x7f50d019f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d0103b40 0x7f50d019f710 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f50c802c630 tx=0x7f50c80346d0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 -- 172.21.15.94:0/2385645836 shutdown_connections 2024-04-21T08:39:59.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f50bc0a3a30 0x7f50bc0a5ef0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d0103180 0x7f50d01986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d0103b40 0x7f50d019f710 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 --2- 172.21.15.94:0/2385645836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50d0101f90 0x7f50d0198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:59.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 -- 172.21.15.94:0/2385645836 >> 172.21.15.94:0/2385645836 conn(0x7f50d00fd500 msgr2=0x7f50d0106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:59.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 -- 172.21.15.94:0/2385645836 shutdown_connections 2024-04-21T08:39:59.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:39:59.963+0000 7f50d496d640 1 -- 172.21.15.94:0/2385645836 wait complete. 2024-04-21T08:39:59.978 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:40:00.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.107+0000 7f57635a4640 1 -- 172.21.15.94:0/1025704122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f575c107cf0 msgr2=0x7f575c10a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.107+0000 7f57635a4640 1 --2- 172.21.15.94:0/1025704122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f575c107cf0 0x7f575c10a0e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5754009ca0 tx=0x7f575402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.107+0000 7f57635a4640 1 -- 172.21.15.94:0/1025704122 shutdown_connections 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.107+0000 7f57635a4640 1 --2- 172.21.15.94:0/1025704122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f575c1053c0 0x7f575c1077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.107+0000 7f57635a4640 1 --2- 172.21.15.94:0/1025704122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f575c107cf0 0x7f575c10a0e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.107+0000 7f57635a4640 1 --2- 172.21.15.94:0/1025704122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f575c102a90 0x7f575c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.107+0000 7f57635a4640 1 -- 172.21.15.94:0/1025704122 >> 172.21.15.94:0/1025704122 conn(0x7f575c0fc400 msgr2=0x7f575c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.107+0000 7f57635a4640 1 -- 172.21.15.94:0/1025704122 shutdown_connections 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f57635a4640 1 -- 172.21.15.94:0/1025704122 wait complete. 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f57635a4640 1 Processor -- start 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f57635a4640 1 -- start start 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f57635a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f575c102a90 0x7f575c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f57635a4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f575c1053c0 0x7f575c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f57635a4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f575c107cf0 0x7f575c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f57635a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f575c10dad0 con 0x7f575c102a90 2024-04-21T08:40:00.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f57635a4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f575c10d950 con 0x7f575c1053c0 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f57635a4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f575c10dc50 con 0x7f575c107cf0 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f5761340640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f575c102a90 0x7f575c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f5761340640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f575c102a90 0x7f575c198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58518/0 (socket says 172.21.15.94:58518) 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.108+0000 7f5761340640 1 -- 172.21.15.94:0/52678743 learned_addr learned my addr 172.21.15.94:0/52678743 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f5761b41640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f575c107cf0 0x7f575c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f5760b3f640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f575c1053c0 0x7f575c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f5761340640 1 -- 172.21.15.94:0/52678743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f575c107cf0 msgr2=0x7f575c19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f5761340640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f575c107cf0 0x7f575c19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f5761340640 1 -- 172.21.15.94:0/52678743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f575c1053c0 msgr2=0x7f575c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f5761340640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f575c1053c0 0x7f575c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f5761340640 1 -- 172.21.15.94:0/52678743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575c19fe20 con 0x7f575c102a90 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f5760b3f640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f575c1053c0 0x7f575c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:00.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f5761340640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f575c102a90 0x7f575c198200 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f575800d950 tx=0x7f575800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:00.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f57527fc640 1 -- 172.21.15.94:0/52678743 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758017030 con 0x7f575c102a90 2024-04-21T08:40:00.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f57635a4640 1 -- 172.21.15.94:0/52678743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f575c1a0110 con 0x7f575c102a90 2024-04-21T08:40:00.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f57635a4640 1 -- 172.21.15.94:0/52678743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f575c1a06b0 con 0x7f575c102a90 2024-04-21T08:40:00.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f57527fc640 1 -- 172.21.15.94:0/52678743 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5758004020 con 0x7f575c102a90 2024-04-21T08:40:00.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.109+0000 7f57527fc640 1 -- 172.21.15.94:0/52678743 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57580047a0 con 0x7f575c102a90 2024-04-21T08:40:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.110+0000 7f57635a4640 1 -- 172.21.15.94:0/52678743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f575c0fdef0 con 0x7f575c102a90 2024-04-21T08:40:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.111+0000 7f57527fc640 1 -- 172.21.15.94:0/52678743 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 22) ==== 138817+0+0 (secure 0 0 0) 0x7f5758009a30 con 0x7f575c102a90 2024-04-21T08:40:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.112+0000 7f57527fc640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57480a3e30 0x7f57480a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.112+0000 7f5760b3f640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57480a3e30 0x7f57480a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.112+0000 7f57527fc640 1 -- 172.21.15.94:0/52678743 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f57580d0eb0 con 0x7f575c102a90 2024-04-21T08:40:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.112+0000 7f5760b3f640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57480a3e30 0x7f57480a62f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f575c199710 tx=0x7f574c005f90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:00.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.120+0000 7f57527fc640 1 -- 172.21.15.94:0/52678743 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f57580ffff0 con 0x7f575c102a90 2024-04-21T08:40:00.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:59 smithi111 ceph-mon[31493]: mgrmap e21: x(active, since 1.02357s), standbys: y, z 2024-04-21T08:40:00.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:59 smithi111 ceph-mon[31493]: [21/Apr/2024:08:39:59] ENGINE Bus STARTING 2024-04-21T08:40:00.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:59 smithi111 ceph-mon[31493]: [21/Apr/2024:08:39:59] ENGINE Serving on https://172.21.15.94:7150 2024-04-21T08:40:00.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:59 smithi111 ceph-mon[31493]: [21/Apr/2024:08:39:59] ENGINE Serving on http://172.21.15.94:8765 2024-04-21T08:40:00.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:59 smithi111 ceph-mon[31493]: [21/Apr/2024:08:39:59] ENGINE Bus STARTED 2024-04-21T08:40:00.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:59 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1241243095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:00.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:39:59 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3434161103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:00.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:59 smithi173 ceph-mon[31428]: mgrmap e21: x(active, since 1.02357s), standbys: y, z 2024-04-21T08:40:00.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:59 smithi173 ceph-mon[31428]: [21/Apr/2024:08:39:59] ENGINE Bus STARTING 2024-04-21T08:40:00.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:59 smithi173 ceph-mon[31428]: [21/Apr/2024:08:39:59] ENGINE Serving on https://172.21.15.94:7150 2024-04-21T08:40:00.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:59 smithi173 ceph-mon[31428]: [21/Apr/2024:08:39:59] ENGINE Serving on http://172.21.15.94:8765 2024-04-21T08:40:00.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:59 smithi173 ceph-mon[31428]: [21/Apr/2024:08:39:59] ENGINE Bus STARTED 2024-04-21T08:40:00.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:59 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1241243095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:00.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:39:59 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3434161103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:00.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:59 smithi094 ceph-mon[27988]: mgrmap e21: x(active, since 1.02357s), standbys: y, z 2024-04-21T08:40:00.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:59 smithi094 ceph-mon[27988]: [21/Apr/2024:08:39:59] ENGINE Bus STARTING 2024-04-21T08:40:00.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:59 smithi094 ceph-mon[27988]: [21/Apr/2024:08:39:59] ENGINE Serving on https://172.21.15.94:7150 2024-04-21T08:40:00.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:59 smithi094 ceph-mon[27988]: [21/Apr/2024:08:39:59] ENGINE Serving on http://172.21.15.94:8765 2024-04-21T08:40:00.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:59 smithi094 ceph-mon[27988]: [21/Apr/2024:08:39:59] ENGINE Bus STARTED 2024-04-21T08:40:00.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:59 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1241243095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:00.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:39:59 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3434161103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:00.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.305+0000 7f57635a4640 1 -- 172.21.15.94:0/52678743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f575c062b50 con 0x7f575c102a90 2024-04-21T08:40:00.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.306+0000 7f57527fc640 1 -- 172.21.15.94:0/52678743 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f575809d080 con 0x7f575c102a90 2024-04-21T08:40:00.305 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:40:00.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 -- 172.21.15.94:0/52678743 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57480a3e30 msgr2=0x7f57480a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57480a3e30 0x7f57480a62f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f575c199710 tx=0x7f574c005f90 comp rx=0 tx=0).stop 2024-04-21T08:40:00.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 -- 172.21.15.94:0/52678743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f575c102a90 msgr2=0x7f575c198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f575c102a90 0x7f575c198200 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f575800d950 tx=0x7f575800de10 comp rx=0 tx=0).stop 2024-04-21T08:40:00.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 -- 172.21.15.94:0/52678743 shutdown_connections 2024-04-21T08:40:00.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57480a3e30 0x7f57480a62f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f575c107cf0 0x7f575c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f575c1053c0 0x7f575c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 --2- 172.21.15.94:0/52678743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f575c102a90 0x7f575c198200 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 -- 172.21.15.94:0/52678743 >> 172.21.15.94:0/52678743 conn(0x7f575c0fc400 msgr2=0x7f575c103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:00.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 -- 172.21.15.94:0/52678743 shutdown_connections 2024-04-21T08:40:00.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.308+0000 7f57635a4640 1 -- 172.21.15.94:0/52678743 wait complete. 2024-04-21T08:40:00.318 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.424+0000 7f3701f75640 1 -- 172.21.15.94:0/2041157073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36fc1029a0 msgr2=0x7f36fc1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.424+0000 7f3701f75640 1 --2- 172.21.15.94:0/2041157073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36fc1029a0 0x7f36fc1092c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009ca0 tx=0x7f36f402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.424+0000 7f3701f75640 1 -- 172.21.15.94:0/2041157073 shutdown_connections 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.424+0000 7f3701f75640 1 --2- 172.21.15.94:0/2041157073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36fc1029a0 0x7f36fc1092c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.424+0000 7f3701f75640 1 --2- 172.21.15.94:0/2041157073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36fc101fe0 0x7f36fc102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.424+0000 7f3701f75640 1 --2- 172.21.15.94:0/2041157073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36fc100e90 0x7f36fc1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.424+0000 7f3701f75640 1 -- 172.21.15.94:0/2041157073 >> 172.21.15.94:0/2041157073 conn(0x7f36fc0fc400 msgr2=0x7f36fc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.424+0000 7f3701f75640 1 -- 172.21.15.94:0/2041157073 shutdown_connections 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.424+0000 7f3701f75640 1 -- 172.21.15.94:0/2041157073 wait complete. 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.424+0000 7f3701f75640 1 Processor -- start 2024-04-21T08:40:00.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f3701f75640 1 -- start start 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f3701f75640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36fc100e90 0x7f36fc069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f3701f75640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36fc101fe0 0x7f36fc19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f3701f75640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36fc1029a0 0x7f36fc19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f3701f75640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc10c9f0 con 0x7f36fc100e90 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f3701f75640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc10c870 con 0x7f36fc1029a0 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f3701f75640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc10cb70 con 0x7f36fc101fe0 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36fc100e90 0x7f36fc069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36fc100e90 0x7f36fc069cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58526/0 (socket says 172.21.15.94:58526) 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fb7fe640 1 -- 172.21.15.94:0/512782034 learned_addr learned my addr 172.21.15.94:0/512782034 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36f3fff640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36fc101fe0 0x7f36fc19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fbfff640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36fc1029a0 0x7f36fc19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fb7fe640 1 -- 172.21.15.94:0/512782034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36fc101fe0 msgr2=0x7f36fc19cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fb7fe640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36fc101fe0 0x7f36fc19cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fb7fe640 1 -- 172.21.15.94:0/512782034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36fc1029a0 msgr2=0x7f36fc19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fb7fe640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36fc1029a0 0x7f36fc19f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fb7fe640 1 -- 172.21.15.94:0/512782034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36fc19fdf0 con 0x7f36fc100e90 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fbfff640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36fc1029a0 0x7f36fc19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36f3fff640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36fc101fe0 0x7f36fc19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36fb7fe640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36fc100e90 0x7f36fc069cb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f36ec00e9f0 tx=0x7f36ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36f97fa640 1 -- 172.21.15.94:0/512782034 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec00cd80 con 0x7f36fc100e90 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f3701f75640 1 -- 172.21.15.94:0/512782034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36fc1a00e0 con 0x7f36fc100e90 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36f97fa640 1 -- 172.21.15.94:0/512782034 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36ec0054d0 con 0x7f36fc100e90 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f36f97fa640 1 -- 172.21.15.94:0/512782034 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec010690 con 0x7f36fc100e90 2024-04-21T08:40:00.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.425+0000 7f3701f75640 1 -- 172.21.15.94:0/512782034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36fc1a05e0 con 0x7f36fc100e90 2024-04-21T08:40:00.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.427+0000 7f3701f75640 1 -- 172.21.15.94:0/512782034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36fc1081c0 con 0x7f36fc100e90 2024-04-21T08:40:00.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.428+0000 7f36f97fa640 1 -- 172.21.15.94:0/512782034 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 22) ==== 138817+0+0 (secure 0 0 0) 0x7f36ec004e30 con 0x7f36fc100e90 2024-04-21T08:40:00.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.428+0000 7f36f97fa640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f36e80a3db0 0x7f36e80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.428+0000 7f36f97fa640 1 -- 172.21.15.94:0/512782034 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f36ec0d14d0 con 0x7f36fc100e90 2024-04-21T08:40:00.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.428+0000 7f36f3fff640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f36e80a3db0 0x7f36e80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.429+0000 7f36f3fff640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f36e80a3db0 0x7f36e80a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f36e4005d00 tx=0x7f36e4005c70 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:00.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.432+0000 7f36f97fa640 1 -- 172.21.15.94:0/512782034 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36ec053ff0 con 0x7f36fc100e90 2024-04-21T08:40:00.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.619+0000 7f3701f75640 1 -- 172.21.15.94:0/512782034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f36fc062b80 con 0x7f36fc100e90 2024-04-21T08:40:00.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.620+0000 7f36f97fa640 1 -- 172.21.15.94:0/512782034 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f36ec01f340 con 0x7f36fc100e90 2024-04-21T08:40:00.619 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:00.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 -- 172.21.15.94:0/512782034 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f36e80a3db0 msgr2=0x7f36e80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f36e80a3db0 0x7f36e80a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f36e4005d00 tx=0x7f36e4005c70 comp rx=0 tx=0).stop 2024-04-21T08:40:00.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 -- 172.21.15.94:0/512782034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36fc100e90 msgr2=0x7f36fc069cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36fc100e90 0x7f36fc069cb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f36ec00e9f0 tx=0x7f36ec00edf0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 -- 172.21.15.94:0/512782034 shutdown_connections 2024-04-21T08:40:00.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f36e80a3db0 0x7f36e80a6270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36fc101fe0 0x7f36fc19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36fc1029a0 0x7f36fc19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 --2- 172.21.15.94:0/512782034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f36fc100e90 0x7f36fc069cb0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 -- 172.21.15.94:0/512782034 >> 172.21.15.94:0/512782034 conn(0x7f36fc0fc400 msgr2=0x7f36fc105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:00.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 -- 172.21.15.94:0/512782034 shutdown_connections 2024-04-21T08:40:00.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.622+0000 7f3701f75640 1 -- 172.21.15.94:0/512782034 wait complete. 2024-04-21T08:40:00.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:40:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.735+0000 7f61bedf4640 1 -- 172.21.15.94:0/3253578824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b8105320 msgr2=0x7f61b8107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.735+0000 7f61bedf4640 1 --2- 172.21.15.94:0/3253578824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b8105320 0x7f61b8107710 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009900 tx=0x7f61a402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.735+0000 7f61bedf4640 1 -- 172.21.15.94:0/3253578824 shutdown_connections 2024-04-21T08:40:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.735+0000 7f61bedf4640 1 --2- 172.21.15.94:0/3253578824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b8107c50 0x7f61b810a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.735+0000 7f61bedf4640 1 --2- 172.21.15.94:0/3253578824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b8105320 0x7f61b8107710 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.735+0000 7f61bedf4640 1 --2- 172.21.15.94:0/3253578824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b81029f0 0x7f61b8104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.735+0000 7f61bedf4640 1 -- 172.21.15.94:0/3253578824 >> 172.21.15.94:0/3253578824 conn(0x7f61b80fc3e0 msgr2=0x7f61b80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.735+0000 7f61bedf4640 1 -- 172.21.15.94:0/3253578824 shutdown_connections 2024-04-21T08:40:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.735+0000 7f61bedf4640 1 -- 172.21.15.94:0/3253578824 wait complete. 2024-04-21T08:40:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bedf4640 1 Processor -- start 2024-04-21T08:40:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bedf4640 1 -- start start 2024-04-21T08:40:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bedf4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b81029f0 0x7f61b81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bedf4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b8105320 0x7f61b81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bedf4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b8107c50 0x7f61b819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bedf4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f61b810dad0 con 0x7f61b8107c50 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bedf4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f61b810d950 con 0x7f61b8105320 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bedf4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f61b810dc50 con 0x7f61b81029f0 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bd391640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b8107c50 0x7f61b819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61b7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b8105320 0x7f61b81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bd391640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b8107c50 0x7f61b819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58556/0 (socket says 172.21.15.94:58556) 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bcb90640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b81029f0 0x7f61b81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61bd391640 1 -- 172.21.15.94:0/2280252634 learned_addr learned my addr 172.21.15.94:0/2280252634 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61b7fff640 1 -- 172.21.15.94:0/2280252634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b81029f0 msgr2=0x7f61b81981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61b7fff640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b81029f0 0x7f61b81981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61b7fff640 1 -- 172.21.15.94:0/2280252634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b8107c50 msgr2=0x7f61b819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61b7fff640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b8107c50 0x7f61b819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.736+0000 7f61b7fff640 1 -- 172.21.15.94:0/2280252634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b819fdd0 con 0x7f61b8105320 2024-04-21T08:40:00.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.737+0000 7f61bd391640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b8107c50 0x7f61b819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:00.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.737+0000 7f61b7fff640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b8105320 0x7f61b81986f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f61a4036040 tx=0x7f61a40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:00.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.737+0000 7f61b5ffb640 1 -- 172.21.15.94:0/2280252634 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a4043020 con 0x7f61b8105320 2024-04-21T08:40:00.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.737+0000 7f61b5ffb640 1 -- 172.21.15.94:0/2280252634 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61a4004a50 con 0x7f61b8105320 2024-04-21T08:40:00.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.737+0000 7f61bedf4640 1 -- 172.21.15.94:0/2280252634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b81a0060 con 0x7f61b8105320 2024-04-21T08:40:00.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.737+0000 7f61b5ffb640 1 -- 172.21.15.94:0/2280252634 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a4003b30 con 0x7f61b8105320 2024-04-21T08:40:00.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.737+0000 7f61bedf4640 1 -- 172.21.15.94:0/2280252634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b81a0580 con 0x7f61b8105320 2024-04-21T08:40:00.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.738+0000 7f61bedf4640 1 -- 172.21.15.94:0/2280252634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b80fded0 con 0x7f61b8105320 2024-04-21T08:40:00.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.740+0000 7f61b5ffb640 1 -- 172.21.15.94:0/2280252634 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) ==== 138817+0+0 (secure 0 0 0) 0x7f61a4046020 con 0x7f61b8105320 2024-04-21T08:40:00.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.740+0000 7f61b5ffb640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61a80a3db0 0x7f61a80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:00.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.740+0000 7f61bcb90640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61a80a3db0 0x7f61a80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:00.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.740+0000 7f61b5ffb640 1 -- 172.21.15.94:0/2280252634 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f61a40f3390 con 0x7f61b8105320 2024-04-21T08:40:00.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.741+0000 7f61bcb90640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61a80a3db0 0x7f61a80a6270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009bd0 tx=0x7f61ac005f90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:00.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.744+0000 7f61b5ffb640 1 -- 172.21.15.94:0/2280252634 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61a404d020 con 0x7f61b8105320 2024-04-21T08:40:00.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.931+0000 7f61bedf4640 1 -- 172.21.15.94:0/2280252634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61b8062b50 con 0x7f61b8105320 2024-04-21T08:40:00.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.933+0000 7f61b5ffb640 1 -- 172.21.15.94:0/2280252634 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25266 (secure 0 0 0) 0x7f61a40bf5f0 con 0x7f61b8105320 2024-04-21T08:40:00.932 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:00.932 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:57.898777+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:40:00.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.935+0000 7f61bedf4640 1 -- 172.21.15.94:0/2280252634 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61a80a3db0 msgr2=0x7f61a80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.935+0000 7f61bedf4640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61a80a3db0 0x7f61a80a6270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009bd0 tx=0x7f61ac005f90 comp rx=0 tx=0).stop 2024-04-21T08:40:00.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.935+0000 7f61bedf4640 1 -- 172.21.15.94:0/2280252634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b8105320 msgr2=0x7f61b81986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:00.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.935+0000 7f61bedf4640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b8105320 0x7f61b81986f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f61a4036040 tx=0x7f61a40042f0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.935+0000 7f61bedf4640 1 -- 172.21.15.94:0/2280252634 shutdown_connections 2024-04-21T08:40:00.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.935+0000 7f61bedf4640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61a80a3db0 0x7f61a80a6270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.935+0000 7f61bedf4640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61b8107c50 0x7f61b819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.935+0000 7f61bedf4640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b8105320 0x7f61b81986f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.935+0000 7f61bedf4640 1 --2- 172.21.15.94:0/2280252634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61b81029f0 0x7f61b81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:00.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.935+0000 7f61bedf4640 1 -- 172.21.15.94:0/2280252634 >> 172.21.15.94:0/2280252634 conn(0x7f61b80fc3e0 msgr2=0x7f61b81034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:00.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.936+0000 7f61bedf4640 1 -- 172.21.15.94:0/2280252634 shutdown_connections 2024-04-21T08:40:00.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:00.936+0000 7f61bedf4640 1 -- 172.21.15.94:0/2280252634 wait complete. 2024-04-21T08:40:00.946 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:40:00.946 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:40:01.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 -- 172.21.15.94:0/562335856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c4100e90 msgr2=0x7fb5c41012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 --2- 172.21.15.94:0/562335856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c4100e90 0x7fb5c41012b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009900 tx=0x7fb5b402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:01.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 -- 172.21.15.94:0/562335856 shutdown_connections 2024-04-21T08:40:01.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 --2- 172.21.15.94:0/562335856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c41029a0 0x7fb5c41092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 --2- 172.21.15.94:0/562335856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c4101fe0 0x7fb5c4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 --2- 172.21.15.94:0/562335856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c4100e90 0x7fb5c41012b0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 -- 172.21.15.94:0/562335856 >> 172.21.15.94:0/562335856 conn(0x7fb5c40fc400 msgr2=0x7fb5c40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:01.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 -- 172.21.15.94:0/562335856 shutdown_connections 2024-04-21T08:40:01.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 -- 172.21.15.94:0/562335856 wait complete. 2024-04-21T08:40:01.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 Processor -- start 2024-04-21T08:40:01.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.051+0000 7fb5ca752640 1 -- start start 2024-04-21T08:40:01.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5ca752640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c4100e90 0x7fb5c4197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5ca752640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c4101fe0 0x7fb5c41976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5ca752640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c41029a0 0x7fb5c419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5ca752640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c410ca90 con 0x7fb5c4101fe0 2024-04-21T08:40:01.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5ca752640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c410c910 con 0x7fb5c4100e90 2024-04-21T08:40:01.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5ca752640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c410cc10 con 0x7fb5c41029a0 2024-04-21T08:40:01.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c4101fe0 0x7fb5c41976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c4101fe0 0x7fb5c41976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58568/0 (socket says 172.21.15.94:58568) 2024-04-21T08:40:01.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c37fe640 1 -- 172.21.15.94:0/1786342660 learned_addr learned my addr 172.21.15.94:0/1786342660 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:01.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c3fff640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c4100e90 0x7fb5c4197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c8cef640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c41029a0 0x7fb5c419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c37fe640 1 -- 172.21.15.94:0/1786342660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c41029a0 msgr2=0x7fb5c419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c37fe640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c41029a0 0x7fb5c419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c37fe640 1 -- 172.21.15.94:0/1786342660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c4100e90 msgr2=0x7fb5c4197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c37fe640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c4100e90 0x7fb5c4197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c37fe640 1 -- 172.21.15.94:0/1786342660 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c419ed80 con 0x7fb5c4101fe0 2024-04-21T08:40:01.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.052+0000 7fb5c37fe640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c4101fe0 0x7fb5c41976a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800d770 tx=0x7fb5b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:01.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.053+0000 7fb5c17fa640 1 -- 172.21.15.94:0/1786342660 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8017070 con 0x7fb5c4101fe0 2024-04-21T08:40:01.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.053+0000 7fb5c17fa640 1 -- 172.21.15.94:0/1786342660 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5b8009c30 con 0x7fb5c4101fe0 2024-04-21T08:40:01.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.053+0000 7fb5ca752640 1 -- 172.21.15.94:0/1786342660 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c419f070 con 0x7fb5c4101fe0 2024-04-21T08:40:01.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.053+0000 7fb5c17fa640 1 -- 172.21.15.94:0/1786342660 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8003cc0 con 0x7fb5c4101fe0 2024-04-21T08:40:01.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.053+0000 7fb5ca752640 1 -- 172.21.15.94:0/1786342660 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5c419f640 con 0x7fb5c4101fe0 2024-04-21T08:40:01.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.054+0000 7fb5ca752640 1 -- 172.21.15.94:0/1786342660 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5c41081f0 con 0x7fb5c4101fe0 2024-04-21T08:40:01.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.055+0000 7fb5c17fa640 1 -- 172.21.15.94:0/1786342660 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 22) ==== 138817+0+0 (secure 0 0 0) 0x7fb5b8003e60 con 0x7fb5c4101fe0 2024-04-21T08:40:01.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.055+0000 7fb5c17fa640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb5ac0a3db0 0x7fb5ac0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.056+0000 7fb5c3fff640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb5ac0a3db0 0x7fb5ac0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.056+0000 7fb5c17fa640 1 -- 172.21.15.94:0/1786342660 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb5b80d0b60 con 0x7fb5c4101fe0 2024-04-21T08:40:01.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.056+0000 7fb5c3fff640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb5ac0a3db0 0x7fb5ac0a6270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4000c00 tx=0x7fb5b402c7e0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:01.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.059+0000 7fb5c17fa640 1 -- 172.21.15.94:0/1786342660 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5b8009de0 con 0x7fb5c4101fe0 2024-04-21T08:40:01.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:40:01.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-04-21T08:40:01.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2385645836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:01.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: overall HEALTH_OK 2024-04-21T08:40:01.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/52678743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:01.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/512782034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.252+0000 7fb5ca752640 1 -- 172.21.15.94:0/1786342660 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb5c402d810 con 0x7fb5c4101fe0 2024-04-21T08:40:01.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.254+0000 7fb5c17fa640 1 -- 172.21.15.94:0/1786342660 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fb5b809ccb0 con 0x7fb5c4101fe0 2024-04-21T08:40:01.253 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:40:01.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 -- 172.21.15.94:0/1786342660 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb5ac0a3db0 msgr2=0x7fb5ac0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb5ac0a3db0 0x7fb5ac0a6270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4000c00 tx=0x7fb5b402c7e0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 -- 172.21.15.94:0/1786342660 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c4101fe0 msgr2=0x7fb5c41976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c4101fe0 0x7fb5c41976a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800d770 tx=0x7fb5b800dc30 comp rx=0 tx=0).stop 2024-04-21T08:40:01.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 -- 172.21.15.94:0/1786342660 shutdown_connections 2024-04-21T08:40:01.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb5ac0a3db0 0x7fb5ac0a6270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c41029a0 0x7fb5c419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c4101fe0 0x7fb5c41976a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 --2- 172.21.15.94:0/1786342660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c4100e90 0x7fb5c4197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 -- 172.21.15.94:0/1786342660 >> 172.21.15.94:0/1786342660 conn(0x7fb5c40fc400 msgr2=0x7fb5c41053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:01.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 -- 172.21.15.94:0/1786342660 shutdown_connections 2024-04-21T08:40:01.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.256+0000 7fb5ca752640 1 -- 172.21.15.94:0/1786342660 wait complete. 2024-04-21T08:40:01.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:40:01.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-04-21T08:40:01.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2385645836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:01.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: overall HEALTH_OK 2024-04-21T08:40:01.263 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.263 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.263 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.263 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.264 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.264 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.264 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/52678743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:01.264 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/512782034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:01.265 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:40:01.265 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-04-21T08:40:01.265 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2385645836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:01.266 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: overall HEALTH_OK 2024-04-21T08:40:01.266 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.266 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.266 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.266 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.266 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.266 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:01.266 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/52678743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:01.267 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/512782034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:01.268 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:40:01.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.383+0000 7f7075d06640 1 -- 172.21.15.94:0/2267770178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7070102a70 msgr2=0x7f7070104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.383+0000 7f7075d06640 1 --2- 172.21.15.94:0/2267770178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7070102a70 0x7f7070104e60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7060009900 tx=0x7f706002bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:01.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.383+0000 7f7075d06640 1 -- 172.21.15.94:0/2267770178 shutdown_connections 2024-04-21T08:40:01.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.383+0000 7f7075d06640 1 --2- 172.21.15.94:0/2267770178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70701053a0 0x7f7070107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.383+0000 7f7075d06640 1 --2- 172.21.15.94:0/2267770178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7070107cd0 0x7f707010a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.383+0000 7f7075d06640 1 --2- 172.21.15.94:0/2267770178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7070102a70 0x7f7070104e60 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 -- 172.21.15.94:0/2267770178 >> 172.21.15.94:0/2267770178 conn(0x7f70700fc400 msgr2=0x7f70700fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:01.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 -- 172.21.15.94:0/2267770178 shutdown_connections 2024-04-21T08:40:01.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 -- 172.21.15.94:0/2267770178 wait complete. 2024-04-21T08:40:01.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 Processor -- start 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 -- start start 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7070102a70 0x7f7070198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70701053a0 0x7f7070198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7070107cd0 0x7f707019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f707010dac0 con 0x7f7070102a70 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f707010d940 con 0x7f70701053a0 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.384+0000 7f7075d06640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f707010dc40 con 0x7f7070107cd0 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7070102a70 0x7f7070198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7070102a70 0x7f7070198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58598/0 (socket says 172.21.15.94:58598) 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706f7fe640 1 -- 172.21.15.94:0/1552073623 learned_addr learned my addr 172.21.15.94:0/1552073623 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706effd640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70701053a0 0x7f7070198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706ffff640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7070107cd0 0x7f707019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706f7fe640 1 -- 172.21.15.94:0/1552073623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7070107cd0 msgr2=0x7f707019f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706f7fe640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7070107cd0 0x7f707019f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706f7fe640 1 -- 172.21.15.94:0/1552073623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70701053a0 msgr2=0x7f7070198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706f7fe640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70701053a0 0x7f7070198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706f7fe640 1 -- 172.21.15.94:0/1552073623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707019fe60 con 0x7f7070102a70 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706effd640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70701053a0 0x7f7070198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706f7fe640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7070102a70 0x7f7070198240 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7060006010 tx=0x7f7060005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706cff9640 1 -- 172.21.15.94:0/1552073623 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060043020 con 0x7f7070102a70 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f7075d06640 1 -- 172.21.15.94:0/1552073623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70701a00f0 con 0x7f7070102a70 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706cff9640 1 -- 172.21.15.94:0/1552073623 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70600043d0 con 0x7f7070102a70 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f706cff9640 1 -- 172.21.15.94:0/1552073623 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060003b80 con 0x7f7070102a70 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.385+0000 7f7075d06640 1 -- 172.21.15.94:0/1552073623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70701a0690 con 0x7f7070102a70 2024-04-21T08:40:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.386+0000 7f7075d06640 1 -- 172.21.15.94:0/1552073623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70700fdef0 con 0x7f7070102a70 2024-04-21T08:40:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.388+0000 7f706cff9640 1 -- 172.21.15.94:0/1552073623 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 22) ==== 138817+0+0 (secure 0 0 0) 0x7f7060046020 con 0x7f7070102a70 2024-04-21T08:40:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.388+0000 7f706cff9640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f70580a3db0 0x7f70580a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.389+0000 7f706effd640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f70580a3db0 0x7f70580a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.389+0000 7f706cff9640 1 -- 172.21.15.94:0/1552073623 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f70600f34f0 con 0x7f7070102a70 2024-04-21T08:40:01.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.389+0000 7f706effd640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f70580a3db0 0x7f70580a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7070199750 tx=0x7f7064005f90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:01.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.392+0000 7f706cff9640 1 -- 172.21.15.94:0/1552073623 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f706004d020 con 0x7f7070102a70 2024-04-21T08:40:01.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.578+0000 7f7075d06640 1 -- 172.21.15.94:0/1552073623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f707002d750 con 0x7f7070102a70 2024-04-21T08:40:01.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.578+0000 7f706cff9640 1 -- 172.21.15.94:0/1552073623 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f707002d750 con 0x7f7070102a70 2024-04-21T08:40:01.577 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.580+0000 7f7075d06640 1 -- 172.21.15.94:0/1552073623 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f70580a3db0 msgr2=0x7f70580a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.580+0000 7f7075d06640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f70580a3db0 0x7f70580a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7070199750 tx=0x7f7064005f90 comp rx=0 tx=0).stop 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.581+0000 7f7075d06640 1 -- 172.21.15.94:0/1552073623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7070102a70 msgr2=0x7f7070198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.581+0000 7f7075d06640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7070102a70 0x7f7070198240 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7060006010 tx=0x7f7060005e40 comp rx=0 tx=0).stop 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.581+0000 7f7075d06640 1 -- 172.21.15.94:0/1552073623 shutdown_connections 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.581+0000 7f7075d06640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f70580a3db0 0x7f70580a6270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.581+0000 7f7075d06640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7070107cd0 0x7f707019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.581+0000 7f7075d06640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70701053a0 0x7f7070198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.581+0000 7f7075d06640 1 --2- 172.21.15.94:0/1552073623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7070102a70 0x7f7070198240 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.581+0000 7f7075d06640 1 -- 172.21.15.94:0/1552073623 >> 172.21.15.94:0/1552073623 conn(0x7f70700fc400 msgr2=0x7f7070104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.581+0000 7f7075d06640 1 -- 172.21.15.94:0/1552073623 shutdown_connections 2024-04-21T08:40:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.581+0000 7f7075d06640 1 -- 172.21.15.94:0/1552073623 wait complete. 2024-04-21T08:40:01.591 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:40:01.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.696+0000 7f8deb115640 1 -- 172.21.15.94:0/3608047251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8de41011e0 msgr2=0x7f8de4101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.696+0000 7f8deb115640 1 --2- 172.21.15.94:0/3608047251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8de41011e0 0x7f8de4101640 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009900 tx=0x7f8dd002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 -- 172.21.15.94:0/3608047251 shutdown_connections 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 --2- 172.21.15.94:0/3608047251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8de4101b80 0x7f8de4102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 --2- 172.21.15.94:0/3608047251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8de41011e0 0x7f8de4101640 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 --2- 172.21.15.94:0/3608047251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8de4100880 0x7f8de4100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 -- 172.21.15.94:0/3608047251 >> 172.21.15.94:0/3608047251 conn(0x7f8de40fc400 msgr2=0x7f8de40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 -- 172.21.15.94:0/3608047251 shutdown_connections 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 -- 172.21.15.94:0/3608047251 wait complete. 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 Processor -- start 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 -- start start 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8de4100880 0x7f8de419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8de41011e0 0x7f8de419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8de4101b80 0x7f8de41a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8de410fd20 con 0x7f8de41011e0 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8de410fba0 con 0x7f8de4101b80 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8deb115640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8de410fea0 con 0x7f8de4100880 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8de3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8de41011e0 0x7f8de419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8de3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8de41011e0 0x7f8de419a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58616/0 (socket says 172.21.15.94:58616) 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.697+0000 7f8de3fff640 1 -- 172.21.15.94:0/1067457797 learned_addr learned my addr 172.21.15.94:0/1067457797 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de96b2640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8de4101b80 0x7f8de41a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de8eb1640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8de4100880 0x7f8de419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de3fff640 1 -- 172.21.15.94:0/1067457797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8de4100880 msgr2=0x7f8de419a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de3fff640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8de4100880 0x7f8de419a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de3fff640 1 -- 172.21.15.94:0/1067457797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8de4101b80 msgr2=0x7f8de41a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de3fff640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8de4101b80 0x7f8de41a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de3fff640 1 -- 172.21.15.94:0/1067457797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de41a2070 con 0x7f8de41011e0 2024-04-21T08:40:01.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de3fff640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8de41011e0 0x7f8de419a9e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0000c00 tx=0x7f8dd0005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:01.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de1ffb640 1 -- 172.21.15.94:0/1067457797 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0043020 con 0x7f8de41011e0 2024-04-21T08:40:01.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de1ffb640 1 -- 172.21.15.94:0/1067457797 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dd002dd20 con 0x7f8de41011e0 2024-04-21T08:40:01.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8deb115640 1 -- 172.21.15.94:0/1067457797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de41a2300 con 0x7f8de41011e0 2024-04-21T08:40:01.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8de1ffb640 1 -- 172.21.15.94:0/1067457797 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0003af0 con 0x7f8de41011e0 2024-04-21T08:40:01.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.698+0000 7f8deb115640 1 -- 172.21.15.94:0/1067457797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de41a28a0 con 0x7f8de41011e0 2024-04-21T08:40:01.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.700+0000 7f8deb115640 1 -- 172.21.15.94:0/1067457797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de410b7f0 con 0x7f8de41011e0 2024-04-21T08:40:01.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.701+0000 7f8de1ffb640 1 -- 172.21.15.94:0/1067457797 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 22) ==== 138817+0+0 (secure 0 0 0) 0x7f8dd0046020 con 0x7f8de41011e0 2024-04-21T08:40:01.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.702+0000 7f8de1ffb640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8dd40a3e30 0x7f8dd40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:01.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.702+0000 7f8de8eb1640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8dd40a3e30 0x7f8dd40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:01.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.702+0000 7f8de1ffb640 1 -- 172.21.15.94:0/1067457797 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f8dd00f2cd0 con 0x7f8de41011e0 2024-04-21T08:40:01.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.702+0000 7f8de8eb1640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8dd40a3e30 0x7f8dd40a62f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8009bd0 tx=0x7f8dd8005f90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:01.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.706+0000 7f8de1ffb640 1 -- 172.21.15.94:0/1067457797 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8dd004d020 con 0x7f8de41011e0 2024-04-21T08:40:01.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.896+0000 7f8deb115640 1 -- 172.21.15.94:0/1067457797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8de402d810 con 0x7f8de41011e0 2024-04-21T08:40:01.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.897+0000 7f8de1ffb640 1 -- 172.21.15.94:0/1067457797 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f8dd00bef30 con 0x7f8de41011e0 2024-04-21T08:40:01.896 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:40:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 -- 172.21.15.94:0/1067457797 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8dd40a3e30 msgr2=0x7f8dd40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8dd40a3e30 0x7f8dd40a62f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8009bd0 tx=0x7f8dd8005f90 comp rx=0 tx=0).stop 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 -- 172.21.15.94:0/1067457797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8de41011e0 msgr2=0x7f8de419a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8de41011e0 0x7f8de419a9e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0000c00 tx=0x7f8dd0005e30 comp rx=0 tx=0).stop 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 -- 172.21.15.94:0/1067457797 shutdown_connections 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8dd40a3e30 0x7f8dd40a62f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8de4101b80 0x7f8de41a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8de41011e0 0x7f8de419a9e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 --2- 172.21.15.94:0/1067457797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8de4100880 0x7f8de419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 -- 172.21.15.94:0/1067457797 >> 172.21.15.94:0/1067457797 conn(0x7f8de40fc400 msgr2=0x7f8de40fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 -- 172.21.15.94:0/1067457797 shutdown_connections 2024-04-21T08:40:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:01.899+0000 7f8deb115640 1 -- 172.21.15.94:0/1067457797 wait complete. 2024-04-21T08:40:01.910 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:40:02.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.019+0000 7f4ea8138640 1 -- 172.21.15.94:0/1609039910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ea0102a30 msgr2=0x7f4ea0104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.019+0000 7f4ea8138640 1 --2- 172.21.15.94:0/1609039910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ea0102a30 0x7f4ea0104e20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009900 tx=0x7f4e9c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:02.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.019+0000 7f4ea8138640 1 -- 172.21.15.94:0/1609039910 shutdown_connections 2024-04-21T08:40:02.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.019+0000 7f4ea8138640 1 --2- 172.21.15.94:0/1609039910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea0107c90 0x7f4ea010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.019+0000 7f4ea8138640 1 --2- 172.21.15.94:0/1609039910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ea0105360 0x7f4ea0107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.019+0000 7f4ea8138640 1 --2- 172.21.15.94:0/1609039910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ea0102a30 0x7f4ea0104e20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.019+0000 7f4ea8138640 1 -- 172.21.15.94:0/1609039910 >> 172.21.15.94:0/1609039910 conn(0x7f4ea00fc420 msgr2=0x7f4ea00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.019+0000 7f4ea8138640 1 -- 172.21.15.94:0/1609039910 shutdown_connections 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.019+0000 7f4ea8138640 1 -- 172.21.15.94:0/1609039910 wait complete. 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea8138640 1 Processor -- start 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea8138640 1 -- start start 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea8138640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ea0102a30 0x7f4ea0198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea8138640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ea0105360 0x7f4ea01987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea8138640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea0107c90 0x7f4ea019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea8138640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea010daf0 con 0x7f4ea0105360 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea8138640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea010d970 con 0x7f4ea0102a30 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea8138640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea010dc70 con 0x7f4ea0107c90 2024-04-21T08:40:02.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea56d3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ea0105360 0x7f4ea01987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea56d3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ea0105360 0x7f4ea01987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58628/0 (socket says 172.21.15.94:58628) 2024-04-21T08:40:02.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea56d3640 1 -- 172.21.15.94:0/2022328867 learned_addr learned my addr 172.21.15.94:0/2022328867 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:02.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea66d5640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea0107c90 0x7f4ea019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.020+0000 7f4ea5ed4640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ea0102a30 0x7f4ea0198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4ea56d3640 1 -- 172.21.15.94:0/2022328867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea0107c90 msgr2=0x7f4ea019f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4ea56d3640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea0107c90 0x7f4ea019f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4ea56d3640 1 -- 172.21.15.94:0/2022328867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ea0102a30 msgr2=0x7f4ea0198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4ea56d3640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ea0102a30 0x7f4ea0198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4ea56d3640 1 -- 172.21.15.94:0/2022328867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea019fe80 con 0x7f4ea0105360 2024-04-21T08:40:02.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4ea56d3640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ea0105360 0x7f4ea01987a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000e9c0 tx=0x7f4e9000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:02.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4e96ffd640 1 -- 172.21.15.94:0/2022328867 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9000cd80 con 0x7f4ea0105360 2024-04-21T08:40:02.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4e96ffd640 1 -- 172.21.15.94:0/2022328867 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e900054d0 con 0x7f4ea0105360 2024-04-21T08:40:02.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4ea8138640 1 -- 172.21.15.94:0/2022328867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ea01a0170 con 0x7f4ea0105360 2024-04-21T08:40:02.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4e96ffd640 1 -- 172.21.15.94:0/2022328867 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90010430 con 0x7f4ea0105360 2024-04-21T08:40:02.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.021+0000 7f4ea8138640 1 -- 172.21.15.94:0/2022328867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ea01a06c0 con 0x7f4ea0105360 2024-04-21T08:40:02.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.022+0000 7f4ea8138640 1 -- 172.21.15.94:0/2022328867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ea00fdf10 con 0x7f4ea0105360 2024-04-21T08:40:02.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.024+0000 7f4e96ffd640 1 -- 172.21.15.94:0/2022328867 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 22) ==== 138817+0+0 (secure 0 0 0) 0x7f4e90004fb0 con 0x7f4ea0105360 2024-04-21T08:40:02.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.024+0000 7f4e96ffd640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4e8c0a3db0 0x7f4e8c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.024+0000 7f4e96ffd640 1 -- 172.21.15.94:0/2022328867 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4e900d1370 con 0x7f4ea0105360 2024-04-21T08:40:02.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.024+0000 7f4ea5ed4640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4e8c0a3db0 0x7f4e8c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.025+0000 7f4ea5ed4640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4e8c0a3db0 0x7f4e8c0a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009230 tx=0x7f4e9c005fd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:02.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.028+0000 7f4e96ffd640 1 -- 172.21.15.94:0/2022328867 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e90053ff0 con 0x7f4ea0105360 2024-04-21T08:40:02.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2280252634' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi094 to 176.2M 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Unable to set osd_memory_target on smithi094 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1786342660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Unable to set osd_memory_target on smithi111 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:02.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Updating smithi094:/etc/ceph/ceph.conf 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1552073623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:02.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:01 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1067457797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.240+0000 7f4ea8138640 1 -- 172.21.15.94:0/2022328867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4ea002d810 con 0x7f4ea0105360 2024-04-21T08:40:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.241+0000 7f4e96ffd640 1 -- 172.21.15.94:0/2022328867 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f4e9009d4c0 con 0x7f4ea0105360 2024-04-21T08:40:02.240 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.242+0000 7f4ea8138640 1 -- 172.21.15.94:0/2022328867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4e8c0a3db0 msgr2=0x7f4e8c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.242+0000 7f4ea8138640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4e8c0a3db0 0x7f4e8c0a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009230 tx=0x7f4e9c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.242+0000 7f4ea8138640 1 -- 172.21.15.94:0/2022328867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ea0105360 msgr2=0x7f4ea01987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.242+0000 7f4ea8138640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ea0105360 0x7f4ea01987a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000e9c0 tx=0x7f4e9000ee80 comp rx=0 tx=0).stop 2024-04-21T08:40:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.243+0000 7f4ea8138640 1 -- 172.21.15.94:0/2022328867 shutdown_connections 2024-04-21T08:40:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.243+0000 7f4ea8138640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4e8c0a3db0 0x7f4e8c0a6270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.243+0000 7f4ea8138640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea0107c90 0x7f4ea019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.243+0000 7f4ea8138640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ea0105360 0x7f4ea01987a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.243+0000 7f4ea8138640 1 --2- 172.21.15.94:0/2022328867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ea0102a30 0x7f4ea0198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.243+0000 7f4ea8138640 1 -- 172.21.15.94:0/2022328867 >> 172.21.15.94:0/2022328867 conn(0x7f4ea00fc420 msgr2=0x7f4ea01048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.243+0000 7f4ea8138640 1 -- 172.21.15.94:0/2022328867 shutdown_connections 2024-04-21T08:40:02.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.243+0000 7f4ea8138640 1 -- 172.21.15.94:0/2022328867 wait complete. 2024-04-21T08:40:02.252 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2280252634' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi094 to 176.2M 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Unable to set osd_memory_target on smithi094 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.253 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1786342660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Unable to set osd_memory_target on smithi111 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Updating smithi094:/etc/ceph/ceph.conf 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1552073623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:02.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:02.256 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:02.256 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:02.256 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:01 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1067457797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:02.257 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-21T08:40:02.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:40:02.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2280252634' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi094 to 176.2M 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Unable to set osd_memory_target on smithi094 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1786342660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:02.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Unable to set osd_memory_target on smithi111 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-21T08:40:02.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-21T08:40:02.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-04-21T08:40:02.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:02.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:02.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Updating smithi094:/etc/ceph/ceph.conf 2024-04-21T08:40:02.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T08:40:02.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-21T08:40:02.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:40:02.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:40:02.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:40:02.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1552073623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:02.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:02.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:02.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:02.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:01 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1067457797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.368+0000 7f979bac8640 1 -- 172.21.15.94:0/2422615295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9794105e30 msgr2=0x7f9794106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.368+0000 7f979bac8640 1 --2- 172.21.15.94:0/2422615295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9794105e30 0x7f9794106210 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9790009930 tx=0x7f979002bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.368+0000 7f979bac8640 1 -- 172.21.15.94:0/2422615295 shutdown_connections 2024-04-21T08:40:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.368+0000 7f979bac8640 1 --2- 172.21.15.94:0/2422615295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9794101b60 0x7f979410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.368+0000 7f979bac8640 1 --2- 172.21.15.94:0/2422615295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97941011a0 0x7f9794101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.368+0000 7f979bac8640 1 --2- 172.21.15.94:0/2422615295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9794105e30 0x7f9794106210 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.368+0000 7f979bac8640 1 -- 172.21.15.94:0/2422615295 >> 172.21.15.94:0/2422615295 conn(0x7f9794077e30 msgr2=0x7f9794078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.368+0000 7f979bac8640 1 -- 172.21.15.94:0/2422615295 shutdown_connections 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.368+0000 7f979bac8640 1 -- 172.21.15.94:0/2422615295 wait complete. 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.369+0000 7f979bac8640 1 Processor -- start 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.369+0000 7f979bac8640 1 -- start start 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.369+0000 7f979bac8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97941011a0 0x7f9794197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.369+0000 7f979bac8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794101b60 0x7f9794197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.369+0000 7f979bac8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9794105e30 0x7f979419bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.369+0000 7f979bac8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f97941110e0 con 0x7f97941011a0 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.369+0000 7f979bac8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9794110f60 con 0x7f9794101b60 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.369+0000 7f979bac8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9794111260 con 0x7f9794105e30 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f9799864640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97941011a0 0x7f9794197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f9799864640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97941011a0 0x7f9794197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58644/0 (socket says 172.21.15.94:58644) 2024-04-21T08:40:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f9799864640 1 -- 172.21.15.94:0/3638975160 learned_addr learned my addr 172.21.15.94:0/3638975160 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f9799063640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794101b60 0x7f9794197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f979a065640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9794105e30 0x7f979419bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f9799864640 1 -- 172.21.15.94:0/3638975160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9794105e30 msgr2=0x7f979419bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f9799864640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9794105e30 0x7f979419bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f9799864640 1 -- 172.21.15.94:0/3638975160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794101b60 msgr2=0x7f9794197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f9799864640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794101b60 0x7f9794197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f9799864640 1 -- 172.21.15.94:0/3638975160 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f979419c610 con 0x7f97941011a0 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f9799864640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97941011a0 0x7f9794197620 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9790009230 tx=0x7f9790004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f978affd640 1 -- 172.21.15.94:0/3638975160 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9790043020 con 0x7f97941011a0 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f979bac8640 1 -- 172.21.15.94:0/3638975160 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f979419c8a0 con 0x7f97941011a0 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f978affd640 1 -- 172.21.15.94:0/3638975160 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9790004350 con 0x7f97941011a0 2024-04-21T08:40:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f978affd640 1 -- 172.21.15.94:0/3638975160 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9790003b50 con 0x7f97941011a0 2024-04-21T08:40:02.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.370+0000 7f979bac8640 1 -- 172.21.15.94:0/3638975160 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97941a4190 con 0x7f97941011a0 2024-04-21T08:40:02.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.371+0000 7f979bac8640 1 -- 172.21.15.94:0/3638975160 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9794068de0 con 0x7f97941011a0 2024-04-21T08:40:02.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.372+0000 7f978affd640 1 -- 172.21.15.94:0/3638975160 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f9790046020 con 0x7f97941011a0 2024-04-21T08:40:02.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.373+0000 7f978affd640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f97800a3e40 0x7f97800a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.373+0000 7f9799063640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f97800a3e40 0x7f97800a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.373+0000 7f978affd640 1 -- 172.21.15.94:0/3638975160 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f97900f2f60 con 0x7f97941011a0 2024-04-21T08:40:02.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.374+0000 7f9799063640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f97800a3e40 0x7f97800a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97940fe5c0 tx=0x7f978400b390 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:02.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.378+0000 7f978affd640 1 -- 172.21.15.94:0/3638975160 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f979004d020 con 0x7f97941011a0 2024-04-21T08:40:02.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.619+0000 7f979bac8640 1 -- 172.21.15.94:0/3638975160 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9794109330 con 0x7f97941011a0 2024-04-21T08:40:02.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.619+0000 7f978affd640 1 -- 172.21.15.94:0/3638975160 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f97900bf0c0 con 0x7f97941011a0 2024-04-21T08:40:02.619 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:02.619 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:40:02.620 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.621+0000 7f979bac8640 1 -- 172.21.15.94:0/3638975160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f97800a3e40 msgr2=0x7f97800a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.621+0000 7f979bac8640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f97800a3e40 0x7f97800a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97940fe5c0 tx=0x7f978400b390 comp rx=0 tx=0).stop 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.621+0000 7f979bac8640 1 -- 172.21.15.94:0/3638975160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97941011a0 msgr2=0x7f9794197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.621+0000 7f979bac8640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97941011a0 0x7f9794197620 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9790009230 tx=0x7f9790004bb0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.622+0000 7f979bac8640 1 -- 172.21.15.94:0/3638975160 shutdown_connections 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.622+0000 7f979bac8640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f97800a3e40 0x7f97800a6300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.622+0000 7f979bac8640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9794105e30 0x7f979419bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.622+0000 7f979bac8640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794101b60 0x7f9794197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.622+0000 7f979bac8640 1 --2- 172.21.15.94:0/3638975160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97941011a0 0x7f9794197620 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.622+0000 7f979bac8640 1 -- 172.21.15.94:0/3638975160 >> 172.21.15.94:0/3638975160 conn(0x7f9794077e30 msgr2=0x7f97940fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.622+0000 7f979bac8640 1 -- 172.21.15.94:0/3638975160 shutdown_connections 2024-04-21T08:40:02.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.622+0000 7f979bac8640 1 -- 172.21.15.94:0/3638975160 wait complete. 2024-04-21T08:40:02.633 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-21T08:40:02.633 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-21T08:40:02.633 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:40:02.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.734+0000 7f375e481640 1 -- 172.21.15.94:0/768003403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3758069aa0 msgr2=0x7f375810edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.734+0000 7f375e481640 1 --2- 172.21.15.94:0/768003403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3758069aa0 0x7f375810edd0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3750009cd0 tx=0x7f375002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.734+0000 7f375e481640 1 -- 172.21.15.94:0/768003403 shutdown_connections 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.734+0000 7f375e481640 1 --2- 172.21.15.94:0/768003403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3758069aa0 0x7f375810edd0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.734+0000 7f375e481640 1 --2- 172.21.15.94:0/768003403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37580690e0 0x7f3758069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.734+0000 7f375e481640 1 --2- 172.21.15.94:0/768003403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3758108030 0x7f3758108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.734+0000 7f375e481640 1 -- 172.21.15.94:0/768003403 >> 172.21.15.94:0/768003403 conn(0x7f3758077e30 msgr2=0x7f3758078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.735+0000 7f375e481640 1 -- 172.21.15.94:0/768003403 shutdown_connections 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.735+0000 7f375e481640 1 -- 172.21.15.94:0/768003403 wait complete. 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.735+0000 7f375e481640 1 Processor -- start 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.735+0000 7f375e481640 1 -- start start 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.735+0000 7f375e481640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37580690e0 0x7f37581986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.735+0000 7f375e481640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3758069aa0 0x7f3758198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.735+0000 7f375e481640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3758108030 0x7f375819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.735+0000 7f375e481640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f37581121c0 con 0x7f3758108030 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.735+0000 7f375e481640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3758112040 con 0x7f37580690e0 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.735+0000 7f375e481640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3758112340 con 0x7f3758069aa0 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375ca1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3758108030 0x7f375819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375ca1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3758108030 0x7f375819cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58670/0 (socket says 172.21.15.94:58670) 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375ca1e640 1 -- 172.21.15.94:0/3925736579 learned_addr learned my addr 172.21.15.94:0/3925736579 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f37577fe640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3758069aa0 0x7f3758198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f3757fff640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37580690e0 0x7f37581986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375ca1e640 1 -- 172.21.15.94:0/3925736579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3758069aa0 msgr2=0x7f3758198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375ca1e640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3758069aa0 0x7f3758198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375ca1e640 1 -- 172.21.15.94:0/3925736579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37580690e0 msgr2=0x7f37581986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375ca1e640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37580690e0 0x7f37581986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375ca1e640 1 -- 172.21.15.94:0/3925736579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375819d6d0 con 0x7f3758108030 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f37577fe640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3758069aa0 0x7f3758198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f3757fff640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37580690e0 0x7f37581986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375ca1e640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3758108030 0x7f375819cfa0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3750033000 tx=0x7f37500346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f37557fa640 1 -- 172.21.15.94:0/3925736579 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3750041020 con 0x7f3758108030 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f37557fa640 1 -- 172.21.15.94:0/3925736579 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3750032970 con 0x7f3758108030 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f37557fa640 1 -- 172.21.15.94:0/3925736579 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3750039040 con 0x7f3758108030 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375e481640 1 -- 172.21.15.94:0/3925736579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375819d960 con 0x7f3758108030 2024-04-21T08:40:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.736+0000 7f375e481640 1 -- 172.21.15.94:0/3925736579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37581a5250 con 0x7f3758108030 2024-04-21T08:40:02.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.737+0000 7f375e481640 1 -- 172.21.15.94:0/3925736579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3758103640 con 0x7f3758108030 2024-04-21T08:40:02.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.738+0000 7f37557fa640 1 -- 172.21.15.94:0/3925736579 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f3750044020 con 0x7f3758108030 2024-04-21T08:40:02.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.739+0000 7f37557fa640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f37400a3e40 0x7f37400a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:02.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.739+0000 7f37557fa640 1 -- 172.21.15.94:0/3925736579 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f37500f2c80 con 0x7f3758108030 2024-04-21T08:40:02.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.739+0000 7f3757fff640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f37400a3e40 0x7f37400a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:02.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.739+0000 7f3757fff640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f37400a3e40 0x7f37400a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3748006fd0 tx=0x7f3748008040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:02.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.743+0000 7f37557fa640 1 -- 172.21.15.94:0/3925736579 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f375004b020 con 0x7f3758108030 2024-04-21T08:40:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.920+0000 7f375e481640 1 -- 172.21.15.94:0/3925736579 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3758062390 con 0x7f37400a3e40 2024-04-21T08:40:02.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.929+0000 7f37557fa640 1 -- 172.21.15.94:0/3925736579 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339690 (secure 0 0 0) 0x7f3758062390 con 0x7f37400a3e40 2024-04-21T08:40:02.933 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:02.935 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-21T08:40:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 -- 172.21.15.94:0/3925736579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f37400a3e40 msgr2=0x7f37400a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f37400a3e40 0x7f37400a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3748006fd0 tx=0x7f3748008040 comp rx=0 tx=0).stop 2024-04-21T08:40:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 -- 172.21.15.94:0/3925736579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3758108030 msgr2=0x7f375819cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3758108030 0x7f375819cfa0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3750033000 tx=0x7f37500346e0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 -- 172.21.15.94:0/3925736579 shutdown_connections 2024-04-21T08:40:02.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f37400a3e40 0x7f37400a6300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3758069aa0 0x7f3758198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3758108030 0x7f375819cfa0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 --2- 172.21.15.94:0/3925736579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37580690e0 0x7f37581986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 -- 172.21.15.94:0/3925736579 >> 172.21.15.94:0/3925736579 conn(0x7f3758077e30 msgr2=0x7f37580fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 -- 172.21.15.94:0/3925736579 shutdown_connections 2024-04-21T08:40:02.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:02.933+0000 7f375e481640 1 -- 172.21.15.94:0/3925736579 wait complete. 2024-04-21T08:40:02.944 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-21T08:40:01.915737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190718,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1000,"num_read_kb":664,"num_write":472,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378544,"kb_used_data":12704,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563280,"statfs":{"total":1151940427776,"available":1151552798720,"internally_reserved":0,"allocated":13008896,"data_stored":9883484,"data_compressed":539507,"data_compressed_allocated":729088,"data_compressed_original":4308992,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.986713"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318569+0000","last_change":"2024-04-21T08:37:31.448345+0000","last_active":"2024-04-21T08:39:58.318569+0000","last_peered":"2024-04-21T08:39:58.318569+0000","last_clean":"2024-04-21T08:39:58.318569+0000","last_became_active":"2024-04-21T08:37:31.447878+0000","last_became_peered":"2024-04-21T08:37:31.447878+0000","last_unstale":"2024-04-21T08:39:58.318569+0000","last_undegraded":"2024-04-21T08:39:58.318569+0000","last_fullsized":"2024-04-21T08:39:58.318569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:45.589165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319782+0000","last_change":"2024-04-21T08:37:31.452895+0000","last_active":"2024-04-21T08:39:58.319782+0000","last_peered":"2024-04-21T08:39:58.319782+0000","last_clean":"2024-04-21T08:39:58.319782+0000","last_became_active":"2024-04-21T08:37:31.452550+0000","last_became_peered":"2024-04-21T08:37:31.452550+0000","last_unstale":"2024-04-21T08:39:58.319782+0000","last_undegraded":"2024-04-21T08:39:58.319782+0000","last_fullsized":"2024-04-21T08:39:58.319782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:02.497802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319443+0000","last_change":"2024-04-21T08:37:31.447552+0000","last_active":"2024-04-21T08:39:58.319443+0000","last_peered":"2024-04-21T08:39:58.319443+0000","last_clean":"2024-04-21T08:39:58.319443+0000","last_became_active":"2024-04-21T08:37:31.446044+0000","last_became_peered":"2024-04-21T08:37:31.446044+0000","last_unstale":"2024-04-21T08:39:58.319443+0000","last_undegraded":"2024-04-21T08:39:58.319443+0000","last_fullsized":"2024-04-21T08:39:58.319443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:22.801977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151531+0000","last_change":"2024-04-21T08:37:31.452400+0000","last_active":"2024-04-21T08:39:58.151531+0000","last_peered":"2024-04-21T08:39:58.151531+0000","last_clean":"2024-04-21T08:39:58.151531+0000","last_became_active":"2024-04-21T08:37:31.451462+0000","last_became_peered":"2024-04-21T08:37:31.451462+0000","last_unstale":"2024-04-21T08:39:58.151531+0000","last_undegraded":"2024-04-21T08:39:58.151531+0000","last_fullsized":"2024-04-21T08:39:58.151531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:22.463481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009035+0000","last_change":"2024-04-21T08:37:31.448942+0000","last_active":"2024-04-21T08:39:58.009035+0000","last_peered":"2024-04-21T08:39:58.009035+0000","last_clean":"2024-04-21T08:39:58.009035+0000","last_became_active":"2024-04-21T08:37:31.448526+0000","last_became_peered":"2024-04-21T08:37:31.448526+0000","last_unstale":"2024-04-21T08:39:58.009035+0000","last_undegraded":"2024-04-21T08:39:58.009035+0000","last_fullsized":"2024-04-21T08:39:58.009035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:29.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017093+0000","last_change":"2024-04-21T08:37:31.452165+0000","last_active":"2024-04-21T08:39:58.017093+0000","last_peered":"2024-04-21T08:39:58.017093+0000","last_clean":"2024-04-21T08:39:58.017093+0000","last_became_active":"2024-04-21T08:37:31.451802+0000","last_became_peered":"2024-04-21T08:37:31.451802+0000","last_unstale":"2024-04-21T08:39:58.017093+0000","last_undegraded":"2024-04-21T08:39:58.017093+0000","last_fullsized":"2024-04-21T08:39:58.017093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:42.210544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904949+0000","last_change":"2024-04-21T08:39:31.681660+0000","last_active":"2024-04-21T08:39:57.904949+0000","last_peered":"2024-04-21T08:39:57.904949+0000","last_clean":"2024-04-21T08:39:57.904949+0000","last_became_active":"2024-04-21T08:39:31.681203+0000","last_became_peered":"2024-04-21T08:39:31.681203+0000","last_unstale":"2024-04-21T08:39:57.904949+0000","last_undegraded":"2024-04-21T08:39:57.904949+0000","last_fullsized":"2024-04-21T08:39:57.904949+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:54.120862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903472+0000","last_change":"2024-04-21T08:37:31.448257+0000","last_active":"2024-04-21T08:39:57.903472+0000","last_peered":"2024-04-21T08:39:57.903472+0000","last_clean":"2024-04-21T08:39:57.903472+0000","last_became_active":"2024-04-21T08:37:31.447845+0000","last_became_peered":"2024-04-21T08:37:31.447845+0000","last_unstale":"2024-04-21T08:39:57.903472+0000","last_undegraded":"2024-04-21T08:39:57.903472+0000","last_fullsized":"2024-04-21T08:39:57.903472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:19.956699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154785+0000","last_change":"2024-04-21T08:37:31.443022+0000","last_active":"2024-04-21T08:39:58.154785+0000","last_peered":"2024-04-21T08:39:58.154785+0000","last_clean":"2024-04-21T08:39:58.154785+0000","last_became_active":"2024-04-21T08:37:31.442645+0000","last_became_peered":"2024-04-21T08:37:31.442645+0000","last_unstale":"2024-04-21T08:39:58.154785+0000","last_undegraded":"2024-04-21T08:39:58.154785+0000","last_fullsized":"2024-04-21T08:39:58.154785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:18:10.861281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017434+0000","last_change":"2024-04-21T08:37:31.453427+0000","last_active":"2024-04-21T08:39:58.017434+0000","last_peered":"2024-04-21T08:39:58.017434+0000","last_clean":"2024-04-21T08:39:58.017434+0000","last_became_active":"2024-04-21T08:37:31.453096+0000","last_became_peered":"2024-04-21T08:37:31.453096+0000","last_unstale":"2024-04-21T08:39:58.017434+0000","last_undegraded":"2024-04-21T08:39:58.017434+0000","last_fullsized":"2024-04-21T08:39:58.017434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:49.450351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905929+0000","last_change":"2024-04-21T08:37:31.449908+0000","last_active":"2024-04-21T08:39:57.905929+0000","last_peered":"2024-04-21T08:39:57.905929+0000","last_clean":"2024-04-21T08:39:57.905929+0000","last_became_active":"2024-04-21T08:37:31.449558+0000","last_became_peered":"2024-04-21T08:37:31.449558+0000","last_unstale":"2024-04-21T08:39:57.905929+0000","last_undegraded":"2024-04-21T08:39:57.905929+0000","last_fullsized":"2024-04-21T08:39:57.905929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:21.538689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151539+0000","last_change":"2024-04-21T08:37:31.452973+0000","last_active":"2024-04-21T08:39:58.151539+0000","last_peered":"2024-04-21T08:39:58.151539+0000","last_clean":"2024-04-21T08:39:58.151539+0000","last_became_active":"2024-04-21T08:37:31.452586+0000","last_became_peered":"2024-04-21T08:37:31.452586+0000","last_unstale":"2024-04-21T08:39:58.151539+0000","last_undegraded":"2024-04-21T08:39:58.151539+0000","last_fullsized":"2024-04-21T08:39:58.151539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:09:11.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319144+0000","last_change":"2024-04-21T08:37:31.445801+0000","last_active":"2024-04-21T08:39:58.319144+0000","last_peered":"2024-04-21T08:39:58.319144+0000","last_clean":"2024-04-21T08:39:58.319144+0000","last_became_active":"2024-04-21T08:37:31.445478+0000","last_became_peered":"2024-04-21T08:37:31.445478+0000","last_unstale":"2024-04-21T08:39:58.319144+0000","last_undegraded":"2024-04-21T08:39:58.319144+0000","last_fullsized":"2024-04-21T08:39:58.319144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:33.352739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319201+0000","last_change":"2024-04-21T08:37:31.450735+0000","last_active":"2024-04-21T08:39:58.319201+0000","last_peered":"2024-04-21T08:39:58.319201+0000","last_clean":"2024-04-21T08:39:58.319201+0000","last_became_active":"2024-04-21T08:37:31.450384+0000","last_became_peered":"2024-04-21T08:37:31.450384+0000","last_unstale":"2024-04-21T08:39:58.319201+0000","last_undegraded":"2024-04-21T08:39:58.319201+0000","last_fullsized":"2024-04-21T08:39:58.319201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:45.950489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319354+0000","last_change":"2024-04-21T08:37:31.454690+0000","last_active":"2024-04-21T08:39:58.319354+0000","last_peered":"2024-04-21T08:39:58.319354+0000","last_clean":"2024-04-21T08:39:58.319354+0000","last_became_active":"2024-04-21T08:37:31.454330+0000","last_became_peered":"2024-04-21T08:37:31.454330+0000","last_unstale":"2024-04-21T08:39:58.319354+0000","last_undegraded":"2024-04-21T08:39:58.319354+0000","last_fullsized":"2024-04-21T08:39:58.319354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:52.644317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153871+0000","last_change":"2024-04-21T08:37:31.441389+0000","last_active":"2024-04-21T08:39:58.153871+0000","last_peered":"2024-04-21T08:39:58.153871+0000","last_clean":"2024-04-21T08:39:58.153871+0000","last_became_active":"2024-04-21T08:37:31.441065+0000","last_became_peered":"2024-04-21T08:37:31.441065+0000","last_unstale":"2024-04-21T08:39:58.153871+0000","last_undegraded":"2024-04-21T08:39:58.153871+0000","last_fullsized":"2024-04-21T08:39:58.153871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:56.528154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318575+0000","last_change":"2024-04-21T08:37:31.446722+0000","last_active":"2024-04-21T08:39:58.318575+0000","last_peered":"2024-04-21T08:39:58.318575+0000","last_clean":"2024-04-21T08:39:58.318575+0000","last_became_active":"2024-04-21T08:37:31.446396+0000","last_became_peered":"2024-04-21T08:37:31.446396+0000","last_unstale":"2024-04-21T08:39:58.318575+0000","last_undegraded":"2024-04-21T08:39:58.318575+0000","last_fullsized":"2024-04-21T08:39:58.318575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:17.488633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007275+0000","last_change":"2024-04-21T08:37:31.444388+0000","last_active":"2024-04-21T08:39:58.007275+0000","last_peered":"2024-04-21T08:39:58.007275+0000","last_clean":"2024-04-21T08:39:58.007275+0000","last_became_active":"2024-04-21T08:37:31.444023+0000","last_became_peered":"2024-04-21T08:37:31.444023+0000","last_unstale":"2024-04-21T08:39:58.007275+0000","last_undegraded":"2024-04-21T08:39:58.007275+0000","last_fullsized":"2024-04-21T08:39:58.007275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:07:46.238941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905278+0000","last_change":"2024-04-21T08:37:31.447850+0000","last_active":"2024-04-21T08:39:57.905278+0000","last_peered":"2024-04-21T08:39:57.905278+0000","last_clean":"2024-04-21T08:39:57.905278+0000","last_became_active":"2024-04-21T08:37:31.447505+0000","last_became_peered":"2024-04-21T08:37:31.447505+0000","last_unstale":"2024-04-21T08:39:57.905278+0000","last_undegraded":"2024-04-21T08:39:57.905278+0000","last_fullsized":"2024-04-21T08:39:57.905278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:55.399410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905032+0000","last_change":"2024-04-21T08:37:35.608747+0000","last_active":"2024-04-21T08:39:57.905032+0000","last_peered":"2024-04-21T08:39:57.905032+0000","last_clean":"2024-04-21T08:39:57.905032+0000","last_became_active":"2024-04-21T08:37:35.608157+0000","last_became_peered":"2024-04-21T08:37:35.608157+0000","last_unstale":"2024-04-21T08:39:57.905032+0000","last_undegraded":"2024-04-21T08:39:57.905032+0000","last_fullsized":"2024-04-21T08:39:57.905032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:53.384190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007037+0000","last_change":"2024-04-21T08:38:30.679843+0000","last_active":"2024-04-21T08:39:58.007037+0000","last_peered":"2024-04-21T08:39:58.007037+0000","last_clean":"2024-04-21T08:39:58.007037+0000","last_became_active":"2024-04-21T08:38:30.670964+0000","last_became_peered":"2024-04-21T08:38:30.670964+0000","last_unstale":"2024-04-21T08:39:58.007037+0000","last_undegraded":"2024-04-21T08:39:58.007037+0000","last_fullsized":"2024-04-21T08:39:58.007037+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:49.652571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904100+0000","last_change":"2024-04-21T08:37:31.451813+0000","last_active":"2024-04-21T08:39:57.904100+0000","last_peered":"2024-04-21T08:39:57.904100+0000","last_clean":"2024-04-21T08:39:57.904100+0000","last_became_active":"2024-04-21T08:37:31.451447+0000","last_became_peered":"2024-04-21T08:37:31.451447+0000","last_unstale":"2024-04-21T08:39:57.904100+0000","last_undegraded":"2024-04-21T08:39:57.904100+0000","last_fullsized":"2024-04-21T08:39:57.904100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:46:45.235326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151221+0000","last_change":"2024-04-21T08:37:35.603518+0000","last_active":"2024-04-21T08:39:58.151221+0000","last_peered":"2024-04-21T08:39:58.151221+0000","last_clean":"2024-04-21T08:39:58.151221+0000","last_became_active":"2024-04-21T08:37:35.603038+0000","last_became_peered":"2024-04-21T08:37:35.603038+0000","last_unstale":"2024-04-21T08:39:58.151221+0000","last_undegraded":"2024-04-21T08:39:58.151221+0000","last_fullsized":"2024-04-21T08:39:58.151221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:06:29.224025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904228+0000","last_change":"2024-04-21T08:37:30.424608+0000","last_active":"2024-04-21T08:39:57.904228+0000","last_peered":"2024-04-21T08:39:57.904228+0000","last_clean":"2024-04-21T08:39:57.904228+0000","last_became_active":"2024-04-21T08:37:30.424259+0000","last_became_peered":"2024-04-21T08:37:30.424259+0000","last_unstale":"2024-04-21T08:39:57.904228+0000","last_undegraded":"2024-04-21T08:39:57.904228+0000","last_fullsized":"2024-04-21T08:39:57.904228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:38.007095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319688+0000","last_change":"2024-04-21T08:37:31.444656+0000","last_active":"2024-04-21T08:39:58.319688+0000","last_peered":"2024-04-21T08:39:58.319688+0000","last_clean":"2024-04-21T08:39:58.319688+0000","last_became_active":"2024-04-21T08:37:31.444300+0000","last_became_peered":"2024-04-21T08:37:31.444300+0000","last_unstale":"2024-04-21T08:39:58.319688+0000","last_undegraded":"2024-04-21T08:39:58.319688+0000","last_fullsized":"2024-04-21T08:39:58.319688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:40.901674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151388+0000","last_change":"2024-04-21T08:37:35.611732+0000","last_active":"2024-04-21T08:39:58.151388+0000","last_peered":"2024-04-21T08:39:58.151388+0000","last_clean":"2024-04-21T08:39:58.151388+0000","last_became_active":"2024-04-21T08:37:35.611060+0000","last_became_peered":"2024-04-21T08:37:35.611060+0000","last_unstale":"2024-04-21T08:39:58.151388+0000","last_undegraded":"2024-04-21T08:39:58.151388+0000","last_fullsized":"2024-04-21T08:39:58.151388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:36:01.491949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318678+0000","last_change":"2024-04-21T08:38:30.689722+0000","last_active":"2024-04-21T08:39:58.318678+0000","last_peered":"2024-04-21T08:39:58.318678+0000","last_clean":"2024-04-21T08:39:58.318678+0000","last_became_active":"2024-04-21T08:38:30.676521+0000","last_became_peered":"2024-04-21T08:38:30.676521+0000","last_unstale":"2024-04-21T08:39:58.318678+0000","last_undegraded":"2024-04-21T08:39:58.318678+0000","last_fullsized":"2024-04-21T08:39:58.318678+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:24:15.799308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154321+0000","last_change":"2024-04-21T08:37:31.443896+0000","last_active":"2024-04-21T08:39:58.154321+0000","last_peered":"2024-04-21T08:39:58.154321+0000","last_clean":"2024-04-21T08:39:58.154321+0000","last_became_active":"2024-04-21T08:37:31.442684+0000","last_became_peered":"2024-04-21T08:37:31.442684+0000","last_unstale":"2024-04-21T08:39:58.154321+0000","last_undegraded":"2024-04-21T08:39:58.154321+0000","last_fullsized":"2024-04-21T08:39:58.154321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:14:18.649565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151231+0000","last_change":"2024-04-21T08:37:35.600468+0000","last_active":"2024-04-21T08:39:58.151231+0000","last_peered":"2024-04-21T08:39:58.151231+0000","last_clean":"2024-04-21T08:39:58.151231+0000","last_became_active":"2024-04-21T08:37:35.599940+0000","last_became_peered":"2024-04-21T08:37:35.599940+0000","last_unstale":"2024-04-21T08:39:58.151231+0000","last_undegraded":"2024-04-21T08:39:58.151231+0000","last_fullsized":"2024-04-21T08:39:58.151231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:48.125434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009395+0000","last_change":"2024-04-21T08:37:30.416480+0000","last_active":"2024-04-21T08:39:58.009395+0000","last_peered":"2024-04-21T08:39:58.009395+0000","last_clean":"2024-04-21T08:39:58.009395+0000","last_became_active":"2024-04-21T08:37:30.416072+0000","last_became_peered":"2024-04-21T08:37:30.416072+0000","last_unstale":"2024-04-21T08:39:58.009395+0000","last_undegraded":"2024-04-21T08:39:58.009395+0000","last_fullsized":"2024-04-21T08:39:58.009395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:28:22.400926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014147+0000","last_change":"2024-04-21T08:37:31.457216+0000","last_active":"2024-04-21T08:39:58.014147+0000","last_peered":"2024-04-21T08:39:58.014147+0000","last_clean":"2024-04-21T08:39:58.014147+0000","last_became_active":"2024-04-21T08:37:31.456856+0000","last_became_peered":"2024-04-21T08:37:31.456856+0000","last_unstale":"2024-04-21T08:39:58.014147+0000","last_undegraded":"2024-04-21T08:39:58.014147+0000","last_fullsized":"2024-04-21T08:39:58.014147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:10.272886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905178+0000","last_change":"2024-04-21T08:37:35.601715+0000","last_active":"2024-04-21T08:39:57.905178+0000","last_peered":"2024-04-21T08:39:57.905178+0000","last_clean":"2024-04-21T08:39:57.905178+0000","last_became_active":"2024-04-21T08:37:35.601269+0000","last_became_peered":"2024-04-21T08:37:35.601269+0000","last_unstale":"2024-04-21T08:39:57.905178+0000","last_undegraded":"2024-04-21T08:39:57.905178+0000","last_fullsized":"2024-04-21T08:39:57.905178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:57.192918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":164,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.890274+0000","last_change":"2024-04-21T08:37:30.415510+0000","last_active":"2024-04-21T08:39:58.890274+0000","last_peered":"2024-04-21T08:39:58.890274+0000","last_clean":"2024-04-21T08:39:58.890274+0000","last_became_active":"2024-04-21T08:37:30.415199+0000","last_became_peered":"2024-04-21T08:37:30.415199+0000","last_unstale":"2024-04-21T08:39:58.890274+0000","last_undegraded":"2024-04-21T08:39:58.890274+0000","last_fullsized":"2024-04-21T08:39:58.890274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:32.336247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009074+0000","last_change":"2024-04-21T08:39:31.681460+0000","last_active":"2024-04-21T08:39:58.009074+0000","last_peered":"2024-04-21T08:39:58.009074+0000","last_clean":"2024-04-21T08:39:58.009074+0000","last_became_active":"2024-04-21T08:39:31.681057+0000","last_became_peered":"2024-04-21T08:39:31.681057+0000","last_unstale":"2024-04-21T08:39:58.009074+0000","last_undegraded":"2024-04-21T08:39:58.009074+0000","last_fullsized":"2024-04-21T08:39:58.009074+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:43.407369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905226+0000","last_change":"2024-04-21T08:38:30.714455+0000","last_active":"2024-04-21T08:39:57.905226+0000","last_peered":"2024-04-21T08:39:57.905226+0000","last_clean":"2024-04-21T08:39:57.905226+0000","last_became_active":"2024-04-21T08:38:30.713910+0000","last_became_peered":"2024-04-21T08:38:30.713910+0000","last_unstale":"2024-04-21T08:39:57.905226+0000","last_undegraded":"2024-04-21T08:39:57.905226+0000","last_fullsized":"2024-04-21T08:39:57.905226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.448652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,10],"acting":[2,6,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017063+0000","last_change":"2024-04-21T08:37:30.419877+0000","last_active":"2024-04-21T08:39:58.017063+0000","last_peered":"2024-04-21T08:39:58.017063+0000","last_clean":"2024-04-21T08:39:58.017063+0000","last_became_active":"2024-04-21T08:37:30.419572+0000","last_became_peered":"2024-04-21T08:37:30.419572+0000","last_unstale":"2024-04-21T08:39:58.017063+0000","last_undegraded":"2024-04-21T08:39:58.017063+0000","last_fullsized":"2024-04-21T08:39:58.017063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:32:35.524089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014037+0000","last_change":"2024-04-21T08:37:31.458590+0000","last_active":"2024-04-21T08:39:58.014037+0000","last_peered":"2024-04-21T08:39:58.014037+0000","last_clean":"2024-04-21T08:39:58.014037+0000","last_became_active":"2024-04-21T08:37:31.458196+0000","last_became_peered":"2024-04-21T08:37:31.458196+0000","last_unstale":"2024-04-21T08:39:58.014037+0000","last_undegraded":"2024-04-21T08:39:58.014037+0000","last_fullsized":"2024-04-21T08:39:58.014037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:00.890859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318442+0000","last_change":"2024-04-21T08:38:30.803663+0000","last_active":"2024-04-21T08:39:58.318442+0000","last_peered":"2024-04-21T08:39:58.318442+0000","last_clean":"2024-04-21T08:39:58.318442+0000","last_became_active":"2024-04-21T08:38:30.803203+0000","last_became_peered":"2024-04-21T08:38:30.803203+0000","last_unstale":"2024-04-21T08:39:58.318442+0000","last_undegraded":"2024-04-21T08:39:58.318442+0000","last_fullsized":"2024-04-21T08:39:58.318442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:06.085380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318695+0000","last_change":"2024-04-21T08:37:30.415786+0000","last_active":"2024-04-21T08:39:58.318695+0000","last_peered":"2024-04-21T08:39:58.318695+0000","last_clean":"2024-04-21T08:39:58.318695+0000","last_became_active":"2024-04-21T08:37:30.415472+0000","last_became_peered":"2024-04-21T08:37:30.415472+0000","last_unstale":"2024-04-21T08:39:58.318695+0000","last_undegraded":"2024-04-21T08:39:58.318695+0000","last_fullsized":"2024-04-21T08:39:58.318695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.417132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903648+0000","last_change":"2024-04-21T08:37:31.455720+0000","last_active":"2024-04-21T08:39:57.903648+0000","last_peered":"2024-04-21T08:39:57.903648+0000","last_clean":"2024-04-21T08:39:57.903648+0000","last_became_active":"2024-04-21T08:37:31.455334+0000","last_became_peered":"2024-04-21T08:37:31.455334+0000","last_unstale":"2024-04-21T08:39:57.903648+0000","last_undegraded":"2024-04-21T08:39:57.903648+0000","last_fullsized":"2024-04-21T08:39:57.903648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:06:55.220656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319675+0000","last_change":"2024-04-21T08:37:35.603305+0000","last_active":"2024-04-21T08:39:58.319675+0000","last_peered":"2024-04-21T08:39:58.319675+0000","last_clean":"2024-04-21T08:39:58.319675+0000","last_became_active":"2024-04-21T08:37:35.602854+0000","last_became_peered":"2024-04-21T08:37:35.602854+0000","last_unstale":"2024-04-21T08:39:58.319675+0000","last_undegraded":"2024-04-21T08:39:58.319675+0000","last_fullsized":"2024-04-21T08:39:58.319675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:10.993864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013890+0000","last_change":"2024-04-21T08:37:30.420797+0000","last_active":"2024-04-21T08:39:58.013890+0000","last_peered":"2024-04-21T08:39:58.013890+0000","last_clean":"2024-04-21T08:39:58.013890+0000","last_became_active":"2024-04-21T08:37:30.420485+0000","last_became_peered":"2024-04-21T08:37:30.420485+0000","last_unstale":"2024-04-21T08:39:58.013890+0000","last_undegraded":"2024-04-21T08:39:58.013890+0000","last_fullsized":"2024-04-21T08:39:58.013890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:03.275677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006844+0000","last_change":"2024-04-21T08:37:31.448406+0000","last_active":"2024-04-21T08:39:58.006844+0000","last_peered":"2024-04-21T08:39:58.006844+0000","last_clean":"2024-04-21T08:39:58.006844+0000","last_became_active":"2024-04-21T08:37:31.448085+0000","last_became_peered":"2024-04-21T08:37:31.448085+0000","last_unstale":"2024-04-21T08:39:58.006844+0000","last_undegraded":"2024-04-21T08:39:58.006844+0000","last_fullsized":"2024-04-21T08:39:58.006844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:49.574064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903948+0000","last_change":"2024-04-21T08:37:35.601338+0000","last_active":"2024-04-21T08:39:57.903948+0000","last_peered":"2024-04-21T08:39:57.903948+0000","last_clean":"2024-04-21T08:39:57.903948+0000","last_became_active":"2024-04-21T08:37:35.600890+0000","last_became_peered":"2024-04-21T08:37:35.600890+0000","last_unstale":"2024-04-21T08:39:57.903948+0000","last_undegraded":"2024-04-21T08:39:57.903948+0000","last_fullsized":"2024-04-21T08:39:57.903948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:30:21.755850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.889748+0000","last_change":"2024-04-21T08:37:30.423320+0000","last_active":"2024-04-21T08:39:58.889748+0000","last_peered":"2024-04-21T08:39:58.889748+0000","last_clean":"2024-04-21T08:39:58.889748+0000","last_became_active":"2024-04-21T08:37:30.423020+0000","last_became_peered":"2024-04-21T08:37:30.423020+0000","last_unstale":"2024-04-21T08:39:58.889748+0000","last_undegraded":"2024-04-21T08:39:58.889748+0000","last_fullsized":"2024-04-21T08:39:58.889748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:08:27.313196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154669+0000","last_change":"2024-04-21T08:37:31.442425+0000","last_active":"2024-04-21T08:39:58.154669+0000","last_peered":"2024-04-21T08:39:58.154669+0000","last_clean":"2024-04-21T08:39:58.154669+0000","last_became_active":"2024-04-21T08:37:31.442028+0000","last_became_peered":"2024-04-21T08:37:31.442028+0000","last_unstale":"2024-04-21T08:39:58.154669+0000","last_undegraded":"2024-04-21T08:39:58.154669+0000","last_fullsized":"2024-04-21T08:39:58.154669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:45.498656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006697+0000","last_change":"2024-04-21T08:37:35.607829+0000","last_active":"2024-04-21T08:39:58.006697+0000","last_peered":"2024-04-21T08:39:58.006697+0000","last_clean":"2024-04-21T08:39:58.006697+0000","last_became_active":"2024-04-21T08:37:35.607150+0000","last_became_peered":"2024-04-21T08:37:35.607150+0000","last_unstale":"2024-04-21T08:39:58.006697+0000","last_undegraded":"2024-04-21T08:39:58.006697+0000","last_fullsized":"2024-04-21T08:39:58.006697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:01:34.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905384+0000","last_change":"2024-04-21T08:37:30.418613+0000","last_active":"2024-04-21T08:39:57.905384+0000","last_peered":"2024-04-21T08:39:57.905384+0000","last_clean":"2024-04-21T08:39:57.905384+0000","last_became_active":"2024-04-21T08:37:30.418300+0000","last_became_peered":"2024-04-21T08:37:30.418300+0000","last_unstale":"2024-04-21T08:39:57.905384+0000","last_undegraded":"2024-04-21T08:39:57.905384+0000","last_fullsized":"2024-04-21T08:39:57.905384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:17:35.370753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319739+0000","last_change":"2024-04-21T08:39:31.682628+0000","last_active":"2024-04-21T08:39:58.319739+0000","last_peered":"2024-04-21T08:39:58.319739+0000","last_clean":"2024-04-21T08:39:58.319739+0000","last_became_active":"2024-04-21T08:39:31.682383+0000","last_became_peered":"2024-04-21T08:39:31.682383+0000","last_unstale":"2024-04-21T08:39:58.319739+0000","last_undegraded":"2024-04-21T08:39:58.319739+0000","last_fullsized":"2024-04-21T08:39:58.319739+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:38.831288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151734+0000","last_change":"2024-04-21T08:37:35.608280+0000","last_active":"2024-04-21T08:39:58.151734+0000","last_peered":"2024-04-21T08:39:58.151734+0000","last_clean":"2024-04-21T08:39:58.151734+0000","last_became_active":"2024-04-21T08:37:35.607898+0000","last_became_peered":"2024-04-21T08:37:35.607898+0000","last_unstale":"2024-04-21T08:39:58.151734+0000","last_undegraded":"2024-04-21T08:39:58.151734+0000","last_fullsized":"2024-04-21T08:39:58.151734+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:55.689519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014140+0000","last_change":"2024-04-21T08:37:30.417352+0000","last_active":"2024-04-21T08:39:58.014140+0000","last_peered":"2024-04-21T08:39:58.014140+0000","last_clean":"2024-04-21T08:39:58.014140+0000","last_became_active":"2024-04-21T08:37:30.417000+0000","last_became_peered":"2024-04-21T08:37:30.417000+0000","last_unstale":"2024-04-21T08:39:58.014140+0000","last_undegraded":"2024-04-21T08:39:58.014140+0000","last_fullsized":"2024-04-21T08:39:58.014140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:56.087334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006804+0000","last_change":"2024-04-21T08:39:31.679434+0000","last_active":"2024-04-21T08:39:58.006804+0000","last_peered":"2024-04-21T08:39:58.006804+0000","last_clean":"2024-04-21T08:39:58.006804+0000","last_became_active":"2024-04-21T08:39:31.679041+0000","last_became_peered":"2024-04-21T08:39:31.679041+0000","last_unstale":"2024-04-21T08:39:58.006804+0000","last_undegraded":"2024-04-21T08:39:58.006804+0000","last_fullsized":"2024-04-21T08:39:58.006804+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:27:11.011449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006731+0000","last_change":"2024-04-21T08:37:35.601077+0000","last_active":"2024-04-21T08:39:58.006731+0000","last_peered":"2024-04-21T08:39:58.006731+0000","last_clean":"2024-04-21T08:39:58.006731+0000","last_became_active":"2024-04-21T08:37:35.600673+0000","last_became_peered":"2024-04-21T08:37:35.600673+0000","last_unstale":"2024-04-21T08:39:58.006731+0000","last_undegraded":"2024-04-21T08:39:58.006731+0000","last_fullsized":"2024-04-21T08:39:58.006731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:04.554208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'41","reported_seq":333,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.890961+0000","last_change":"2024-04-21T08:37:30.421440+0000","last_active":"2024-04-21T08:39:58.890961+0000","last_peered":"2024-04-21T08:39:58.890961+0000","last_clean":"2024-04-21T08:39:58.890961+0000","last_became_active":"2024-04-21T08:37:30.420988+0000","last_became_peered":"2024-04-21T08:37:30.420988+0000","last_unstale":"2024-04-21T08:39:58.890961+0000","last_undegraded":"2024-04-21T08:39:58.890961+0000","last_fullsized":"2024-04-21T08:39:58.890961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:38:03.850705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":258,"num_read_kb":198,"num_write":42,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009331+0000","last_change":"2024-04-21T08:37:31.447208+0000","last_active":"2024-04-21T08:39:58.009331+0000","last_peered":"2024-04-21T08:39:58.009331+0000","last_clean":"2024-04-21T08:39:58.009331+0000","last_became_active":"2024-04-21T08:37:31.444464+0000","last_became_peered":"2024-04-21T08:37:31.444464+0000","last_unstale":"2024-04-21T08:39:58.009331+0000","last_undegraded":"2024-04-21T08:39:58.009331+0000","last_fullsized":"2024-04-21T08:39:58.009331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:33.805676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014409+0000","last_change":"2024-04-21T08:37:35.603937+0000","last_active":"2024-04-21T08:39:58.014409+0000","last_peered":"2024-04-21T08:39:58.014409+0000","last_clean":"2024-04-21T08:39:58.014409+0000","last_became_active":"2024-04-21T08:37:35.603477+0000","last_became_peered":"2024-04-21T08:37:35.603477+0000","last_unstale":"2024-04-21T08:39:58.014409+0000","last_undegraded":"2024-04-21T08:39:58.014409+0000","last_fullsized":"2024-04-21T08:39:58.014409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:40:33.901570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905391+0000","last_change":"2024-04-21T08:37:30.416941+0000","last_active":"2024-04-21T08:39:57.905391+0000","last_peered":"2024-04-21T08:39:57.905391+0000","last_clean":"2024-04-21T08:39:57.905391+0000","last_became_active":"2024-04-21T08:37:30.416625+0000","last_became_peered":"2024-04-21T08:37:30.416625+0000","last_unstale":"2024-04-21T08:39:57.905391+0000","last_undegraded":"2024-04-21T08:39:57.905391+0000","last_fullsized":"2024-04-21T08:39:57.905391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:07.782779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318733+0000","last_change":"2024-04-21T08:37:31.441707+0000","last_active":"2024-04-21T08:39:58.318733+0000","last_peered":"2024-04-21T08:39:58.318733+0000","last_clean":"2024-04-21T08:39:58.318733+0000","last_became_active":"2024-04-21T08:37:31.441366+0000","last_became_peered":"2024-04-21T08:37:31.441366+0000","last_unstale":"2024-04-21T08:39:58.318733+0000","last_undegraded":"2024-04-21T08:39:58.318733+0000","last_fullsized":"2024-04-21T08:39:58.318733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:16.573804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318906+0000","last_change":"2024-04-21T08:37:35.596214+0000","last_active":"2024-04-21T08:39:58.318906+0000","last_peered":"2024-04-21T08:39:58.318906+0000","last_clean":"2024-04-21T08:39:58.318906+0000","last_became_active":"2024-04-21T08:37:35.595773+0000","last_became_peered":"2024-04-21T08:37:35.595773+0000","last_unstale":"2024-04-21T08:39:58.318906+0000","last_undegraded":"2024-04-21T08:39:58.318906+0000","last_fullsized":"2024-04-21T08:39:58.318906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:07.538190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318273+0000","last_change":"2024-04-21T08:38:30.684828+0000","last_active":"2024-04-21T08:39:58.318273+0000","last_peered":"2024-04-21T08:39:58.318273+0000","last_clean":"2024-04-21T08:39:58.318273+0000","last_became_active":"2024-04-21T08:38:30.673923+0000","last_became_peered":"2024-04-21T08:38:30.673923+0000","last_unstale":"2024-04-21T08:39:58.318273+0000","last_undegraded":"2024-04-21T08:39:58.318273+0000","last_fullsized":"2024-04-21T08:39:58.318273+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:51:37.906150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905472+0000","last_change":"2024-04-21T08:37:31.453680+0000","last_active":"2024-04-21T08:39:57.905472+0000","last_peered":"2024-04-21T08:39:57.905472+0000","last_clean":"2024-04-21T08:39:57.905472+0000","last_became_active":"2024-04-21T08:37:31.453319+0000","last_became_peered":"2024-04-21T08:37:31.453319+0000","last_unstale":"2024-04-21T08:39:57.905472+0000","last_undegraded":"2024-04-21T08:39:57.905472+0000","last_fullsized":"2024-04-21T08:39:57.905472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:43.139789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006992+0000","last_change":"2024-04-21T08:37:35.608906+0000","last_active":"2024-04-21T08:39:58.006992+0000","last_peered":"2024-04-21T08:39:58.006992+0000","last_clean":"2024-04-21T08:39:58.006992+0000","last_became_active":"2024-04-21T08:37:35.608535+0000","last_became_peered":"2024-04-21T08:37:35.608535+0000","last_unstale":"2024-04-21T08:39:58.006992+0000","last_undegraded":"2024-04-21T08:39:58.006992+0000","last_fullsized":"2024-04-21T08:39:58.006992+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:58:01.422308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":171,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.890194+0000","last_change":"2024-04-21T08:37:30.414865+0000","last_active":"2024-04-21T08:39:58.890194+0000","last_peered":"2024-04-21T08:39:58.890194+0000","last_clean":"2024-04-21T08:39:58.890194+0000","last_became_active":"2024-04-21T08:37:30.414471+0000","last_became_peered":"2024-04-21T08:37:30.414471+0000","last_unstale":"2024-04-21T08:39:58.890194+0000","last_undegraded":"2024-04-21T08:39:58.890194+0000","last_fullsized":"2024-04-21T08:39:58.890194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:46.943574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905326+0000","last_change":"2024-04-21T08:37:31.456634+0000","last_active":"2024-04-21T08:39:57.905326+0000","last_peered":"2024-04-21T08:39:57.905326+0000","last_clean":"2024-04-21T08:39:57.905326+0000","last_became_active":"2024-04-21T08:37:31.456278+0000","last_became_peered":"2024-04-21T08:37:31.456278+0000","last_unstale":"2024-04-21T08:39:57.905326+0000","last_undegraded":"2024-04-21T08:39:57.905326+0000","last_fullsized":"2024-04-21T08:39:57.905326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:04.870024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319167+0000","last_change":"2024-04-21T08:37:35.601760+0000","last_active":"2024-04-21T08:39:58.319167+0000","last_peered":"2024-04-21T08:39:58.319167+0000","last_clean":"2024-04-21T08:39:58.319167+0000","last_became_active":"2024-04-21T08:37:35.601332+0000","last_became_peered":"2024-04-21T08:37:35.601332+0000","last_unstale":"2024-04-21T08:39:58.319167+0000","last_undegraded":"2024-04-21T08:39:58.319167+0000","last_fullsized":"2024-04-21T08:39:58.319167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:04:03.694581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008961+0000","last_change":"2024-04-21T08:37:30.422419+0000","last_active":"2024-04-21T08:39:58.008961+0000","last_peered":"2024-04-21T08:39:58.008961+0000","last_clean":"2024-04-21T08:39:58.008961+0000","last_became_active":"2024-04-21T08:37:30.422123+0000","last_became_peered":"2024-04-21T08:37:30.422123+0000","last_unstale":"2024-04-21T08:39:58.008961+0000","last_undegraded":"2024-04-21T08:39:58.008961+0000","last_fullsized":"2024-04-21T08:39:58.008961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:32:54.029041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013848+0000","last_change":"2024-04-21T08:37:31.458992+0000","last_active":"2024-04-21T08:39:58.013848+0000","last_peered":"2024-04-21T08:39:58.013848+0000","last_clean":"2024-04-21T08:39:58.013848+0000","last_became_active":"2024-04-21T08:37:31.458635+0000","last_became_peered":"2024-04-21T08:37:31.458635+0000","last_unstale":"2024-04-21T08:39:58.013848+0000","last_undegraded":"2024-04-21T08:39:58.013848+0000","last_fullsized":"2024-04-21T08:39:58.013848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:48:14.325571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013750+0000","last_change":"2024-04-21T08:37:35.603739+0000","last_active":"2024-04-21T08:39:58.013750+0000","last_peered":"2024-04-21T08:39:58.013750+0000","last_clean":"2024-04-21T08:39:58.013750+0000","last_became_active":"2024-04-21T08:37:35.603310+0000","last_became_peered":"2024-04-21T08:37:35.603310+0000","last_unstale":"2024-04-21T08:39:58.013750+0000","last_undegraded":"2024-04-21T08:39:58.013750+0000","last_fullsized":"2024-04-21T08:39:58.013750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:19.162520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009465+0000","last_change":"2024-04-21T08:37:30.418104+0000","last_active":"2024-04-21T08:39:58.009465+0000","last_peered":"2024-04-21T08:39:58.009465+0000","last_clean":"2024-04-21T08:39:58.009465+0000","last_became_active":"2024-04-21T08:37:30.417814+0000","last_became_peered":"2024-04-21T08:37:30.417814+0000","last_unstale":"2024-04-21T08:39:58.009465+0000","last_undegraded":"2024-04-21T08:39:58.009465+0000","last_fullsized":"2024-04-21T08:39:58.009465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:50:56.696513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318582+0000","last_change":"2024-04-21T08:37:31.451952+0000","last_active":"2024-04-21T08:39:58.318582+0000","last_peered":"2024-04-21T08:39:58.318582+0000","last_clean":"2024-04-21T08:39:58.318582+0000","last_became_active":"2024-04-21T08:37:31.451588+0000","last_became_peered":"2024-04-21T08:37:31.451588+0000","last_unstale":"2024-04-21T08:39:58.318582+0000","last_undegraded":"2024-04-21T08:39:58.318582+0000","last_fullsized":"2024-04-21T08:39:58.318582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:06.506769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009071+0000","last_change":"2024-04-21T08:37:35.760661+0000","last_active":"2024-04-21T08:39:58.009071+0000","last_peered":"2024-04-21T08:39:58.009071+0000","last_clean":"2024-04-21T08:39:58.009071+0000","last_became_active":"2024-04-21T08:37:35.759925+0000","last_became_peered":"2024-04-21T08:37:35.759925+0000","last_unstale":"2024-04-21T08:39:58.009071+0000","last_undegraded":"2024-04-21T08:39:58.009071+0000","last_fullsized":"2024-04-21T08:39:58.009071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:11:53.062251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007012+0000","last_change":"2024-04-21T08:37:30.416487+0000","last_active":"2024-04-21T08:39:58.007012+0000","last_peered":"2024-04-21T08:39:58.007012+0000","last_clean":"2024-04-21T08:39:58.007012+0000","last_became_active":"2024-04-21T08:37:30.416076+0000","last_became_peered":"2024-04-21T08:37:30.416076+0000","last_unstale":"2024-04-21T08:39:58.007012+0000","last_undegraded":"2024-04-21T08:39:58.007012+0000","last_fullsized":"2024-04-21T08:39:58.007012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:18:33.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017205+0000","last_change":"2024-04-21T08:37:31.448824+0000","last_active":"2024-04-21T08:39:58.017205+0000","last_peered":"2024-04-21T08:39:58.017205+0000","last_clean":"2024-04-21T08:39:58.017205+0000","last_became_active":"2024-04-21T08:37:31.448432+0000","last_became_peered":"2024-04-21T08:37:31.448432+0000","last_unstale":"2024-04-21T08:39:58.017205+0000","last_undegraded":"2024-04-21T08:39:58.017205+0000","last_fullsized":"2024-04-21T08:39:58.017205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:41.111272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318815+0000","last_change":"2024-04-21T08:37:35.604539+0000","last_active":"2024-04-21T08:39:58.318815+0000","last_peered":"2024-04-21T08:39:58.318815+0000","last_clean":"2024-04-21T08:39:58.318815+0000","last_became_active":"2024-04-21T08:37:35.604132+0000","last_became_peered":"2024-04-21T08:37:35.604132+0000","last_unstale":"2024-04-21T08:39:58.318815+0000","last_undegraded":"2024-04-21T08:39:58.318815+0000","last_fullsized":"2024-04-21T08:39:58.318815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:12.180730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905085+0000","last_change":"2024-04-21T08:37:30.418854+0000","last_active":"2024-04-21T08:39:57.905085+0000","last_peered":"2024-04-21T08:39:57.905085+0000","last_clean":"2024-04-21T08:39:57.905085+0000","last_became_active":"2024-04-21T08:37:30.418511+0000","last_became_peered":"2024-04-21T08:37:30.418511+0000","last_unstale":"2024-04-21T08:39:57.905085+0000","last_undegraded":"2024-04-21T08:39:57.905085+0000","last_fullsized":"2024-04-21T08:39:57.905085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:26.751924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.016918+0000","last_change":"2024-04-21T08:37:31.453365+0000","last_active":"2024-04-21T08:39:58.016918+0000","last_peered":"2024-04-21T08:39:58.016918+0000","last_clean":"2024-04-21T08:39:58.016918+0000","last_became_active":"2024-04-21T08:37:31.453042+0000","last_became_peered":"2024-04-21T08:37:31.453042+0000","last_unstale":"2024-04-21T08:39:58.016918+0000","last_undegraded":"2024-04-21T08:39:58.016918+0000","last_fullsized":"2024-04-21T08:39:58.016918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:49:46.648754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318806+0000","last_change":"2024-04-21T08:37:35.760626+0000","last_active":"2024-04-21T08:39:58.318806+0000","last_peered":"2024-04-21T08:39:58.318806+0000","last_clean":"2024-04-21T08:39:58.318806+0000","last_became_active":"2024-04-21T08:37:35.760125+0000","last_became_peered":"2024-04-21T08:37:35.760125+0000","last_unstale":"2024-04-21T08:39:58.318806+0000","last_undegraded":"2024-04-21T08:39:58.318806+0000","last_fullsized":"2024-04-21T08:39:58.318806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:58.366557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013932+0000","last_change":"2024-04-21T08:37:31.457654+0000","last_active":"2024-04-21T08:39:58.013932+0000","last_peered":"2024-04-21T08:39:58.013932+0000","last_clean":"2024-04-21T08:39:58.013932+0000","last_became_active":"2024-04-21T08:37:31.456950+0000","last_became_peered":"2024-04-21T08:37:31.456950+0000","last_unstale":"2024-04-21T08:39:58.013932+0000","last_undegraded":"2024-04-21T08:39:58.013932+0000","last_fullsized":"2024-04-21T08:39:58.013932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:02.863000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318817+0000","last_change":"2024-04-21T08:37:30.422021+0000","last_active":"2024-04-21T08:39:58.318817+0000","last_peered":"2024-04-21T08:39:58.318817+0000","last_clean":"2024-04-21T08:39:58.318817+0000","last_became_active":"2024-04-21T08:37:30.421711+0000","last_became_peered":"2024-04-21T08:37:30.421711+0000","last_unstale":"2024-04-21T08:39:58.318817+0000","last_undegraded":"2024-04-21T08:39:58.318817+0000","last_fullsized":"2024-04-21T08:39:58.318817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:01.612205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905863+0000","last_change":"2024-04-21T08:37:35.598041+0000","last_active":"2024-04-21T08:39:57.905863+0000","last_peered":"2024-04-21T08:39:57.905863+0000","last_clean":"2024-04-21T08:39:57.905863+0000","last_became_active":"2024-04-21T08:37:35.597635+0000","last_became_peered":"2024-04-21T08:37:35.597635+0000","last_unstale":"2024-04-21T08:39:57.905863+0000","last_undegraded":"2024-04-21T08:39:57.905863+0000","last_fullsized":"2024-04-21T08:39:57.905863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:35:27.452340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903604+0000","last_change":"2024-04-21T08:37:31.450423+0000","last_active":"2024-04-21T08:39:57.903604+0000","last_peered":"2024-04-21T08:39:57.903604+0000","last_clean":"2024-04-21T08:39:57.903604+0000","last_became_active":"2024-04-21T08:37:31.450112+0000","last_became_peered":"2024-04-21T08:37:31.450112+0000","last_unstale":"2024-04-21T08:39:57.903604+0000","last_undegraded":"2024-04-21T08:39:57.903604+0000","last_fullsized":"2024-04-21T08:39:57.903604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:10:47.006769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151568+0000","last_change":"2024-04-21T08:37:30.416519+0000","last_active":"2024-04-21T08:39:58.151568+0000","last_peered":"2024-04-21T08:39:58.151568+0000","last_clean":"2024-04-21T08:39:58.151568+0000","last_became_active":"2024-04-21T08:37:30.416131+0000","last_became_peered":"2024-04-21T08:37:30.416131+0000","last_unstale":"2024-04-21T08:39:58.151568+0000","last_undegraded":"2024-04-21T08:39:58.151568+0000","last_fullsized":"2024-04-21T08:39:58.151568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:12.280847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":368,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904918+0000","last_change":"2024-04-21T08:36:20.257759+0000","last_active":"2024-04-21T08:39:57.904918+0000","last_peered":"2024-04-21T08:39:57.904918+0000","last_clean":"2024-04-21T08:39:57.904918+0000","last_became_active":"2024-04-21T08:36:20.257313+0000","last_became_peered":"2024-04-21T08:36:20.257313+0000","last_unstale":"2024-04-21T08:39:57.904918+0000","last_undegraded":"2024-04-21T08:39:57.904918+0000","last_fullsized":"2024-04-21T08:39:57.904918+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_clean_scrub_stamp":"2024-04-21T08:32:56.546295+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:42:44.393447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014207+0000","last_change":"2024-04-21T08:37:35.761562+0000","last_active":"2024-04-21T08:39:58.014207+0000","last_peered":"2024-04-21T08:39:58.014207+0000","last_clean":"2024-04-21T08:39:58.014207+0000","last_became_active":"2024-04-21T08:37:35.760703+0000","last_became_peered":"2024-04-21T08:37:35.760703+0000","last_unstale":"2024-04-21T08:39:58.014207+0000","last_undegraded":"2024-04-21T08:39:58.014207+0000","last_fullsized":"2024-04-21T08:39:58.014207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:12:47.511117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318506+0000","last_change":"2024-04-21T08:39:31.678868+0000","last_active":"2024-04-21T08:39:58.318506+0000","last_peered":"2024-04-21T08:39:58.318506+0000","last_clean":"2024-04-21T08:39:58.318506+0000","last_became_active":"2024-04-21T08:39:31.678457+0000","last_became_peered":"2024-04-21T08:39:31.678457+0000","last_unstale":"2024-04-21T08:39:58.318506+0000","last_undegraded":"2024-04-21T08:39:58.318506+0000","last_fullsized":"2024-04-21T08:39:58.318506+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:49.918106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905042+0000","last_change":"2024-04-21T08:37:30.420560+0000","last_active":"2024-04-21T08:39:57.905042+0000","last_peered":"2024-04-21T08:39:57.905042+0000","last_clean":"2024-04-21T08:39:57.905042+0000","last_became_active":"2024-04-21T08:37:30.420223+0000","last_became_peered":"2024-04-21T08:37:30.420223+0000","last_unstale":"2024-04-21T08:39:57.905042+0000","last_undegraded":"2024-04-21T08:39:57.905042+0000","last_fullsized":"2024-04-21T08:39:57.905042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:43:55.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318381+0000","last_change":"2024-04-21T08:37:35.598294+0000","last_active":"2024-04-21T08:39:58.318381+0000","last_peered":"2024-04-21T08:39:58.318381+0000","last_clean":"2024-04-21T08:39:58.318381+0000","last_became_active":"2024-04-21T08:37:35.597883+0000","last_became_peered":"2024-04-21T08:37:35.597883+0000","last_unstale":"2024-04-21T08:39:58.318381+0000","last_undegraded":"2024-04-21T08:39:58.318381+0000","last_fullsized":"2024-04-21T08:39:58.318381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:24:05.144763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009321+0000","last_change":"2024-04-21T08:37:30.423182+0000","last_active":"2024-04-21T08:39:58.009321+0000","last_peered":"2024-04-21T08:39:58.009321+0000","last_clean":"2024-04-21T08:39:58.009321+0000","last_became_active":"2024-04-21T08:37:30.422792+0000","last_became_peered":"2024-04-21T08:37:30.422792+0000","last_unstale":"2024-04-21T08:39:58.009321+0000","last_undegraded":"2024-04-21T08:39:58.009321+0000","last_fullsized":"2024-04-21T08:39:58.009321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:44:00.727269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318924+0000","last_change":"2024-04-21T08:37:31.444013+0000","last_active":"2024-04-21T08:39:58.318924+0000","last_peered":"2024-04-21T08:39:58.318924+0000","last_clean":"2024-04-21T08:39:58.318924+0000","last_became_active":"2024-04-21T08:37:31.443684+0000","last_became_peered":"2024-04-21T08:37:31.443684+0000","last_unstale":"2024-04-21T08:39:58.318924+0000","last_undegraded":"2024-04-21T08:39:58.318924+0000","last_fullsized":"2024-04-21T08:39:58.318924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:00:35.400818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903525+0000","last_change":"2024-04-21T08:37:35.759817+0000","last_active":"2024-04-21T08:39:57.903525+0000","last_peered":"2024-04-21T08:39:57.903525+0000","last_clean":"2024-04-21T08:39:57.903525+0000","last_became_active":"2024-04-21T08:37:35.759396+0000","last_became_peered":"2024-04-21T08:37:35.759396+0000","last_unstale":"2024-04-21T08:39:57.903525+0000","last_undegraded":"2024-04-21T08:39:57.903525+0000","last_fullsized":"2024-04-21T08:39:57.903525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:20:36.170880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'44","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.114287+0000","last_change":"2024-04-21T08:37:30.414102+0000","last_active":"2024-04-21T08:39:58.114287+0000","last_peered":"2024-04-21T08:39:58.114287+0000","last_clean":"2024-04-21T08:39:58.114287+0000","last_became_active":"2024-04-21T08:37:30.413670+0000","last_became_peered":"2024-04-21T08:37:30.413670+0000","last_unstale":"2024-04-21T08:39:58.114287+0000","last_undegraded":"2024-04-21T08:39:58.114287+0000","last_fullsized":"2024-04-21T08:39:58.114287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:12:49.696516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904320+0000","last_change":"2024-04-21T08:37:31.457451+0000","last_active":"2024-04-21T08:39:57.904320+0000","last_peered":"2024-04-21T08:39:57.904320+0000","last_clean":"2024-04-21T08:39:57.904320+0000","last_became_active":"2024-04-21T08:37:31.457120+0000","last_became_peered":"2024-04-21T08:37:31.457120+0000","last_unstale":"2024-04-21T08:39:57.904320+0000","last_undegraded":"2024-04-21T08:39:57.904320+0000","last_fullsized":"2024-04-21T08:39:57.904320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:07:48.025843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904850+0000","last_change":"2024-04-21T08:37:35.600815+0000","last_active":"2024-04-21T08:39:57.904850+0000","last_peered":"2024-04-21T08:39:57.904850+0000","last_clean":"2024-04-21T08:39:57.904850+0000","last_became_active":"2024-04-21T08:37:35.600380+0000","last_became_peered":"2024-04-21T08:37:35.600380+0000","last_unstale":"2024-04-21T08:39:57.904850+0000","last_undegraded":"2024-04-21T08:39:57.904850+0000","last_fullsized":"2024-04-21T08:39:57.904850+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:19:33.710431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151729+0000","last_change":"2024-04-21T08:37:30.425592+0000","last_active":"2024-04-21T08:39:58.151729+0000","last_peered":"2024-04-21T08:39:58.151729+0000","last_clean":"2024-04-21T08:39:58.151729+0000","last_became_active":"2024-04-21T08:37:30.425274+0000","last_became_peered":"2024-04-21T08:37:30.425274+0000","last_unstale":"2024-04-21T08:39:58.151729+0000","last_undegraded":"2024-04-21T08:39:58.151729+0000","last_fullsized":"2024-04-21T08:39:58.151729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:27:29.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319672+0000","last_change":"2024-04-21T08:39:31.680028+0000","last_active":"2024-04-21T08:39:58.319672+0000","last_peered":"2024-04-21T08:39:58.319672+0000","last_clean":"2024-04-21T08:39:58.319672+0000","last_became_active":"2024-04-21T08:39:31.679162+0000","last_became_peered":"2024-04-21T08:39:31.679162+0000","last_unstale":"2024-04-21T08:39:58.319672+0000","last_undegraded":"2024-04-21T08:39:58.319672+0000","last_fullsized":"2024-04-21T08:39:58.319672+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:32.824411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151564+0000","last_change":"2024-04-21T08:38:30.713521+0000","last_active":"2024-04-21T08:39:58.151564+0000","last_peered":"2024-04-21T08:39:58.151564+0000","last_clean":"2024-04-21T08:39:58.151564+0000","last_became_active":"2024-04-21T08:38:30.713096+0000","last_became_peered":"2024-04-21T08:38:30.713096+0000","last_unstale":"2024-04-21T08:39:58.151564+0000","last_undegraded":"2024-04-21T08:39:58.151564+0000","last_fullsized":"2024-04-21T08:39:58.151564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:54:59.078776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153818+0000","last_change":"2024-04-21T08:37:30.416696+0000","last_active":"2024-04-21T08:39:58.153818+0000","last_peered":"2024-04-21T08:39:58.153818+0000","last_clean":"2024-04-21T08:39:58.153818+0000","last_became_active":"2024-04-21T08:37:30.415868+0000","last_became_peered":"2024-04-21T08:37:30.415868+0000","last_unstale":"2024-04-21T08:39:58.153818+0000","last_undegraded":"2024-04-21T08:39:58.153818+0000","last_fullsized":"2024-04-21T08:39:58.153818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:48:08.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008945+0000","last_change":"2024-04-21T08:37:31.453861+0000","last_active":"2024-04-21T08:39:58.008945+0000","last_peered":"2024-04-21T08:39:58.008945+0000","last_clean":"2024-04-21T08:39:58.008945+0000","last_became_active":"2024-04-21T08:37:31.453562+0000","last_became_peered":"2024-04-21T08:37:31.453562+0000","last_unstale":"2024-04-21T08:39:58.008945+0000","last_undegraded":"2024-04-21T08:39:58.008945+0000","last_fullsized":"2024-04-21T08:39:58.008945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:17:50.513917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006977+0000","last_change":"2024-04-21T08:37:35.609217+0000","last_active":"2024-04-21T08:39:58.006977+0000","last_peered":"2024-04-21T08:39:58.006977+0000","last_clean":"2024-04-21T08:39:58.006977+0000","last_became_active":"2024-04-21T08:37:35.608786+0000","last_became_peered":"2024-04-21T08:37:35.608786+0000","last_unstale":"2024-04-21T08:39:58.006977+0000","last_undegraded":"2024-04-21T08:39:58.006977+0000","last_fullsized":"2024-04-21T08:39:58.006977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:45:50.945845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903818+0000","last_change":"2024-04-21T08:37:30.417156+0000","last_active":"2024-04-21T08:39:57.903818+0000","last_peered":"2024-04-21T08:39:57.903818+0000","last_clean":"2024-04-21T08:39:57.903818+0000","last_became_active":"2024-04-21T08:37:30.415717+0000","last_became_peered":"2024-04-21T08:37:30.415717+0000","last_unstale":"2024-04-21T08:39:57.903818+0000","last_undegraded":"2024-04-21T08:39:57.903818+0000","last_fullsized":"2024-04-21T08:39:57.903818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:37.819082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318260+0000","last_change":"2024-04-21T08:37:31.444948+0000","last_active":"2024-04-21T08:39:58.318260+0000","last_peered":"2024-04-21T08:39:58.318260+0000","last_clean":"2024-04-21T08:39:58.318260+0000","last_became_active":"2024-04-21T08:37:31.444604+0000","last_became_peered":"2024-04-21T08:37:31.444604+0000","last_unstale":"2024-04-21T08:39:58.318260+0000","last_undegraded":"2024-04-21T08:39:58.318260+0000","last_fullsized":"2024-04-21T08:39:58.318260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.820503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151386+0000","last_change":"2024-04-21T08:37:35.611541+0000","last_active":"2024-04-21T08:39:58.151386+0000","last_peered":"2024-04-21T08:39:58.151386+0000","last_clean":"2024-04-21T08:39:58.151386+0000","last_became_active":"2024-04-21T08:37:35.611136+0000","last_became_peered":"2024-04-21T08:37:35.611136+0000","last_unstale":"2024-04-21T08:39:58.151386+0000","last_undegraded":"2024-04-21T08:39:58.151386+0000","last_fullsized":"2024-04-21T08:39:58.151386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:28.254529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904963+0000","last_change":"2024-04-21T08:37:30.414216+0000","last_active":"2024-04-21T08:39:57.904963+0000","last_peered":"2024-04-21T08:39:57.904963+0000","last_clean":"2024-04-21T08:39:57.904963+0000","last_became_active":"2024-04-21T08:37:30.413890+0000","last_became_peered":"2024-04-21T08:37:30.413890+0000","last_unstale":"2024-04-21T08:39:57.904963+0000","last_undegraded":"2024-04-21T08:39:57.904963+0000","last_fullsized":"2024-04-21T08:39:57.904963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:16.972618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903800+0000","last_change":"2024-04-21T08:37:31.449307+0000","last_active":"2024-04-21T08:39:57.903800+0000","last_peered":"2024-04-21T08:39:57.903800+0000","last_clean":"2024-04-21T08:39:57.903800+0000","last_became_active":"2024-04-21T08:37:31.448980+0000","last_became_peered":"2024-04-21T08:37:31.448980+0000","last_unstale":"2024-04-21T08:39:57.903800+0000","last_undegraded":"2024-04-21T08:39:57.903800+0000","last_fullsized":"2024-04-21T08:39:57.903800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:30:31.939274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151476+0000","last_change":"2024-04-21T08:37:35.760616+0000","last_active":"2024-04-21T08:39:58.151476+0000","last_peered":"2024-04-21T08:39:58.151476+0000","last_clean":"2024-04-21T08:39:58.151476+0000","last_became_active":"2024-04-21T08:37:35.760131+0000","last_became_peered":"2024-04-21T08:37:35.760131+0000","last_unstale":"2024-04-21T08:39:58.151476+0000","last_undegraded":"2024-04-21T08:39:58.151476+0000","last_fullsized":"2024-04-21T08:39:58.151476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:51.510259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009114+0000","last_change":"2024-04-21T08:38:30.682791+0000","last_active":"2024-04-21T08:39:58.009114+0000","last_peered":"2024-04-21T08:39:58.009114+0000","last_clean":"2024-04-21T08:39:58.009114+0000","last_became_active":"2024-04-21T08:38:30.671977+0000","last_became_peered":"2024-04-21T08:38:30.671977+0000","last_unstale":"2024-04-21T08:39:58.009114+0000","last_undegraded":"2024-04-21T08:39:58.009114+0000","last_fullsized":"2024-04-21T08:39:58.009114+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:55:15.899399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905569+0000","last_change":"2024-04-21T08:37:31.447234+0000","last_active":"2024-04-21T08:39:57.905569+0000","last_peered":"2024-04-21T08:39:57.905569+0000","last_clean":"2024-04-21T08:39:57.905569+0000","last_became_active":"2024-04-21T08:37:31.446876+0000","last_became_peered":"2024-04-21T08:37:31.446876+0000","last_unstale":"2024-04-21T08:39:57.905569+0000","last_undegraded":"2024-04-21T08:39:57.905569+0000","last_fullsized":"2024-04-21T08:39:57.905569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:51.784126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153981+0000","last_change":"2024-04-21T08:37:35.598577+0000","last_active":"2024-04-21T08:39:58.153981+0000","last_peered":"2024-04-21T08:39:58.153981+0000","last_clean":"2024-04-21T08:39:58.153981+0000","last_became_active":"2024-04-21T08:37:35.598119+0000","last_became_peered":"2024-04-21T08:37:35.598119+0000","last_unstale":"2024-04-21T08:39:58.153981+0000","last_undegraded":"2024-04-21T08:39:58.153981+0000","last_fullsized":"2024-04-21T08:39:58.153981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:41.986674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905332+0000","last_change":"2024-04-21T08:37:30.425737+0000","last_active":"2024-04-21T08:39:57.905332+0000","last_peered":"2024-04-21T08:39:57.905332+0000","last_clean":"2024-04-21T08:39:57.905332+0000","last_became_active":"2024-04-21T08:37:30.425180+0000","last_became_peered":"2024-04-21T08:37:30.425180+0000","last_unstale":"2024-04-21T08:39:57.905332+0000","last_undegraded":"2024-04-21T08:39:57.905332+0000","last_fullsized":"2024-04-21T08:39:57.905332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:08.646106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.906011+0000","last_change":"2024-04-21T08:37:31.451505+0000","last_active":"2024-04-21T08:39:57.906011+0000","last_peered":"2024-04-21T08:39:57.906011+0000","last_clean":"2024-04-21T08:39:57.906011+0000","last_became_active":"2024-04-21T08:37:31.451146+0000","last_became_peered":"2024-04-21T08:37:31.451146+0000","last_unstale":"2024-04-21T08:39:57.906011+0000","last_undegraded":"2024-04-21T08:39:57.906011+0000","last_fullsized":"2024-04-21T08:39:57.906011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:27:01.793910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319583+0000","last_change":"2024-04-21T08:37:35.608995+0000","last_active":"2024-04-21T08:39:58.319583+0000","last_peered":"2024-04-21T08:39:58.319583+0000","last_clean":"2024-04-21T08:39:58.319583+0000","last_became_active":"2024-04-21T08:37:35.608587+0000","last_became_peered":"2024-04-21T08:37:35.608587+0000","last_unstale":"2024-04-21T08:39:58.319583+0000","last_undegraded":"2024-04-21T08:39:58.319583+0000","last_fullsized":"2024-04-21T08:39:58.319583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.829418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.112239+0000","last_change":"2024-04-21T08:37:30.419026+0000","last_active":"2024-04-21T08:39:58.112239+0000","last_peered":"2024-04-21T08:39:58.112239+0000","last_clean":"2024-04-21T08:39:58.112239+0000","last_became_active":"2024-04-21T08:37:30.418706+0000","last_became_peered":"2024-04-21T08:37:30.418706+0000","last_unstale":"2024-04-21T08:39:58.112239+0000","last_undegraded":"2024-04-21T08:39:58.112239+0000","last_fullsized":"2024-04-21T08:39:58.112239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:24.441918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905192+0000","last_change":"2024-04-21T08:37:31.446561+0000","last_active":"2024-04-21T08:39:57.905192+0000","last_peered":"2024-04-21T08:39:57.905192+0000","last_clean":"2024-04-21T08:39:57.905192+0000","last_became_active":"2024-04-21T08:37:31.446162+0000","last_became_peered":"2024-04-21T08:37:31.446162+0000","last_unstale":"2024-04-21T08:39:57.905192+0000","last_undegraded":"2024-04-21T08:39:57.905192+0000","last_fullsized":"2024-04-21T08:39:57.905192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:38:24.684537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013588+0000","last_change":"2024-04-21T08:37:35.598748+0000","last_active":"2024-04-21T08:39:58.013588+0000","last_peered":"2024-04-21T08:39:58.013588+0000","last_clean":"2024-04-21T08:39:58.013588+0000","last_became_active":"2024-04-21T08:37:35.598353+0000","last_became_peered":"2024-04-21T08:37:35.598353+0000","last_unstale":"2024-04-21T08:39:58.013588+0000","last_undegraded":"2024-04-21T08:39:58.013588+0000","last_fullsized":"2024-04-21T08:39:58.013588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:47:36.519694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319054+0000","last_change":"2024-04-21T08:38:30.691292+0000","last_active":"2024-04-21T08:39:58.319054+0000","last_peered":"2024-04-21T08:39:58.319054+0000","last_clean":"2024-04-21T08:39:58.319054+0000","last_became_active":"2024-04-21T08:38:30.676580+0000","last_became_peered":"2024-04-21T08:38:30.676580+0000","last_unstale":"2024-04-21T08:39:58.319054+0000","last_undegraded":"2024-04-21T08:39:58.319054+0000","last_fullsized":"2024-04-21T08:39:58.319054+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:43:53.150369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905478+0000","last_change":"2024-04-21T08:37:31.452367+0000","last_active":"2024-04-21T08:39:57.905478+0000","last_peered":"2024-04-21T08:39:57.905478+0000","last_clean":"2024-04-21T08:39:57.905478+0000","last_became_active":"2024-04-21T08:37:31.452031+0000","last_became_peered":"2024-04-21T08:37:31.452031+0000","last_unstale":"2024-04-21T08:39:57.905478+0000","last_undegraded":"2024-04-21T08:39:57.905478+0000","last_fullsized":"2024-04-21T08:39:57.905478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:58.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318378+0000","last_change":"2024-04-21T08:37:31.451222+0000","last_active":"2024-04-21T08:39:58.318378+0000","last_peered":"2024-04-21T08:39:58.318378+0000","last_clean":"2024-04-21T08:39:58.318378+0000","last_became_active":"2024-04-21T08:37:31.450869+0000","last_became_peered":"2024-04-21T08:37:31.450869+0000","last_unstale":"2024-04-21T08:39:58.318378+0000","last_undegraded":"2024-04-21T08:39:58.318378+0000","last_fullsized":"2024-04-21T08:39:58.318378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:17.051330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905137+0000","last_change":"2024-04-21T08:37:31.453415+0000","last_active":"2024-04-21T08:39:57.905137+0000","last_peered":"2024-04-21T08:39:57.905137+0000","last_clean":"2024-04-21T08:39:57.905137+0000","last_became_active":"2024-04-21T08:37:31.453089+0000","last_became_peered":"2024-04-21T08:37:31.453089+0000","last_unstale":"2024-04-21T08:39:57.905137+0000","last_undegraded":"2024-04-21T08:39:57.905137+0000","last_fullsized":"2024-04-21T08:39:57.905137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:25:44.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904904+0000","last_change":"2024-04-21T08:37:31.446707+0000","last_active":"2024-04-21T08:39:57.904904+0000","last_peered":"2024-04-21T08:39:57.904904+0000","last_clean":"2024-04-21T08:39:57.904904+0000","last_became_active":"2024-04-21T08:37:31.446340+0000","last_became_peered":"2024-04-21T08:37:31.446340+0000","last_unstale":"2024-04-21T08:39:57.904904+0000","last_undegraded":"2024-04-21T08:39:57.904904+0000","last_fullsized":"2024-04-21T08:39:57.904904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:13.649607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319500+0000","last_change":"2024-04-21T08:37:31.455532+0000","last_active":"2024-04-21T08:39:58.319500+0000","last_peered":"2024-04-21T08:39:58.319500+0000","last_clean":"2024-04-21T08:39:58.319500+0000","last_became_active":"2024-04-21T08:37:31.453547+0000","last_became_peered":"2024-04-21T08:37:31.453547+0000","last_unstale":"2024-04-21T08:39:58.319500+0000","last_undegraded":"2024-04-21T08:39:58.319500+0000","last_fullsized":"2024-04-21T08:39:58.319500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:39:47.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318944+0000","last_change":"2024-04-21T08:39:31.679882+0000","last_active":"2024-04-21T08:39:58.318944+0000","last_peered":"2024-04-21T08:39:58.318944+0000","last_clean":"2024-04-21T08:39:58.318944+0000","last_became_active":"2024-04-21T08:39:31.679440+0000","last_became_peered":"2024-04-21T08:39:31.679440+0000","last_unstale":"2024-04-21T08:39:58.318944+0000","last_undegraded":"2024-04-21T08:39:58.318944+0000","last_fullsized":"2024-04-21T08:39:58.318944+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:57.679038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318685+0000","last_change":"2024-04-21T08:37:31.451806+0000","last_active":"2024-04-21T08:39:58.318685+0000","last_peered":"2024-04-21T08:39:58.318685+0000","last_clean":"2024-04-21T08:39:58.318685+0000","last_became_active":"2024-04-21T08:37:31.451442+0000","last_became_peered":"2024-04-21T08:37:31.451442+0000","last_unstale":"2024-04-21T08:39:58.318685+0000","last_undegraded":"2024-04-21T08:39:58.318685+0000","last_fullsized":"2024-04-21T08:39:58.318685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:07.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905544+0000","last_change":"2024-04-21T08:37:31.444440+0000","last_active":"2024-04-21T08:39:57.905544+0000","last_peered":"2024-04-21T08:39:57.905544+0000","last_clean":"2024-04-21T08:39:57.905544+0000","last_became_active":"2024-04-21T08:37:31.444079+0000","last_became_peered":"2024-04-21T08:37:31.444079+0000","last_unstale":"2024-04-21T08:39:57.905544+0000","last_undegraded":"2024-04-21T08:39:57.905544+0000","last_fullsized":"2024-04-21T08:39:57.905544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:13.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904413+0000","last_change":"2024-04-21T08:37:31.453698+0000","last_active":"2024-04-21T08:39:57.904413+0000","last_peered":"2024-04-21T08:39:57.904413+0000","last_clean":"2024-04-21T08:39:57.904413+0000","last_became_active":"2024-04-21T08:37:31.453379+0000","last_became_peered":"2024-04-21T08:37:31.453379+0000","last_unstale":"2024-04-21T08:39:57.904413+0000","last_undegraded":"2024-04-21T08:39:57.904413+0000","last_fullsized":"2024-04-21T08:39:57.904413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:46:29.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013777+0000","last_change":"2024-04-21T08:37:31.444480+0000","last_active":"2024-04-21T08:39:58.013777+0000","last_peered":"2024-04-21T08:39:58.013777+0000","last_clean":"2024-04-21T08:39:58.013777+0000","last_became_active":"2024-04-21T08:37:31.444030+0000","last_became_peered":"2024-04-21T08:37:31.444030+0000","last_unstale":"2024-04-21T08:39:58.013777+0000","last_undegraded":"2024-04-21T08:39:58.013777+0000","last_fullsized":"2024-04-21T08:39:58.013777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:15:05.668885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903935+0000","last_change":"2024-04-21T08:37:31.456602+0000","last_active":"2024-04-21T08:39:57.903935+0000","last_peered":"2024-04-21T08:39:57.903935+0000","last_clean":"2024-04-21T08:39:57.903935+0000","last_became_active":"2024-04-21T08:37:31.456286+0000","last_became_peered":"2024-04-21T08:37:31.456286+0000","last_unstale":"2024-04-21T08:39:57.903935+0000","last_undegraded":"2024-04-21T08:39:57.903935+0000","last_fullsized":"2024-04-21T08:39:57.903935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:48:12.462037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154176+0000","last_change":"2024-04-21T08:37:31.447008+0000","last_active":"2024-04-21T08:39:58.154176+0000","last_peered":"2024-04-21T08:39:58.154176+0000","last_clean":"2024-04-21T08:39:58.154176+0000","last_became_active":"2024-04-21T08:37:31.445192+0000","last_became_peered":"2024-04-21T08:37:31.445192+0000","last_unstale":"2024-04-21T08:39:58.154176+0000","last_undegraded":"2024-04-21T08:39:58.154176+0000","last_fullsized":"2024-04-21T08:39:58.154176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:08.025647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318698+0000","last_change":"2024-04-21T08:37:31.443288+0000","last_active":"2024-04-21T08:39:58.318698+0000","last_peered":"2024-04-21T08:39:58.318698+0000","last_clean":"2024-04-21T08:39:58.318698+0000","last_became_active":"2024-04-21T08:37:31.442586+0000","last_became_peered":"2024-04-21T08:37:31.442586+0000","last_unstale":"2024-04-21T08:39:58.318698+0000","last_undegraded":"2024-04-21T08:39:58.318698+0000","last_fullsized":"2024-04-21T08:39:58.318698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:01.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318501+0000","last_change":"2024-04-21T08:37:31.453899+0000","last_active":"2024-04-21T08:39:58.318501+0000","last_peered":"2024-04-21T08:39:58.318501+0000","last_clean":"2024-04-21T08:39:58.318501+0000","last_became_active":"2024-04-21T08:37:31.453527+0000","last_became_peered":"2024-04-21T08:37:31.453527+0000","last_unstale":"2024-04-21T08:39:58.318501+0000","last_undegraded":"2024-04-21T08:39:58.318501+0000","last_fullsized":"2024-04-21T08:39:58.318501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:23:18.429618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108246,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600116,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":864,"num_read_kb":545,"num_write":289,"num_write_kb":1443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1800348,"data_compressed":80889,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":529807,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32348,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712804,"statfs":{"total":95995035648,"available":95961911296,"internally_reserved":0,"allocated":1200128,"data_stored":1320365,"data_compressed":73597,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":927355,"data_compressed":48844,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":922597,"data_compressed":48844,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1127103,"data_compressed":65780,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1028096,"data_stored":525219,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":585028,"data_compressed":28357,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1052672,"data_stored":526607,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1122304,"data_stored":595765,"data_compressed":31359,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1050229,"data_compressed":55000,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":520740,"data_compressed":26696,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1126400,"data_stored":1252669,"data_compressed":71936,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70650,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71418,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133076,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393372,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":204868,"data_compressed":7817,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:40:02.950 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:40:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 -- 172.21.15.94:0/4232199322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd454103af0 msgr2=0x7fd45410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 --2- 172.21.15.94:0/4232199322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd454103af0 0x7fd45410bfd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd448009900 tx=0x7fd44802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 -- 172.21.15.94:0/4232199322 shutdown_connections 2024-04-21T08:40:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 --2- 172.21.15.94:0/4232199322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45410c5a0 0x7fd45410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 --2- 172.21.15.94:0/4232199322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd454103af0 0x7fd45410bfd0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 --2- 172.21.15.94:0/4232199322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4541031d0 0x7fd4541035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 -- 172.21.15.94:0/4232199322 >> 172.21.15.94:0/4232199322 conn(0x7fd4540771d0 msgr2=0x7fd4540775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 -- 172.21.15.94:0/4232199322 shutdown_connections 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 -- 172.21.15.94:0/4232199322 wait complete. 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 Processor -- start 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.083+0000 7fd45a3d0640 1 -- start start 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd45a3d0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4541031d0 0x7fd45419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd45a3d0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd454103af0 0x7fd45419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd45a3d0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45410c5a0 0x7fd4541985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd45a3d0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd454112240 con 0x7fd4541031d0 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd45a3d0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd4541120c0 con 0x7fd45410c5a0 2024-04-21T08:40:03.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd45a3d0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd4541123c0 con 0x7fd454103af0 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd453fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4541031d0 0x7fd45419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd453fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4541031d0 0x7fd45419e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58696/0 (socket says 172.21.15.94:58696) 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd453fff640 1 -- 172.21.15.94:0/2681280658 learned_addr learned my addr 172.21.15.94:0/2681280658 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd4537fe640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd454103af0 0x7fd45419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd45896d640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45410c5a0 0x7fd4541985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd453fff640 1 -- 172.21.15.94:0/2681280658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd454103af0 msgr2=0x7fd45419e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd453fff640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd454103af0 0x7fd45419e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd453fff640 1 -- 172.21.15.94:0/2681280658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45410c5a0 msgr2=0x7fd4541985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd453fff640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45410c5a0 0x7fd4541985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd453fff640 1 -- 172.21.15.94:0/2681280658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd454198d60 con 0x7fd4541031d0 2024-04-21T08:40:03.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd45896d640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45410c5a0 0x7fd4541985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd453fff640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4541031d0 0x7fd45419e430 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd44400d950 tx=0x7fd44400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.084+0000 7fd4517fa640 1 -- 172.21.15.94:0/2681280658 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd444017070 con 0x7fd4541031d0 2024-04-21T08:40:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.085+0000 7fd45a3d0640 1 -- 172.21.15.94:0/2681280658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd454199050 con 0x7fd4541031d0 2024-04-21T08:40:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.085+0000 7fd4517fa640 1 -- 172.21.15.94:0/2681280658 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd444009c30 con 0x7fd4541031d0 2024-04-21T08:40:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.085+0000 7fd4517fa640 1 -- 172.21.15.94:0/2681280658 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44400f620 con 0x7fd4541031d0 2024-04-21T08:40:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.085+0000 7fd45a3d0640 1 -- 172.21.15.94:0/2681280658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4541a52f0 con 0x7fd4541031d0 2024-04-21T08:40:03.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.085+0000 7fd45a3d0640 1 -- 172.21.15.94:0/2681280658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd454069230 con 0x7fd4541031d0 2024-04-21T08:40:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.086+0000 7fd4517fa640 1 -- 172.21.15.94:0/2681280658 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fd444010460 con 0x7fd4541031d0 2024-04-21T08:40:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.087+0000 7fd4517fa640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd43c0a3e40 0x7fd43c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.087+0000 7fd4517fa640 1 -- 172.21.15.94:0/2681280658 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd4440d0ad0 con 0x7fd4541031d0 2024-04-21T08:40:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.087+0000 7fd4537fe640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd43c0a3e40 0x7fd43c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.088+0000 7fd4537fe640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd43c0a3e40 0x7fd43c0a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd448006010 tx=0x7fd448005e40 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:03.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.092+0000 7fd4517fa640 1 -- 172.21.15.94:0/2681280658 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd44409cb40 con 0x7fd4541031d0 2024-04-21T08:40:03.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.268+0000 7fd45a3d0640 1 -- 172.21.15.94:0/2681280658 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd454062390 con 0x7fd43c0a3e40 2024-04-21T08:40:03.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.277+0000 7fd4517fa640 1 -- 172.21.15.94:0/2681280658 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339690 (secure 0 0 0) 0x7fd454062390 con 0x7fd43c0a3e40 2024-04-21T08:40:03.280 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:03.282 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-21T08:40:03.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 -- 172.21.15.94:0/2681280658 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd43c0a3e40 msgr2=0x7fd43c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd43c0a3e40 0x7fd43c0a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd448006010 tx=0x7fd448005e40 comp rx=0 tx=0).stop 2024-04-21T08:40:03.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 -- 172.21.15.94:0/2681280658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4541031d0 msgr2=0x7fd45419e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4541031d0 0x7fd45419e430 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd44400d950 tx=0x7fd44400de10 comp rx=0 tx=0).stop 2024-04-21T08:40:03.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 -- 172.21.15.94:0/2681280658 shutdown_connections 2024-04-21T08:40:03.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd43c0a3e40 0x7fd43c0a6300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd454103af0 0x7fd45419e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45410c5a0 0x7fd4541985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 --2- 172.21.15.94:0/2681280658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4541031d0 0x7fd45419e430 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 -- 172.21.15.94:0/2681280658 >> 172.21.15.94:0/2681280658 conn(0x7fd4540771d0 msgr2=0x7fd4540fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 -- 172.21.15.94:0/2681280658 shutdown_connections 2024-04-21T08:40:03.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.280+0000 7fd45a3d0640 1 -- 172.21.15.94:0/2681280658 wait complete. 2024-04-21T08:40:03.291 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-21T08:40:01.915737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190718,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1000,"num_read_kb":664,"num_write":472,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378544,"kb_used_data":12704,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563280,"statfs":{"total":1151940427776,"available":1151552798720,"internally_reserved":0,"allocated":13008896,"data_stored":9883484,"data_compressed":539507,"data_compressed_allocated":729088,"data_compressed_original":4308992,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.986713"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318569+0000","last_change":"2024-04-21T08:37:31.448345+0000","last_active":"2024-04-21T08:39:58.318569+0000","last_peered":"2024-04-21T08:39:58.318569+0000","last_clean":"2024-04-21T08:39:58.318569+0000","last_became_active":"2024-04-21T08:37:31.447878+0000","last_became_peered":"2024-04-21T08:37:31.447878+0000","last_unstale":"2024-04-21T08:39:58.318569+0000","last_undegraded":"2024-04-21T08:39:58.318569+0000","last_fullsized":"2024-04-21T08:39:58.318569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:45.589165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319782+0000","last_change":"2024-04-21T08:37:31.452895+0000","last_active":"2024-04-21T08:39:58.319782+0000","last_peered":"2024-04-21T08:39:58.319782+0000","last_clean":"2024-04-21T08:39:58.319782+0000","last_became_active":"2024-04-21T08:37:31.452550+0000","last_became_peered":"2024-04-21T08:37:31.452550+0000","last_unstale":"2024-04-21T08:39:58.319782+0000","last_undegraded":"2024-04-21T08:39:58.319782+0000","last_fullsized":"2024-04-21T08:39:58.319782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:02.497802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319443+0000","last_change":"2024-04-21T08:37:31.447552+0000","last_active":"2024-04-21T08:39:58.319443+0000","last_peered":"2024-04-21T08:39:58.319443+0000","last_clean":"2024-04-21T08:39:58.319443+0000","last_became_active":"2024-04-21T08:37:31.446044+0000","last_became_peered":"2024-04-21T08:37:31.446044+0000","last_unstale":"2024-04-21T08:39:58.319443+0000","last_undegraded":"2024-04-21T08:39:58.319443+0000","last_fullsized":"2024-04-21T08:39:58.319443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:22.801977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151531+0000","last_change":"2024-04-21T08:37:31.452400+0000","last_active":"2024-04-21T08:39:58.151531+0000","last_peered":"2024-04-21T08:39:58.151531+0000","last_clean":"2024-04-21T08:39:58.151531+0000","last_became_active":"2024-04-21T08:37:31.451462+0000","last_became_peered":"2024-04-21T08:37:31.451462+0000","last_unstale":"2024-04-21T08:39:58.151531+0000","last_undegraded":"2024-04-21T08:39:58.151531+0000","last_fullsized":"2024-04-21T08:39:58.151531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:22.463481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009035+0000","last_change":"2024-04-21T08:37:31.448942+0000","last_active":"2024-04-21T08:39:58.009035+0000","last_peered":"2024-04-21T08:39:58.009035+0000","last_clean":"2024-04-21T08:39:58.009035+0000","last_became_active":"2024-04-21T08:37:31.448526+0000","last_became_peered":"2024-04-21T08:37:31.448526+0000","last_unstale":"2024-04-21T08:39:58.009035+0000","last_undegraded":"2024-04-21T08:39:58.009035+0000","last_fullsized":"2024-04-21T08:39:58.009035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:29.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017093+0000","last_change":"2024-04-21T08:37:31.452165+0000","last_active":"2024-04-21T08:39:58.017093+0000","last_peered":"2024-04-21T08:39:58.017093+0000","last_clean":"2024-04-21T08:39:58.017093+0000","last_became_active":"2024-04-21T08:37:31.451802+0000","last_became_peered":"2024-04-21T08:37:31.451802+0000","last_unstale":"2024-04-21T08:39:58.017093+0000","last_undegraded":"2024-04-21T08:39:58.017093+0000","last_fullsized":"2024-04-21T08:39:58.017093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:42.210544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904949+0000","last_change":"2024-04-21T08:39:31.681660+0000","last_active":"2024-04-21T08:39:57.904949+0000","last_peered":"2024-04-21T08:39:57.904949+0000","last_clean":"2024-04-21T08:39:57.904949+0000","last_became_active":"2024-04-21T08:39:31.681203+0000","last_became_peered":"2024-04-21T08:39:31.681203+0000","last_unstale":"2024-04-21T08:39:57.904949+0000","last_undegraded":"2024-04-21T08:39:57.904949+0000","last_fullsized":"2024-04-21T08:39:57.904949+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:54.120862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903472+0000","last_change":"2024-04-21T08:37:31.448257+0000","last_active":"2024-04-21T08:39:57.903472+0000","last_peered":"2024-04-21T08:39:57.903472+0000","last_clean":"2024-04-21T08:39:57.903472+0000","last_became_active":"2024-04-21T08:37:31.447845+0000","last_became_peered":"2024-04-21T08:37:31.447845+0000","last_unstale":"2024-04-21T08:39:57.903472+0000","last_undegraded":"2024-04-21T08:39:57.903472+0000","last_fullsized":"2024-04-21T08:39:57.903472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:19.956699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154785+0000","last_change":"2024-04-21T08:37:31.443022+0000","last_active":"2024-04-21T08:39:58.154785+0000","last_peered":"2024-04-21T08:39:58.154785+0000","last_clean":"2024-04-21T08:39:58.154785+0000","last_became_active":"2024-04-21T08:37:31.442645+0000","last_became_peered":"2024-04-21T08:37:31.442645+0000","last_unstale":"2024-04-21T08:39:58.154785+0000","last_undegraded":"2024-04-21T08:39:58.154785+0000","last_fullsized":"2024-04-21T08:39:58.154785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:18:10.861281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017434+0000","last_change":"2024-04-21T08:37:31.453427+0000","last_active":"2024-04-21T08:39:58.017434+0000","last_peered":"2024-04-21T08:39:58.017434+0000","last_clean":"2024-04-21T08:39:58.017434+0000","last_became_active":"2024-04-21T08:37:31.453096+0000","last_became_peered":"2024-04-21T08:37:31.453096+0000","last_unstale":"2024-04-21T08:39:58.017434+0000","last_undegraded":"2024-04-21T08:39:58.017434+0000","last_fullsized":"2024-04-21T08:39:58.017434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:49.450351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905929+0000","last_change":"2024-04-21T08:37:31.449908+0000","last_active":"2024-04-21T08:39:57.905929+0000","last_peered":"2024-04-21T08:39:57.905929+0000","last_clean":"2024-04-21T08:39:57.905929+0000","last_became_active":"2024-04-21T08:37:31.449558+0000","last_became_peered":"2024-04-21T08:37:31.449558+0000","last_unstale":"2024-04-21T08:39:57.905929+0000","last_undegraded":"2024-04-21T08:39:57.905929+0000","last_fullsized":"2024-04-21T08:39:57.905929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:21.538689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151539+0000","last_change":"2024-04-21T08:37:31.452973+0000","last_active":"2024-04-21T08:39:58.151539+0000","last_peered":"2024-04-21T08:39:58.151539+0000","last_clean":"2024-04-21T08:39:58.151539+0000","last_became_active":"2024-04-21T08:37:31.452586+0000","last_became_peered":"2024-04-21T08:37:31.452586+0000","last_unstale":"2024-04-21T08:39:58.151539+0000","last_undegraded":"2024-04-21T08:39:58.151539+0000","last_fullsized":"2024-04-21T08:39:58.151539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:09:11.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319144+0000","last_change":"2024-04-21T08:37:31.445801+0000","last_active":"2024-04-21T08:39:58.319144+0000","last_peered":"2024-04-21T08:39:58.319144+0000","last_clean":"2024-04-21T08:39:58.319144+0000","last_became_active":"2024-04-21T08:37:31.445478+0000","last_became_peered":"2024-04-21T08:37:31.445478+0000","last_unstale":"2024-04-21T08:39:58.319144+0000","last_undegraded":"2024-04-21T08:39:58.319144+0000","last_fullsized":"2024-04-21T08:39:58.319144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:33.352739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319201+0000","last_change":"2024-04-21T08:37:31.450735+0000","last_active":"2024-04-21T08:39:58.319201+0000","last_peered":"2024-04-21T08:39:58.319201+0000","last_clean":"2024-04-21T08:39:58.319201+0000","last_became_active":"2024-04-21T08:37:31.450384+0000","last_became_peered":"2024-04-21T08:37:31.450384+0000","last_unstale":"2024-04-21T08:39:58.319201+0000","last_undegraded":"2024-04-21T08:39:58.319201+0000","last_fullsized":"2024-04-21T08:39:58.319201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:45.950489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319354+0000","last_change":"2024-04-21T08:37:31.454690+0000","last_active":"2024-04-21T08:39:58.319354+0000","last_peered":"2024-04-21T08:39:58.319354+0000","last_clean":"2024-04-21T08:39:58.319354+0000","last_became_active":"2024-04-21T08:37:31.454330+0000","last_became_peered":"2024-04-21T08:37:31.454330+0000","last_unstale":"2024-04-21T08:39:58.319354+0000","last_undegraded":"2024-04-21T08:39:58.319354+0000","last_fullsized":"2024-04-21T08:39:58.319354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:52.644317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153871+0000","last_change":"2024-04-21T08:37:31.441389+0000","last_active":"2024-04-21T08:39:58.153871+0000","last_peered":"2024-04-21T08:39:58.153871+0000","last_clean":"2024-04-21T08:39:58.153871+0000","last_became_active":"2024-04-21T08:37:31.441065+0000","last_became_peered":"2024-04-21T08:37:31.441065+0000","last_unstale":"2024-04-21T08:39:58.153871+0000","last_undegraded":"2024-04-21T08:39:58.153871+0000","last_fullsized":"2024-04-21T08:39:58.153871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:56.528154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318575+0000","last_change":"2024-04-21T08:37:31.446722+0000","last_active":"2024-04-21T08:39:58.318575+0000","last_peered":"2024-04-21T08:39:58.318575+0000","last_clean":"2024-04-21T08:39:58.318575+0000","last_became_active":"2024-04-21T08:37:31.446396+0000","last_became_peered":"2024-04-21T08:37:31.446396+0000","last_unstale":"2024-04-21T08:39:58.318575+0000","last_undegraded":"2024-04-21T08:39:58.318575+0000","last_fullsized":"2024-04-21T08:39:58.318575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:17.488633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007275+0000","last_change":"2024-04-21T08:37:31.444388+0000","last_active":"2024-04-21T08:39:58.007275+0000","last_peered":"2024-04-21T08:39:58.007275+0000","last_clean":"2024-04-21T08:39:58.007275+0000","last_became_active":"2024-04-21T08:37:31.444023+0000","last_became_peered":"2024-04-21T08:37:31.444023+0000","last_unstale":"2024-04-21T08:39:58.007275+0000","last_undegraded":"2024-04-21T08:39:58.007275+0000","last_fullsized":"2024-04-21T08:39:58.007275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:07:46.238941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905278+0000","last_change":"2024-04-21T08:37:31.447850+0000","last_active":"2024-04-21T08:39:57.905278+0000","last_peered":"2024-04-21T08:39:57.905278+0000","last_clean":"2024-04-21T08:39:57.905278+0000","last_became_active":"2024-04-21T08:37:31.447505+0000","last_became_peered":"2024-04-21T08:37:31.447505+0000","last_unstale":"2024-04-21T08:39:57.905278+0000","last_undegraded":"2024-04-21T08:39:57.905278+0000","last_fullsized":"2024-04-21T08:39:57.905278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:55.399410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905032+0000","last_change":"2024-04-21T08:37:35.608747+0000","last_active":"2024-04-21T08:39:57.905032+0000","last_peered":"2024-04-21T08:39:57.905032+0000","last_clean":"2024-04-21T08:39:57.905032+0000","last_became_active":"2024-04-21T08:37:35.608157+0000","last_became_peered":"2024-04-21T08:37:35.608157+0000","last_unstale":"2024-04-21T08:39:57.905032+0000","last_undegraded":"2024-04-21T08:39:57.905032+0000","last_fullsized":"2024-04-21T08:39:57.905032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:53.384190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007037+0000","last_change":"2024-04-21T08:38:30.679843+0000","last_active":"2024-04-21T08:39:58.007037+0000","last_peered":"2024-04-21T08:39:58.007037+0000","last_clean":"2024-04-21T08:39:58.007037+0000","last_became_active":"2024-04-21T08:38:30.670964+0000","last_became_peered":"2024-04-21T08:38:30.670964+0000","last_unstale":"2024-04-21T08:39:58.007037+0000","last_undegraded":"2024-04-21T08:39:58.007037+0000","last_fullsized":"2024-04-21T08:39:58.007037+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:49.652571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904100+0000","last_change":"2024-04-21T08:37:31.451813+0000","last_active":"2024-04-21T08:39:57.904100+0000","last_peered":"2024-04-21T08:39:57.904100+0000","last_clean":"2024-04-21T08:39:57.904100+0000","last_became_active":"2024-04-21T08:37:31.451447+0000","last_became_peered":"2024-04-21T08:37:31.451447+0000","last_unstale":"2024-04-21T08:39:57.904100+0000","last_undegraded":"2024-04-21T08:39:57.904100+0000","last_fullsized":"2024-04-21T08:39:57.904100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:46:45.235326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151221+0000","last_change":"2024-04-21T08:37:35.603518+0000","last_active":"2024-04-21T08:39:58.151221+0000","last_peered":"2024-04-21T08:39:58.151221+0000","last_clean":"2024-04-21T08:39:58.151221+0000","last_became_active":"2024-04-21T08:37:35.603038+0000","last_became_peered":"2024-04-21T08:37:35.603038+0000","last_unstale":"2024-04-21T08:39:58.151221+0000","last_undegraded":"2024-04-21T08:39:58.151221+0000","last_fullsized":"2024-04-21T08:39:58.151221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:06:29.224025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904228+0000","last_change":"2024-04-21T08:37:30.424608+0000","last_active":"2024-04-21T08:39:57.904228+0000","last_peered":"2024-04-21T08:39:57.904228+0000","last_clean":"2024-04-21T08:39:57.904228+0000","last_became_active":"2024-04-21T08:37:30.424259+0000","last_became_peered":"2024-04-21T08:37:30.424259+0000","last_unstale":"2024-04-21T08:39:57.904228+0000","last_undegraded":"2024-04-21T08:39:57.904228+0000","last_fullsized":"2024-04-21T08:39:57.904228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:38.007095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319688+0000","last_change":"2024-04-21T08:37:31.444656+0000","last_active":"2024-04-21T08:39:58.319688+0000","last_peered":"2024-04-21T08:39:58.319688+0000","last_clean":"2024-04-21T08:39:58.319688+0000","last_became_active":"2024-04-21T08:37:31.444300+0000","last_became_peered":"2024-04-21T08:37:31.444300+0000","last_unstale":"2024-04-21T08:39:58.319688+0000","last_undegraded":"2024-04-21T08:39:58.319688+0000","last_fullsized":"2024-04-21T08:39:58.319688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:40.901674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151388+0000","last_change":"2024-04-21T08:37:35.611732+0000","last_active":"2024-04-21T08:39:58.151388+0000","last_peered":"2024-04-21T08:39:58.151388+0000","last_clean":"2024-04-21T08:39:58.151388+0000","last_became_active":"2024-04-21T08:37:35.611060+0000","last_became_peered":"2024-04-21T08:37:35.611060+0000","last_unstale":"2024-04-21T08:39:58.151388+0000","last_undegraded":"2024-04-21T08:39:58.151388+0000","last_fullsized":"2024-04-21T08:39:58.151388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:36:01.491949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318678+0000","last_change":"2024-04-21T08:38:30.689722+0000","last_active":"2024-04-21T08:39:58.318678+0000","last_peered":"2024-04-21T08:39:58.318678+0000","last_clean":"2024-04-21T08:39:58.318678+0000","last_became_active":"2024-04-21T08:38:30.676521+0000","last_became_peered":"2024-04-21T08:38:30.676521+0000","last_unstale":"2024-04-21T08:39:58.318678+0000","last_undegraded":"2024-04-21T08:39:58.318678+0000","last_fullsized":"2024-04-21T08:39:58.318678+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:24:15.799308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154321+0000","last_change":"2024-04-21T08:37:31.443896+0000","last_active":"2024-04-21T08:39:58.154321+0000","last_peered":"2024-04-21T08:39:58.154321+0000","last_clean":"2024-04-21T08:39:58.154321+0000","last_became_active":"2024-04-21T08:37:31.442684+0000","last_became_peered":"2024-04-21T08:37:31.442684+0000","last_unstale":"2024-04-21T08:39:58.154321+0000","last_undegraded":"2024-04-21T08:39:58.154321+0000","last_fullsized":"2024-04-21T08:39:58.154321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:14:18.649565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151231+0000","last_change":"2024-04-21T08:37:35.600468+0000","last_active":"2024-04-21T08:39:58.151231+0000","last_peered":"2024-04-21T08:39:58.151231+0000","last_clean":"2024-04-21T08:39:58.151231+0000","last_became_active":"2024-04-21T08:37:35.599940+0000","last_became_peered":"2024-04-21T08:37:35.599940+0000","last_unstale":"2024-04-21T08:39:58.151231+0000","last_undegraded":"2024-04-21T08:39:58.151231+0000","last_fullsized":"2024-04-21T08:39:58.151231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:48.125434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009395+0000","last_change":"2024-04-21T08:37:30.416480+0000","last_active":"2024-04-21T08:39:58.009395+0000","last_peered":"2024-04-21T08:39:58.009395+0000","last_clean":"2024-04-21T08:39:58.009395+0000","last_became_active":"2024-04-21T08:37:30.416072+0000","last_became_peered":"2024-04-21T08:37:30.416072+0000","last_unstale":"2024-04-21T08:39:58.009395+0000","last_undegraded":"2024-04-21T08:39:58.009395+0000","last_fullsized":"2024-04-21T08:39:58.009395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:28:22.400926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014147+0000","last_change":"2024-04-21T08:37:31.457216+0000","last_active":"2024-04-21T08:39:58.014147+0000","last_peered":"2024-04-21T08:39:58.014147+0000","last_clean":"2024-04-21T08:39:58.014147+0000","last_became_active":"2024-04-21T08:37:31.456856+0000","last_became_peered":"2024-04-21T08:37:31.456856+0000","last_unstale":"2024-04-21T08:39:58.014147+0000","last_undegraded":"2024-04-21T08:39:58.014147+0000","last_fullsized":"2024-04-21T08:39:58.014147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:10.272886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905178+0000","last_change":"2024-04-21T08:37:35.601715+0000","last_active":"2024-04-21T08:39:57.905178+0000","last_peered":"2024-04-21T08:39:57.905178+0000","last_clean":"2024-04-21T08:39:57.905178+0000","last_became_active":"2024-04-21T08:37:35.601269+0000","last_became_peered":"2024-04-21T08:37:35.601269+0000","last_unstale":"2024-04-21T08:39:57.905178+0000","last_undegraded":"2024-04-21T08:39:57.905178+0000","last_fullsized":"2024-04-21T08:39:57.905178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:57.192918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":164,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.890274+0000","last_change":"2024-04-21T08:37:30.415510+0000","last_active":"2024-04-21T08:39:58.890274+0000","last_peered":"2024-04-21T08:39:58.890274+0000","last_clean":"2024-04-21T08:39:58.890274+0000","last_became_active":"2024-04-21T08:37:30.415199+0000","last_became_peered":"2024-04-21T08:37:30.415199+0000","last_unstale":"2024-04-21T08:39:58.890274+0000","last_undegraded":"2024-04-21T08:39:58.890274+0000","last_fullsized":"2024-04-21T08:39:58.890274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:32.336247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009074+0000","last_change":"2024-04-21T08:39:31.681460+0000","last_active":"2024-04-21T08:39:58.009074+0000","last_peered":"2024-04-21T08:39:58.009074+0000","last_clean":"2024-04-21T08:39:58.009074+0000","last_became_active":"2024-04-21T08:39:31.681057+0000","last_became_peered":"2024-04-21T08:39:31.681057+0000","last_unstale":"2024-04-21T08:39:58.009074+0000","last_undegraded":"2024-04-21T08:39:58.009074+0000","last_fullsized":"2024-04-21T08:39:58.009074+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:43.407369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905226+0000","last_change":"2024-04-21T08:38:30.714455+0000","last_active":"2024-04-21T08:39:57.905226+0000","last_peered":"2024-04-21T08:39:57.905226+0000","last_clean":"2024-04-21T08:39:57.905226+0000","last_became_active":"2024-04-21T08:38:30.713910+0000","last_became_peered":"2024-04-21T08:38:30.713910+0000","last_unstale":"2024-04-21T08:39:57.905226+0000","last_undegraded":"2024-04-21T08:39:57.905226+0000","last_fullsized":"2024-04-21T08:39:57.905226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.448652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,10],"acting":[2,6,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017063+0000","last_change":"2024-04-21T08:37:30.419877+0000","last_active":"2024-04-21T08:39:58.017063+0000","last_peered":"2024-04-21T08:39:58.017063+0000","last_clean":"2024-04-21T08:39:58.017063+0000","last_became_active":"2024-04-21T08:37:30.419572+0000","last_became_peered":"2024-04-21T08:37:30.419572+0000","last_unstale":"2024-04-21T08:39:58.017063+0000","last_undegraded":"2024-04-21T08:39:58.017063+0000","last_fullsized":"2024-04-21T08:39:58.017063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:32:35.524089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014037+0000","last_change":"2024-04-21T08:37:31.458590+0000","last_active":"2024-04-21T08:39:58.014037+0000","last_peered":"2024-04-21T08:39:58.014037+0000","last_clean":"2024-04-21T08:39:58.014037+0000","last_became_active":"2024-04-21T08:37:31.458196+0000","last_became_peered":"2024-04-21T08:37:31.458196+0000","last_unstale":"2024-04-21T08:39:58.014037+0000","last_undegraded":"2024-04-21T08:39:58.014037+0000","last_fullsized":"2024-04-21T08:39:58.014037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:00.890859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318442+0000","last_change":"2024-04-21T08:38:30.803663+0000","last_active":"2024-04-21T08:39:58.318442+0000","last_peered":"2024-04-21T08:39:58.318442+0000","last_clean":"2024-04-21T08:39:58.318442+0000","last_became_active":"2024-04-21T08:38:30.803203+0000","last_became_peered":"2024-04-21T08:38:30.803203+0000","last_unstale":"2024-04-21T08:39:58.318442+0000","last_undegraded":"2024-04-21T08:39:58.318442+0000","last_fullsized":"2024-04-21T08:39:58.318442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:06.085380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318695+0000","last_change":"2024-04-21T08:37:30.415786+0000","last_active":"2024-04-21T08:39:58.318695+0000","last_peered":"2024-04-21T08:39:58.318695+0000","last_clean":"2024-04-21T08:39:58.318695+0000","last_became_active":"2024-04-21T08:37:30.415472+0000","last_became_peered":"2024-04-21T08:37:30.415472+0000","last_unstale":"2024-04-21T08:39:58.318695+0000","last_undegraded":"2024-04-21T08:39:58.318695+0000","last_fullsized":"2024-04-21T08:39:58.318695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.417132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903648+0000","last_change":"2024-04-21T08:37:31.455720+0000","last_active":"2024-04-21T08:39:57.903648+0000","last_peered":"2024-04-21T08:39:57.903648+0000","last_clean":"2024-04-21T08:39:57.903648+0000","last_became_active":"2024-04-21T08:37:31.455334+0000","last_became_peered":"2024-04-21T08:37:31.455334+0000","last_unstale":"2024-04-21T08:39:57.903648+0000","last_undegraded":"2024-04-21T08:39:57.903648+0000","last_fullsized":"2024-04-21T08:39:57.903648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:06:55.220656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319675+0000","last_change":"2024-04-21T08:37:35.603305+0000","last_active":"2024-04-21T08:39:58.319675+0000","last_peered":"2024-04-21T08:39:58.319675+0000","last_clean":"2024-04-21T08:39:58.319675+0000","last_became_active":"2024-04-21T08:37:35.602854+0000","last_became_peered":"2024-04-21T08:37:35.602854+0000","last_unstale":"2024-04-21T08:39:58.319675+0000","last_undegraded":"2024-04-21T08:39:58.319675+0000","last_fullsized":"2024-04-21T08:39:58.319675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:10.993864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013890+0000","last_change":"2024-04-21T08:37:30.420797+0000","last_active":"2024-04-21T08:39:58.013890+0000","last_peered":"2024-04-21T08:39:58.013890+0000","last_clean":"2024-04-21T08:39:58.013890+0000","last_became_active":"2024-04-21T08:37:30.420485+0000","last_became_peered":"2024-04-21T08:37:30.420485+0000","last_unstale":"2024-04-21T08:39:58.013890+0000","last_undegraded":"2024-04-21T08:39:58.013890+0000","last_fullsized":"2024-04-21T08:39:58.013890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:03.275677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006844+0000","last_change":"2024-04-21T08:37:31.448406+0000","last_active":"2024-04-21T08:39:58.006844+0000","last_peered":"2024-04-21T08:39:58.006844+0000","last_clean":"2024-04-21T08:39:58.006844+0000","last_became_active":"2024-04-21T08:37:31.448085+0000","last_became_peered":"2024-04-21T08:37:31.448085+0000","last_unstale":"2024-04-21T08:39:58.006844+0000","last_undegraded":"2024-04-21T08:39:58.006844+0000","last_fullsized":"2024-04-21T08:39:58.006844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:49.574064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903948+0000","last_change":"2024-04-21T08:37:35.601338+0000","last_active":"2024-04-21T08:39:57.903948+0000","last_peered":"2024-04-21T08:39:57.903948+0000","last_clean":"2024-04-21T08:39:57.903948+0000","last_became_active":"2024-04-21T08:37:35.600890+0000","last_became_peered":"2024-04-21T08:37:35.600890+0000","last_unstale":"2024-04-21T08:39:57.903948+0000","last_undegraded":"2024-04-21T08:39:57.903948+0000","last_fullsized":"2024-04-21T08:39:57.903948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:30:21.755850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.889748+0000","last_change":"2024-04-21T08:37:30.423320+0000","last_active":"2024-04-21T08:39:58.889748+0000","last_peered":"2024-04-21T08:39:58.889748+0000","last_clean":"2024-04-21T08:39:58.889748+0000","last_became_active":"2024-04-21T08:37:30.423020+0000","last_became_peered":"2024-04-21T08:37:30.423020+0000","last_unstale":"2024-04-21T08:39:58.889748+0000","last_undegraded":"2024-04-21T08:39:58.889748+0000","last_fullsized":"2024-04-21T08:39:58.889748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:08:27.313196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154669+0000","last_change":"2024-04-21T08:37:31.442425+0000","last_active":"2024-04-21T08:39:58.154669+0000","last_peered":"2024-04-21T08:39:58.154669+0000","last_clean":"2024-04-21T08:39:58.154669+0000","last_became_active":"2024-04-21T08:37:31.442028+0000","last_became_peered":"2024-04-21T08:37:31.442028+0000","last_unstale":"2024-04-21T08:39:58.154669+0000","last_undegraded":"2024-04-21T08:39:58.154669+0000","last_fullsized":"2024-04-21T08:39:58.154669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:45.498656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006697+0000","last_change":"2024-04-21T08:37:35.607829+0000","last_active":"2024-04-21T08:39:58.006697+0000","last_peered":"2024-04-21T08:39:58.006697+0000","last_clean":"2024-04-21T08:39:58.006697+0000","last_became_active":"2024-04-21T08:37:35.607150+0000","last_became_peered":"2024-04-21T08:37:35.607150+0000","last_unstale":"2024-04-21T08:39:58.006697+0000","last_undegraded":"2024-04-21T08:39:58.006697+0000","last_fullsized":"2024-04-21T08:39:58.006697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:01:34.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905384+0000","last_change":"2024-04-21T08:37:30.418613+0000","last_active":"2024-04-21T08:39:57.905384+0000","last_peered":"2024-04-21T08:39:57.905384+0000","last_clean":"2024-04-21T08:39:57.905384+0000","last_became_active":"2024-04-21T08:37:30.418300+0000","last_became_peered":"2024-04-21T08:37:30.418300+0000","last_unstale":"2024-04-21T08:39:57.905384+0000","last_undegraded":"2024-04-21T08:39:57.905384+0000","last_fullsized":"2024-04-21T08:39:57.905384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:17:35.370753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319739+0000","last_change":"2024-04-21T08:39:31.682628+0000","last_active":"2024-04-21T08:39:58.319739+0000","last_peered":"2024-04-21T08:39:58.319739+0000","last_clean":"2024-04-21T08:39:58.319739+0000","last_became_active":"2024-04-21T08:39:31.682383+0000","last_became_peered":"2024-04-21T08:39:31.682383+0000","last_unstale":"2024-04-21T08:39:58.319739+0000","last_undegraded":"2024-04-21T08:39:58.319739+0000","last_fullsized":"2024-04-21T08:39:58.319739+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:38.831288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151734+0000","last_change":"2024-04-21T08:37:35.608280+0000","last_active":"2024-04-21T08:39:58.151734+0000","last_peered":"2024-04-21T08:39:58.151734+0000","last_clean":"2024-04-21T08:39:58.151734+0000","last_became_active":"2024-04-21T08:37:35.607898+0000","last_became_peered":"2024-04-21T08:37:35.607898+0000","last_unstale":"2024-04-21T08:39:58.151734+0000","last_undegraded":"2024-04-21T08:39:58.151734+0000","last_fullsized":"2024-04-21T08:39:58.151734+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:55.689519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014140+0000","last_change":"2024-04-21T08:37:30.417352+0000","last_active":"2024-04-21T08:39:58.014140+0000","last_peered":"2024-04-21T08:39:58.014140+0000","last_clean":"2024-04-21T08:39:58.014140+0000","last_became_active":"2024-04-21T08:37:30.417000+0000","last_became_peered":"2024-04-21T08:37:30.417000+0000","last_unstale":"2024-04-21T08:39:58.014140+0000","last_undegraded":"2024-04-21T08:39:58.014140+0000","last_fullsized":"2024-04-21T08:39:58.014140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:56.087334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006804+0000","last_change":"2024-04-21T08:39:31.679434+0000","last_active":"2024-04-21T08:39:58.006804+0000","last_peered":"2024-04-21T08:39:58.006804+0000","last_clean":"2024-04-21T08:39:58.006804+0000","last_became_active":"2024-04-21T08:39:31.679041+0000","last_became_peered":"2024-04-21T08:39:31.679041+0000","last_unstale":"2024-04-21T08:39:58.006804+0000","last_undegraded":"2024-04-21T08:39:58.006804+0000","last_fullsized":"2024-04-21T08:39:58.006804+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:27:11.011449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006731+0000","last_change":"2024-04-21T08:37:35.601077+0000","last_active":"2024-04-21T08:39:58.006731+0000","last_peered":"2024-04-21T08:39:58.006731+0000","last_clean":"2024-04-21T08:39:58.006731+0000","last_became_active":"2024-04-21T08:37:35.600673+0000","last_became_peered":"2024-04-21T08:37:35.600673+0000","last_unstale":"2024-04-21T08:39:58.006731+0000","last_undegraded":"2024-04-21T08:39:58.006731+0000","last_fullsized":"2024-04-21T08:39:58.006731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:04.554208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'41","reported_seq":333,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.890961+0000","last_change":"2024-04-21T08:37:30.421440+0000","last_active":"2024-04-21T08:39:58.890961+0000","last_peered":"2024-04-21T08:39:58.890961+0000","last_clean":"2024-04-21T08:39:58.890961+0000","last_became_active":"2024-04-21T08:37:30.420988+0000","last_became_peered":"2024-04-21T08:37:30.420988+0000","last_unstale":"2024-04-21T08:39:58.890961+0000","last_undegraded":"2024-04-21T08:39:58.890961+0000","last_fullsized":"2024-04-21T08:39:58.890961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:38:03.850705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":258,"num_read_kb":198,"num_write":42,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009331+0000","last_change":"2024-04-21T08:37:31.447208+0000","last_active":"2024-04-21T08:39:58.009331+0000","last_peered":"2024-04-21T08:39:58.009331+0000","last_clean":"2024-04-21T08:39:58.009331+0000","last_became_active":"2024-04-21T08:37:31.444464+0000","last_became_peered":"2024-04-21T08:37:31.444464+0000","last_unstale":"2024-04-21T08:39:58.009331+0000","last_undegraded":"2024-04-21T08:39:58.009331+0000","last_fullsized":"2024-04-21T08:39:58.009331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:33.805676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014409+0000","last_change":"2024-04-21T08:37:35.603937+0000","last_active":"2024-04-21T08:39:58.014409+0000","last_peered":"2024-04-21T08:39:58.014409+0000","last_clean":"2024-04-21T08:39:58.014409+0000","last_became_active":"2024-04-21T08:37:35.603477+0000","last_became_peered":"2024-04-21T08:37:35.603477+0000","last_unstale":"2024-04-21T08:39:58.014409+0000","last_undegraded":"2024-04-21T08:39:58.014409+0000","last_fullsized":"2024-04-21T08:39:58.014409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:40:33.901570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905391+0000","last_change":"2024-04-21T08:37:30.416941+0000","last_active":"2024-04-21T08:39:57.905391+0000","last_peered":"2024-04-21T08:39:57.905391+0000","last_clean":"2024-04-21T08:39:57.905391+0000","last_became_active":"2024-04-21T08:37:30.416625+0000","last_became_peered":"2024-04-21T08:37:30.416625+0000","last_unstale":"2024-04-21T08:39:57.905391+0000","last_undegraded":"2024-04-21T08:39:57.905391+0000","last_fullsized":"2024-04-21T08:39:57.905391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:07.782779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318733+0000","last_change":"2024-04-21T08:37:31.441707+0000","last_active":"2024-04-21T08:39:58.318733+0000","last_peered":"2024-04-21T08:39:58.318733+0000","last_clean":"2024-04-21T08:39:58.318733+0000","last_became_active":"2024-04-21T08:37:31.441366+0000","last_became_peered":"2024-04-21T08:37:31.441366+0000","last_unstale":"2024-04-21T08:39:58.318733+0000","last_undegraded":"2024-04-21T08:39:58.318733+0000","last_fullsized":"2024-04-21T08:39:58.318733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:16.573804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318906+0000","last_change":"2024-04-21T08:37:35.596214+0000","last_active":"2024-04-21T08:39:58.318906+0000","last_peered":"2024-04-21T08:39:58.318906+0000","last_clean":"2024-04-21T08:39:58.318906+0000","last_became_active":"2024-04-21T08:37:35.595773+0000","last_became_peered":"2024-04-21T08:37:35.595773+0000","last_unstale":"2024-04-21T08:39:58.318906+0000","last_undegraded":"2024-04-21T08:39:58.318906+0000","last_fullsized":"2024-04-21T08:39:58.318906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:07.538190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318273+0000","last_change":"2024-04-21T08:38:30.684828+0000","last_active":"2024-04-21T08:39:58.318273+0000","last_peered":"2024-04-21T08:39:58.318273+0000","last_clean":"2024-04-21T08:39:58.318273+0000","last_became_active":"2024-04-21T08:38:30.673923+0000","last_became_peered":"2024-04-21T08:38:30.673923+0000","last_unstale":"2024-04-21T08:39:58.318273+0000","last_undegraded":"2024-04-21T08:39:58.318273+0000","last_fullsized":"2024-04-21T08:39:58.318273+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:51:37.906150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905472+0000","last_change":"2024-04-21T08:37:31.453680+0000","last_active":"2024-04-21T08:39:57.905472+0000","last_peered":"2024-04-21T08:39:57.905472+0000","last_clean":"2024-04-21T08:39:57.905472+0000","last_became_active":"2024-04-21T08:37:31.453319+0000","last_became_peered":"2024-04-21T08:37:31.453319+0000","last_unstale":"2024-04-21T08:39:57.905472+0000","last_undegraded":"2024-04-21T08:39:57.905472+0000","last_fullsized":"2024-04-21T08:39:57.905472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:43.139789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006992+0000","last_change":"2024-04-21T08:37:35.608906+0000","last_active":"2024-04-21T08:39:58.006992+0000","last_peered":"2024-04-21T08:39:58.006992+0000","last_clean":"2024-04-21T08:39:58.006992+0000","last_became_active":"2024-04-21T08:37:35.608535+0000","last_became_peered":"2024-04-21T08:37:35.608535+0000","last_unstale":"2024-04-21T08:39:58.006992+0000","last_undegraded":"2024-04-21T08:39:58.006992+0000","last_fullsized":"2024-04-21T08:39:58.006992+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:58:01.422308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":171,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.890194+0000","last_change":"2024-04-21T08:37:30.414865+0000","last_active":"2024-04-21T08:39:58.890194+0000","last_peered":"2024-04-21T08:39:58.890194+0000","last_clean":"2024-04-21T08:39:58.890194+0000","last_became_active":"2024-04-21T08:37:30.414471+0000","last_became_peered":"2024-04-21T08:37:30.414471+0000","last_unstale":"2024-04-21T08:39:58.890194+0000","last_undegraded":"2024-04-21T08:39:58.890194+0000","last_fullsized":"2024-04-21T08:39:58.890194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:46.943574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905326+0000","last_change":"2024-04-21T08:37:31.456634+0000","last_active":"2024-04-21T08:39:57.905326+0000","last_peered":"2024-04-21T08:39:57.905326+0000","last_clean":"2024-04-21T08:39:57.905326+0000","last_became_active":"2024-04-21T08:37:31.456278+0000","last_became_peered":"2024-04-21T08:37:31.456278+0000","last_unstale":"2024-04-21T08:39:57.905326+0000","last_undegraded":"2024-04-21T08:39:57.905326+0000","last_fullsized":"2024-04-21T08:39:57.905326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:04.870024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319167+0000","last_change":"2024-04-21T08:37:35.601760+0000","last_active":"2024-04-21T08:39:58.319167+0000","last_peered":"2024-04-21T08:39:58.319167+0000","last_clean":"2024-04-21T08:39:58.319167+0000","last_became_active":"2024-04-21T08:37:35.601332+0000","last_became_peered":"2024-04-21T08:37:35.601332+0000","last_unstale":"2024-04-21T08:39:58.319167+0000","last_undegraded":"2024-04-21T08:39:58.319167+0000","last_fullsized":"2024-04-21T08:39:58.319167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:04:03.694581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008961+0000","last_change":"2024-04-21T08:37:30.422419+0000","last_active":"2024-04-21T08:39:58.008961+0000","last_peered":"2024-04-21T08:39:58.008961+0000","last_clean":"2024-04-21T08:39:58.008961+0000","last_became_active":"2024-04-21T08:37:30.422123+0000","last_became_peered":"2024-04-21T08:37:30.422123+0000","last_unstale":"2024-04-21T08:39:58.008961+0000","last_undegraded":"2024-04-21T08:39:58.008961+0000","last_fullsized":"2024-04-21T08:39:58.008961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:32:54.029041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013848+0000","last_change":"2024-04-21T08:37:31.458992+0000","last_active":"2024-04-21T08:39:58.013848+0000","last_peered":"2024-04-21T08:39:58.013848+0000","last_clean":"2024-04-21T08:39:58.013848+0000","last_became_active":"2024-04-21T08:37:31.458635+0000","last_became_peered":"2024-04-21T08:37:31.458635+0000","last_unstale":"2024-04-21T08:39:58.013848+0000","last_undegraded":"2024-04-21T08:39:58.013848+0000","last_fullsized":"2024-04-21T08:39:58.013848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:48:14.325571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013750+0000","last_change":"2024-04-21T08:37:35.603739+0000","last_active":"2024-04-21T08:39:58.013750+0000","last_peered":"2024-04-21T08:39:58.013750+0000","last_clean":"2024-04-21T08:39:58.013750+0000","last_became_active":"2024-04-21T08:37:35.603310+0000","last_became_peered":"2024-04-21T08:37:35.603310+0000","last_unstale":"2024-04-21T08:39:58.013750+0000","last_undegraded":"2024-04-21T08:39:58.013750+0000","last_fullsized":"2024-04-21T08:39:58.013750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:19.162520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009465+0000","last_change":"2024-04-21T08:37:30.418104+0000","last_active":"2024-04-21T08:39:58.009465+0000","last_peered":"2024-04-21T08:39:58.009465+0000","last_clean":"2024-04-21T08:39:58.009465+0000","last_became_active":"2024-04-21T08:37:30.417814+0000","last_became_peered":"2024-04-21T08:37:30.417814+0000","last_unstale":"2024-04-21T08:39:58.009465+0000","last_undegraded":"2024-04-21T08:39:58.009465+0000","last_fullsized":"2024-04-21T08:39:58.009465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:50:56.696513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318582+0000","last_change":"2024-04-21T08:37:31.451952+0000","last_active":"2024-04-21T08:39:58.318582+0000","last_peered":"2024-04-21T08:39:58.318582+0000","last_clean":"2024-04-21T08:39:58.318582+0000","last_became_active":"2024-04-21T08:37:31.451588+0000","last_became_peered":"2024-04-21T08:37:31.451588+0000","last_unstale":"2024-04-21T08:39:58.318582+0000","last_undegraded":"2024-04-21T08:39:58.318582+0000","last_fullsized":"2024-04-21T08:39:58.318582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:06.506769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009071+0000","last_change":"2024-04-21T08:37:35.760661+0000","last_active":"2024-04-21T08:39:58.009071+0000","last_peered":"2024-04-21T08:39:58.009071+0000","last_clean":"2024-04-21T08:39:58.009071+0000","last_became_active":"2024-04-21T08:37:35.759925+0000","last_became_peered":"2024-04-21T08:37:35.759925+0000","last_unstale":"2024-04-21T08:39:58.009071+0000","last_undegraded":"2024-04-21T08:39:58.009071+0000","last_fullsized":"2024-04-21T08:39:58.009071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:11:53.062251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007012+0000","last_change":"2024-04-21T08:37:30.416487+0000","last_active":"2024-04-21T08:39:58.007012+0000","last_peered":"2024-04-21T08:39:58.007012+0000","last_clean":"2024-04-21T08:39:58.007012+0000","last_became_active":"2024-04-21T08:37:30.416076+0000","last_became_peered":"2024-04-21T08:37:30.416076+0000","last_unstale":"2024-04-21T08:39:58.007012+0000","last_undegraded":"2024-04-21T08:39:58.007012+0000","last_fullsized":"2024-04-21T08:39:58.007012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:18:33.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017205+0000","last_change":"2024-04-21T08:37:31.448824+0000","last_active":"2024-04-21T08:39:58.017205+0000","last_peered":"2024-04-21T08:39:58.017205+0000","last_clean":"2024-04-21T08:39:58.017205+0000","last_became_active":"2024-04-21T08:37:31.448432+0000","last_became_peered":"2024-04-21T08:37:31.448432+0000","last_unstale":"2024-04-21T08:39:58.017205+0000","last_undegraded":"2024-04-21T08:39:58.017205+0000","last_fullsized":"2024-04-21T08:39:58.017205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:41.111272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318815+0000","last_change":"2024-04-21T08:37:35.604539+0000","last_active":"2024-04-21T08:39:58.318815+0000","last_peered":"2024-04-21T08:39:58.318815+0000","last_clean":"2024-04-21T08:39:58.318815+0000","last_became_active":"2024-04-21T08:37:35.604132+0000","last_became_peered":"2024-04-21T08:37:35.604132+0000","last_unstale":"2024-04-21T08:39:58.318815+0000","last_undegraded":"2024-04-21T08:39:58.318815+0000","last_fullsized":"2024-04-21T08:39:58.318815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:12.180730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905085+0000","last_change":"2024-04-21T08:37:30.418854+0000","last_active":"2024-04-21T08:39:57.905085+0000","last_peered":"2024-04-21T08:39:57.905085+0000","last_clean":"2024-04-21T08:39:57.905085+0000","last_became_active":"2024-04-21T08:37:30.418511+0000","last_became_peered":"2024-04-21T08:37:30.418511+0000","last_unstale":"2024-04-21T08:39:57.905085+0000","last_undegraded":"2024-04-21T08:39:57.905085+0000","last_fullsized":"2024-04-21T08:39:57.905085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:26.751924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.016918+0000","last_change":"2024-04-21T08:37:31.453365+0000","last_active":"2024-04-21T08:39:58.016918+0000","last_peered":"2024-04-21T08:39:58.016918+0000","last_clean":"2024-04-21T08:39:58.016918+0000","last_became_active":"2024-04-21T08:37:31.453042+0000","last_became_peered":"2024-04-21T08:37:31.453042+0000","last_unstale":"2024-04-21T08:39:58.016918+0000","last_undegraded":"2024-04-21T08:39:58.016918+0000","last_fullsized":"2024-04-21T08:39:58.016918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:49:46.648754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318806+0000","last_change":"2024-04-21T08:37:35.760626+0000","last_active":"2024-04-21T08:39:58.318806+0000","last_peered":"2024-04-21T08:39:58.318806+0000","last_clean":"2024-04-21T08:39:58.318806+0000","last_became_active":"2024-04-21T08:37:35.760125+0000","last_became_peered":"2024-04-21T08:37:35.760125+0000","last_unstale":"2024-04-21T08:39:58.318806+0000","last_undegraded":"2024-04-21T08:39:58.318806+0000","last_fullsized":"2024-04-21T08:39:58.318806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:58.366557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013932+0000","last_change":"2024-04-21T08:37:31.457654+0000","last_active":"2024-04-21T08:39:58.013932+0000","last_peered":"2024-04-21T08:39:58.013932+0000","last_clean":"2024-04-21T08:39:58.013932+0000","last_became_active":"2024-04-21T08:37:31.456950+0000","last_became_peered":"2024-04-21T08:37:31.456950+0000","last_unstale":"2024-04-21T08:39:58.013932+0000","last_undegraded":"2024-04-21T08:39:58.013932+0000","last_fullsized":"2024-04-21T08:39:58.013932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:02.863000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318817+0000","last_change":"2024-04-21T08:37:30.422021+0000","last_active":"2024-04-21T08:39:58.318817+0000","last_peered":"2024-04-21T08:39:58.318817+0000","last_clean":"2024-04-21T08:39:58.318817+0000","last_became_active":"2024-04-21T08:37:30.421711+0000","last_became_peered":"2024-04-21T08:37:30.421711+0000","last_unstale":"2024-04-21T08:39:58.318817+0000","last_undegraded":"2024-04-21T08:39:58.318817+0000","last_fullsized":"2024-04-21T08:39:58.318817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:01.612205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905863+0000","last_change":"2024-04-21T08:37:35.598041+0000","last_active":"2024-04-21T08:39:57.905863+0000","last_peered":"2024-04-21T08:39:57.905863+0000","last_clean":"2024-04-21T08:39:57.905863+0000","last_became_active":"2024-04-21T08:37:35.597635+0000","last_became_peered":"2024-04-21T08:37:35.597635+0000","last_unstale":"2024-04-21T08:39:57.905863+0000","last_undegraded":"2024-04-21T08:39:57.905863+0000","last_fullsized":"2024-04-21T08:39:57.905863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:35:27.452340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903604+0000","last_change":"2024-04-21T08:37:31.450423+0000","last_active":"2024-04-21T08:39:57.903604+0000","last_peered":"2024-04-21T08:39:57.903604+0000","last_clean":"2024-04-21T08:39:57.903604+0000","last_became_active":"2024-04-21T08:37:31.450112+0000","last_became_peered":"2024-04-21T08:37:31.450112+0000","last_unstale":"2024-04-21T08:39:57.903604+0000","last_undegraded":"2024-04-21T08:39:57.903604+0000","last_fullsized":"2024-04-21T08:39:57.903604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:10:47.006769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151568+0000","last_change":"2024-04-21T08:37:30.416519+0000","last_active":"2024-04-21T08:39:58.151568+0000","last_peered":"2024-04-21T08:39:58.151568+0000","last_clean":"2024-04-21T08:39:58.151568+0000","last_became_active":"2024-04-21T08:37:30.416131+0000","last_became_peered":"2024-04-21T08:37:30.416131+0000","last_unstale":"2024-04-21T08:39:58.151568+0000","last_undegraded":"2024-04-21T08:39:58.151568+0000","last_fullsized":"2024-04-21T08:39:58.151568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:12.280847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":368,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904918+0000","last_change":"2024-04-21T08:36:20.257759+0000","last_active":"2024-04-21T08:39:57.904918+0000","last_peered":"2024-04-21T08:39:57.904918+0000","last_clean":"2024-04-21T08:39:57.904918+0000","last_became_active":"2024-04-21T08:36:20.257313+0000","last_became_peered":"2024-04-21T08:36:20.257313+0000","last_unstale":"2024-04-21T08:39:57.904918+0000","last_undegraded":"2024-04-21T08:39:57.904918+0000","last_fullsized":"2024-04-21T08:39:57.904918+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_clean_scrub_stamp":"2024-04-21T08:32:56.546295+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:42:44.393447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014207+0000","last_change":"2024-04-21T08:37:35.761562+0000","last_active":"2024-04-21T08:39:58.014207+0000","last_peered":"2024-04-21T08:39:58.014207+0000","last_clean":"2024-04-21T08:39:58.014207+0000","last_became_active":"2024-04-21T08:37:35.760703+0000","last_became_peered":"2024-04-21T08:37:35.760703+0000","last_unstale":"2024-04-21T08:39:58.014207+0000","last_undegraded":"2024-04-21T08:39:58.014207+0000","last_fullsized":"2024-04-21T08:39:58.014207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:12:47.511117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318506+0000","last_change":"2024-04-21T08:39:31.678868+0000","last_active":"2024-04-21T08:39:58.318506+0000","last_peered":"2024-04-21T08:39:58.318506+0000","last_clean":"2024-04-21T08:39:58.318506+0000","last_became_active":"2024-04-21T08:39:31.678457+0000","last_became_peered":"2024-04-21T08:39:31.678457+0000","last_unstale":"2024-04-21T08:39:58.318506+0000","last_undegraded":"2024-04-21T08:39:58.318506+0000","last_fullsized":"2024-04-21T08:39:58.318506+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:49.918106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905042+0000","last_change":"2024-04-21T08:37:30.420560+0000","last_active":"2024-04-21T08:39:57.905042+0000","last_peered":"2024-04-21T08:39:57.905042+0000","last_clean":"2024-04-21T08:39:57.905042+0000","last_became_active":"2024-04-21T08:37:30.420223+0000","last_became_peered":"2024-04-21T08:37:30.420223+0000","last_unstale":"2024-04-21T08:39:57.905042+0000","last_undegraded":"2024-04-21T08:39:57.905042+0000","last_fullsized":"2024-04-21T08:39:57.905042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:43:55.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318381+0000","last_change":"2024-04-21T08:37:35.598294+0000","last_active":"2024-04-21T08:39:58.318381+0000","last_peered":"2024-04-21T08:39:58.318381+0000","last_clean":"2024-04-21T08:39:58.318381+0000","last_became_active":"2024-04-21T08:37:35.597883+0000","last_became_peered":"2024-04-21T08:37:35.597883+0000","last_unstale":"2024-04-21T08:39:58.318381+0000","last_undegraded":"2024-04-21T08:39:58.318381+0000","last_fullsized":"2024-04-21T08:39:58.318381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:24:05.144763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009321+0000","last_change":"2024-04-21T08:37:30.423182+0000","last_active":"2024-04-21T08:39:58.009321+0000","last_peered":"2024-04-21T08:39:58.009321+0000","last_clean":"2024-04-21T08:39:58.009321+0000","last_became_active":"2024-04-21T08:37:30.422792+0000","last_became_peered":"2024-04-21T08:37:30.422792+0000","last_unstale":"2024-04-21T08:39:58.009321+0000","last_undegraded":"2024-04-21T08:39:58.009321+0000","last_fullsized":"2024-04-21T08:39:58.009321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:44:00.727269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318924+0000","last_change":"2024-04-21T08:37:31.444013+0000","last_active":"2024-04-21T08:39:58.318924+0000","last_peered":"2024-04-21T08:39:58.318924+0000","last_clean":"2024-04-21T08:39:58.318924+0000","last_became_active":"2024-04-21T08:37:31.443684+0000","last_became_peered":"2024-04-21T08:37:31.443684+0000","last_unstale":"2024-04-21T08:39:58.318924+0000","last_undegraded":"2024-04-21T08:39:58.318924+0000","last_fullsized":"2024-04-21T08:39:58.318924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:00:35.400818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903525+0000","last_change":"2024-04-21T08:37:35.759817+0000","last_active":"2024-04-21T08:39:57.903525+0000","last_peered":"2024-04-21T08:39:57.903525+0000","last_clean":"2024-04-21T08:39:57.903525+0000","last_became_active":"2024-04-21T08:37:35.759396+0000","last_became_peered":"2024-04-21T08:37:35.759396+0000","last_unstale":"2024-04-21T08:39:57.903525+0000","last_undegraded":"2024-04-21T08:39:57.903525+0000","last_fullsized":"2024-04-21T08:39:57.903525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:20:36.170880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'44","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.114287+0000","last_change":"2024-04-21T08:37:30.414102+0000","last_active":"2024-04-21T08:39:58.114287+0000","last_peered":"2024-04-21T08:39:58.114287+0000","last_clean":"2024-04-21T08:39:58.114287+0000","last_became_active":"2024-04-21T08:37:30.413670+0000","last_became_peered":"2024-04-21T08:37:30.413670+0000","last_unstale":"2024-04-21T08:39:58.114287+0000","last_undegraded":"2024-04-21T08:39:58.114287+0000","last_fullsized":"2024-04-21T08:39:58.114287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:12:49.696516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904320+0000","last_change":"2024-04-21T08:37:31.457451+0000","last_active":"2024-04-21T08:39:57.904320+0000","last_peered":"2024-04-21T08:39:57.904320+0000","last_clean":"2024-04-21T08:39:57.904320+0000","last_became_active":"2024-04-21T08:37:31.457120+0000","last_became_peered":"2024-04-21T08:37:31.457120+0000","last_unstale":"2024-04-21T08:39:57.904320+0000","last_undegraded":"2024-04-21T08:39:57.904320+0000","last_fullsized":"2024-04-21T08:39:57.904320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:07:48.025843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904850+0000","last_change":"2024-04-21T08:37:35.600815+0000","last_active":"2024-04-21T08:39:57.904850+0000","last_peered":"2024-04-21T08:39:57.904850+0000","last_clean":"2024-04-21T08:39:57.904850+0000","last_became_active":"2024-04-21T08:37:35.600380+0000","last_became_peered":"2024-04-21T08:37:35.600380+0000","last_unstale":"2024-04-21T08:39:57.904850+0000","last_undegraded":"2024-04-21T08:39:57.904850+0000","last_fullsized":"2024-04-21T08:39:57.904850+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:19:33.710431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151729+0000","last_change":"2024-04-21T08:37:30.425592+0000","last_active":"2024-04-21T08:39:58.151729+0000","last_peered":"2024-04-21T08:39:58.151729+0000","last_clean":"2024-04-21T08:39:58.151729+0000","last_became_active":"2024-04-21T08:37:30.425274+0000","last_became_peered":"2024-04-21T08:37:30.425274+0000","last_unstale":"2024-04-21T08:39:58.151729+0000","last_undegraded":"2024-04-21T08:39:58.151729+0000","last_fullsized":"2024-04-21T08:39:58.151729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:27:29.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319672+0000","last_change":"2024-04-21T08:39:31.680028+0000","last_active":"2024-04-21T08:39:58.319672+0000","last_peered":"2024-04-21T08:39:58.319672+0000","last_clean":"2024-04-21T08:39:58.319672+0000","last_became_active":"2024-04-21T08:39:31.679162+0000","last_became_peered":"2024-04-21T08:39:31.679162+0000","last_unstale":"2024-04-21T08:39:58.319672+0000","last_undegraded":"2024-04-21T08:39:58.319672+0000","last_fullsized":"2024-04-21T08:39:58.319672+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:32.824411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151564+0000","last_change":"2024-04-21T08:38:30.713521+0000","last_active":"2024-04-21T08:39:58.151564+0000","last_peered":"2024-04-21T08:39:58.151564+0000","last_clean":"2024-04-21T08:39:58.151564+0000","last_became_active":"2024-04-21T08:38:30.713096+0000","last_became_peered":"2024-04-21T08:38:30.713096+0000","last_unstale":"2024-04-21T08:39:58.151564+0000","last_undegraded":"2024-04-21T08:39:58.151564+0000","last_fullsized":"2024-04-21T08:39:58.151564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:54:59.078776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153818+0000","last_change":"2024-04-21T08:37:30.416696+0000","last_active":"2024-04-21T08:39:58.153818+0000","last_peered":"2024-04-21T08:39:58.153818+0000","last_clean":"2024-04-21T08:39:58.153818+0000","last_became_active":"2024-04-21T08:37:30.415868+0000","last_became_peered":"2024-04-21T08:37:30.415868+0000","last_unstale":"2024-04-21T08:39:58.153818+0000","last_undegraded":"2024-04-21T08:39:58.153818+0000","last_fullsized":"2024-04-21T08:39:58.153818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:48:08.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008945+0000","last_change":"2024-04-21T08:37:31.453861+0000","last_active":"2024-04-21T08:39:58.008945+0000","last_peered":"2024-04-21T08:39:58.008945+0000","last_clean":"2024-04-21T08:39:58.008945+0000","last_became_active":"2024-04-21T08:37:31.453562+0000","last_became_peered":"2024-04-21T08:37:31.453562+0000","last_unstale":"2024-04-21T08:39:58.008945+0000","last_undegraded":"2024-04-21T08:39:58.008945+0000","last_fullsized":"2024-04-21T08:39:58.008945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:17:50.513917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006977+0000","last_change":"2024-04-21T08:37:35.609217+0000","last_active":"2024-04-21T08:39:58.006977+0000","last_peered":"2024-04-21T08:39:58.006977+0000","last_clean":"2024-04-21T08:39:58.006977+0000","last_became_active":"2024-04-21T08:37:35.608786+0000","last_became_peered":"2024-04-21T08:37:35.608786+0000","last_unstale":"2024-04-21T08:39:58.006977+0000","last_undegraded":"2024-04-21T08:39:58.006977+0000","last_fullsized":"2024-04-21T08:39:58.006977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:45:50.945845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903818+0000","last_change":"2024-04-21T08:37:30.417156+0000","last_active":"2024-04-21T08:39:57.903818+0000","last_peered":"2024-04-21T08:39:57.903818+0000","last_clean":"2024-04-21T08:39:57.903818+0000","last_became_active":"2024-04-21T08:37:30.415717+0000","last_became_peered":"2024-04-21T08:37:30.415717+0000","last_unstale":"2024-04-21T08:39:57.903818+0000","last_undegraded":"2024-04-21T08:39:57.903818+0000","last_fullsized":"2024-04-21T08:39:57.903818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:37.819082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318260+0000","last_change":"2024-04-21T08:37:31.444948+0000","last_active":"2024-04-21T08:39:58.318260+0000","last_peered":"2024-04-21T08:39:58.318260+0000","last_clean":"2024-04-21T08:39:58.318260+0000","last_became_active":"2024-04-21T08:37:31.444604+0000","last_became_peered":"2024-04-21T08:37:31.444604+0000","last_unstale":"2024-04-21T08:39:58.318260+0000","last_undegraded":"2024-04-21T08:39:58.318260+0000","last_fullsized":"2024-04-21T08:39:58.318260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.820503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151386+0000","last_change":"2024-04-21T08:37:35.611541+0000","last_active":"2024-04-21T08:39:58.151386+0000","last_peered":"2024-04-21T08:39:58.151386+0000","last_clean":"2024-04-21T08:39:58.151386+0000","last_became_active":"2024-04-21T08:37:35.611136+0000","last_became_peered":"2024-04-21T08:37:35.611136+0000","last_unstale":"2024-04-21T08:39:58.151386+0000","last_undegraded":"2024-04-21T08:39:58.151386+0000","last_fullsized":"2024-04-21T08:39:58.151386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:28.254529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904963+0000","last_change":"2024-04-21T08:37:30.414216+0000","last_active":"2024-04-21T08:39:57.904963+0000","last_peered":"2024-04-21T08:39:57.904963+0000","last_clean":"2024-04-21T08:39:57.904963+0000","last_became_active":"2024-04-21T08:37:30.413890+0000","last_became_peered":"2024-04-21T08:37:30.413890+0000","last_unstale":"2024-04-21T08:39:57.904963+0000","last_undegraded":"2024-04-21T08:39:57.904963+0000","last_fullsized":"2024-04-21T08:39:57.904963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:16.972618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903800+0000","last_change":"2024-04-21T08:37:31.449307+0000","last_active":"2024-04-21T08:39:57.903800+0000","last_peered":"2024-04-21T08:39:57.903800+0000","last_clean":"2024-04-21T08:39:57.903800+0000","last_became_active":"2024-04-21T08:37:31.448980+0000","last_became_peered":"2024-04-21T08:37:31.448980+0000","last_unstale":"2024-04-21T08:39:57.903800+0000","last_undegraded":"2024-04-21T08:39:57.903800+0000","last_fullsized":"2024-04-21T08:39:57.903800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:30:31.939274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151476+0000","last_change":"2024-04-21T08:37:35.760616+0000","last_active":"2024-04-21T08:39:58.151476+0000","last_peered":"2024-04-21T08:39:58.151476+0000","last_clean":"2024-04-21T08:39:58.151476+0000","last_became_active":"2024-04-21T08:37:35.760131+0000","last_became_peered":"2024-04-21T08:37:35.760131+0000","last_unstale":"2024-04-21T08:39:58.151476+0000","last_undegraded":"2024-04-21T08:39:58.151476+0000","last_fullsized":"2024-04-21T08:39:58.151476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:51.510259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009114+0000","last_change":"2024-04-21T08:38:30.682791+0000","last_active":"2024-04-21T08:39:58.009114+0000","last_peered":"2024-04-21T08:39:58.009114+0000","last_clean":"2024-04-21T08:39:58.009114+0000","last_became_active":"2024-04-21T08:38:30.671977+0000","last_became_peered":"2024-04-21T08:38:30.671977+0000","last_unstale":"2024-04-21T08:39:58.009114+0000","last_undegraded":"2024-04-21T08:39:58.009114+0000","last_fullsized":"2024-04-21T08:39:58.009114+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:55:15.899399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905569+0000","last_change":"2024-04-21T08:37:31.447234+0000","last_active":"2024-04-21T08:39:57.905569+0000","last_peered":"2024-04-21T08:39:57.905569+0000","last_clean":"2024-04-21T08:39:57.905569+0000","last_became_active":"2024-04-21T08:37:31.446876+0000","last_became_peered":"2024-04-21T08:37:31.446876+0000","last_unstale":"2024-04-21T08:39:57.905569+0000","last_undegraded":"2024-04-21T08:39:57.905569+0000","last_fullsized":"2024-04-21T08:39:57.905569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:51.784126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153981+0000","last_change":"2024-04-21T08:37:35.598577+0000","last_active":"2024-04-21T08:39:58.153981+0000","last_peered":"2024-04-21T08:39:58.153981+0000","last_clean":"2024-04-21T08:39:58.153981+0000","last_became_active":"2024-04-21T08:37:35.598119+0000","last_became_peered":"2024-04-21T08:37:35.598119+0000","last_unstale":"2024-04-21T08:39:58.153981+0000","last_undegraded":"2024-04-21T08:39:58.153981+0000","last_fullsized":"2024-04-21T08:39:58.153981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:41.986674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905332+0000","last_change":"2024-04-21T08:37:30.425737+0000","last_active":"2024-04-21T08:39:57.905332+0000","last_peered":"2024-04-21T08:39:57.905332+0000","last_clean":"2024-04-21T08:39:57.905332+0000","last_became_active":"2024-04-21T08:37:30.425180+0000","last_became_peered":"2024-04-21T08:37:30.425180+0000","last_unstale":"2024-04-21T08:39:57.905332+0000","last_undegraded":"2024-04-21T08:39:57.905332+0000","last_fullsized":"2024-04-21T08:39:57.905332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:08.646106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.906011+0000","last_change":"2024-04-21T08:37:31.451505+0000","last_active":"2024-04-21T08:39:57.906011+0000","last_peered":"2024-04-21T08:39:57.906011+0000","last_clean":"2024-04-21T08:39:57.906011+0000","last_became_active":"2024-04-21T08:37:31.451146+0000","last_became_peered":"2024-04-21T08:37:31.451146+0000","last_unstale":"2024-04-21T08:39:57.906011+0000","last_undegraded":"2024-04-21T08:39:57.906011+0000","last_fullsized":"2024-04-21T08:39:57.906011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:27:01.793910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319583+0000","last_change":"2024-04-21T08:37:35.608995+0000","last_active":"2024-04-21T08:39:58.319583+0000","last_peered":"2024-04-21T08:39:58.319583+0000","last_clean":"2024-04-21T08:39:58.319583+0000","last_became_active":"2024-04-21T08:37:35.608587+0000","last_became_peered":"2024-04-21T08:37:35.608587+0000","last_unstale":"2024-04-21T08:39:58.319583+0000","last_undegraded":"2024-04-21T08:39:58.319583+0000","last_fullsized":"2024-04-21T08:39:58.319583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.829418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.112239+0000","last_change":"2024-04-21T08:37:30.419026+0000","last_active":"2024-04-21T08:39:58.112239+0000","last_peered":"2024-04-21T08:39:58.112239+0000","last_clean":"2024-04-21T08:39:58.112239+0000","last_became_active":"2024-04-21T08:37:30.418706+0000","last_became_peered":"2024-04-21T08:37:30.418706+0000","last_unstale":"2024-04-21T08:39:58.112239+0000","last_undegraded":"2024-04-21T08:39:58.112239+0000","last_fullsized":"2024-04-21T08:39:58.112239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:24.441918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905192+0000","last_change":"2024-04-21T08:37:31.446561+0000","last_active":"2024-04-21T08:39:57.905192+0000","last_peered":"2024-04-21T08:39:57.905192+0000","last_clean":"2024-04-21T08:39:57.905192+0000","last_became_active":"2024-04-21T08:37:31.446162+0000","last_became_peered":"2024-04-21T08:37:31.446162+0000","last_unstale":"2024-04-21T08:39:57.905192+0000","last_undegraded":"2024-04-21T08:39:57.905192+0000","last_fullsized":"2024-04-21T08:39:57.905192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:38:24.684537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013588+0000","last_change":"2024-04-21T08:37:35.598748+0000","last_active":"2024-04-21T08:39:58.013588+0000","last_peered":"2024-04-21T08:39:58.013588+0000","last_clean":"2024-04-21T08:39:58.013588+0000","last_became_active":"2024-04-21T08:37:35.598353+0000","last_became_peered":"2024-04-21T08:37:35.598353+0000","last_unstale":"2024-04-21T08:39:58.013588+0000","last_undegraded":"2024-04-21T08:39:58.013588+0000","last_fullsized":"2024-04-21T08:39:58.013588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:47:36.519694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319054+0000","last_change":"2024-04-21T08:38:30.691292+0000","last_active":"2024-04-21T08:39:58.319054+0000","last_peered":"2024-04-21T08:39:58.319054+0000","last_clean":"2024-04-21T08:39:58.319054+0000","last_became_active":"2024-04-21T08:38:30.676580+0000","last_became_peered":"2024-04-21T08:38:30.676580+0000","last_unstale":"2024-04-21T08:39:58.319054+0000","last_undegraded":"2024-04-21T08:39:58.319054+0000","last_fullsized":"2024-04-21T08:39:58.319054+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:43:53.150369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905478+0000","last_change":"2024-04-21T08:37:31.452367+0000","last_active":"2024-04-21T08:39:57.905478+0000","last_peered":"2024-04-21T08:39:57.905478+0000","last_clean":"2024-04-21T08:39:57.905478+0000","last_became_active":"2024-04-21T08:37:31.452031+0000","last_became_peered":"2024-04-21T08:37:31.452031+0000","last_unstale":"2024-04-21T08:39:57.905478+0000","last_undegraded":"2024-04-21T08:39:57.905478+0000","last_fullsized":"2024-04-21T08:39:57.905478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:58.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318378+0000","last_change":"2024-04-21T08:37:31.451222+0000","last_active":"2024-04-21T08:39:58.318378+0000","last_peered":"2024-04-21T08:39:58.318378+0000","last_clean":"2024-04-21T08:39:58.318378+0000","last_became_active":"2024-04-21T08:37:31.450869+0000","last_became_peered":"2024-04-21T08:37:31.450869+0000","last_unstale":"2024-04-21T08:39:58.318378+0000","last_undegraded":"2024-04-21T08:39:58.318378+0000","last_fullsized":"2024-04-21T08:39:58.318378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:17.051330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905137+0000","last_change":"2024-04-21T08:37:31.453415+0000","last_active":"2024-04-21T08:39:57.905137+0000","last_peered":"2024-04-21T08:39:57.905137+0000","last_clean":"2024-04-21T08:39:57.905137+0000","last_became_active":"2024-04-21T08:37:31.453089+0000","last_became_peered":"2024-04-21T08:37:31.453089+0000","last_unstale":"2024-04-21T08:39:57.905137+0000","last_undegraded":"2024-04-21T08:39:57.905137+0000","last_fullsized":"2024-04-21T08:39:57.905137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:25:44.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904904+0000","last_change":"2024-04-21T08:37:31.446707+0000","last_active":"2024-04-21T08:39:57.904904+0000","last_peered":"2024-04-21T08:39:57.904904+0000","last_clean":"2024-04-21T08:39:57.904904+0000","last_became_active":"2024-04-21T08:37:31.446340+0000","last_became_peered":"2024-04-21T08:37:31.446340+0000","last_unstale":"2024-04-21T08:39:57.904904+0000","last_undegraded":"2024-04-21T08:39:57.904904+0000","last_fullsized":"2024-04-21T08:39:57.904904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:13.649607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319500+0000","last_change":"2024-04-21T08:37:31.455532+0000","last_active":"2024-04-21T08:39:58.319500+0000","last_peered":"2024-04-21T08:39:58.319500+0000","last_clean":"2024-04-21T08:39:58.319500+0000","last_became_active":"2024-04-21T08:37:31.453547+0000","last_became_peered":"2024-04-21T08:37:31.453547+0000","last_unstale":"2024-04-21T08:39:58.319500+0000","last_undegraded":"2024-04-21T08:39:58.319500+0000","last_fullsized":"2024-04-21T08:39:58.319500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:39:47.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318944+0000","last_change":"2024-04-21T08:39:31.679882+0000","last_active":"2024-04-21T08:39:58.318944+0000","last_peered":"2024-04-21T08:39:58.318944+0000","last_clean":"2024-04-21T08:39:58.318944+0000","last_became_active":"2024-04-21T08:39:31.679440+0000","last_became_peered":"2024-04-21T08:39:31.679440+0000","last_unstale":"2024-04-21T08:39:58.318944+0000","last_undegraded":"2024-04-21T08:39:58.318944+0000","last_fullsized":"2024-04-21T08:39:58.318944+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:57.679038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318685+0000","last_change":"2024-04-21T08:37:31.451806+0000","last_active":"2024-04-21T08:39:58.318685+0000","last_peered":"2024-04-21T08:39:58.318685+0000","last_clean":"2024-04-21T08:39:58.318685+0000","last_became_active":"2024-04-21T08:37:31.451442+0000","last_became_peered":"2024-04-21T08:37:31.451442+0000","last_unstale":"2024-04-21T08:39:58.318685+0000","last_undegraded":"2024-04-21T08:39:58.318685+0000","last_fullsized":"2024-04-21T08:39:58.318685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:07.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905544+0000","last_change":"2024-04-21T08:37:31.444440+0000","last_active":"2024-04-21T08:39:57.905544+0000","last_peered":"2024-04-21T08:39:57.905544+0000","last_clean":"2024-04-21T08:39:57.905544+0000","last_became_active":"2024-04-21T08:37:31.444079+0000","last_became_peered":"2024-04-21T08:37:31.444079+0000","last_unstale":"2024-04-21T08:39:57.905544+0000","last_undegraded":"2024-04-21T08:39:57.905544+0000","last_fullsized":"2024-04-21T08:39:57.905544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:13.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904413+0000","last_change":"2024-04-21T08:37:31.453698+0000","last_active":"2024-04-21T08:39:57.904413+0000","last_peered":"2024-04-21T08:39:57.904413+0000","last_clean":"2024-04-21T08:39:57.904413+0000","last_became_active":"2024-04-21T08:37:31.453379+0000","last_became_peered":"2024-04-21T08:37:31.453379+0000","last_unstale":"2024-04-21T08:39:57.904413+0000","last_undegraded":"2024-04-21T08:39:57.904413+0000","last_fullsized":"2024-04-21T08:39:57.904413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:46:29.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013777+0000","last_change":"2024-04-21T08:37:31.444480+0000","last_active":"2024-04-21T08:39:58.013777+0000","last_peered":"2024-04-21T08:39:58.013777+0000","last_clean":"2024-04-21T08:39:58.013777+0000","last_became_active":"2024-04-21T08:37:31.444030+0000","last_became_peered":"2024-04-21T08:37:31.444030+0000","last_unstale":"2024-04-21T08:39:58.013777+0000","last_undegraded":"2024-04-21T08:39:58.013777+0000","last_fullsized":"2024-04-21T08:39:58.013777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:15:05.668885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903935+0000","last_change":"2024-04-21T08:37:31.456602+0000","last_active":"2024-04-21T08:39:57.903935+0000","last_peered":"2024-04-21T08:39:57.903935+0000","last_clean":"2024-04-21T08:39:57.903935+0000","last_became_active":"2024-04-21T08:37:31.456286+0000","last_became_peered":"2024-04-21T08:37:31.456286+0000","last_unstale":"2024-04-21T08:39:57.903935+0000","last_undegraded":"2024-04-21T08:39:57.903935+0000","last_fullsized":"2024-04-21T08:39:57.903935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:48:12.462037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154176+0000","last_change":"2024-04-21T08:37:31.447008+0000","last_active":"2024-04-21T08:39:58.154176+0000","last_peered":"2024-04-21T08:39:58.154176+0000","last_clean":"2024-04-21T08:39:58.154176+0000","last_became_active":"2024-04-21T08:37:31.445192+0000","last_became_peered":"2024-04-21T08:37:31.445192+0000","last_unstale":"2024-04-21T08:39:58.154176+0000","last_undegraded":"2024-04-21T08:39:58.154176+0000","last_fullsized":"2024-04-21T08:39:58.154176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:08.025647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318698+0000","last_change":"2024-04-21T08:37:31.443288+0000","last_active":"2024-04-21T08:39:58.318698+0000","last_peered":"2024-04-21T08:39:58.318698+0000","last_clean":"2024-04-21T08:39:58.318698+0000","last_became_active":"2024-04-21T08:37:31.442586+0000","last_became_peered":"2024-04-21T08:37:31.442586+0000","last_unstale":"2024-04-21T08:39:58.318698+0000","last_undegraded":"2024-04-21T08:39:58.318698+0000","last_fullsized":"2024-04-21T08:39:58.318698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:01.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318501+0000","last_change":"2024-04-21T08:37:31.453899+0000","last_active":"2024-04-21T08:39:58.318501+0000","last_peered":"2024-04-21T08:39:58.318501+0000","last_clean":"2024-04-21T08:39:58.318501+0000","last_became_active":"2024-04-21T08:37:31.453527+0000","last_became_peered":"2024-04-21T08:37:31.453527+0000","last_unstale":"2024-04-21T08:39:58.318501+0000","last_undegraded":"2024-04-21T08:39:58.318501+0000","last_fullsized":"2024-04-21T08:39:58.318501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:23:18.429618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108246,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600116,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":864,"num_read_kb":545,"num_write":289,"num_write_kb":1443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1800348,"data_compressed":80889,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":529807,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32348,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712804,"statfs":{"total":95995035648,"available":95961911296,"internally_reserved":0,"allocated":1200128,"data_stored":1320365,"data_compressed":73597,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":927355,"data_compressed":48844,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":922597,"data_compressed":48844,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1127103,"data_compressed":65780,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1028096,"data_stored":525219,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":585028,"data_compressed":28357,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1052672,"data_stored":526607,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1122304,"data_stored":595765,"data_compressed":31359,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1050229,"data_compressed":55000,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":520740,"data_compressed":26696,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1126400,"data_stored":1252669,"data_compressed":71936,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70650,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71418,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133076,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393372,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":204868,"data_compressed":7817,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:40:03.296 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-21T08:40:03.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:40:03.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.429+0000 7f4a99515640 1 -- 172.21.15.94:0/1305321985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a94105e30 msgr2=0x7f4a94106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.429+0000 7f4a99515640 1 --2- 172.21.15.94:0/1305321985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a94105e30 0x7f4a94106210 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c009930 tx=0x7f4a8c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.429+0000 7f4a99515640 1 -- 172.21.15.94:0/1305321985 shutdown_connections 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.429+0000 7f4a99515640 1 --2- 172.21.15.94:0/1305321985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a94101b60 0x7f4a9410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.429+0000 7f4a99515640 1 --2- 172.21.15.94:0/1305321985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a941011a0 0x7f4a94101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.429+0000 7f4a99515640 1 --2- 172.21.15.94:0/1305321985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a94105e30 0x7f4a94106210 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.430+0000 7f4a99515640 1 -- 172.21.15.94:0/1305321985 >> 172.21.15.94:0/1305321985 conn(0x7f4a94077e30 msgr2=0x7f4a94078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.430+0000 7f4a99515640 1 -- 172.21.15.94:0/1305321985 shutdown_connections 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.430+0000 7f4a99515640 1 -- 172.21.15.94:0/1305321985 wait complete. 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.430+0000 7f4a99515640 1 Processor -- start 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.430+0000 7f4a99515640 1 -- start start 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.430+0000 7f4a99515640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a941011a0 0x7f4a94197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.430+0000 7f4a99515640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a94101b60 0x7f4a94197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.430+0000 7f4a99515640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a94105e30 0x7f4a9419bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a99515640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94111160 con 0x7f4a94105e30 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a99515640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94110fe0 con 0x7f4a94101b60 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a99515640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941112e0 con 0x7f4a941011a0 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a937fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a94105e30 0x7f4a9419bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a937fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a94105e30 0x7f4a9419bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58702/0 (socket says 172.21.15.94:58702) 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a937fe640 1 -- 172.21.15.94:0/475272807 learned_addr learned my addr 172.21.15.94:0/475272807 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a92ffd640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a941011a0 0x7f4a94197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a927fc640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a94101b60 0x7f4a94197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a937fe640 1 -- 172.21.15.94:0/475272807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a941011a0 msgr2=0x7f4a94197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a937fe640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a941011a0 0x7f4a94197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a937fe640 1 -- 172.21.15.94:0/475272807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a94101b60 msgr2=0x7f4a94197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a937fe640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a94101b60 0x7f4a94197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a937fe640 1 -- 172.21.15.94:0/475272807 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9419c6a0 con 0x7f4a94105e30 2024-04-21T08:40:03.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a92ffd640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a941011a0 0x7f4a94197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a937fe640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a94105e30 0x7f4a9419bf70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4a8400d750 tx=0x7f4a8400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a7bfff640 1 -- 172.21.15.94:0/475272807 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a84017070 con 0x7f4a94105e30 2024-04-21T08:40:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a7bfff640 1 -- 172.21.15.94:0/475272807 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a840054d0 con 0x7f4a94105e30 2024-04-21T08:40:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a99515640 1 -- 172.21.15.94:0/475272807 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9419c990 con 0x7f4a94105e30 2024-04-21T08:40:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a7bfff640 1 -- 172.21.15.94:0/475272807 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a84004c60 con 0x7f4a94105e30 2024-04-21T08:40:03.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.431+0000 7f4a99515640 1 -- 172.21.15.94:0/475272807 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a941a4220 con 0x7f4a94105e30 2024-04-21T08:40:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.432+0000 7f4a99515640 1 -- 172.21.15.94:0/475272807 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a94102570 con 0x7f4a94105e30 2024-04-21T08:40:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.433+0000 7f4a7bfff640 1 -- 172.21.15.94:0/475272807 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f4a84009bd0 con 0x7f4a94105e30 2024-04-21T08:40:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.434+0000 7f4a7bfff640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a7c0a3e40 0x7f4a7c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.434+0000 7f4a92ffd640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a7c0a3e40 0x7f4a7c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.434+0000 7f4a7bfff640 1 -- 172.21.15.94:0/475272807 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4a840d0900 con 0x7f4a94105e30 2024-04-21T08:40:03.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.435+0000 7f4a92ffd640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a7c0a3e40 0x7f4a7c0a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006010 tx=0x7f4a8c009610 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:03.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.439+0000 7f4a7bfff640 1 -- 172.21.15.94:0/475272807 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a8409c970 con 0x7f4a94105e30 2024-04-21T08:40:03.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:40:03.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:40:03.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:40:03.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:40:03.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2022328867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:03 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3638975160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2022328867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:03.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:03.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:03.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:03 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3638975160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:03.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.617+0000 7f4a99515640 1 -- 172.21.15.94:0/475272807 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4a94062390 con 0x7f4a7c0a3e40 2024-04-21T08:40:03.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.626+0000 7f4a7bfff640 1 -- 172.21.15.94:0/475272807 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339690 (secure 0 0 0) 0x7f4a94062390 con 0x7f4a7c0a3e40 2024-04-21T08:40:03.630 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-21T08:40:03.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 -- 172.21.15.94:0/475272807 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a7c0a3e40 msgr2=0x7f4a7c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a7c0a3e40 0x7f4a7c0a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006010 tx=0x7f4a8c009610 comp rx=0 tx=0).stop 2024-04-21T08:40:03.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 -- 172.21.15.94:0/475272807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a94105e30 msgr2=0x7f4a9419bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a94105e30 0x7f4a9419bf70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4a8400d750 tx=0x7f4a8400dc10 comp rx=0 tx=0).stop 2024-04-21T08:40:03.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 -- 172.21.15.94:0/475272807 shutdown_connections 2024-04-21T08:40:03.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a7c0a3e40 0x7f4a7c0a6300 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a94105e30 0x7f4a9419bf70 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a94101b60 0x7f4a94197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 --2- 172.21.15.94:0/475272807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a941011a0 0x7f4a94197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 -- 172.21.15.94:0/475272807 >> 172.21.15.94:0/475272807 conn(0x7f4a94077e30 msgr2=0x7f4a940feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:03.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 -- 172.21.15.94:0/475272807 shutdown_connections 2024-04-21T08:40:03.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.630+0000 7f4a99515640 1 -- 172.21.15.94:0/475272807 wait complete. 2024-04-21T08:40:03.641 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-21T08:40:01.915737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190718,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1000,"num_read_kb":664,"num_write":472,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378544,"kb_used_data":12704,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563280,"statfs":{"total":1151940427776,"available":1151552798720,"internally_reserved":0,"allocated":13008896,"data_stored":9883484,"data_compressed":539507,"data_compressed_allocated":729088,"data_compressed_original":4308992,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.986713"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318569+0000","last_change":"2024-04-21T08:37:31.448345+0000","last_active":"2024-04-21T08:39:58.318569+0000","last_peered":"2024-04-21T08:39:58.318569+0000","last_clean":"2024-04-21T08:39:58.318569+0000","last_became_active":"2024-04-21T08:37:31.447878+0000","last_became_peered":"2024-04-21T08:37:31.447878+0000","last_unstale":"2024-04-21T08:39:58.318569+0000","last_undegraded":"2024-04-21T08:39:58.318569+0000","last_fullsized":"2024-04-21T08:39:58.318569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:45.589165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319782+0000","last_change":"2024-04-21T08:37:31.452895+0000","last_active":"2024-04-21T08:39:58.319782+0000","last_peered":"2024-04-21T08:39:58.319782+0000","last_clean":"2024-04-21T08:39:58.319782+0000","last_became_active":"2024-04-21T08:37:31.452550+0000","last_became_peered":"2024-04-21T08:37:31.452550+0000","last_unstale":"2024-04-21T08:39:58.319782+0000","last_undegraded":"2024-04-21T08:39:58.319782+0000","last_fullsized":"2024-04-21T08:39:58.319782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:02.497802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319443+0000","last_change":"2024-04-21T08:37:31.447552+0000","last_active":"2024-04-21T08:39:58.319443+0000","last_peered":"2024-04-21T08:39:58.319443+0000","last_clean":"2024-04-21T08:39:58.319443+0000","last_became_active":"2024-04-21T08:37:31.446044+0000","last_became_peered":"2024-04-21T08:37:31.446044+0000","last_unstale":"2024-04-21T08:39:58.319443+0000","last_undegraded":"2024-04-21T08:39:58.319443+0000","last_fullsized":"2024-04-21T08:39:58.319443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:22.801977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151531+0000","last_change":"2024-04-21T08:37:31.452400+0000","last_active":"2024-04-21T08:39:58.151531+0000","last_peered":"2024-04-21T08:39:58.151531+0000","last_clean":"2024-04-21T08:39:58.151531+0000","last_became_active":"2024-04-21T08:37:31.451462+0000","last_became_peered":"2024-04-21T08:37:31.451462+0000","last_unstale":"2024-04-21T08:39:58.151531+0000","last_undegraded":"2024-04-21T08:39:58.151531+0000","last_fullsized":"2024-04-21T08:39:58.151531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:22.463481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009035+0000","last_change":"2024-04-21T08:37:31.448942+0000","last_active":"2024-04-21T08:39:58.009035+0000","last_peered":"2024-04-21T08:39:58.009035+0000","last_clean":"2024-04-21T08:39:58.009035+0000","last_became_active":"2024-04-21T08:37:31.448526+0000","last_became_peered":"2024-04-21T08:37:31.448526+0000","last_unstale":"2024-04-21T08:39:58.009035+0000","last_undegraded":"2024-04-21T08:39:58.009035+0000","last_fullsized":"2024-04-21T08:39:58.009035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:29.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017093+0000","last_change":"2024-04-21T08:37:31.452165+0000","last_active":"2024-04-21T08:39:58.017093+0000","last_peered":"2024-04-21T08:39:58.017093+0000","last_clean":"2024-04-21T08:39:58.017093+0000","last_became_active":"2024-04-21T08:37:31.451802+0000","last_became_peered":"2024-04-21T08:37:31.451802+0000","last_unstale":"2024-04-21T08:39:58.017093+0000","last_undegraded":"2024-04-21T08:39:58.017093+0000","last_fullsized":"2024-04-21T08:39:58.017093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:42.210544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904949+0000","last_change":"2024-04-21T08:39:31.681660+0000","last_active":"2024-04-21T08:39:57.904949+0000","last_peered":"2024-04-21T08:39:57.904949+0000","last_clean":"2024-04-21T08:39:57.904949+0000","last_became_active":"2024-04-21T08:39:31.681203+0000","last_became_peered":"2024-04-21T08:39:31.681203+0000","last_unstale":"2024-04-21T08:39:57.904949+0000","last_undegraded":"2024-04-21T08:39:57.904949+0000","last_fullsized":"2024-04-21T08:39:57.904949+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:54.120862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903472+0000","last_change":"2024-04-21T08:37:31.448257+0000","last_active":"2024-04-21T08:39:57.903472+0000","last_peered":"2024-04-21T08:39:57.903472+0000","last_clean":"2024-04-21T08:39:57.903472+0000","last_became_active":"2024-04-21T08:37:31.447845+0000","last_became_peered":"2024-04-21T08:37:31.447845+0000","last_unstale":"2024-04-21T08:39:57.903472+0000","last_undegraded":"2024-04-21T08:39:57.903472+0000","last_fullsized":"2024-04-21T08:39:57.903472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:19.956699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154785+0000","last_change":"2024-04-21T08:37:31.443022+0000","last_active":"2024-04-21T08:39:58.154785+0000","last_peered":"2024-04-21T08:39:58.154785+0000","last_clean":"2024-04-21T08:39:58.154785+0000","last_became_active":"2024-04-21T08:37:31.442645+0000","last_became_peered":"2024-04-21T08:37:31.442645+0000","last_unstale":"2024-04-21T08:39:58.154785+0000","last_undegraded":"2024-04-21T08:39:58.154785+0000","last_fullsized":"2024-04-21T08:39:58.154785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:18:10.861281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017434+0000","last_change":"2024-04-21T08:37:31.453427+0000","last_active":"2024-04-21T08:39:58.017434+0000","last_peered":"2024-04-21T08:39:58.017434+0000","last_clean":"2024-04-21T08:39:58.017434+0000","last_became_active":"2024-04-21T08:37:31.453096+0000","last_became_peered":"2024-04-21T08:37:31.453096+0000","last_unstale":"2024-04-21T08:39:58.017434+0000","last_undegraded":"2024-04-21T08:39:58.017434+0000","last_fullsized":"2024-04-21T08:39:58.017434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:49.450351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905929+0000","last_change":"2024-04-21T08:37:31.449908+0000","last_active":"2024-04-21T08:39:57.905929+0000","last_peered":"2024-04-21T08:39:57.905929+0000","last_clean":"2024-04-21T08:39:57.905929+0000","last_became_active":"2024-04-21T08:37:31.449558+0000","last_became_peered":"2024-04-21T08:37:31.449558+0000","last_unstale":"2024-04-21T08:39:57.905929+0000","last_undegraded":"2024-04-21T08:39:57.905929+0000","last_fullsized":"2024-04-21T08:39:57.905929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:21.538689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151539+0000","last_change":"2024-04-21T08:37:31.452973+0000","last_active":"2024-04-21T08:39:58.151539+0000","last_peered":"2024-04-21T08:39:58.151539+0000","last_clean":"2024-04-21T08:39:58.151539+0000","last_became_active":"2024-04-21T08:37:31.452586+0000","last_became_peered":"2024-04-21T08:37:31.452586+0000","last_unstale":"2024-04-21T08:39:58.151539+0000","last_undegraded":"2024-04-21T08:39:58.151539+0000","last_fullsized":"2024-04-21T08:39:58.151539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:09:11.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319144+0000","last_change":"2024-04-21T08:37:31.445801+0000","last_active":"2024-04-21T08:39:58.319144+0000","last_peered":"2024-04-21T08:39:58.319144+0000","last_clean":"2024-04-21T08:39:58.319144+0000","last_became_active":"2024-04-21T08:37:31.445478+0000","last_became_peered":"2024-04-21T08:37:31.445478+0000","last_unstale":"2024-04-21T08:39:58.319144+0000","last_undegraded":"2024-04-21T08:39:58.319144+0000","last_fullsized":"2024-04-21T08:39:58.319144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:33.352739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319201+0000","last_change":"2024-04-21T08:37:31.450735+0000","last_active":"2024-04-21T08:39:58.319201+0000","last_peered":"2024-04-21T08:39:58.319201+0000","last_clean":"2024-04-21T08:39:58.319201+0000","last_became_active":"2024-04-21T08:37:31.450384+0000","last_became_peered":"2024-04-21T08:37:31.450384+0000","last_unstale":"2024-04-21T08:39:58.319201+0000","last_undegraded":"2024-04-21T08:39:58.319201+0000","last_fullsized":"2024-04-21T08:39:58.319201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:45.950489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319354+0000","last_change":"2024-04-21T08:37:31.454690+0000","last_active":"2024-04-21T08:39:58.319354+0000","last_peered":"2024-04-21T08:39:58.319354+0000","last_clean":"2024-04-21T08:39:58.319354+0000","last_became_active":"2024-04-21T08:37:31.454330+0000","last_became_peered":"2024-04-21T08:37:31.454330+0000","last_unstale":"2024-04-21T08:39:58.319354+0000","last_undegraded":"2024-04-21T08:39:58.319354+0000","last_fullsized":"2024-04-21T08:39:58.319354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:52.644317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153871+0000","last_change":"2024-04-21T08:37:31.441389+0000","last_active":"2024-04-21T08:39:58.153871+0000","last_peered":"2024-04-21T08:39:58.153871+0000","last_clean":"2024-04-21T08:39:58.153871+0000","last_became_active":"2024-04-21T08:37:31.441065+0000","last_became_peered":"2024-04-21T08:37:31.441065+0000","last_unstale":"2024-04-21T08:39:58.153871+0000","last_undegraded":"2024-04-21T08:39:58.153871+0000","last_fullsized":"2024-04-21T08:39:58.153871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:56.528154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318575+0000","last_change":"2024-04-21T08:37:31.446722+0000","last_active":"2024-04-21T08:39:58.318575+0000","last_peered":"2024-04-21T08:39:58.318575+0000","last_clean":"2024-04-21T08:39:58.318575+0000","last_became_active":"2024-04-21T08:37:31.446396+0000","last_became_peered":"2024-04-21T08:37:31.446396+0000","last_unstale":"2024-04-21T08:39:58.318575+0000","last_undegraded":"2024-04-21T08:39:58.318575+0000","last_fullsized":"2024-04-21T08:39:58.318575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:17.488633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007275+0000","last_change":"2024-04-21T08:37:31.444388+0000","last_active":"2024-04-21T08:39:58.007275+0000","last_peered":"2024-04-21T08:39:58.007275+0000","last_clean":"2024-04-21T08:39:58.007275+0000","last_became_active":"2024-04-21T08:37:31.444023+0000","last_became_peered":"2024-04-21T08:37:31.444023+0000","last_unstale":"2024-04-21T08:39:58.007275+0000","last_undegraded":"2024-04-21T08:39:58.007275+0000","last_fullsized":"2024-04-21T08:39:58.007275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:07:46.238941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905278+0000","last_change":"2024-04-21T08:37:31.447850+0000","last_active":"2024-04-21T08:39:57.905278+0000","last_peered":"2024-04-21T08:39:57.905278+0000","last_clean":"2024-04-21T08:39:57.905278+0000","last_became_active":"2024-04-21T08:37:31.447505+0000","last_became_peered":"2024-04-21T08:37:31.447505+0000","last_unstale":"2024-04-21T08:39:57.905278+0000","last_undegraded":"2024-04-21T08:39:57.905278+0000","last_fullsized":"2024-04-21T08:39:57.905278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:55.399410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905032+0000","last_change":"2024-04-21T08:37:35.608747+0000","last_active":"2024-04-21T08:39:57.905032+0000","last_peered":"2024-04-21T08:39:57.905032+0000","last_clean":"2024-04-21T08:39:57.905032+0000","last_became_active":"2024-04-21T08:37:35.608157+0000","last_became_peered":"2024-04-21T08:37:35.608157+0000","last_unstale":"2024-04-21T08:39:57.905032+0000","last_undegraded":"2024-04-21T08:39:57.905032+0000","last_fullsized":"2024-04-21T08:39:57.905032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:53.384190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007037+0000","last_change":"2024-04-21T08:38:30.679843+0000","last_active":"2024-04-21T08:39:58.007037+0000","last_peered":"2024-04-21T08:39:58.007037+0000","last_clean":"2024-04-21T08:39:58.007037+0000","last_became_active":"2024-04-21T08:38:30.670964+0000","last_became_peered":"2024-04-21T08:38:30.670964+0000","last_unstale":"2024-04-21T08:39:58.007037+0000","last_undegraded":"2024-04-21T08:39:58.007037+0000","last_fullsized":"2024-04-21T08:39:58.007037+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:49.652571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904100+0000","last_change":"2024-04-21T08:37:31.451813+0000","last_active":"2024-04-21T08:39:57.904100+0000","last_peered":"2024-04-21T08:39:57.904100+0000","last_clean":"2024-04-21T08:39:57.904100+0000","last_became_active":"2024-04-21T08:37:31.451447+0000","last_became_peered":"2024-04-21T08:37:31.451447+0000","last_unstale":"2024-04-21T08:39:57.904100+0000","last_undegraded":"2024-04-21T08:39:57.904100+0000","last_fullsized":"2024-04-21T08:39:57.904100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:46:45.235326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151221+0000","last_change":"2024-04-21T08:37:35.603518+0000","last_active":"2024-04-21T08:39:58.151221+0000","last_peered":"2024-04-21T08:39:58.151221+0000","last_clean":"2024-04-21T08:39:58.151221+0000","last_became_active":"2024-04-21T08:37:35.603038+0000","last_became_peered":"2024-04-21T08:37:35.603038+0000","last_unstale":"2024-04-21T08:39:58.151221+0000","last_undegraded":"2024-04-21T08:39:58.151221+0000","last_fullsized":"2024-04-21T08:39:58.151221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:06:29.224025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904228+0000","last_change":"2024-04-21T08:37:30.424608+0000","last_active":"2024-04-21T08:39:57.904228+0000","last_peered":"2024-04-21T08:39:57.904228+0000","last_clean":"2024-04-21T08:39:57.904228+0000","last_became_active":"2024-04-21T08:37:30.424259+0000","last_became_peered":"2024-04-21T08:37:30.424259+0000","last_unstale":"2024-04-21T08:39:57.904228+0000","last_undegraded":"2024-04-21T08:39:57.904228+0000","last_fullsized":"2024-04-21T08:39:57.904228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:38.007095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319688+0000","last_change":"2024-04-21T08:37:31.444656+0000","last_active":"2024-04-21T08:39:58.319688+0000","last_peered":"2024-04-21T08:39:58.319688+0000","last_clean":"2024-04-21T08:39:58.319688+0000","last_became_active":"2024-04-21T08:37:31.444300+0000","last_became_peered":"2024-04-21T08:37:31.444300+0000","last_unstale":"2024-04-21T08:39:58.319688+0000","last_undegraded":"2024-04-21T08:39:58.319688+0000","last_fullsized":"2024-04-21T08:39:58.319688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:40.901674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151388+0000","last_change":"2024-04-21T08:37:35.611732+0000","last_active":"2024-04-21T08:39:58.151388+0000","last_peered":"2024-04-21T08:39:58.151388+0000","last_clean":"2024-04-21T08:39:58.151388+0000","last_became_active":"2024-04-21T08:37:35.611060+0000","last_became_peered":"2024-04-21T08:37:35.611060+0000","last_unstale":"2024-04-21T08:39:58.151388+0000","last_undegraded":"2024-04-21T08:39:58.151388+0000","last_fullsized":"2024-04-21T08:39:58.151388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:36:01.491949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318678+0000","last_change":"2024-04-21T08:38:30.689722+0000","last_active":"2024-04-21T08:39:58.318678+0000","last_peered":"2024-04-21T08:39:58.318678+0000","last_clean":"2024-04-21T08:39:58.318678+0000","last_became_active":"2024-04-21T08:38:30.676521+0000","last_became_peered":"2024-04-21T08:38:30.676521+0000","last_unstale":"2024-04-21T08:39:58.318678+0000","last_undegraded":"2024-04-21T08:39:58.318678+0000","last_fullsized":"2024-04-21T08:39:58.318678+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:24:15.799308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154321+0000","last_change":"2024-04-21T08:37:31.443896+0000","last_active":"2024-04-21T08:39:58.154321+0000","last_peered":"2024-04-21T08:39:58.154321+0000","last_clean":"2024-04-21T08:39:58.154321+0000","last_became_active":"2024-04-21T08:37:31.442684+0000","last_became_peered":"2024-04-21T08:37:31.442684+0000","last_unstale":"2024-04-21T08:39:58.154321+0000","last_undegraded":"2024-04-21T08:39:58.154321+0000","last_fullsized":"2024-04-21T08:39:58.154321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:14:18.649565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151231+0000","last_change":"2024-04-21T08:37:35.600468+0000","last_active":"2024-04-21T08:39:58.151231+0000","last_peered":"2024-04-21T08:39:58.151231+0000","last_clean":"2024-04-21T08:39:58.151231+0000","last_became_active":"2024-04-21T08:37:35.599940+0000","last_became_peered":"2024-04-21T08:37:35.599940+0000","last_unstale":"2024-04-21T08:39:58.151231+0000","last_undegraded":"2024-04-21T08:39:58.151231+0000","last_fullsized":"2024-04-21T08:39:58.151231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:48.125434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009395+0000","last_change":"2024-04-21T08:37:30.416480+0000","last_active":"2024-04-21T08:39:58.009395+0000","last_peered":"2024-04-21T08:39:58.009395+0000","last_clean":"2024-04-21T08:39:58.009395+0000","last_became_active":"2024-04-21T08:37:30.416072+0000","last_became_peered":"2024-04-21T08:37:30.416072+0000","last_unstale":"2024-04-21T08:39:58.009395+0000","last_undegraded":"2024-04-21T08:39:58.009395+0000","last_fullsized":"2024-04-21T08:39:58.009395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:28:22.400926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014147+0000","last_change":"2024-04-21T08:37:31.457216+0000","last_active":"2024-04-21T08:39:58.014147+0000","last_peered":"2024-04-21T08:39:58.014147+0000","last_clean":"2024-04-21T08:39:58.014147+0000","last_became_active":"2024-04-21T08:37:31.456856+0000","last_became_peered":"2024-04-21T08:37:31.456856+0000","last_unstale":"2024-04-21T08:39:58.014147+0000","last_undegraded":"2024-04-21T08:39:58.014147+0000","last_fullsized":"2024-04-21T08:39:58.014147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:10.272886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905178+0000","last_change":"2024-04-21T08:37:35.601715+0000","last_active":"2024-04-21T08:39:57.905178+0000","last_peered":"2024-04-21T08:39:57.905178+0000","last_clean":"2024-04-21T08:39:57.905178+0000","last_became_active":"2024-04-21T08:37:35.601269+0000","last_became_peered":"2024-04-21T08:37:35.601269+0000","last_unstale":"2024-04-21T08:39:57.905178+0000","last_undegraded":"2024-04-21T08:39:57.905178+0000","last_fullsized":"2024-04-21T08:39:57.905178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:57.192918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":164,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.890274+0000","last_change":"2024-04-21T08:37:30.415510+0000","last_active":"2024-04-21T08:39:58.890274+0000","last_peered":"2024-04-21T08:39:58.890274+0000","last_clean":"2024-04-21T08:39:58.890274+0000","last_became_active":"2024-04-21T08:37:30.415199+0000","last_became_peered":"2024-04-21T08:37:30.415199+0000","last_unstale":"2024-04-21T08:39:58.890274+0000","last_undegraded":"2024-04-21T08:39:58.890274+0000","last_fullsized":"2024-04-21T08:39:58.890274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:32.336247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009074+0000","last_change":"2024-04-21T08:39:31.681460+0000","last_active":"2024-04-21T08:39:58.009074+0000","last_peered":"2024-04-21T08:39:58.009074+0000","last_clean":"2024-04-21T08:39:58.009074+0000","last_became_active":"2024-04-21T08:39:31.681057+0000","last_became_peered":"2024-04-21T08:39:31.681057+0000","last_unstale":"2024-04-21T08:39:58.009074+0000","last_undegraded":"2024-04-21T08:39:58.009074+0000","last_fullsized":"2024-04-21T08:39:58.009074+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:43.407369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905226+0000","last_change":"2024-04-21T08:38:30.714455+0000","last_active":"2024-04-21T08:39:57.905226+0000","last_peered":"2024-04-21T08:39:57.905226+0000","last_clean":"2024-04-21T08:39:57.905226+0000","last_became_active":"2024-04-21T08:38:30.713910+0000","last_became_peered":"2024-04-21T08:38:30.713910+0000","last_unstale":"2024-04-21T08:39:57.905226+0000","last_undegraded":"2024-04-21T08:39:57.905226+0000","last_fullsized":"2024-04-21T08:39:57.905226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.448652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,10],"acting":[2,6,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017063+0000","last_change":"2024-04-21T08:37:30.419877+0000","last_active":"2024-04-21T08:39:58.017063+0000","last_peered":"2024-04-21T08:39:58.017063+0000","last_clean":"2024-04-21T08:39:58.017063+0000","last_became_active":"2024-04-21T08:37:30.419572+0000","last_became_peered":"2024-04-21T08:37:30.419572+0000","last_unstale":"2024-04-21T08:39:58.017063+0000","last_undegraded":"2024-04-21T08:39:58.017063+0000","last_fullsized":"2024-04-21T08:39:58.017063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:32:35.524089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014037+0000","last_change":"2024-04-21T08:37:31.458590+0000","last_active":"2024-04-21T08:39:58.014037+0000","last_peered":"2024-04-21T08:39:58.014037+0000","last_clean":"2024-04-21T08:39:58.014037+0000","last_became_active":"2024-04-21T08:37:31.458196+0000","last_became_peered":"2024-04-21T08:37:31.458196+0000","last_unstale":"2024-04-21T08:39:58.014037+0000","last_undegraded":"2024-04-21T08:39:58.014037+0000","last_fullsized":"2024-04-21T08:39:58.014037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:00.890859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318442+0000","last_change":"2024-04-21T08:38:30.803663+0000","last_active":"2024-04-21T08:39:58.318442+0000","last_peered":"2024-04-21T08:39:58.318442+0000","last_clean":"2024-04-21T08:39:58.318442+0000","last_became_active":"2024-04-21T08:38:30.803203+0000","last_became_peered":"2024-04-21T08:38:30.803203+0000","last_unstale":"2024-04-21T08:39:58.318442+0000","last_undegraded":"2024-04-21T08:39:58.318442+0000","last_fullsized":"2024-04-21T08:39:58.318442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:06.085380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318695+0000","last_change":"2024-04-21T08:37:30.415786+0000","last_active":"2024-04-21T08:39:58.318695+0000","last_peered":"2024-04-21T08:39:58.318695+0000","last_clean":"2024-04-21T08:39:58.318695+0000","last_became_active":"2024-04-21T08:37:30.415472+0000","last_became_peered":"2024-04-21T08:37:30.415472+0000","last_unstale":"2024-04-21T08:39:58.318695+0000","last_undegraded":"2024-04-21T08:39:58.318695+0000","last_fullsized":"2024-04-21T08:39:58.318695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.417132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903648+0000","last_change":"2024-04-21T08:37:31.455720+0000","last_active":"2024-04-21T08:39:57.903648+0000","last_peered":"2024-04-21T08:39:57.903648+0000","last_clean":"2024-04-21T08:39:57.903648+0000","last_became_active":"2024-04-21T08:37:31.455334+0000","last_became_peered":"2024-04-21T08:37:31.455334+0000","last_unstale":"2024-04-21T08:39:57.903648+0000","last_undegraded":"2024-04-21T08:39:57.903648+0000","last_fullsized":"2024-04-21T08:39:57.903648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:06:55.220656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319675+0000","last_change":"2024-04-21T08:37:35.603305+0000","last_active":"2024-04-21T08:39:58.319675+0000","last_peered":"2024-04-21T08:39:58.319675+0000","last_clean":"2024-04-21T08:39:58.319675+0000","last_became_active":"2024-04-21T08:37:35.602854+0000","last_became_peered":"2024-04-21T08:37:35.602854+0000","last_unstale":"2024-04-21T08:39:58.319675+0000","last_undegraded":"2024-04-21T08:39:58.319675+0000","last_fullsized":"2024-04-21T08:39:58.319675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:10.993864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013890+0000","last_change":"2024-04-21T08:37:30.420797+0000","last_active":"2024-04-21T08:39:58.013890+0000","last_peered":"2024-04-21T08:39:58.013890+0000","last_clean":"2024-04-21T08:39:58.013890+0000","last_became_active":"2024-04-21T08:37:30.420485+0000","last_became_peered":"2024-04-21T08:37:30.420485+0000","last_unstale":"2024-04-21T08:39:58.013890+0000","last_undegraded":"2024-04-21T08:39:58.013890+0000","last_fullsized":"2024-04-21T08:39:58.013890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:03.275677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006844+0000","last_change":"2024-04-21T08:37:31.448406+0000","last_active":"2024-04-21T08:39:58.006844+0000","last_peered":"2024-04-21T08:39:58.006844+0000","last_clean":"2024-04-21T08:39:58.006844+0000","last_became_active":"2024-04-21T08:37:31.448085+0000","last_became_peered":"2024-04-21T08:37:31.448085+0000","last_unstale":"2024-04-21T08:39:58.006844+0000","last_undegraded":"2024-04-21T08:39:58.006844+0000","last_fullsized":"2024-04-21T08:39:58.006844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:49.574064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903948+0000","last_change":"2024-04-21T08:37:35.601338+0000","last_active":"2024-04-21T08:39:57.903948+0000","last_peered":"2024-04-21T08:39:57.903948+0000","last_clean":"2024-04-21T08:39:57.903948+0000","last_became_active":"2024-04-21T08:37:35.600890+0000","last_became_peered":"2024-04-21T08:37:35.600890+0000","last_unstale":"2024-04-21T08:39:57.903948+0000","last_undegraded":"2024-04-21T08:39:57.903948+0000","last_fullsized":"2024-04-21T08:39:57.903948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:30:21.755850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.889748+0000","last_change":"2024-04-21T08:37:30.423320+0000","last_active":"2024-04-21T08:39:58.889748+0000","last_peered":"2024-04-21T08:39:58.889748+0000","last_clean":"2024-04-21T08:39:58.889748+0000","last_became_active":"2024-04-21T08:37:30.423020+0000","last_became_peered":"2024-04-21T08:37:30.423020+0000","last_unstale":"2024-04-21T08:39:58.889748+0000","last_undegraded":"2024-04-21T08:39:58.889748+0000","last_fullsized":"2024-04-21T08:39:58.889748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:08:27.313196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154669+0000","last_change":"2024-04-21T08:37:31.442425+0000","last_active":"2024-04-21T08:39:58.154669+0000","last_peered":"2024-04-21T08:39:58.154669+0000","last_clean":"2024-04-21T08:39:58.154669+0000","last_became_active":"2024-04-21T08:37:31.442028+0000","last_became_peered":"2024-04-21T08:37:31.442028+0000","last_unstale":"2024-04-21T08:39:58.154669+0000","last_undegraded":"2024-04-21T08:39:58.154669+0000","last_fullsized":"2024-04-21T08:39:58.154669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:45.498656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006697+0000","last_change":"2024-04-21T08:37:35.607829+0000","last_active":"2024-04-21T08:39:58.006697+0000","last_peered":"2024-04-21T08:39:58.006697+0000","last_clean":"2024-04-21T08:39:58.006697+0000","last_became_active":"2024-04-21T08:37:35.607150+0000","last_became_peered":"2024-04-21T08:37:35.607150+0000","last_unstale":"2024-04-21T08:39:58.006697+0000","last_undegraded":"2024-04-21T08:39:58.006697+0000","last_fullsized":"2024-04-21T08:39:58.006697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:01:34.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905384+0000","last_change":"2024-04-21T08:37:30.418613+0000","last_active":"2024-04-21T08:39:57.905384+0000","last_peered":"2024-04-21T08:39:57.905384+0000","last_clean":"2024-04-21T08:39:57.905384+0000","last_became_active":"2024-04-21T08:37:30.418300+0000","last_became_peered":"2024-04-21T08:37:30.418300+0000","last_unstale":"2024-04-21T08:39:57.905384+0000","last_undegraded":"2024-04-21T08:39:57.905384+0000","last_fullsized":"2024-04-21T08:39:57.905384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:17:35.370753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319739+0000","last_change":"2024-04-21T08:39:31.682628+0000","last_active":"2024-04-21T08:39:58.319739+0000","last_peered":"2024-04-21T08:39:58.319739+0000","last_clean":"2024-04-21T08:39:58.319739+0000","last_became_active":"2024-04-21T08:39:31.682383+0000","last_became_peered":"2024-04-21T08:39:31.682383+0000","last_unstale":"2024-04-21T08:39:58.319739+0000","last_undegraded":"2024-04-21T08:39:58.319739+0000","last_fullsized":"2024-04-21T08:39:58.319739+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:38.831288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151734+0000","last_change":"2024-04-21T08:37:35.608280+0000","last_active":"2024-04-21T08:39:58.151734+0000","last_peered":"2024-04-21T08:39:58.151734+0000","last_clean":"2024-04-21T08:39:58.151734+0000","last_became_active":"2024-04-21T08:37:35.607898+0000","last_became_peered":"2024-04-21T08:37:35.607898+0000","last_unstale":"2024-04-21T08:39:58.151734+0000","last_undegraded":"2024-04-21T08:39:58.151734+0000","last_fullsized":"2024-04-21T08:39:58.151734+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:55.689519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014140+0000","last_change":"2024-04-21T08:37:30.417352+0000","last_active":"2024-04-21T08:39:58.014140+0000","last_peered":"2024-04-21T08:39:58.014140+0000","last_clean":"2024-04-21T08:39:58.014140+0000","last_became_active":"2024-04-21T08:37:30.417000+0000","last_became_peered":"2024-04-21T08:37:30.417000+0000","last_unstale":"2024-04-21T08:39:58.014140+0000","last_undegraded":"2024-04-21T08:39:58.014140+0000","last_fullsized":"2024-04-21T08:39:58.014140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:56.087334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006804+0000","last_change":"2024-04-21T08:39:31.679434+0000","last_active":"2024-04-21T08:39:58.006804+0000","last_peered":"2024-04-21T08:39:58.006804+0000","last_clean":"2024-04-21T08:39:58.006804+0000","last_became_active":"2024-04-21T08:39:31.679041+0000","last_became_peered":"2024-04-21T08:39:31.679041+0000","last_unstale":"2024-04-21T08:39:58.006804+0000","last_undegraded":"2024-04-21T08:39:58.006804+0000","last_fullsized":"2024-04-21T08:39:58.006804+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:27:11.011449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006731+0000","last_change":"2024-04-21T08:37:35.601077+0000","last_active":"2024-04-21T08:39:58.006731+0000","last_peered":"2024-04-21T08:39:58.006731+0000","last_clean":"2024-04-21T08:39:58.006731+0000","last_became_active":"2024-04-21T08:37:35.600673+0000","last_became_peered":"2024-04-21T08:37:35.600673+0000","last_unstale":"2024-04-21T08:39:58.006731+0000","last_undegraded":"2024-04-21T08:39:58.006731+0000","last_fullsized":"2024-04-21T08:39:58.006731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:04.554208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'41","reported_seq":333,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.890961+0000","last_change":"2024-04-21T08:37:30.421440+0000","last_active":"2024-04-21T08:39:58.890961+0000","last_peered":"2024-04-21T08:39:58.890961+0000","last_clean":"2024-04-21T08:39:58.890961+0000","last_became_active":"2024-04-21T08:37:30.420988+0000","last_became_peered":"2024-04-21T08:37:30.420988+0000","last_unstale":"2024-04-21T08:39:58.890961+0000","last_undegraded":"2024-04-21T08:39:58.890961+0000","last_fullsized":"2024-04-21T08:39:58.890961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:38:03.850705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":258,"num_read_kb":198,"num_write":42,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009331+0000","last_change":"2024-04-21T08:37:31.447208+0000","last_active":"2024-04-21T08:39:58.009331+0000","last_peered":"2024-04-21T08:39:58.009331+0000","last_clean":"2024-04-21T08:39:58.009331+0000","last_became_active":"2024-04-21T08:37:31.444464+0000","last_became_peered":"2024-04-21T08:37:31.444464+0000","last_unstale":"2024-04-21T08:39:58.009331+0000","last_undegraded":"2024-04-21T08:39:58.009331+0000","last_fullsized":"2024-04-21T08:39:58.009331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:33.805676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014409+0000","last_change":"2024-04-21T08:37:35.603937+0000","last_active":"2024-04-21T08:39:58.014409+0000","last_peered":"2024-04-21T08:39:58.014409+0000","last_clean":"2024-04-21T08:39:58.014409+0000","last_became_active":"2024-04-21T08:37:35.603477+0000","last_became_peered":"2024-04-21T08:37:35.603477+0000","last_unstale":"2024-04-21T08:39:58.014409+0000","last_undegraded":"2024-04-21T08:39:58.014409+0000","last_fullsized":"2024-04-21T08:39:58.014409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:40:33.901570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905391+0000","last_change":"2024-04-21T08:37:30.416941+0000","last_active":"2024-04-21T08:39:57.905391+0000","last_peered":"2024-04-21T08:39:57.905391+0000","last_clean":"2024-04-21T08:39:57.905391+0000","last_became_active":"2024-04-21T08:37:30.416625+0000","last_became_peered":"2024-04-21T08:37:30.416625+0000","last_unstale":"2024-04-21T08:39:57.905391+0000","last_undegraded":"2024-04-21T08:39:57.905391+0000","last_fullsized":"2024-04-21T08:39:57.905391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:07.782779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318733+0000","last_change":"2024-04-21T08:37:31.441707+0000","last_active":"2024-04-21T08:39:58.318733+0000","last_peered":"2024-04-21T08:39:58.318733+0000","last_clean":"2024-04-21T08:39:58.318733+0000","last_became_active":"2024-04-21T08:37:31.441366+0000","last_became_peered":"2024-04-21T08:37:31.441366+0000","last_unstale":"2024-04-21T08:39:58.318733+0000","last_undegraded":"2024-04-21T08:39:58.318733+0000","last_fullsized":"2024-04-21T08:39:58.318733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:16.573804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318906+0000","last_change":"2024-04-21T08:37:35.596214+0000","last_active":"2024-04-21T08:39:58.318906+0000","last_peered":"2024-04-21T08:39:58.318906+0000","last_clean":"2024-04-21T08:39:58.318906+0000","last_became_active":"2024-04-21T08:37:35.595773+0000","last_became_peered":"2024-04-21T08:37:35.595773+0000","last_unstale":"2024-04-21T08:39:58.318906+0000","last_undegraded":"2024-04-21T08:39:58.318906+0000","last_fullsized":"2024-04-21T08:39:58.318906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:07.538190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318273+0000","last_change":"2024-04-21T08:38:30.684828+0000","last_active":"2024-04-21T08:39:58.318273+0000","last_peered":"2024-04-21T08:39:58.318273+0000","last_clean":"2024-04-21T08:39:58.318273+0000","last_became_active":"2024-04-21T08:38:30.673923+0000","last_became_peered":"2024-04-21T08:38:30.673923+0000","last_unstale":"2024-04-21T08:39:58.318273+0000","last_undegraded":"2024-04-21T08:39:58.318273+0000","last_fullsized":"2024-04-21T08:39:58.318273+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:51:37.906150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905472+0000","last_change":"2024-04-21T08:37:31.453680+0000","last_active":"2024-04-21T08:39:57.905472+0000","last_peered":"2024-04-21T08:39:57.905472+0000","last_clean":"2024-04-21T08:39:57.905472+0000","last_became_active":"2024-04-21T08:37:31.453319+0000","last_became_peered":"2024-04-21T08:37:31.453319+0000","last_unstale":"2024-04-21T08:39:57.905472+0000","last_undegraded":"2024-04-21T08:39:57.905472+0000","last_fullsized":"2024-04-21T08:39:57.905472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:43.139789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006992+0000","last_change":"2024-04-21T08:37:35.608906+0000","last_active":"2024-04-21T08:39:58.006992+0000","last_peered":"2024-04-21T08:39:58.006992+0000","last_clean":"2024-04-21T08:39:58.006992+0000","last_became_active":"2024-04-21T08:37:35.608535+0000","last_became_peered":"2024-04-21T08:37:35.608535+0000","last_unstale":"2024-04-21T08:39:58.006992+0000","last_undegraded":"2024-04-21T08:39:58.006992+0000","last_fullsized":"2024-04-21T08:39:58.006992+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:58:01.422308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":171,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.890194+0000","last_change":"2024-04-21T08:37:30.414865+0000","last_active":"2024-04-21T08:39:58.890194+0000","last_peered":"2024-04-21T08:39:58.890194+0000","last_clean":"2024-04-21T08:39:58.890194+0000","last_became_active":"2024-04-21T08:37:30.414471+0000","last_became_peered":"2024-04-21T08:37:30.414471+0000","last_unstale":"2024-04-21T08:39:58.890194+0000","last_undegraded":"2024-04-21T08:39:58.890194+0000","last_fullsized":"2024-04-21T08:39:58.890194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:46.943574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905326+0000","last_change":"2024-04-21T08:37:31.456634+0000","last_active":"2024-04-21T08:39:57.905326+0000","last_peered":"2024-04-21T08:39:57.905326+0000","last_clean":"2024-04-21T08:39:57.905326+0000","last_became_active":"2024-04-21T08:37:31.456278+0000","last_became_peered":"2024-04-21T08:37:31.456278+0000","last_unstale":"2024-04-21T08:39:57.905326+0000","last_undegraded":"2024-04-21T08:39:57.905326+0000","last_fullsized":"2024-04-21T08:39:57.905326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:04.870024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319167+0000","last_change":"2024-04-21T08:37:35.601760+0000","last_active":"2024-04-21T08:39:58.319167+0000","last_peered":"2024-04-21T08:39:58.319167+0000","last_clean":"2024-04-21T08:39:58.319167+0000","last_became_active":"2024-04-21T08:37:35.601332+0000","last_became_peered":"2024-04-21T08:37:35.601332+0000","last_unstale":"2024-04-21T08:39:58.319167+0000","last_undegraded":"2024-04-21T08:39:58.319167+0000","last_fullsized":"2024-04-21T08:39:58.319167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:04:03.694581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008961+0000","last_change":"2024-04-21T08:37:30.422419+0000","last_active":"2024-04-21T08:39:58.008961+0000","last_peered":"2024-04-21T08:39:58.008961+0000","last_clean":"2024-04-21T08:39:58.008961+0000","last_became_active":"2024-04-21T08:37:30.422123+0000","last_became_peered":"2024-04-21T08:37:30.422123+0000","last_unstale":"2024-04-21T08:39:58.008961+0000","last_undegraded":"2024-04-21T08:39:58.008961+0000","last_fullsized":"2024-04-21T08:39:58.008961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:32:54.029041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013848+0000","last_change":"2024-04-21T08:37:31.458992+0000","last_active":"2024-04-21T08:39:58.013848+0000","last_peered":"2024-04-21T08:39:58.013848+0000","last_clean":"2024-04-21T08:39:58.013848+0000","last_became_active":"2024-04-21T08:37:31.458635+0000","last_became_peered":"2024-04-21T08:37:31.458635+0000","last_unstale":"2024-04-21T08:39:58.013848+0000","last_undegraded":"2024-04-21T08:39:58.013848+0000","last_fullsized":"2024-04-21T08:39:58.013848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:48:14.325571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013750+0000","last_change":"2024-04-21T08:37:35.603739+0000","last_active":"2024-04-21T08:39:58.013750+0000","last_peered":"2024-04-21T08:39:58.013750+0000","last_clean":"2024-04-21T08:39:58.013750+0000","last_became_active":"2024-04-21T08:37:35.603310+0000","last_became_peered":"2024-04-21T08:37:35.603310+0000","last_unstale":"2024-04-21T08:39:58.013750+0000","last_undegraded":"2024-04-21T08:39:58.013750+0000","last_fullsized":"2024-04-21T08:39:58.013750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:19.162520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009465+0000","last_change":"2024-04-21T08:37:30.418104+0000","last_active":"2024-04-21T08:39:58.009465+0000","last_peered":"2024-04-21T08:39:58.009465+0000","last_clean":"2024-04-21T08:39:58.009465+0000","last_became_active":"2024-04-21T08:37:30.417814+0000","last_became_peered":"2024-04-21T08:37:30.417814+0000","last_unstale":"2024-04-21T08:39:58.009465+0000","last_undegraded":"2024-04-21T08:39:58.009465+0000","last_fullsized":"2024-04-21T08:39:58.009465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:50:56.696513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318582+0000","last_change":"2024-04-21T08:37:31.451952+0000","last_active":"2024-04-21T08:39:58.318582+0000","last_peered":"2024-04-21T08:39:58.318582+0000","last_clean":"2024-04-21T08:39:58.318582+0000","last_became_active":"2024-04-21T08:37:31.451588+0000","last_became_peered":"2024-04-21T08:37:31.451588+0000","last_unstale":"2024-04-21T08:39:58.318582+0000","last_undegraded":"2024-04-21T08:39:58.318582+0000","last_fullsized":"2024-04-21T08:39:58.318582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:06.506769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009071+0000","last_change":"2024-04-21T08:37:35.760661+0000","last_active":"2024-04-21T08:39:58.009071+0000","last_peered":"2024-04-21T08:39:58.009071+0000","last_clean":"2024-04-21T08:39:58.009071+0000","last_became_active":"2024-04-21T08:37:35.759925+0000","last_became_peered":"2024-04-21T08:37:35.759925+0000","last_unstale":"2024-04-21T08:39:58.009071+0000","last_undegraded":"2024-04-21T08:39:58.009071+0000","last_fullsized":"2024-04-21T08:39:58.009071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:11:53.062251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007012+0000","last_change":"2024-04-21T08:37:30.416487+0000","last_active":"2024-04-21T08:39:58.007012+0000","last_peered":"2024-04-21T08:39:58.007012+0000","last_clean":"2024-04-21T08:39:58.007012+0000","last_became_active":"2024-04-21T08:37:30.416076+0000","last_became_peered":"2024-04-21T08:37:30.416076+0000","last_unstale":"2024-04-21T08:39:58.007012+0000","last_undegraded":"2024-04-21T08:39:58.007012+0000","last_fullsized":"2024-04-21T08:39:58.007012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:18:33.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017205+0000","last_change":"2024-04-21T08:37:31.448824+0000","last_active":"2024-04-21T08:39:58.017205+0000","last_peered":"2024-04-21T08:39:58.017205+0000","last_clean":"2024-04-21T08:39:58.017205+0000","last_became_active":"2024-04-21T08:37:31.448432+0000","last_became_peered":"2024-04-21T08:37:31.448432+0000","last_unstale":"2024-04-21T08:39:58.017205+0000","last_undegraded":"2024-04-21T08:39:58.017205+0000","last_fullsized":"2024-04-21T08:39:58.017205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:41.111272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318815+0000","last_change":"2024-04-21T08:37:35.604539+0000","last_active":"2024-04-21T08:39:58.318815+0000","last_peered":"2024-04-21T08:39:58.318815+0000","last_clean":"2024-04-21T08:39:58.318815+0000","last_became_active":"2024-04-21T08:37:35.604132+0000","last_became_peered":"2024-04-21T08:37:35.604132+0000","last_unstale":"2024-04-21T08:39:58.318815+0000","last_undegraded":"2024-04-21T08:39:58.318815+0000","last_fullsized":"2024-04-21T08:39:58.318815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:12.180730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905085+0000","last_change":"2024-04-21T08:37:30.418854+0000","last_active":"2024-04-21T08:39:57.905085+0000","last_peered":"2024-04-21T08:39:57.905085+0000","last_clean":"2024-04-21T08:39:57.905085+0000","last_became_active":"2024-04-21T08:37:30.418511+0000","last_became_peered":"2024-04-21T08:37:30.418511+0000","last_unstale":"2024-04-21T08:39:57.905085+0000","last_undegraded":"2024-04-21T08:39:57.905085+0000","last_fullsized":"2024-04-21T08:39:57.905085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:26.751924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.016918+0000","last_change":"2024-04-21T08:37:31.453365+0000","last_active":"2024-04-21T08:39:58.016918+0000","last_peered":"2024-04-21T08:39:58.016918+0000","last_clean":"2024-04-21T08:39:58.016918+0000","last_became_active":"2024-04-21T08:37:31.453042+0000","last_became_peered":"2024-04-21T08:37:31.453042+0000","last_unstale":"2024-04-21T08:39:58.016918+0000","last_undegraded":"2024-04-21T08:39:58.016918+0000","last_fullsized":"2024-04-21T08:39:58.016918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:49:46.648754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318806+0000","last_change":"2024-04-21T08:37:35.760626+0000","last_active":"2024-04-21T08:39:58.318806+0000","last_peered":"2024-04-21T08:39:58.318806+0000","last_clean":"2024-04-21T08:39:58.318806+0000","last_became_active":"2024-04-21T08:37:35.760125+0000","last_became_peered":"2024-04-21T08:37:35.760125+0000","last_unstale":"2024-04-21T08:39:58.318806+0000","last_undegraded":"2024-04-21T08:39:58.318806+0000","last_fullsized":"2024-04-21T08:39:58.318806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:58.366557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013932+0000","last_change":"2024-04-21T08:37:31.457654+0000","last_active":"2024-04-21T08:39:58.013932+0000","last_peered":"2024-04-21T08:39:58.013932+0000","last_clean":"2024-04-21T08:39:58.013932+0000","last_became_active":"2024-04-21T08:37:31.456950+0000","last_became_peered":"2024-04-21T08:37:31.456950+0000","last_unstale":"2024-04-21T08:39:58.013932+0000","last_undegraded":"2024-04-21T08:39:58.013932+0000","last_fullsized":"2024-04-21T08:39:58.013932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:02.863000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318817+0000","last_change":"2024-04-21T08:37:30.422021+0000","last_active":"2024-04-21T08:39:58.318817+0000","last_peered":"2024-04-21T08:39:58.318817+0000","last_clean":"2024-04-21T08:39:58.318817+0000","last_became_active":"2024-04-21T08:37:30.421711+0000","last_became_peered":"2024-04-21T08:37:30.421711+0000","last_unstale":"2024-04-21T08:39:58.318817+0000","last_undegraded":"2024-04-21T08:39:58.318817+0000","last_fullsized":"2024-04-21T08:39:58.318817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:01.612205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905863+0000","last_change":"2024-04-21T08:37:35.598041+0000","last_active":"2024-04-21T08:39:57.905863+0000","last_peered":"2024-04-21T08:39:57.905863+0000","last_clean":"2024-04-21T08:39:57.905863+0000","last_became_active":"2024-04-21T08:37:35.597635+0000","last_became_peered":"2024-04-21T08:37:35.597635+0000","last_unstale":"2024-04-21T08:39:57.905863+0000","last_undegraded":"2024-04-21T08:39:57.905863+0000","last_fullsized":"2024-04-21T08:39:57.905863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:35:27.452340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903604+0000","last_change":"2024-04-21T08:37:31.450423+0000","last_active":"2024-04-21T08:39:57.903604+0000","last_peered":"2024-04-21T08:39:57.903604+0000","last_clean":"2024-04-21T08:39:57.903604+0000","last_became_active":"2024-04-21T08:37:31.450112+0000","last_became_peered":"2024-04-21T08:37:31.450112+0000","last_unstale":"2024-04-21T08:39:57.903604+0000","last_undegraded":"2024-04-21T08:39:57.903604+0000","last_fullsized":"2024-04-21T08:39:57.903604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:10:47.006769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151568+0000","last_change":"2024-04-21T08:37:30.416519+0000","last_active":"2024-04-21T08:39:58.151568+0000","last_peered":"2024-04-21T08:39:58.151568+0000","last_clean":"2024-04-21T08:39:58.151568+0000","last_became_active":"2024-04-21T08:37:30.416131+0000","last_became_peered":"2024-04-21T08:37:30.416131+0000","last_unstale":"2024-04-21T08:39:58.151568+0000","last_undegraded":"2024-04-21T08:39:58.151568+0000","last_fullsized":"2024-04-21T08:39:58.151568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:12.280847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":368,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904918+0000","last_change":"2024-04-21T08:36:20.257759+0000","last_active":"2024-04-21T08:39:57.904918+0000","last_peered":"2024-04-21T08:39:57.904918+0000","last_clean":"2024-04-21T08:39:57.904918+0000","last_became_active":"2024-04-21T08:36:20.257313+0000","last_became_peered":"2024-04-21T08:36:20.257313+0000","last_unstale":"2024-04-21T08:39:57.904918+0000","last_undegraded":"2024-04-21T08:39:57.904918+0000","last_fullsized":"2024-04-21T08:39:57.904918+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_clean_scrub_stamp":"2024-04-21T08:32:56.546295+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:42:44.393447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014207+0000","last_change":"2024-04-21T08:37:35.761562+0000","last_active":"2024-04-21T08:39:58.014207+0000","last_peered":"2024-04-21T08:39:58.014207+0000","last_clean":"2024-04-21T08:39:58.014207+0000","last_became_active":"2024-04-21T08:37:35.760703+0000","last_became_peered":"2024-04-21T08:37:35.760703+0000","last_unstale":"2024-04-21T08:39:58.014207+0000","last_undegraded":"2024-04-21T08:39:58.014207+0000","last_fullsized":"2024-04-21T08:39:58.014207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:12:47.511117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318506+0000","last_change":"2024-04-21T08:39:31.678868+0000","last_active":"2024-04-21T08:39:58.318506+0000","last_peered":"2024-04-21T08:39:58.318506+0000","last_clean":"2024-04-21T08:39:58.318506+0000","last_became_active":"2024-04-21T08:39:31.678457+0000","last_became_peered":"2024-04-21T08:39:31.678457+0000","last_unstale":"2024-04-21T08:39:58.318506+0000","last_undegraded":"2024-04-21T08:39:58.318506+0000","last_fullsized":"2024-04-21T08:39:58.318506+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:49.918106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905042+0000","last_change":"2024-04-21T08:37:30.420560+0000","last_active":"2024-04-21T08:39:57.905042+0000","last_peered":"2024-04-21T08:39:57.905042+0000","last_clean":"2024-04-21T08:39:57.905042+0000","last_became_active":"2024-04-21T08:37:30.420223+0000","last_became_peered":"2024-04-21T08:37:30.420223+0000","last_unstale":"2024-04-21T08:39:57.905042+0000","last_undegraded":"2024-04-21T08:39:57.905042+0000","last_fullsized":"2024-04-21T08:39:57.905042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:43:55.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318381+0000","last_change":"2024-04-21T08:37:35.598294+0000","last_active":"2024-04-21T08:39:58.318381+0000","last_peered":"2024-04-21T08:39:58.318381+0000","last_clean":"2024-04-21T08:39:58.318381+0000","last_became_active":"2024-04-21T08:37:35.597883+0000","last_became_peered":"2024-04-21T08:37:35.597883+0000","last_unstale":"2024-04-21T08:39:58.318381+0000","last_undegraded":"2024-04-21T08:39:58.318381+0000","last_fullsized":"2024-04-21T08:39:58.318381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:24:05.144763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009321+0000","last_change":"2024-04-21T08:37:30.423182+0000","last_active":"2024-04-21T08:39:58.009321+0000","last_peered":"2024-04-21T08:39:58.009321+0000","last_clean":"2024-04-21T08:39:58.009321+0000","last_became_active":"2024-04-21T08:37:30.422792+0000","last_became_peered":"2024-04-21T08:37:30.422792+0000","last_unstale":"2024-04-21T08:39:58.009321+0000","last_undegraded":"2024-04-21T08:39:58.009321+0000","last_fullsized":"2024-04-21T08:39:58.009321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:44:00.727269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318924+0000","last_change":"2024-04-21T08:37:31.444013+0000","last_active":"2024-04-21T08:39:58.318924+0000","last_peered":"2024-04-21T08:39:58.318924+0000","last_clean":"2024-04-21T08:39:58.318924+0000","last_became_active":"2024-04-21T08:37:31.443684+0000","last_became_peered":"2024-04-21T08:37:31.443684+0000","last_unstale":"2024-04-21T08:39:58.318924+0000","last_undegraded":"2024-04-21T08:39:58.318924+0000","last_fullsized":"2024-04-21T08:39:58.318924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:00:35.400818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903525+0000","last_change":"2024-04-21T08:37:35.759817+0000","last_active":"2024-04-21T08:39:57.903525+0000","last_peered":"2024-04-21T08:39:57.903525+0000","last_clean":"2024-04-21T08:39:57.903525+0000","last_became_active":"2024-04-21T08:37:35.759396+0000","last_became_peered":"2024-04-21T08:37:35.759396+0000","last_unstale":"2024-04-21T08:39:57.903525+0000","last_undegraded":"2024-04-21T08:39:57.903525+0000","last_fullsized":"2024-04-21T08:39:57.903525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:20:36.170880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'44","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.114287+0000","last_change":"2024-04-21T08:37:30.414102+0000","last_active":"2024-04-21T08:39:58.114287+0000","last_peered":"2024-04-21T08:39:58.114287+0000","last_clean":"2024-04-21T08:39:58.114287+0000","last_became_active":"2024-04-21T08:37:30.413670+0000","last_became_peered":"2024-04-21T08:37:30.413670+0000","last_unstale":"2024-04-21T08:39:58.114287+0000","last_undegraded":"2024-04-21T08:39:58.114287+0000","last_fullsized":"2024-04-21T08:39:58.114287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:12:49.696516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904320+0000","last_change":"2024-04-21T08:37:31.457451+0000","last_active":"2024-04-21T08:39:57.904320+0000","last_peered":"2024-04-21T08:39:57.904320+0000","last_clean":"2024-04-21T08:39:57.904320+0000","last_became_active":"2024-04-21T08:37:31.457120+0000","last_became_peered":"2024-04-21T08:37:31.457120+0000","last_unstale":"2024-04-21T08:39:57.904320+0000","last_undegraded":"2024-04-21T08:39:57.904320+0000","last_fullsized":"2024-04-21T08:39:57.904320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:07:48.025843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904850+0000","last_change":"2024-04-21T08:37:35.600815+0000","last_active":"2024-04-21T08:39:57.904850+0000","last_peered":"2024-04-21T08:39:57.904850+0000","last_clean":"2024-04-21T08:39:57.904850+0000","last_became_active":"2024-04-21T08:37:35.600380+0000","last_became_peered":"2024-04-21T08:37:35.600380+0000","last_unstale":"2024-04-21T08:39:57.904850+0000","last_undegraded":"2024-04-21T08:39:57.904850+0000","last_fullsized":"2024-04-21T08:39:57.904850+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:19:33.710431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151729+0000","last_change":"2024-04-21T08:37:30.425592+0000","last_active":"2024-04-21T08:39:58.151729+0000","last_peered":"2024-04-21T08:39:58.151729+0000","last_clean":"2024-04-21T08:39:58.151729+0000","last_became_active":"2024-04-21T08:37:30.425274+0000","last_became_peered":"2024-04-21T08:37:30.425274+0000","last_unstale":"2024-04-21T08:39:58.151729+0000","last_undegraded":"2024-04-21T08:39:58.151729+0000","last_fullsized":"2024-04-21T08:39:58.151729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:27:29.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319672+0000","last_change":"2024-04-21T08:39:31.680028+0000","last_active":"2024-04-21T08:39:58.319672+0000","last_peered":"2024-04-21T08:39:58.319672+0000","last_clean":"2024-04-21T08:39:58.319672+0000","last_became_active":"2024-04-21T08:39:31.679162+0000","last_became_peered":"2024-04-21T08:39:31.679162+0000","last_unstale":"2024-04-21T08:39:58.319672+0000","last_undegraded":"2024-04-21T08:39:58.319672+0000","last_fullsized":"2024-04-21T08:39:58.319672+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:32.824411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151564+0000","last_change":"2024-04-21T08:38:30.713521+0000","last_active":"2024-04-21T08:39:58.151564+0000","last_peered":"2024-04-21T08:39:58.151564+0000","last_clean":"2024-04-21T08:39:58.151564+0000","last_became_active":"2024-04-21T08:38:30.713096+0000","last_became_peered":"2024-04-21T08:38:30.713096+0000","last_unstale":"2024-04-21T08:39:58.151564+0000","last_undegraded":"2024-04-21T08:39:58.151564+0000","last_fullsized":"2024-04-21T08:39:58.151564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:54:59.078776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153818+0000","last_change":"2024-04-21T08:37:30.416696+0000","last_active":"2024-04-21T08:39:58.153818+0000","last_peered":"2024-04-21T08:39:58.153818+0000","last_clean":"2024-04-21T08:39:58.153818+0000","last_became_active":"2024-04-21T08:37:30.415868+0000","last_became_peered":"2024-04-21T08:37:30.415868+0000","last_unstale":"2024-04-21T08:39:58.153818+0000","last_undegraded":"2024-04-21T08:39:58.153818+0000","last_fullsized":"2024-04-21T08:39:58.153818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:48:08.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008945+0000","last_change":"2024-04-21T08:37:31.453861+0000","last_active":"2024-04-21T08:39:58.008945+0000","last_peered":"2024-04-21T08:39:58.008945+0000","last_clean":"2024-04-21T08:39:58.008945+0000","last_became_active":"2024-04-21T08:37:31.453562+0000","last_became_peered":"2024-04-21T08:37:31.453562+0000","last_unstale":"2024-04-21T08:39:58.008945+0000","last_undegraded":"2024-04-21T08:39:58.008945+0000","last_fullsized":"2024-04-21T08:39:58.008945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:17:50.513917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006977+0000","last_change":"2024-04-21T08:37:35.609217+0000","last_active":"2024-04-21T08:39:58.006977+0000","last_peered":"2024-04-21T08:39:58.006977+0000","last_clean":"2024-04-21T08:39:58.006977+0000","last_became_active":"2024-04-21T08:37:35.608786+0000","last_became_peered":"2024-04-21T08:37:35.608786+0000","last_unstale":"2024-04-21T08:39:58.006977+0000","last_undegraded":"2024-04-21T08:39:58.006977+0000","last_fullsized":"2024-04-21T08:39:58.006977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:45:50.945845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903818+0000","last_change":"2024-04-21T08:37:30.417156+0000","last_active":"2024-04-21T08:39:57.903818+0000","last_peered":"2024-04-21T08:39:57.903818+0000","last_clean":"2024-04-21T08:39:57.903818+0000","last_became_active":"2024-04-21T08:37:30.415717+0000","last_became_peered":"2024-04-21T08:37:30.415717+0000","last_unstale":"2024-04-21T08:39:57.903818+0000","last_undegraded":"2024-04-21T08:39:57.903818+0000","last_fullsized":"2024-04-21T08:39:57.903818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:37.819082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318260+0000","last_change":"2024-04-21T08:37:31.444948+0000","last_active":"2024-04-21T08:39:58.318260+0000","last_peered":"2024-04-21T08:39:58.318260+0000","last_clean":"2024-04-21T08:39:58.318260+0000","last_became_active":"2024-04-21T08:37:31.444604+0000","last_became_peered":"2024-04-21T08:37:31.444604+0000","last_unstale":"2024-04-21T08:39:58.318260+0000","last_undegraded":"2024-04-21T08:39:58.318260+0000","last_fullsized":"2024-04-21T08:39:58.318260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.820503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151386+0000","last_change":"2024-04-21T08:37:35.611541+0000","last_active":"2024-04-21T08:39:58.151386+0000","last_peered":"2024-04-21T08:39:58.151386+0000","last_clean":"2024-04-21T08:39:58.151386+0000","last_became_active":"2024-04-21T08:37:35.611136+0000","last_became_peered":"2024-04-21T08:37:35.611136+0000","last_unstale":"2024-04-21T08:39:58.151386+0000","last_undegraded":"2024-04-21T08:39:58.151386+0000","last_fullsized":"2024-04-21T08:39:58.151386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:28.254529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904963+0000","last_change":"2024-04-21T08:37:30.414216+0000","last_active":"2024-04-21T08:39:57.904963+0000","last_peered":"2024-04-21T08:39:57.904963+0000","last_clean":"2024-04-21T08:39:57.904963+0000","last_became_active":"2024-04-21T08:37:30.413890+0000","last_became_peered":"2024-04-21T08:37:30.413890+0000","last_unstale":"2024-04-21T08:39:57.904963+0000","last_undegraded":"2024-04-21T08:39:57.904963+0000","last_fullsized":"2024-04-21T08:39:57.904963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:16.972618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903800+0000","last_change":"2024-04-21T08:37:31.449307+0000","last_active":"2024-04-21T08:39:57.903800+0000","last_peered":"2024-04-21T08:39:57.903800+0000","last_clean":"2024-04-21T08:39:57.903800+0000","last_became_active":"2024-04-21T08:37:31.448980+0000","last_became_peered":"2024-04-21T08:37:31.448980+0000","last_unstale":"2024-04-21T08:39:57.903800+0000","last_undegraded":"2024-04-21T08:39:57.903800+0000","last_fullsized":"2024-04-21T08:39:57.903800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:30:31.939274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151476+0000","last_change":"2024-04-21T08:37:35.760616+0000","last_active":"2024-04-21T08:39:58.151476+0000","last_peered":"2024-04-21T08:39:58.151476+0000","last_clean":"2024-04-21T08:39:58.151476+0000","last_became_active":"2024-04-21T08:37:35.760131+0000","last_became_peered":"2024-04-21T08:37:35.760131+0000","last_unstale":"2024-04-21T08:39:58.151476+0000","last_undegraded":"2024-04-21T08:39:58.151476+0000","last_fullsized":"2024-04-21T08:39:58.151476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:51.510259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009114+0000","last_change":"2024-04-21T08:38:30.682791+0000","last_active":"2024-04-21T08:39:58.009114+0000","last_peered":"2024-04-21T08:39:58.009114+0000","last_clean":"2024-04-21T08:39:58.009114+0000","last_became_active":"2024-04-21T08:38:30.671977+0000","last_became_peered":"2024-04-21T08:38:30.671977+0000","last_unstale":"2024-04-21T08:39:58.009114+0000","last_undegraded":"2024-04-21T08:39:58.009114+0000","last_fullsized":"2024-04-21T08:39:58.009114+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:55:15.899399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905569+0000","last_change":"2024-04-21T08:37:31.447234+0000","last_active":"2024-04-21T08:39:57.905569+0000","last_peered":"2024-04-21T08:39:57.905569+0000","last_clean":"2024-04-21T08:39:57.905569+0000","last_became_active":"2024-04-21T08:37:31.446876+0000","last_became_peered":"2024-04-21T08:37:31.446876+0000","last_unstale":"2024-04-21T08:39:57.905569+0000","last_undegraded":"2024-04-21T08:39:57.905569+0000","last_fullsized":"2024-04-21T08:39:57.905569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:51.784126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153981+0000","last_change":"2024-04-21T08:37:35.598577+0000","last_active":"2024-04-21T08:39:58.153981+0000","last_peered":"2024-04-21T08:39:58.153981+0000","last_clean":"2024-04-21T08:39:58.153981+0000","last_became_active":"2024-04-21T08:37:35.598119+0000","last_became_peered":"2024-04-21T08:37:35.598119+0000","last_unstale":"2024-04-21T08:39:58.153981+0000","last_undegraded":"2024-04-21T08:39:58.153981+0000","last_fullsized":"2024-04-21T08:39:58.153981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:41.986674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905332+0000","last_change":"2024-04-21T08:37:30.425737+0000","last_active":"2024-04-21T08:39:57.905332+0000","last_peered":"2024-04-21T08:39:57.905332+0000","last_clean":"2024-04-21T08:39:57.905332+0000","last_became_active":"2024-04-21T08:37:30.425180+0000","last_became_peered":"2024-04-21T08:37:30.425180+0000","last_unstale":"2024-04-21T08:39:57.905332+0000","last_undegraded":"2024-04-21T08:39:57.905332+0000","last_fullsized":"2024-04-21T08:39:57.905332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:08.646106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.906011+0000","last_change":"2024-04-21T08:37:31.451505+0000","last_active":"2024-04-21T08:39:57.906011+0000","last_peered":"2024-04-21T08:39:57.906011+0000","last_clean":"2024-04-21T08:39:57.906011+0000","last_became_active":"2024-04-21T08:37:31.451146+0000","last_became_peered":"2024-04-21T08:37:31.451146+0000","last_unstale":"2024-04-21T08:39:57.906011+0000","last_undegraded":"2024-04-21T08:39:57.906011+0000","last_fullsized":"2024-04-21T08:39:57.906011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:27:01.793910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319583+0000","last_change":"2024-04-21T08:37:35.608995+0000","last_active":"2024-04-21T08:39:58.319583+0000","last_peered":"2024-04-21T08:39:58.319583+0000","last_clean":"2024-04-21T08:39:58.319583+0000","last_became_active":"2024-04-21T08:37:35.608587+0000","last_became_peered":"2024-04-21T08:37:35.608587+0000","last_unstale":"2024-04-21T08:39:58.319583+0000","last_undegraded":"2024-04-21T08:39:58.319583+0000","last_fullsized":"2024-04-21T08:39:58.319583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.829418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.112239+0000","last_change":"2024-04-21T08:37:30.419026+0000","last_active":"2024-04-21T08:39:58.112239+0000","last_peered":"2024-04-21T08:39:58.112239+0000","last_clean":"2024-04-21T08:39:58.112239+0000","last_became_active":"2024-04-21T08:37:30.418706+0000","last_became_peered":"2024-04-21T08:37:30.418706+0000","last_unstale":"2024-04-21T08:39:58.112239+0000","last_undegraded":"2024-04-21T08:39:58.112239+0000","last_fullsized":"2024-04-21T08:39:58.112239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:24.441918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905192+0000","last_change":"2024-04-21T08:37:31.446561+0000","last_active":"2024-04-21T08:39:57.905192+0000","last_peered":"2024-04-21T08:39:57.905192+0000","last_clean":"2024-04-21T08:39:57.905192+0000","last_became_active":"2024-04-21T08:37:31.446162+0000","last_became_peered":"2024-04-21T08:37:31.446162+0000","last_unstale":"2024-04-21T08:39:57.905192+0000","last_undegraded":"2024-04-21T08:39:57.905192+0000","last_fullsized":"2024-04-21T08:39:57.905192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:38:24.684537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013588+0000","last_change":"2024-04-21T08:37:35.598748+0000","last_active":"2024-04-21T08:39:58.013588+0000","last_peered":"2024-04-21T08:39:58.013588+0000","last_clean":"2024-04-21T08:39:58.013588+0000","last_became_active":"2024-04-21T08:37:35.598353+0000","last_became_peered":"2024-04-21T08:37:35.598353+0000","last_unstale":"2024-04-21T08:39:58.013588+0000","last_undegraded":"2024-04-21T08:39:58.013588+0000","last_fullsized":"2024-04-21T08:39:58.013588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:47:36.519694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319054+0000","last_change":"2024-04-21T08:38:30.691292+0000","last_active":"2024-04-21T08:39:58.319054+0000","last_peered":"2024-04-21T08:39:58.319054+0000","last_clean":"2024-04-21T08:39:58.319054+0000","last_became_active":"2024-04-21T08:38:30.676580+0000","last_became_peered":"2024-04-21T08:38:30.676580+0000","last_unstale":"2024-04-21T08:39:58.319054+0000","last_undegraded":"2024-04-21T08:39:58.319054+0000","last_fullsized":"2024-04-21T08:39:58.319054+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:43:53.150369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905478+0000","last_change":"2024-04-21T08:37:31.452367+0000","last_active":"2024-04-21T08:39:57.905478+0000","last_peered":"2024-04-21T08:39:57.905478+0000","last_clean":"2024-04-21T08:39:57.905478+0000","last_became_active":"2024-04-21T08:37:31.452031+0000","last_became_peered":"2024-04-21T08:37:31.452031+0000","last_unstale":"2024-04-21T08:39:57.905478+0000","last_undegraded":"2024-04-21T08:39:57.905478+0000","last_fullsized":"2024-04-21T08:39:57.905478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:58.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318378+0000","last_change":"2024-04-21T08:37:31.451222+0000","last_active":"2024-04-21T08:39:58.318378+0000","last_peered":"2024-04-21T08:39:58.318378+0000","last_clean":"2024-04-21T08:39:58.318378+0000","last_became_active":"2024-04-21T08:37:31.450869+0000","last_became_peered":"2024-04-21T08:37:31.450869+0000","last_unstale":"2024-04-21T08:39:58.318378+0000","last_undegraded":"2024-04-21T08:39:58.318378+0000","last_fullsized":"2024-04-21T08:39:58.318378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:17.051330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905137+0000","last_change":"2024-04-21T08:37:31.453415+0000","last_active":"2024-04-21T08:39:57.905137+0000","last_peered":"2024-04-21T08:39:57.905137+0000","last_clean":"2024-04-21T08:39:57.905137+0000","last_became_active":"2024-04-21T08:37:31.453089+0000","last_became_peered":"2024-04-21T08:37:31.453089+0000","last_unstale":"2024-04-21T08:39:57.905137+0000","last_undegraded":"2024-04-21T08:39:57.905137+0000","last_fullsized":"2024-04-21T08:39:57.905137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:25:44.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904904+0000","last_change":"2024-04-21T08:37:31.446707+0000","last_active":"2024-04-21T08:39:57.904904+0000","last_peered":"2024-04-21T08:39:57.904904+0000","last_clean":"2024-04-21T08:39:57.904904+0000","last_became_active":"2024-04-21T08:37:31.446340+0000","last_became_peered":"2024-04-21T08:37:31.446340+0000","last_unstale":"2024-04-21T08:39:57.904904+0000","last_undegraded":"2024-04-21T08:39:57.904904+0000","last_fullsized":"2024-04-21T08:39:57.904904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:13.649607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319500+0000","last_change":"2024-04-21T08:37:31.455532+0000","last_active":"2024-04-21T08:39:58.319500+0000","last_peered":"2024-04-21T08:39:58.319500+0000","last_clean":"2024-04-21T08:39:58.319500+0000","last_became_active":"2024-04-21T08:37:31.453547+0000","last_became_peered":"2024-04-21T08:37:31.453547+0000","last_unstale":"2024-04-21T08:39:58.319500+0000","last_undegraded":"2024-04-21T08:39:58.319500+0000","last_fullsized":"2024-04-21T08:39:58.319500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:39:47.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318944+0000","last_change":"2024-04-21T08:39:31.679882+0000","last_active":"2024-04-21T08:39:58.318944+0000","last_peered":"2024-04-21T08:39:58.318944+0000","last_clean":"2024-04-21T08:39:58.318944+0000","last_became_active":"2024-04-21T08:39:31.679440+0000","last_became_peered":"2024-04-21T08:39:31.679440+0000","last_unstale":"2024-04-21T08:39:58.318944+0000","last_undegraded":"2024-04-21T08:39:58.318944+0000","last_fullsized":"2024-04-21T08:39:58.318944+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:57.679038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318685+0000","last_change":"2024-04-21T08:37:31.451806+0000","last_active":"2024-04-21T08:39:58.318685+0000","last_peered":"2024-04-21T08:39:58.318685+0000","last_clean":"2024-04-21T08:39:58.318685+0000","last_became_active":"2024-04-21T08:37:31.451442+0000","last_became_peered":"2024-04-21T08:37:31.451442+0000","last_unstale":"2024-04-21T08:39:58.318685+0000","last_undegraded":"2024-04-21T08:39:58.318685+0000","last_fullsized":"2024-04-21T08:39:58.318685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:07.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905544+0000","last_change":"2024-04-21T08:37:31.444440+0000","last_active":"2024-04-21T08:39:57.905544+0000","last_peered":"2024-04-21T08:39:57.905544+0000","last_clean":"2024-04-21T08:39:57.905544+0000","last_became_active":"2024-04-21T08:37:31.444079+0000","last_became_peered":"2024-04-21T08:37:31.444079+0000","last_unstale":"2024-04-21T08:39:57.905544+0000","last_undegraded":"2024-04-21T08:39:57.905544+0000","last_fullsized":"2024-04-21T08:39:57.905544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:13.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904413+0000","last_change":"2024-04-21T08:37:31.453698+0000","last_active":"2024-04-21T08:39:57.904413+0000","last_peered":"2024-04-21T08:39:57.904413+0000","last_clean":"2024-04-21T08:39:57.904413+0000","last_became_active":"2024-04-21T08:37:31.453379+0000","last_became_peered":"2024-04-21T08:37:31.453379+0000","last_unstale":"2024-04-21T08:39:57.904413+0000","last_undegraded":"2024-04-21T08:39:57.904413+0000","last_fullsized":"2024-04-21T08:39:57.904413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:46:29.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013777+0000","last_change":"2024-04-21T08:37:31.444480+0000","last_active":"2024-04-21T08:39:58.013777+0000","last_peered":"2024-04-21T08:39:58.013777+0000","last_clean":"2024-04-21T08:39:58.013777+0000","last_became_active":"2024-04-21T08:37:31.444030+0000","last_became_peered":"2024-04-21T08:37:31.444030+0000","last_unstale":"2024-04-21T08:39:58.013777+0000","last_undegraded":"2024-04-21T08:39:58.013777+0000","last_fullsized":"2024-04-21T08:39:58.013777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:15:05.668885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903935+0000","last_change":"2024-04-21T08:37:31.456602+0000","last_active":"2024-04-21T08:39:57.903935+0000","last_peered":"2024-04-21T08:39:57.903935+0000","last_clean":"2024-04-21T08:39:57.903935+0000","last_became_active":"2024-04-21T08:37:31.456286+0000","last_became_peered":"2024-04-21T08:37:31.456286+0000","last_unstale":"2024-04-21T08:39:57.903935+0000","last_undegraded":"2024-04-21T08:39:57.903935+0000","last_fullsized":"2024-04-21T08:39:57.903935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:48:12.462037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154176+0000","last_change":"2024-04-21T08:37:31.447008+0000","last_active":"2024-04-21T08:39:58.154176+0000","last_peered":"2024-04-21T08:39:58.154176+0000","last_clean":"2024-04-21T08:39:58.154176+0000","last_became_active":"2024-04-21T08:37:31.445192+0000","last_became_peered":"2024-04-21T08:37:31.445192+0000","last_unstale":"2024-04-21T08:39:58.154176+0000","last_undegraded":"2024-04-21T08:39:58.154176+0000","last_fullsized":"2024-04-21T08:39:58.154176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:08.025647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318698+0000","last_change":"2024-04-21T08:37:31.443288+0000","last_active":"2024-04-21T08:39:58.318698+0000","last_peered":"2024-04-21T08:39:58.318698+0000","last_clean":"2024-04-21T08:39:58.318698+0000","last_became_active":"2024-04-21T08:37:31.442586+0000","last_became_peered":"2024-04-21T08:37:31.442586+0000","last_unstale":"2024-04-21T08:39:58.318698+0000","last_undegraded":"2024-04-21T08:39:58.318698+0000","last_fullsized":"2024-04-21T08:39:58.318698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:01.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318501+0000","last_change":"2024-04-21T08:37:31.453899+0000","last_active":"2024-04-21T08:39:58.318501+0000","last_peered":"2024-04-21T08:39:58.318501+0000","last_clean":"2024-04-21T08:39:58.318501+0000","last_became_active":"2024-04-21T08:37:31.453527+0000","last_became_peered":"2024-04-21T08:37:31.453527+0000","last_unstale":"2024-04-21T08:39:58.318501+0000","last_undegraded":"2024-04-21T08:39:58.318501+0000","last_fullsized":"2024-04-21T08:39:58.318501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:23:18.429618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108246,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600116,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":864,"num_read_kb":545,"num_write":289,"num_write_kb":1443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1800348,"data_compressed":80889,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":529807,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32348,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712804,"statfs":{"total":95995035648,"available":95961911296,"internally_reserved":0,"allocated":1200128,"data_stored":1320365,"data_compressed":73597,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":927355,"data_compressed":48844,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":922597,"data_compressed":48844,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1127103,"data_compressed":65780,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1028096,"data_stored":525219,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":585028,"data_compressed":28357,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1052672,"data_stored":526607,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1122304,"data_stored":595765,"data_compressed":31359,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1050229,"data_compressed":55000,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":520740,"data_compressed":26696,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1126400,"data_stored":1252669,"data_compressed":71936,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70650,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71418,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133076,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393372,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":204868,"data_compressed":7817,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:40:03.646 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:40:03.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:40:03.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: Updating smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:40:03.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: Updating smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: Updating smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2022328867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:03.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:40:03.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:03 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3638975160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:03.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.780+0000 7fe10876d640 1 -- 172.21.15.94:0/3000297826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe100106f50 msgr2=0x7fe100107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.780+0000 7fe10876d640 1 --2- 172.21.15.94:0/3000297826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe100106f50 0x7fe100107330 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009930 tx=0x7fe0fc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.780+0000 7fe10876d640 1 -- 172.21.15.94:0/3000297826 shutdown_connections 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.780+0000 7fe10876d640 1 --2- 172.21.15.94:0/3000297826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe100100e80 0x7fe10010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.780+0000 7fe10876d640 1 --2- 172.21.15.94:0/3000297826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1001004c0 0x7fe100100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.780+0000 7fe10876d640 1 --2- 172.21.15.94:0/3000297826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe100106f50 0x7fe100107330 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.780+0000 7fe10876d640 1 -- 172.21.15.94:0/3000297826 >> 172.21.15.94:0/3000297826 conn(0x7fe100077450 msgr2=0x7fe1000fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.780+0000 7fe10876d640 1 -- 172.21.15.94:0/3000297826 shutdown_connections 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.780+0000 7fe10876d640 1 -- 172.21.15.94:0/3000297826 wait complete. 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe10876d640 1 Processor -- start 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe10876d640 1 -- start start 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe10876d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1001004c0 0x7fe1001016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe10876d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe100100e80 0x7fe100101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe10876d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe100106f50 0x7fe100102120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe10876d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe100112280 con 0x7fe100106f50 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe10876d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe100112100 con 0x7fe100100e80 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe10876d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe100112400 con 0x7fe1001004c0 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe106d0a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe100106f50 0x7fe100102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe106d0a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe100106f50 0x7fe100102120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36466/0 (socket says 172.21.15.94:36466) 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe106d0a640 1 -- 172.21.15.94:0/1969915160 learned_addr learned my addr 172.21.15.94:0/1969915160 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe105d08640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe100100e80 0x7fe100101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe106d0a640 1 -- 172.21.15.94:0/1969915160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1001004c0 msgr2=0x7fe1001016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe106d0a640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1001004c0 0x7fe1001016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe106d0a640 1 -- 172.21.15.94:0/1969915160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe100100e80 msgr2=0x7fe100101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe106d0a640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe100100e80 0x7fe100101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe106d0a640 1 -- 172.21.15.94:0/1969915160 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1001a6cc0 con 0x7fe100106f50 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.781+0000 7fe106d0a640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe100106f50 0x7fe100102120 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800eaa0 tx=0x7fe0f800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.782+0000 7fe0f77fe640 1 -- 172.21.15.94:0/1969915160 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f800cdb0 con 0x7fe100106f50 2024-04-21T08:40:03.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.782+0000 7fe10876d640 1 -- 172.21.15.94:0/1969915160 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1001a6fb0 con 0x7fe100106f50 2024-04-21T08:40:03.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.782+0000 7fe0f77fe640 1 -- 172.21.15.94:0/1969915160 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0f8005020 con 0x7fe100106f50 2024-04-21T08:40:03.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.782+0000 7fe0f77fe640 1 -- 172.21.15.94:0/1969915160 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f80116d0 con 0x7fe100106f50 2024-04-21T08:40:03.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.782+0000 7fe10876d640 1 -- 172.21.15.94:0/1969915160 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1001a74d0 con 0x7fe100106f50 2024-04-21T08:40:03.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.783+0000 7fe10876d640 1 -- 172.21.15.94:0/1969915160 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe100069230 con 0x7fe100106f50 2024-04-21T08:40:03.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.784+0000 7fe0f77fe640 1 -- 172.21.15.94:0/1969915160 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fe0f80054a0 con 0x7fe100106f50 2024-04-21T08:40:03.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.784+0000 7fe0f77fe640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe0ec0a3e40 0x7fe0ec0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:03.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.785+0000 7fe106509640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe0ec0a3e40 0x7fe0ec0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:03.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.785+0000 7fe0f77fe640 1 -- 172.21.15.94:0/1969915160 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe0f80d13a0 con 0x7fe100106f50 2024-04-21T08:40:03.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.785+0000 7fe106509640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe0ec0a3e40 0x7fe0ec0a6300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc036040 tx=0x7fe0fc000bc0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:03.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.788+0000 7fe0f77fe640 1 -- 172.21.15.94:0/1969915160 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe0f8016030 con 0x7fe100106f50 2024-04-21T08:40:03.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.968+0000 7fe10876d640 1 -- 172.21.15.94:0/1969915160 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe100102bd0 con 0x7fe100106f50 2024-04-21T08:40:03.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.970+0000 7fe0f77fe640 1 -- 172.21.15.94:0/1969915160 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25266 (secure 0 0 0) 0x7fe0f809d410 con 0x7fe100106f50 2024-04-21T08:40:03.970 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:03.970 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:57.898777+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:40:03.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 -- 172.21.15.94:0/1969915160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe0ec0a3e40 msgr2=0x7fe0ec0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe0ec0a3e40 0x7fe0ec0a6300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc036040 tx=0x7fe0fc000bc0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 -- 172.21.15.94:0/1969915160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe100106f50 msgr2=0x7fe100102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:03.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe100106f50 0x7fe100102120 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800eaa0 tx=0x7fe0f800ef60 comp rx=0 tx=0).stop 2024-04-21T08:40:03.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 -- 172.21.15.94:0/1969915160 shutdown_connections 2024-04-21T08:40:03.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe0ec0a3e40 0x7fe0ec0a6300 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe100106f50 0x7fe100102120 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe100100e80 0x7fe100101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 --2- 172.21.15.94:0/1969915160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1001004c0 0x7fe1001016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:03.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 -- 172.21.15.94:0/1969915160 >> 172.21.15.94:0/1969915160 conn(0x7fe100077450 msgr2=0x7fe100105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:03.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 -- 172.21.15.94:0/1969915160 shutdown_connections 2024-04-21T08:40:03.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:03.972+0000 7fe10876d640 1 -- 172.21.15.94:0/1969915160 wait complete. 2024-04-21T08:40:03.983 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:40:03.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:40:04.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.084+0000 7f8edd08c640 1 -- 172.21.15.94:0/3304975930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed8102190 msgr2=0x7f8ed8102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.084+0000 7f8edd08c640 1 --2- 172.21.15.94:0/3304975930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed8102190 0x7f8ed8102570 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009900 tx=0x7f8ecc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.084+0000 7f8edd08c640 1 -- 172.21.15.94:0/3304975930 shutdown_connections 2024-04-21T08:40:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.084+0000 7f8edd08c640 1 --2- 172.21.15.94:0/3304975930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed8107260 0x7f8ed810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.084+0000 7f8edd08c640 1 --2- 172.21.15.94:0/3304975930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed8102b40 0x7f8ed8106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.084+0000 7f8edd08c640 1 --2- 172.21.15.94:0/3304975930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed8102190 0x7f8ed8102570 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.084+0000 7f8edd08c640 1 -- 172.21.15.94:0/3304975930 >> 172.21.15.94:0/3304975930 conn(0x7f8ed80fd990 msgr2=0x7f8ed80ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.084+0000 7f8edd08c640 1 -- 172.21.15.94:0/3304975930 shutdown_connections 2024-04-21T08:40:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.084+0000 7f8edd08c640 1 -- 172.21.15.94:0/3304975930 wait complete. 2024-04-21T08:40:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.085+0000 7f8edd08c640 1 Processor -- start 2024-04-21T08:40:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.085+0000 7f8edd08c640 1 -- start start 2024-04-21T08:40:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.085+0000 7f8edd08c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed8102190 0x7f8ed819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.085+0000 7f8edd08c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed8107260 0x7f8ed819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.085+0000 7f8edd08c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed81985b0 0x7f8ed8198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.085+0000 7f8edd08c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed81121a0 con 0x7f8ed81985b0 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.085+0000 7f8edd08c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed8112020 con 0x7f8ed8102190 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.085+0000 7f8edd08c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed8112320 con 0x7f8ed8107260 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed81985b0 0x7f8ed8198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed81985b0 0x7f8ed8198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36476/0 (socket says 172.21.15.94:36476) 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed759e640 1 -- 172.21.15.94:0/4033889897 learned_addr learned my addr 172.21.15.94:0/4033889897 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed659c640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed8107260 0x7f8ed819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed6d9d640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed8102190 0x7f8ed819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed759e640 1 -- 172.21.15.94:0/4033889897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed8107260 msgr2=0x7f8ed819e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed759e640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed8107260 0x7f8ed819e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed759e640 1 -- 172.21.15.94:0/4033889897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed8102190 msgr2=0x7f8ed819e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed759e640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed8102190 0x7f8ed819e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed759e640 1 -- 172.21.15.94:0/4033889897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed81990d0 con 0x7f8ed81985b0 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed659c640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed8107260 0x7f8ed819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed6d9d640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed8102190 0x7f8ed819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ed759e640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed81985b0 0x7f8ed8198a60 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000eaa0 tx=0x7f8ed000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ebffff640 1 -- 172.21.15.94:0/4033889897 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed000cdb0 con 0x7f8ed81985b0 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ebffff640 1 -- 172.21.15.94:0/4033889897 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ed0005020 con 0x7f8ed81985b0 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8edd08c640 1 -- 172.21.15.94:0/4033889897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed81993c0 con 0x7f8ed81985b0 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8ebffff640 1 -- 172.21.15.94:0/4033889897 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed00116d0 con 0x7f8ed81985b0 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.086+0000 7f8edd08c640 1 -- 172.21.15.94:0/4033889897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed81a55c0 con 0x7f8ed81985b0 2024-04-21T08:40:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.087+0000 7f8edd08c640 1 -- 172.21.15.94:0/4033889897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed80691c0 con 0x7f8ed81985b0 2024-04-21T08:40:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.088+0000 7f8ebffff640 1 -- 172.21.15.94:0/4033889897 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f8ed00051d0 con 0x7f8ed81985b0 2024-04-21T08:40:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.089+0000 7f8ebffff640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8ec00a3e40 0x7f8ec00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.089+0000 7f8ed6d9d640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8ec00a3e40 0x7f8ec00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.089+0000 7f8ebffff640 1 -- 172.21.15.94:0/4033889897 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f8ed00d10e0 con 0x7f8ed81985b0 2024-04-21T08:40:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.089+0000 7f8ed6d9d640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8ec00a3e40 0x7f8ec00a6300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc036040 tx=0x7f8ecc000bc0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.093+0000 7f8ebffff640 1 -- 172.21.15.94:0/4033889897 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ed0016030 con 0x7f8ed81985b0 2024-04-21T08:40:04.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:04 smithi094 ceph-mon[27988]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:04.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:04 smithi094 ceph-mon[27988]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:04.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:04 smithi094 ceph-mon[27988]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:04.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:04 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1969915160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:04.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.270+0000 7f8edd08c640 1 -- 172.21.15.94:0/4033889897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8ed8102b40 con 0x7f8ed81985b0 2024-04-21T08:40:04.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.270+0000 7f8ebffff640 1 -- 172.21.15.94:0/4033889897 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f8ed009d150 con 0x7f8ed81985b0 2024-04-21T08:40:04.270 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:40:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 -- 172.21.15.94:0/4033889897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8ec00a3e40 msgr2=0x7f8ec00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8ec00a3e40 0x7f8ec00a6300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc036040 tx=0x7f8ecc000bc0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 -- 172.21.15.94:0/4033889897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed81985b0 msgr2=0x7f8ed8198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed81985b0 0x7f8ed8198a60 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000eaa0 tx=0x7f8ed000ef60 comp rx=0 tx=0).stop 2024-04-21T08:40:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 -- 172.21.15.94:0/4033889897 shutdown_connections 2024-04-21T08:40:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8ec00a3e40 0x7f8ec00a6300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ed8107260 0x7f8ed819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed81985b0 0x7f8ed8198a60 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 --2- 172.21.15.94:0/4033889897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed8102190 0x7f8ed819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 -- 172.21.15.94:0/4033889897 >> 172.21.15.94:0/4033889897 conn(0x7f8ed80fd990 msgr2=0x7f8ed80fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 -- 172.21.15.94:0/4033889897 shutdown_connections 2024-04-21T08:40:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.272+0000 7f8edd08c640 1 -- 172.21.15.94:0/4033889897 wait complete. 2024-04-21T08:40:04.283 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 -- 172.21.15.94:0/713163824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77d4106160 msgr2=0x7f77d410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 --2- 172.21.15.94:0/713163824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77d4106160 0x7f77d410dd30 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f77cc009900 tx=0x7f77cc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 -- 172.21.15.94:0/713163824 shutdown_connections 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 --2- 172.21.15.94:0/713163824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77d4101a40 0x7f77d4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 --2- 172.21.15.94:0/713163824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77d4106160 0x7f77d410dd30 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 --2- 172.21.15.94:0/713163824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77d4101090 0x7f77d4101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 -- 172.21.15.94:0/713163824 >> 172.21.15.94:0/713163824 conn(0x7f77d4077f50 msgr2=0x7f77d4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 -- 172.21.15.94:0/713163824 shutdown_connections 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 -- 172.21.15.94:0/713163824 wait complete. 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 Processor -- start 2024-04-21T08:40:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 -- start start 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77d4101090 0x7f77d41995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77d4106160 0x7f77d4199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77d419a090 0x7f77d41a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4111120 con 0x7f77d4106160 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4110fa0 con 0x7f77d4101090 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.385+0000 7f77db6b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f77d41112a0 con 0x7f77d419a090 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d8c4d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77d4106160 0x7f77d4199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d8c4d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77d4106160 0x7f77d4199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36500/0 (socket says 172.21.15.94:36500) 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d8c4d640 1 -- 172.21.15.94:0/1588087154 learned_addr learned my addr 172.21.15.94:0/1588087154 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d9c4f640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77d419a090 0x7f77d41a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d944e640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77d4101090 0x7f77d41995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d8c4d640 1 -- 172.21.15.94:0/1588087154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77d419a090 msgr2=0x7f77d41a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d8c4d640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77d419a090 0x7f77d41a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d8c4d640 1 -- 172.21.15.94:0/1588087154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77d4101090 msgr2=0x7f77d41995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d8c4d640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77d4101090 0x7f77d41995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d8c4d640 1 -- 172.21.15.94:0/1588087154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d41a4e10 con 0x7f77d4106160 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d944e640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77d4101090 0x7f77d41995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77d8c4d640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77d4106160 0x7f77d4199b00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f77c400b450 tx=0x7f77c400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77ca7fc640 1 -- 172.21.15.94:0/1588087154 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c4015020 con 0x7f77d4106160 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77db6b2640 1 -- 172.21.15.94:0/1588087154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d41a5100 con 0x7f77d4106160 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77ca7fc640 1 -- 172.21.15.94:0/1588087154 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77c4011d40 con 0x7f77d4106160 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77ca7fc640 1 -- 172.21.15.94:0/1588087154 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c400f340 con 0x7f77d4106160 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.386+0000 7f77db6b2640 1 -- 172.21.15.94:0/1588087154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d41a5600 con 0x7f77d4106160 2024-04-21T08:40:04.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.387+0000 7f77db6b2640 1 -- 172.21.15.94:0/1588087154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77d4068d70 con 0x7f77d4106160 2024-04-21T08:40:04.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.388+0000 7f77ca7fc640 1 -- 172.21.15.94:0/1588087154 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f77c4018020 con 0x7f77d4106160 2024-04-21T08:40:04.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.389+0000 7f77ca7fc640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f77c00a3e40 0x7f77c00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.389+0000 7f77d944e640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f77c00a3e40 0x7f77c00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.389+0000 7f77ca7fc640 1 -- 172.21.15.94:0/1588087154 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f77c40d0890 con 0x7f77d4106160 2024-04-21T08:40:04.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.389+0000 7f77d944e640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f77c00a3e40 0x7f77c00a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77d0005fd0 tx=0x7f77d0009520 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:04.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.393+0000 7f77ca7fc640 1 -- 172.21.15.94:0/1588087154 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77c401f020 con 0x7f77d4106160 2024-04-21T08:40:04.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:04 smithi173 ceph-mon[31428]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:04.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:04 smithi173 ceph-mon[31428]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:04.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:04 smithi173 ceph-mon[31428]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:04.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:04 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1969915160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.568+0000 7f77db6b2640 1 -- 172.21.15.94:0/1588087154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f77d4109390 con 0x7f77d4106160 2024-04-21T08:40:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.568+0000 7f77ca7fc640 1 -- 172.21.15.94:0/1588087154 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f77c400d040 con 0x7f77d4106160 2024-04-21T08:40:04.568 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 -- 172.21.15.94:0/1588087154 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f77c00a3e40 msgr2=0x7f77c00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f77c00a3e40 0x7f77c00a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77d0005fd0 tx=0x7f77d0009520 comp rx=0 tx=0).stop 2024-04-21T08:40:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 -- 172.21.15.94:0/1588087154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77d4106160 msgr2=0x7f77d4199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77d4106160 0x7f77d4199b00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f77c400b450 tx=0x7f77c400b910 comp rx=0 tx=0).stop 2024-04-21T08:40:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 -- 172.21.15.94:0/1588087154 shutdown_connections 2024-04-21T08:40:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f77c00a3e40 0x7f77c00a6300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77d419a090 0x7f77d41a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77d4106160 0x7f77d4199b00 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 --2- 172.21.15.94:0/1588087154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77d4101090 0x7f77d41995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 -- 172.21.15.94:0/1588087154 >> 172.21.15.94:0/1588087154 conn(0x7f77d4077f50 msgr2=0x7f77d40ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 -- 172.21.15.94:0/1588087154 shutdown_connections 2024-04-21T08:40:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.570+0000 7f77db6b2640 1 -- 172.21.15.94:0/1588087154 wait complete. 2024-04-21T08:40:04.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:40:04.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.683+0000 7ff298575640 1 -- 172.21.15.94:0/1819193039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff290107260 msgr2=0x7ff29010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.683+0000 7ff298575640 1 --2- 172.21.15.94:0/1819193039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff290107260 0x7ff29010ee30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff288009ca0 tx=0x7ff28802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.683+0000 7ff298575640 1 -- 172.21.15.94:0/1819193039 shutdown_connections 2024-04-21T08:40:04.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.683+0000 7ff298575640 1 --2- 172.21.15.94:0/1819193039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff290102b40 0x7ff290106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.683+0000 7ff298575640 1 --2- 172.21.15.94:0/1819193039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff290107260 0x7ff29010ee30 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.683+0000 7ff298575640 1 --2- 172.21.15.94:0/1819193039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff290102190 0x7ff290102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.683+0000 7ff298575640 1 -- 172.21.15.94:0/1819193039 >> 172.21.15.94:0/1819193039 conn(0x7ff2900793a0 msgr2=0x7ff2900797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff298575640 1 -- 172.21.15.94:0/1819193039 shutdown_connections 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff298575640 1 -- 172.21.15.94:0/1819193039 wait complete. 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff298575640 1 Processor -- start 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff298575640 1 -- start start 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff298575640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff290102190 0x7ff290198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff298575640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff290107260 0x7ff290198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff298575640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff29019cfd0 0x7ff29019d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff298575640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901121b0 con 0x7ff290102190 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff298575640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff290112030 con 0x7ff290107260 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff298575640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff290112330 con 0x7ff29019cfd0 2024-04-21T08:40:04.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff296311640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff290102190 0x7ff290198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff296311640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff290102190 0x7ff290198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36520/0 (socket says 172.21.15.94:36520) 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.684+0000 7ff296311640 1 -- 172.21.15.94:0/815627707 learned_addr learned my addr 172.21.15.94:0/815627707 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff296b12640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff29019cfd0 0x7ff29019d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff296311640 1 -- 172.21.15.94:0/815627707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff29019cfd0 msgr2=0x7ff29019d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff296311640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff29019cfd0 0x7ff29019d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff296311640 1 -- 172.21.15.94:0/815627707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff290107260 msgr2=0x7ff290198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff296311640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff290107260 0x7ff290198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff296311640 1 -- 172.21.15.94:0/815627707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29019dac0 con 0x7ff290102190 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff296311640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff290102190 0x7ff290198710 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00e9f0 tx=0x7ff28c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff2877fe640 1 -- 172.21.15.94:0/815627707 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c00cd80 con 0x7ff290102190 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff2877fe640 1 -- 172.21.15.94:0/815627707 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff28c0108e0 con 0x7ff290102190 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff298575640 1 -- 172.21.15.94:0/815627707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29019ddb0 con 0x7ff290102190 2024-04-21T08:40:04.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff2877fe640 1 -- 172.21.15.94:0/815627707 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c018980 con 0x7ff290102190 2024-04-21T08:40:04.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.685+0000 7ff298575640 1 -- 172.21.15.94:0/815627707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2901a55f0 con 0x7ff290102190 2024-04-21T08:40:04.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:04 smithi111 ceph-mon[31493]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:04.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:04 smithi111 ceph-mon[31493]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:04.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:04 smithi111 ceph-mon[31493]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:04.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:04 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1969915160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:04.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.686+0000 7ff298575640 1 -- 172.21.15.94:0/815627707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2900691c0 con 0x7ff290102190 2024-04-21T08:40:04.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.687+0000 7ff2877fe640 1 -- 172.21.15.94:0/815627707 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7ff28c010430 con 0x7ff290102190 2024-04-21T08:40:04.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.687+0000 7ff2877fe640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff27c0a3ec0 0x7ff27c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.688+0000 7ff295b10640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff27c0a3ec0 0x7ff27c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.688+0000 7ff2877fe640 1 -- 172.21.15.94:0/815627707 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff28c0d12c0 con 0x7ff290102190 2024-04-21T08:40:04.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.688+0000 7ff295b10640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff27c0a3ec0 0x7ff27c0a6380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff280005fd0 tx=0x7ff280005f40 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.692+0000 7ff2877fe640 1 -- 172.21.15.94:0/815627707 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff28c09d330 con 0x7ff290102190 2024-04-21T08:40:04.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.871+0000 7ff298575640 1 -- 172.21.15.94:0/815627707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff29010a4f0 con 0x7ff290102190 2024-04-21T08:40:04.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.871+0000 7ff2877fe640 1 -- 172.21.15.94:0/815627707 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7ff28c09cc30 con 0x7ff290102190 2024-04-21T08:40:04.871 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:40:04.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.873+0000 7ff298575640 1 -- 172.21.15.94:0/815627707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff27c0a3ec0 msgr2=0x7ff27c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.873+0000 7ff298575640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff27c0a3ec0 0x7ff27c0a6380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff280005fd0 tx=0x7ff280005f40 comp rx=0 tx=0).stop 2024-04-21T08:40:04.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.873+0000 7ff298575640 1 -- 172.21.15.94:0/815627707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff290102190 msgr2=0x7ff290198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.873+0000 7ff298575640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff290102190 0x7ff290198710 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00e9f0 tx=0x7ff28c00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.873+0000 7ff298575640 1 -- 172.21.15.94:0/815627707 shutdown_connections 2024-04-21T08:40:04.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.873+0000 7ff298575640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff27c0a3ec0 0x7ff27c0a6380 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.873+0000 7ff298575640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff29019cfd0 0x7ff29019d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.873+0000 7ff298575640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff290107260 0x7ff290198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.873+0000 7ff298575640 1 --2- 172.21.15.94:0/815627707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff290102190 0x7ff290198710 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.873+0000 7ff298575640 1 -- 172.21.15.94:0/815627707 >> 172.21.15.94:0/815627707 conn(0x7ff2900793a0 msgr2=0x7ff2900fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:04.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.874+0000 7ff298575640 1 -- 172.21.15.94:0/815627707 shutdown_connections 2024-04-21T08:40:04.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.874+0000 7ff298575640 1 -- 172.21.15.94:0/815627707 wait complete. 2024-04-21T08:40:04.884 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:40:04.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 -- 172.21.15.94:0/2866154856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec0103af0 msgr2=0x7f9ec010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 --2- 172.21.15.94:0/2866154856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec0103af0 0x7f9ec010bfd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009900 tx=0x7f9eb002bd60 comp rx=0 tx=0).stop 2024-04-21T08:40:04.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 -- 172.21.15.94:0/2866154856 shutdown_connections 2024-04-21T08:40:04.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 --2- 172.21.15.94:0/2866154856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec010c510 0x7f9ec010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 --2- 172.21.15.94:0/2866154856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec0103af0 0x7f9ec010bfd0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 --2- 172.21.15.94:0/2866154856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec01031d0 0x7f9ec01035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 -- 172.21.15.94:0/2866154856 >> 172.21.15.94:0/2866154856 conn(0x7f9ec0077f50 msgr2=0x7f9ec0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 -- 172.21.15.94:0/2866154856 shutdown_connections 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 -- 172.21.15.94:0/2866154856 wait complete. 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 Processor -- start 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.986+0000 7f9ec744e640 1 -- start start 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec744e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec01031d0 0x7f9ec01a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec744e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec0103af0 0x7f9ec01a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec744e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec010c510 0x7f9ec019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec744e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec01121a0 con 0x7f9ec010c510 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec744e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec0112020 con 0x7f9ec01031d0 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec744e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec0112320 con 0x7f9ec0103af0 2024-04-21T08:40:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec59eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec010c510 0x7f9ec019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec59eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec010c510 0x7f9ec019a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36544/0 (socket says 172.21.15.94:36544) 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec59eb640 1 -- 172.21.15.94:0/4117860937 learned_addr learned my addr 172.21.15.94:0/4117860937 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec51ea640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec01031d0 0x7f9ec01a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec49e9640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec0103af0 0x7f9ec01a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec59eb640 1 -- 172.21.15.94:0/4117860937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec0103af0 msgr2=0x7f9ec01a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec59eb640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec0103af0 0x7f9ec01a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec59eb640 1 -- 172.21.15.94:0/4117860937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec01031d0 msgr2=0x7f9ec01a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec59eb640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec01031d0 0x7f9ec01a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec59eb640 1 -- 172.21.15.94:0/4117860937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec019af70 con 0x7f9ec010c510 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec59eb640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec010c510 0x7f9ec019a7a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800cad0 tx=0x7f9eb800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9eb67fc640 1 -- 172.21.15.94:0/4117860937 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb800ae20 con 0x7f9ec010c510 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9eb67fc640 1 -- 172.21.15.94:0/4117860937 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eb8005020 con 0x7f9ec010c510 2024-04-21T08:40:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec744e640 1 -- 172.21.15.94:0/4117860937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ec019b260 con 0x7f9ec010c510 2024-04-21T08:40:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9eb67fc640 1 -- 172.21.15.94:0/4117860937 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb800f6b0 con 0x7f9ec010c510 2024-04-21T08:40:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.987+0000 7f9ec744e640 1 -- 172.21.15.94:0/4117860937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ec01a7480 con 0x7f9ec010c510 2024-04-21T08:40:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.988+0000 7f9ec744e640 1 -- 172.21.15.94:0/4117860937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ec00698e0 con 0x7f9ec010c510 2024-04-21T08:40:04.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.990+0000 7f9eb67fc640 1 -- 172.21.15.94:0/4117860937 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f9eb80051d0 con 0x7f9ec010c510 2024-04-21T08:40:04.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.990+0000 7f9eb67fc640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9eac0a3e40 0x7f9eac0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:04.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.990+0000 7f9ec51ea640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9eac0a3e40 0x7f9eac0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:04.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.990+0000 7f9eb67fc640 1 -- 172.21.15.94:0/4117860937 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9eb80d1230 con 0x7f9ec010c510 2024-04-21T08:40:04.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.991+0000 7f9ec51ea640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9eac0a3e40 0x7f9eac0a6300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc005fd0 tx=0x7f9ebc00b390 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:04.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:04.994+0000 7f9eb67fc640 1 -- 172.21.15.94:0/4117860937 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9eb8014030 con 0x7f9ec010c510 2024-04-21T08:40:05.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.169+0000 7f9ec744e640 1 -- 172.21.15.94:0/4117860937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9ec01a77b0 con 0x7f9ec010c510 2024-04-21T08:40:05.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.170+0000 7f9eb67fc640 1 -- 172.21.15.94:0/4117860937 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f9eb809d2a0 con 0x7f9ec010c510 2024-04-21T08:40:05.170 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 -- 172.21.15.94:0/4117860937 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9eac0a3e40 msgr2=0x7f9eac0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9eac0a3e40 0x7f9eac0a6300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc005fd0 tx=0x7f9ebc00b390 comp rx=0 tx=0).stop 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 -- 172.21.15.94:0/4117860937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec010c510 msgr2=0x7f9ec019a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec010c510 0x7f9ec019a7a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800cad0 tx=0x7f9eb800cf90 comp rx=0 tx=0).stop 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 -- 172.21.15.94:0/4117860937 shutdown_connections 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9eac0a3e40 0x7f9eac0a6300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ec0103af0 0x7f9ec01a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ec010c510 0x7f9ec019a7a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 --2- 172.21.15.94:0/4117860937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec01031d0 0x7f9ec01a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 -- 172.21.15.94:0/4117860937 >> 172.21.15.94:0/4117860937 conn(0x7f9ec0077f50 msgr2=0x7f9ec00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 -- 172.21.15.94:0/4117860937 shutdown_connections 2024-04-21T08:40:05.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.172+0000 7f9ec744e640 1 -- 172.21.15.94:0/4117860937 wait complete. 2024-04-21T08:40:05.184 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:40:05.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.283+0000 7f1fad401640 1 -- 172.21.15.94:0/3363652732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa81018d0 msgr2=0x7f1fa8105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.283+0000 7f1fad401640 1 --2- 172.21.15.94:0/3363652732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa81018d0 0x7f1fa8105910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1f90009900 tx=0x7f1f9002bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:05.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.283+0000 7f1fad401640 1 -- 172.21.15.94:0/3363652732 shutdown_connections 2024-04-21T08:40:05.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.283+0000 7f1fad401640 1 --2- 172.21.15.94:0/3363652732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fa8105ff0 0x7f1fa810dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.283+0000 7f1fad401640 1 --2- 172.21.15.94:0/3363652732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa81018d0 0x7f1fa8105910 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.283+0000 7f1fad401640 1 --2- 172.21.15.94:0/3363652732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fa8100f20 0x7f1fa8101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.283+0000 7f1fad401640 1 -- 172.21.15.94:0/3363652732 >> 172.21.15.94:0/3363652732 conn(0x7f1fa80fc7a0 msgr2=0x7f1fa80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.284+0000 7f1fad401640 1 -- 172.21.15.94:0/3363652732 shutdown_connections 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.284+0000 7f1fad401640 1 -- 172.21.15.94:0/3363652732 wait complete. 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.284+0000 7f1fad401640 1 Processor -- start 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.284+0000 7f1fad401640 1 -- start start 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.284+0000 7f1fad401640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fa8100f20 0x7f1fa8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.284+0000 7f1fad401640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fa8105ff0 0x7f1fa8198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.284+0000 7f1fad401640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa819cf00 0x7f1fa819d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fad401640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa81120b0 con 0x7f1fa819cf00 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fad401640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa8111f30 con 0x7f1fa8100f20 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fad401640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa8112230 con 0x7f1fa8105ff0 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa819cf00 0x7f1fa819d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa819cf00 0x7f1fa819d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36558/0 (socket says 172.21.15.94:36558) 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa77fe640 1 -- 172.21.15.94:0/1433594137 learned_addr learned my addr 172.21.15.94:0/1433594137 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa67fc640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fa8105ff0 0x7f1fa8198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa6ffd640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fa8100f20 0x7f1fa8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa77fe640 1 -- 172.21.15.94:0/1433594137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fa8105ff0 msgr2=0x7f1fa8198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa77fe640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fa8105ff0 0x7f1fa8198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa77fe640 1 -- 172.21.15.94:0/1433594137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fa8100f20 msgr2=0x7f1fa8198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa77fe640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fa8100f20 0x7f1fa8198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa77fe640 1 -- 172.21.15.94:0/1433594137 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fa819d9f0 con 0x7f1fa819cf00 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa67fc640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fa8105ff0 0x7f1fa8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa6ffd640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fa8100f20 0x7f1fa8198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fa77fe640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa819cf00 0x7f1fa819d3b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1f9800cad0 tx=0x7f1f9800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1f9ffff640 1 -- 172.21.15.94:0/1433594137 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9800ae20 con 0x7f1fa819cf00 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fad401640 1 -- 172.21.15.94:0/1433594137 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fa819dce0 con 0x7f1fa819cf00 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1f9ffff640 1 -- 172.21.15.94:0/1433594137 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f98005020 con 0x7f1fa819cf00 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1f9ffff640 1 -- 172.21.15.94:0/1433594137 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9800f6b0 con 0x7f1fa819cf00 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.285+0000 7f1fad401640 1 -- 172.21.15.94:0/1433594137 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fa81a5520 con 0x7f1fa819cf00 2024-04-21T08:40:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.286+0000 7f1fad401640 1 -- 172.21.15.94:0/1433594137 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fa8068c90 con 0x7f1fa819cf00 2024-04-21T08:40:05.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.287+0000 7f1f9ffff640 1 -- 172.21.15.94:0/1433594137 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f1f980051d0 con 0x7f1fa819cf00 2024-04-21T08:40:05.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.288+0000 7f1f9ffff640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1f940a3e40 0x7f1f940a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.288+0000 7f1fa6ffd640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1f940a3e40 0x7f1f940a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.288+0000 7f1f9ffff640 1 -- 172.21.15.94:0/1433594137 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f1f980d0fd0 con 0x7f1fa819cf00 2024-04-21T08:40:05.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.288+0000 7f1fa6ffd640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1f940a3e40 0x7f1f940a6300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0006fd0 tx=0x7f1fa0008040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.292+0000 7f1f9ffff640 1 -- 172.21.15.94:0/1433594137 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f98014030 con 0x7f1fa819cf00 2024-04-21T08:40:05.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.508+0000 7f1fad401640 1 -- 172.21.15.94:0/1433594137 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1fa81018d0 con 0x7f1fa819cf00 2024-04-21T08:40:05.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.508+0000 7f1f9ffff640 1 -- 172.21.15.94:0/1433594137 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f1f9809d040 con 0x7f1fa819cf00 2024-04-21T08:40:05.508 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:05.508 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:40:05.509 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:40:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:05 smithi094 ceph-mon[27988]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 410 B/s wr, 0 op/s 2024-04-21T08:40:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:05 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4033889897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:05 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1588087154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:05 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/815627707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:05 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4117860937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:05 smithi173 ceph-mon[31428]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 410 B/s wr, 0 op/s 2024-04-21T08:40:05.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:05 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4033889897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:05 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1588087154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:05 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/815627707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:05 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4117860937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 -- 172.21.15.94:0/1433594137 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1f940a3e40 msgr2=0x7f1f940a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1f940a3e40 0x7f1f940a6300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0006fd0 tx=0x7f1fa0008040 comp rx=0 tx=0).stop 2024-04-21T08:40:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 -- 172.21.15.94:0/1433594137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa819cf00 msgr2=0x7f1fa819d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa819cf00 0x7f1fa819d3b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1f9800cad0 tx=0x7f1f9800cf90 comp rx=0 tx=0).stop 2024-04-21T08:40:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 -- 172.21.15.94:0/1433594137 shutdown_connections 2024-04-21T08:40:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1f940a3e40 0x7f1f940a6300 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fa8105ff0 0x7f1fa8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa819cf00 0x7f1fa819d3b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 --2- 172.21.15.94:0/1433594137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fa8100f20 0x7f1fa8198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 -- 172.21.15.94:0/1433594137 >> 172.21.15.94:0/1433594137 conn(0x7f1fa80fc7a0 msgr2=0x7f1fa80fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 -- 172.21.15.94:0/1433594137 shutdown_connections 2024-04-21T08:40:05.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.511+0000 7f1fad401640 1 -- 172.21.15.94:0/1433594137 wait complete. 2024-04-21T08:40:05.522 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:40:05.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.620+0000 7fa820814640 1 -- 172.21.15.94:0/1187126389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa81c101b60 msgr2=0x7fa81c10dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.620+0000 7fa820814640 1 --2- 172.21.15.94:0/1187126389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa81c101b60 0x7fa81c10dc90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa814009930 tx=0x7fa81402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:05.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 -- 172.21.15.94:0/1187126389 shutdown_connections 2024-04-21T08:40:05.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 --2- 172.21.15.94:0/1187126389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa81c1011a0 0x7fa81c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 --2- 172.21.15.94:0/1187126389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa81c101b60 0x7fa81c10dc90 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 --2- 172.21.15.94:0/1187126389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa81c105e30 0x7fa81c106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 -- 172.21.15.94:0/1187126389 >> 172.21.15.94:0/1187126389 conn(0x7fa81c077e30 msgr2=0x7fa81c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 -- 172.21.15.94:0/1187126389 shutdown_connections 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 -- 172.21.15.94:0/1187126389 wait complete. 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 Processor -- start 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 -- start start 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa81c1011a0 0x7fa81c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa81c101b60 0x7fa81c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa81c105e30 0x7fa81c19cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.621+0000 7fa820814640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c112250 con 0x7fa81c105e30 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa820814640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1120d0 con 0x7fa81c1011a0 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa820814640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1123d0 con 0x7fa81c101b60 2024-04-21T08:40:05.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa81c105e30 0x7fa81c19cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa81c105e30 0x7fa81c19cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36564/0 (socket says 172.21.15.94:36564) 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81ad9d640 1 -- 172.21.15.94:0/2818554077 learned_addr learned my addr 172.21.15.94:0/2818554077 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa819d9b640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa81c101b60 0x7fa81c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81a59c640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa81c1011a0 0x7fa81c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81ad9d640 1 -- 172.21.15.94:0/2818554077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa81c101b60 msgr2=0x7fa81c198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81ad9d640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa81c101b60 0x7fa81c198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81ad9d640 1 -- 172.21.15.94:0/2818554077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa81c1011a0 msgr2=0x7fa81c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81ad9d640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa81c1011a0 0x7fa81c1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81ad9d640 1 -- 172.21.15.94:0/2818554077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c19d6a0 con 0x7fa81c105e30 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81a59c640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa81c1011a0 0x7fa81c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa81ad9d640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa81c105e30 0x7fa81c19cf70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa814005940 tx=0x7fa81402c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa80b7fe640 1 -- 172.21.15.94:0/2818554077 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814043020 con 0x7fa81c105e30 2024-04-21T08:40:05.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa820814640 1 -- 172.21.15.94:0/2818554077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81c19d930 con 0x7fa81c105e30 2024-04-21T08:40:05.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa80b7fe640 1 -- 172.21.15.94:0/2818554077 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8140043d0 con 0x7fa81c105e30 2024-04-21T08:40:05.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa80b7fe640 1 -- 172.21.15.94:0/2818554077 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814003b50 con 0x7fa81c105e30 2024-04-21T08:40:05.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.622+0000 7fa820814640 1 -- 172.21.15.94:0/2818554077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81c1a5220 con 0x7fa81c105e30 2024-04-21T08:40:05.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.623+0000 7fa820814640 1 -- 172.21.15.94:0/2818554077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa81c068de0 con 0x7fa81c105e30 2024-04-21T08:40:05.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.624+0000 7fa80b7fe640 1 -- 172.21.15.94:0/2818554077 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fa814046020 con 0x7fa81c105e30 2024-04-21T08:40:05.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.625+0000 7fa80b7fe640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa8040a3e40 0x7fa8040a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.625+0000 7fa81a59c640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa8040a3e40 0x7fa8040a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.625+0000 7fa80b7fe640 1 -- 172.21.15.94:0/2818554077 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fa8140f30e0 con 0x7fa81c105e30 2024-04-21T08:40:05.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.625+0000 7fa81a59c640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa8040a3e40 0x7fa8040a6300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa80c005c40 tx=0x7fa80c005bd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.629+0000 7fa80b7fe640 1 -- 172.21.15.94:0/2818554077 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa81404d020 con 0x7fa81c105e30 2024-04-21T08:40:05.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:05 smithi111 ceph-mon[31493]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 410 B/s wr, 0 op/s 2024-04-21T08:40:05.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:05 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4033889897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:05 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1588087154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:05 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/815627707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:05 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4117860937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:05.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.808+0000 7fa820814640 1 -- 172.21.15.94:0/2818554077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa81c109330 con 0x7fa81c105e30 2024-04-21T08:40:05.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.810+0000 7fa80b7fe640 1 -- 172.21.15.94:0/2818554077 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25266 (secure 0 0 0) 0x7fa81403c350 con 0x7fa81c105e30 2024-04-21T08:40:05.810 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:05.810 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:57.898777+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:40:05.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 -- 172.21.15.94:0/2818554077 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa8040a3e40 msgr2=0x7fa8040a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa8040a3e40 0x7fa8040a6300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa80c005c40 tx=0x7fa80c005bd0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 -- 172.21.15.94:0/2818554077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa81c105e30 msgr2=0x7fa81c19cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa81c105e30 0x7fa81c19cf70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa814005940 tx=0x7fa81402c710 comp rx=0 tx=0).stop 2024-04-21T08:40:05.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 -- 172.21.15.94:0/2818554077 shutdown_connections 2024-04-21T08:40:05.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa8040a3e40 0x7fa8040a6300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa81c101b60 0x7fa81c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa81c105e30 0x7fa81c19cf70 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 --2- 172.21.15.94:0/2818554077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa81c1011a0 0x7fa81c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 -- 172.21.15.94:0/2818554077 >> 172.21.15.94:0/2818554077 conn(0x7fa81c077e30 msgr2=0x7fa81c0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 -- 172.21.15.94:0/2818554077 shutdown_connections 2024-04-21T08:40:05.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.812+0000 7fa820814640 1 -- 172.21.15.94:0/2818554077 wait complete. 2024-04-21T08:40:05.824 DEBUG:tasks.fwd_scrub:Yielding 2024-04-21T08:40:05.824 INFO:teuthology.run_tasks:Running task exec... 2024-04-21T08:40:05.834 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:40:05.834 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:40:05.834 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-21T08:40:05.835 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-21T08:40:05.835 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:40:05.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:40:05.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:40:05.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.938+0000 7f61a3e25640 1 -- 172.21.15.94:0/1417263864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f619c10c520 msgr2=0x7f619c10e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.938+0000 7f61a3e25640 1 --2- 172.21.15.94:0/1417263864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f619c10c520 0x7f619c10e9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6198009ca0 tx=0x7f619802c230 comp rx=0 tx=0).stop 2024-04-21T08:40:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.938+0000 7f61a3e25640 1 -- 172.21.15.94:0/1417263864 shutdown_connections 2024-04-21T08:40:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.938+0000 7f61a3e25640 1 --2- 172.21.15.94:0/1417263864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f619c103b00 0x7f619c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.938+0000 7f61a3e25640 1 --2- 172.21.15.94:0/1417263864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f619c10c520 0x7f619c10e9c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.938+0000 7f61a3e25640 1 --2- 172.21.15.94:0/1417263864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f619c1031e0 0x7f619c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.938+0000 7f61a3e25640 1 -- 172.21.15.94:0/1417263864 >> 172.21.15.94:0/1417263864 conn(0x7f619c077fd0 msgr2=0x7f619c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.938+0000 7f61a3e25640 1 -- 172.21.15.94:0/1417263864 shutdown_connections 2024-04-21T08:40:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.938+0000 7f61a3e25640 1 -- 172.21.15.94:0/1417263864 wait complete. 2024-04-21T08:40:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a3e25640 1 Processor -- start 2024-04-21T08:40:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a3e25640 1 -- start start 2024-04-21T08:40:05.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f25ac4d2640 1 -- 172.21.15.94:0/3400807070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a4106160 msgr2=0x7f25a410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f25ac4d2640 1 --2- 172.21.15.94:0/3400807070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a4106160 0x7f25a410dd30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f259c009ca0 tx=0x7f259c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a3e25640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f619c1031e0 0x7f619c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a3e25640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f619c103b00 0x7f619c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a3e25640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f619c10c520 0x7f619c19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a3e25640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f619c112190 con 0x7f619c103b00 2024-04-21T08:40:05.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f25ac4d2640 1 -- 172.21.15.94:0/3400807070 shutdown_connections 2024-04-21T08:40:05.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f25ac4d2640 1 --2- 172.21.15.94:0/3400807070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a4106160 0x7f25a410dd30 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a3e25640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f619c112010 con 0x7f619c1031e0 2024-04-21T08:40:05.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a3e25640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f619c112310 con 0x7f619c10c520 2024-04-21T08:40:05.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f25ac4d2640 1 --2- 172.21.15.94:0/3400807070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a4101a40 0x7f25a4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f25ac4d2640 1 --2- 172.21.15.94:0/3400807070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a4101090 0x7f25a4101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f25ac4d2640 1 -- 172.21.15.94:0/3400807070 >> 172.21.15.94:0/3400807070 conn(0x7f25a4077f50 msgr2=0x7f25a4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a13c0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f619c103b00 0x7f619c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a13c0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f619c103b00 0x7f619c198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36604/0 (socket says 172.21.15.94:36604) 2024-04-21T08:40:05.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a13c0640 1 -- 172.21.15.94:0/2178886823 learned_addr learned my addr 172.21.15.94:0/2178886823 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:05.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f25ac4d2640 1 -- 172.21.15.94:0/3400807070 shutdown_connections 2024-04-21T08:40:05.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f25ac4d2640 1 -- 172.21.15.94:0/3400807070 wait complete. 2024-04-21T08:40:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a23c2640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f619c10c520 0x7f619c19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a1bc1640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f619c1031e0 0x7f619c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a13c0640 1 -- 172.21.15.94:0/2178886823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f619c10c520 msgr2=0x7f619c19ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a13c0640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f619c10c520 0x7f619c19ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a13c0640 1 -- 172.21.15.94:0/2178886823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f619c1031e0 msgr2=0x7f619c198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a13c0640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f619c1031e0 0x7f619c198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.939+0000 7f61a13c0640 1 -- 172.21.15.94:0/2178886823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619c19d600 con 0x7f619c103b00 2024-04-21T08:40:05.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f61a13c0640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f619c103b00 0x7f619c198b50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f618c00d950 tx=0x7f618c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f6192ffd640 1 -- 172.21.15.94:0/2178886823 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c017030 con 0x7f619c103b00 2024-04-21T08:40:05.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25ac4d2640 1 Processor -- start 2024-04-21T08:40:05.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f6192ffd640 1 -- 172.21.15.94:0/2178886823 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f618c004020 con 0x7f619c103b00 2024-04-21T08:40:05.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f61a3e25640 1 -- 172.21.15.94:0/2178886823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f619c19d8f0 con 0x7f619c103b00 2024-04-21T08:40:05.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f6192ffd640 1 -- 172.21.15.94:0/2178886823 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c0047a0 con 0x7f619c103b00 2024-04-21T08:40:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25ac4d2640 1 -- start start 2024-04-21T08:40:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f61a3e25640 1 -- 172.21.15.94:0/2178886823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f619c1a5180 con 0x7f619c103b00 2024-04-21T08:40:05.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25ac4d2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a4101090 0x7f25a41a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25ac4d2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a4106160 0x7f25a41a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25ac4d2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a419a710 0x7f25a419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25ac4d2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410ffa0 con 0x7f25a4101090 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25ac4d2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410fe20 con 0x7f25a4106160 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25ac4d2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4110120 con 0x7f25a419a710 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25aa26e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a4101090 0x7f25a41a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25aa26e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a4101090 0x7f25a41a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36620/0 (socket says 172.21.15.94:36620) 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25aa26e640 1 -- 172.21.15.94:0/1253445300 learned_addr learned my addr 172.21.15.94:0/1253445300 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a4106160 0x7f25a41a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25aaa6f640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a419a710 0x7f25a419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25aa26e640 1 -- 172.21.15.94:0/1253445300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a419a710 msgr2=0x7f25a419ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25aa26e640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a419a710 0x7f25a419ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25aa26e640 1 -- 172.21.15.94:0/1253445300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a4106160 msgr2=0x7f25a41a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25aa26e640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a4106160 0x7f25a41a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25aa26e640 1 -- 172.21.15.94:0/1253445300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a419b310 con 0x7f25a4101090 2024-04-21T08:40:05.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a4106160 0x7f25a41a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:05.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.940+0000 7f25aaa6f640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a419a710 0x7f25a419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:05.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.941+0000 7f25aa26e640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a4101090 0x7f25a41a0640 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f25a000d950 tx=0x7f25a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.941+0000 7f259b7fe640 1 -- 172.21.15.94:0/1253445300 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0017070 con 0x7f25a4101090 2024-04-21T08:40:05.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.941+0000 7f259b7fe640 1 -- 172.21.15.94:0/1253445300 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25a0009c30 con 0x7f25a4101090 2024-04-21T08:40:05.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.941+0000 7f25ac4d2640 1 -- 172.21.15.94:0/1253445300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a41a73e0 con 0x7f25a4101090 2024-04-21T08:40:05.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.941+0000 7f259b7fe640 1 -- 172.21.15.94:0/1253445300 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a000f620 con 0x7f25a4101090 2024-04-21T08:40:05.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.941+0000 7f25ac4d2640 1 -- 172.21.15.94:0/1253445300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a41a77e0 con 0x7f25a4101090 2024-04-21T08:40:05.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.941+0000 7f61a3e25640 1 -- 172.21.15.94:0/2178886823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f619c04fce0 con 0x7f619c103b00 2024-04-21T08:40:05.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.942+0000 7f6192ffd640 1 -- 172.21.15.94:0/2178886823 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f618c009a30 con 0x7f619c103b00 2024-04-21T08:40:05.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.943+0000 7f6192ffd640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61880a3ee0 0x7f61880a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.943+0000 7f259b7fe640 1 -- 172.21.15.94:0/1253445300 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f25a000f7c0 con 0x7f25a4101090 2024-04-21T08:40:05.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.943+0000 7f61a1bc1640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61880a3ee0 0x7f61880a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.943+0000 7f6192ffd640 1 -- 172.21.15.94:0/2178886823 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f618c0d0e80 con 0x7f619c103b00 2024-04-21T08:40:05.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f61a1bc1640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61880a3ee0 0x7f61880a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6194005fd0 tx=0x7f6194005850 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f259b7fe640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f25900a3e40 0x7f25900a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25af1a1740 1 Processor -- start 2024-04-21T08:40:05.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f25900a3e40 0x7f25900a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f259b7fe640 1 -- 172.21.15.94:0/1253445300 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f25a00d0bb0 con 0x7f25a4101090 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25af1a1740 1 -- start start 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8510cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8511290 0x5577a8519840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8519e10 0x5577a851c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25af1a1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5577a8474f00 con 0x5577a85108e0 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25af1a1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5577a8412590 con 0x5577a8519e10 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25af1a1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5577a8470ec0 con 0x5577a8511290 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f25900a3e40 0x7f25900a6300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2594005e10 tx=0x7f2594005da0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25aa26e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8510cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25aa26e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8510cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36634/0 (socket says 172.21.15.94:36634) 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.944+0000 7f25aa26e640 1 -- 172.21.15.94:0/1432134978 learned_addr learned my addr 172.21.15.94:0/1432134978 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.945+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/1432134978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8511290 0x5577a8519840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.945+0000 7f25aa26e640 1 -- 172.21.15.94:0/1432134978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8511290 msgr2=0x5577a8519840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:05.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.945+0000 7f25aa26e640 1 --2- 172.21.15.94:0/1432134978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8511290 0x5577a8519840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.945+0000 7f25aa26e640 1 -- 172.21.15.94:0/1432134978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8519e10 msgr2=0x5577a851c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:05.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.945+0000 7f25aa26e640 1 --2- 172.21.15.94:0/1432134978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8519e10 0x5577a851c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.945+0000 7f25aa26e640 1 -- 172.21.15.94:0/1432134978 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577a824f7e0 con 0x5577a85108e0 2024-04-21T08:40:05.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.945+0000 7f25aa26e640 1 --2- 172.21.15.94:0/1432134978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8510cc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f25a009c4b0 tx=0x7f25a009cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7409170e316ed4f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.947+0000 7f2598bfe640 1 -- 172.21.15.94:0/1432134978 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0020070 con 0x5577a85108e0 2024-04-21T08:40:05.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.947+0000 7f2598bfe640 1 -- 172.21.15.94:0/1432134978 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25a00a08d0 con 0x5577a85108e0 2024-04-21T08:40:05.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.947+0000 7f2598bfe640 1 -- 172.21.15.94:0/1432134978 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0098270 con 0x5577a85108e0 2024-04-21T08:40:05.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.947+0000 7f6192ffd640 1 -- 172.21.15.94:0/2178886823 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f618c09cef0 con 0x7f619c103b00 2024-04-21T08:40:05.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.948+0000 7f25af1a1740 1 -- 172.21.15.94:0/1432134978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 msgr2=0x5577a8510cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.948+0000 7f25af1a1740 1 --2- 172.21.15.94:0/1432134978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8510cc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f25a009c4b0 tx=0x7f25a009cc20 comp rx=0 tx=0).stop 2024-04-21T08:40:05.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.948+0000 7f25af1a1740 1 -- 172.21.15.94:0/1432134978 shutdown_connections 2024-04-21T08:40:05.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.948+0000 7f25af1a1740 1 --2- 172.21.15.94:0/1432134978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8511290 0x5577a8519840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.948+0000 7f25af1a1740 1 --2- 172.21.15.94:0/1432134978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8519e10 0x5577a851c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.948+0000 7f25af1a1740 1 --2- 172.21.15.94:0/1432134978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8510cc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.948+0000 7f25af1a1740 1 -- 172.21.15.94:0/1432134978 >> 172.21.15.94:0/1432134978 conn(0x5577a850afb0 msgr2=0x5577a850b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.948+0000 7f25af1a1740 1 -- 172.21.15.94:0/1432134978 shutdown_connections 2024-04-21T08:40:05.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.948+0000 7f25af1a1740 1 -- 172.21.15.94:0/1432134978 wait complete. 2024-04-21T08:40:05.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25af1a1740 1 -- start start 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25af1a1740 1 Processor -- start 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a85108e0 0x5577a8516df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8511290 0x5577a8517330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8519e10 0x5577a8517870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25af1a1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5577a851cd20 con 0x5577a8511290 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25af1a1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5577a842e580 con 0x5577a8519e10 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25af1a1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5577a851cea0 con 0x5577a85108e0 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25af1a1740 10 client.0 _ll_register_callbacks cb 0x5577a8434650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25a9a6d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8511290 0x5577a8517330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25a9a6d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8511290 0x5577a8517330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36638/0 (socket says 172.21.15.94:36638) 2024-04-21T08:40:05.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25a9a6d640 1 -- 172.21.15.94:0/2223703455 learned_addr learned my addr 172.21.15.94:0/2223703455 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25aa26e640 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a85108e0 0x5577a8516df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25aaa6f640 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8519e10 0x5577a8517870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25a9a6d640 1 -- 172.21.15.94:0/2223703455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a85108e0 msgr2=0x5577a8516df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a85108e0 0x5577a8516df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25a9a6d640 1 -- 172.21.15.94:0/2223703455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8519e10 msgr2=0x5577a8517870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8519e10 0x5577a8517870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25a9a6d640 1 -- 172.21.15.94:0/2223703455 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577a82aff70 con 0x5577a8511290 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25aa26e640 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a85108e0 0x5577a8516df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.949+0000 7f25aaa6f640 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8519e10 0x5577a8517870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8511290 0x5577a8517330 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f25a406b6e0 tx=0x7f25940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ee86480daa0510 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f258dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f258dffb640 1 -- 172.21.15.94:0/2223703455 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259400fd50 con 0x5577a8511290 2024-04-21T08:40:05.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f258dffb640 1 -- 172.21.15.94:0/2223703455 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f259401e430 con 0x5577a8511290 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f25af1a1740 1 -- 172.21.15.94:0/2223703455 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25a00a08d0 con 0x5577a8511290 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f258dffb640 1 -- 172.21.15.94:0/2223703455 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f259402f050 con 0x5577a8511290 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f25af1a1740 10 client.15678 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f25af1a1740 1 -- 172.21.15.94:0/2223703455 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5577a8147170 con 0x5577a8511290 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f258dffb640 1 -- 172.21.15.94:0/2223703455 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25a00a08d0 con 0x5577a8511290 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f258dffb640 1 -- 172.21.15.94:0/2223703455 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f259401d620 con 0x5577a8511290 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f25af1a1740 10 client.15678 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f25af1a1740 10 client.15678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f25af1a1740 20 client.15678 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f25af1a1740 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5577a852a3e0 0x5577a852c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f25af1a1740 4 client.15678 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.950+0000 7f25af1a1740 1 -- 172.21.15.94:0/2223703455 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577a8482920 con 0x5577a852a3e0 2024-04-21T08:40:05.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.951+0000 7f25aa26e640 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5577a852a3e0 0x5577a852c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.951+0000 7f258dffb640 10 client.15678 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.951+0000 7f25aa26e640 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5577a852a3e0 0x5577a852c830 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=582dc3b66862574d server_cookie=eb8ce381af28cbaf in_seq=0 out_seq=0 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f258dffb640 1 -- 172.21.15.94:0/2223703455 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5577a8482920 con 0x5577a852a3e0 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f258dffb640 10 client.15678 handle_command_reply: tid=0 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 client.15678 shutdown 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 -- 172.21.15.94:0/2223703455 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8511290 msgr2=0x5577a8517330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8511290 0x5577a8517330 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f25a406b6e0 tx=0x7f25940046c0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 -- 172.21.15.94:0/2223703455 shutdown_connections 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5577a852a3e0 0x5577a852c830 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8519e10 0x5577a8517870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f258dffb640 0 client.15678 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:05.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8511290 0x5577a8517330 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 --2- 172.21.15.94:0/2223703455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a85108e0 0x5577a8516df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 -- 172.21.15.94:0/2223703455 >> 172.21.15.94:0/2223703455 conn(0x5577a850afb0 msgr2=0x5577a850cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 -- 172.21.15.94:0/2223703455 shutdown_connections 2024-04-21T08:40:05.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 1 -- 172.21.15.94:0/2223703455 wait complete. 2024-04-21T08:40:05.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.952+0000 7f25af1a1740 20 client.15678 trim_cache size 0 max 16384 2024-04-21T08:40:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.982+0000 7f25af1a1740 1 Processor -- start 2024-04-21T08:40:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.982+0000 7f25af1a1740 1 -- start start 2024-04-21T08:40:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.982+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a85108e0 0x5577a8519520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.982+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8511290 0x5577a8525c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.982+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8519e10 0x5577a8509e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.982+0000 7f25af1a1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f25a41b3f60 con 0x5577a8519e10 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.982+0000 7f25af1a1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f25a41b1830 con 0x5577a85108e0 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.982+0000 7f25af1a1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f25a41af680 con 0x5577a8511290 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25aaa6f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8519e10 0x5577a8509e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25aaa6f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8519e10 0x5577a8509e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36640/0 (socket says 172.21.15.94:36640) 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25aaa6f640 1 -- 172.21.15.94:0/2678779131 learned_addr learned my addr 172.21.15.94:0/2678779131 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25aa26e640 1 --2- 172.21.15.94:0/2678779131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a85108e0 0x5577a8519520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/2678779131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8511290 0x5577a8525c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25aaa6f640 1 -- 172.21.15.94:0/2678779131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8511290 msgr2=0x5577a8525c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25aaa6f640 1 --2- 172.21.15.94:0/2678779131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8511290 0x5577a8525c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25aaa6f640 1 -- 172.21.15.94:0/2678779131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a85108e0 msgr2=0x5577a8519520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25aaa6f640 1 --2- 172.21.15.94:0/2678779131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a85108e0 0x5577a8519520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25aaa6f640 1 -- 172.21.15.94:0/2678779131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577a8511030 con 0x5577a8519e10 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f25aaa6f640 1 --2- 172.21.15.94:0/2678779131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8519e10 0x5577a8509e90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f259c009ca0 tx=0x7f259c0323d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb2501caa8d82aa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f258ffff640 1 -- 172.21.15.94:0/2678779131 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c039070 con 0x5577a8519e10 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f258ffff640 1 -- 172.21.15.94:0/2678779131 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f259c03a040 con 0x5577a8519e10 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.983+0000 7f258ffff640 1 -- 172.21.15.94:0/2678779131 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c003bb0 con 0x5577a8519e10 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 -- 172.21.15.94:0/2678779131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8519e10 msgr2=0x5577a8509e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 --2- 172.21.15.94:0/2678779131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8519e10 0x5577a8509e90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f259c009ca0 tx=0x7f259c0323d0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 -- 172.21.15.94:0/2678779131 shutdown_connections 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 --2- 172.21.15.94:0/2678779131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8511290 0x5577a8525c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 --2- 172.21.15.94:0/2678779131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a8519e10 0x5577a8509e90 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 --2- 172.21.15.94:0/2678779131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a85108e0 0x5577a8519520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 -- 172.21.15.94:0/2678779131 >> 172.21.15.94:0/2678779131 conn(0x5577a850afb0 msgr2=0x5577a8509780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 -- 172.21.15.94:0/2678779131 shutdown_connections 2024-04-21T08:40:05.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 -- 172.21.15.94:0/2678779131 wait complete. 2024-04-21T08:40:05.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 -- start start 2024-04-21T08:40:05.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.984+0000 7f25af1a1740 1 Processor -- start 2024-04-21T08:40:05.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8588cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8511290 0x5577a8589230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25af1a1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8519e10 0x5577a851e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25af1a1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f25a41afcf0 con 0x5577a85108e0 2024-04-21T08:40:05.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25af1a1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f25a41b1160 con 0x5577a8511290 2024-04-21T08:40:05.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25af1a1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f25a41b03c0 con 0x5577a8519e10 2024-04-21T08:40:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25af1a1740 10 client.0 _ll_register_callbacks cb 0x7f2594014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25aa26e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8588cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25aa26e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8588cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36652/0 (socket says 172.21.15.94:36652) 2024-04-21T08:40:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25aa26e640 1 -- 172.21.15.94:0/3988683466 learned_addr learned my addr 172.21.15.94:0/3988683466 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8511290 0x5577a8589230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25aa26e640 1 -- 172.21.15.94:0/3988683466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8519e10 msgr2=0x5577a851e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25aa26e640 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8519e10 0x5577a851e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25aa26e640 1 -- 172.21.15.94:0/3988683466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8511290 msgr2=0x5577a8589230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25aa26e640 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8511290 0x5577a8589230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25aa26e640 1 -- 172.21.15.94:0/3988683466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577a851ec10 con 0x5577a85108e0 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8511290 0x5577a8589230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25aa26e640 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8588cf0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f25a009cdd0 tx=0x7f25a009c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f0dfb5021d153d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f258dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f258dffb640 1 -- 172.21.15.94:0/3988683466 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0020040 con 0x5577a85108e0 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f258dffb640 1 -- 172.21.15.94:0/3988683466 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f25a00ce080 con 0x5577a85108e0 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25af1a1740 1 -- 172.21.15.94:0/3988683466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f259c03a040 con 0x5577a85108e0 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f258dffb640 1 -- 172.21.15.94:0/3988683466 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f25a009c520 con 0x5577a85108e0 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25af1a1740 10 client.15684 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.985+0000 7f25af1a1740 1 -- 172.21.15.94:0/3988683466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5577a85915c0 con 0x5577a85108e0 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f258dffb640 1 -- 172.21.15.94:0/3988683466 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f259c03a040 con 0x5577a85108e0 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f258dffb640 1 -- 172.21.15.94:0/3988683466 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f25a0098300 con 0x5577a85108e0 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f25af1a1740 10 client.15684 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f25af1a1740 10 client.15684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f25af1a1740 20 client.15684 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f25af1a1740 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5577a8591840 0x5577a8593c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f25af1a1740 4 client.15684 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f25af1a1740 1 -- 172.21.15.94:0/3988683466 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5577a847c790 con 0x5577a8591840 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5577a8591840 0x5577a8593c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f25a9a6d640 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5577a8591840 0x5577a8593c30 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70cf4e2130e9347f server_cookie=cac83d9115665f68 in_seq=0 out_seq=0 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.986+0000 7f258dffb640 10 client.15684 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f258dffb640 1 -- 172.21.15.94:0/3988683466 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5577a847c790 con 0x5577a8591840 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f258dffb640 10 client.15684 handle_command_reply: tid=0 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 client.15684 shutdown 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 -- 172.21.15.94:0/3988683466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 msgr2=0x5577a8588cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8588cf0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f25a009cdd0 tx=0x7f25a009c380 comp rx=0 tx=0).stop 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 -- 172.21.15.94:0/3988683466 shutdown_connections 2024-04-21T08:40:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5577a8591840 0x5577a8593c30 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5577a8519e10 0x5577a851e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577a8511290 0x5577a8589230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 --2- 172.21.15.94:0/3988683466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5577a85108e0 0x5577a8588cf0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 -- 172.21.15.94:0/3988683466 >> 172.21.15.94:0/3988683466 conn(0x5577a850afb0 msgr2=0x5577a850a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f258dffb640 0 client.15684 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 -- 172.21.15.94:0/3988683466 shutdown_connections 2024-04-21T08:40:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 1 -- 172.21.15.94:0/3988683466 wait complete. 2024-04-21T08:40:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.988+0000 7f25af1a1740 20 client.15684 trim_cache size 0 max 16384 2024-04-21T08:40:05.995 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:40:05.996 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:40:05.996 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "3fcafc27-00c9-46df-b375-68348f334901", 2024-04-21T08:40:05.996 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:40:05.996 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:40:05.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 -- 172.21.15.94:0/1253445300 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f25900a3e40 msgr2=0x7f25900a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f25900a3e40 0x7f25900a6300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2594005e10 tx=0x7f2594005da0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 -- 172.21.15.94:0/1253445300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a4101090 msgr2=0x7f25a41a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:05.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a4101090 0x7f25a41a0640 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f25a000d950 tx=0x7f25a000de10 comp rx=0 tx=0).stop 2024-04-21T08:40:05.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 -- 172.21.15.94:0/1253445300 shutdown_connections 2024-04-21T08:40:05.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f25900a3e40 0x7f25900a6300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a419a710 0x7f25a419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a4106160 0x7f25a41a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 --2- 172.21.15.94:0/1253445300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a4101090 0x7f25a41a0640 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:05.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 -- 172.21.15.94:0/1253445300 >> 172.21.15.94:0/1253445300 conn(0x7f25a4077f50 msgr2=0x7f25a40fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:05.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 -- 172.21.15.94:0/1253445300 shutdown_connections 2024-04-21T08:40:05.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:05.989+0000 7f25ac4d2640 1 -- 172.21.15.94:0/1253445300 wait complete. 2024-04-21T08:40:06.003 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:40:06.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:40:06.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.104+0000 7f27b97ce640 1 -- 172.21.15.94:0/3185305626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b41031c0 msgr2=0x7f27b41035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.104+0000 7f27b97ce640 1 --2- 172.21.15.94:0/3185305626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b41031c0 0x7f27b41035a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f27a8009900 tx=0x7f27a802bd60 comp rx=0 tx=0).stop 2024-04-21T08:40:06.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.104+0000 7f27b97ce640 1 -- 172.21.15.94:0/3185305626 shutdown_connections 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.104+0000 7f27b97ce640 1 --2- 172.21.15.94:0/3185305626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27b410c500 0x7f27b410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.104+0000 7f27b97ce640 1 --2- 172.21.15.94:0/3185305626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27b4103ae0 0x7f27b410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.104+0000 7f27b97ce640 1 --2- 172.21.15.94:0/3185305626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b41031c0 0x7f27b41035a0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.104+0000 7f27b97ce640 1 -- 172.21.15.94:0/3185305626 >> 172.21.15.94:0/3185305626 conn(0x7f27b4078000 msgr2=0x7f27b4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.104+0000 7f27b97ce640 1 -- 172.21.15.94:0/3185305626 shutdown_connections 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.104+0000 7f27b97ce640 1 -- 172.21.15.94:0/3185305626 wait complete. 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b97ce640 1 Processor -- start 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b97ce640 1 -- start start 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b97ce640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27b41031c0 0x7f27b4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b97ce640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b4103ae0 0x7f27b4198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b97ce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27b410c500 0x7f27b419cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b97ce640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f27b4112190 con 0x7f27b4103ae0 2024-04-21T08:40:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b97ce640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f27b4112010 con 0x7f27b41031c0 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b97ce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f27b4112310 con 0x7f27b410c500 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b4103ae0 0x7f27b4198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b4103ae0 0x7f27b4198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36670/0 (socket says 172.21.15.94:36670) 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.105+0000 7f27b27fc640 1 -- 172.21.15.94:0/622233557 learned_addr learned my addr 172.21.15.94:0/622233557 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f27b37fe640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27b410c500 0x7f27b419cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27b41031c0 0x7f27b4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f27b27fc640 1 -- 172.21.15.94:0/622233557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27b410c500 msgr2=0x7f27b419cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f27b27fc640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27b410c500 0x7f27b419cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f27b27fc640 1 -- 172.21.15.94:0/622233557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27b41031c0 msgr2=0x7f27b4198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f27b27fc640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27b41031c0 0x7f27b4198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f27b27fc640 1 -- 172.21.15.94:0/622233557 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b419d700 con 0x7f27b4103ae0 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f27b27fc640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b4103ae0 0x7f27b4198c50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f27a400e9f0 tx=0x7f27a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:06.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f279bfff640 1 -- 172.21.15.94:0/622233557 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a400cd80 con 0x7f27b4103ae0 2024-04-21T08:40:06.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f279bfff640 1 -- 172.21.15.94:0/622233557 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27a40108e0 con 0x7f27b4103ae0 2024-04-21T08:40:06.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f27b97ce640 1 -- 172.21.15.94:0/622233557 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b419d9f0 con 0x7f27b4103ae0 2024-04-21T08:40:06.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f279bfff640 1 -- 172.21.15.94:0/622233557 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4018980 con 0x7f27b4103ae0 2024-04-21T08:40:06.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.106+0000 7f27b97ce640 1 -- 172.21.15.94:0/622233557 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b41a5280 con 0x7f27b4103ae0 2024-04-21T08:40:06.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.109+0000 7f27bc3bf740 1 Processor -- start 2024-04-21T08:40:06.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.109+0000 7f27bc3bf740 1 -- start start 2024-04-21T08:40:06.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.109+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0580 0x55e0b20f0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.109+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.109+0000 7f279bfff640 1 -- 172.21.15.94:0/622233557 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f27a4010430 con 0x7f27b4103ae0 2024-04-21T08:40:06.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.109+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f9a80 0x55e0b20fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.109+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55e0b20551e0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.109+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e0b1ff2390 con 0x55e0b20f9a80 2024-04-21T08:40:06.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.109+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e0b20511a0 con 0x55e0b20f0580 2024-04-21T08:40:06.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.109+0000 7f27b2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36684/0 (socket says 172.21.15.94:36684) 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b2ffd640 1 -- 172.21.15.94:0/1101416910 learned_addr learned my addr 172.21.15.94:0/1101416910 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b37fe640 1 --2- 172.21.15.94:0/1101416910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0580 0x55e0b20f0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b27fc640 1 --2- 172.21.15.94:0/1101416910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f9a80 0x55e0b20fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b2ffd640 1 -- 172.21.15.94:0/1101416910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0580 msgr2=0x55e0b20f0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/1101416910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0580 0x55e0b20f0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b2ffd640 1 -- 172.21.15.94:0/1101416910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f9a80 msgr2=0x55e0b20fbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/1101416910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f9a80 0x55e0b20fbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b2ffd640 1 -- 172.21.15.94:0/1101416910 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0b1e317d0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b27fc640 1 --2- 172.21.15.94:0/1101416910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f9a80 0x55e0b20fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f279bfff640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f279c0a3e40 0x7f279c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/1101416910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f9540 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f27a8009900 tx=0x7f27a80095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92b885c4d6091f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27b37fe640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f279c0a3e40 0x7f279c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f279bfff640 1 -- 172.21.15.94:0/622233557 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f27a40d1150 con 0x7f27b4103ae0 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27997fa640 1 -- 172.21.15.94:0/1101416910 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a8004040 con 0x55e0b20f0ea0 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27997fa640 1 -- 172.21.15.94:0/1101416910 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27a802cdc0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.110+0000 7f27997fa640 1 -- 172.21.15.94:0/1101416910 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a80046c0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1101416910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 msgr2=0x55e0b20f9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27b37fe640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f279c0a3e40 0x7f279c0a6300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f27ac005fd0 tx=0x7f27ac009500 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/1101416910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f9540 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f27a8009900 tx=0x7f27a80095f0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1101416910 shutdown_connections 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/1101416910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f9a80 0x55e0b20fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/1101416910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f9540 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/1101416910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0580 0x55e0b20f0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1101416910 >> 172.21.15.94:0/1101416910 conn(0x55e0b20eacb0 msgr2=0x55e0b20eb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:06.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1101416910 shutdown_connections 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1101416910 wait complete. 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 -- start start 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.111+0000 7f27bc3bf740 1 Processor -- start 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f9a80 0x55e0b20f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55e0b20fc810 con 0x55e0b20f0ea0 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e0b200e2f0 con 0x55e0b20f0580 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e0b20fc990 con 0x55e0b20f9a80 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27bc3bf740 10 client.0 _ll_register_callbacks cb 0x55e0b201b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36692/0 (socket says 172.21.15.94:36692) 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b27fc640 1 -- 172.21.15.94:0/1278943974 learned_addr learned my addr 172.21.15.94:0/1278943974 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:06.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b37fe640 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f9a80 0x55e0b20f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b27fc640 1 -- 172.21.15.94:0/1278943974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f9a80 msgr2=0x55e0b20f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b27fc640 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f9a80 0x55e0b20f74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b27fc640 1 -- 172.21.15.94:0/1278943974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 msgr2=0x55e0b20f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b27fc640 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b27fc640 1 -- 172.21.15.94:0/1278943974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0b1d673f0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27b27fc640 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f6f90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f27a400c6a0 tx=0x7f27a409ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d037051859097 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f2796ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f2796ff5640 1 -- 172.21.15.94:0/1278943974 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40a5730 con 0x55e0b20f0ea0 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f2796ff5640 1 -- 172.21.15.94:0/1278943974 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f27a40ce620 con 0x55e0b20f0ea0 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1278943974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27a802cdc0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f2796ff5640 1 -- 172.21.15.94:0/1278943974 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f27a40ce820 con 0x55e0b20f0ea0 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27bc3bf740 10 client.15696 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.112+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1278943974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e0b1d270f0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f2796ff5640 1 -- 172.21.15.94:0/1278943974 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27a802cdc0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f2796ff5640 1 -- 172.21.15.94:0/1278943974 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f27a40d32a0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f27bc3bf740 10 client.15696 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f27bc3bf740 10 client.15696 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f27bc3bf740 20 client.15696 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e0b2109970 0x55e0b210bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f27bc3bf740 4 client.15696 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1278943974 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0b2062c00 con 0x55e0b2109970 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e0b2109970 0x55e0b210bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e0b2109970 0x55e0b210bd60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efdca60bbec3a9ff server_cookie=2b74a4433fc6743d in_seq=0 out_seq=0 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.113+0000 7f2796ff5640 10 client.15696 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f2796ff5640 1 -- 172.21.15.94:0/1278943974 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0b2062c00 con 0x55e0b2109970 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f2796ff5640 10 client.15696 handle_command_reply: tid=0 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 client.15696 shutdown 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1278943974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 msgr2=0x55e0b20f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f6f90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f27a400c6a0 tx=0x7f27a409ca70 comp rx=0 tx=0).stop 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1278943974 shutdown_connections 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e0b2109970 0x55e0b210bd60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f9a80 0x55e0b20f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f6f90 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/1278943974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f2796ff5640 0 client.15696 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:06.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1278943974 >> 172.21.15.94:0/1278943974 conn(0x55e0b20eacb0 msgr2=0x55e0b20eca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:06.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1278943974 shutdown_connections 2024-04-21T08:40:06.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 1 -- 172.21.15.94:0/1278943974 wait complete. 2024-04-21T08:40:06.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.114+0000 7f27bc3bf740 20 client.15696 trim_cache size 0 max 16384 2024-04-21T08:40:06.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27bc3bf740 1 Processor -- start 2024-04-21T08:40:06.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27bc3bf740 1 -- start start 2024-04-21T08:40:06.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0ea0 0x55e0b21056d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b20e9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41b1c90 con 0x55e0b20f9a80 2024-04-21T08:40:06.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41af560 con 0x55e0b20f0580 2024-04-21T08:40:06.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41ad3b0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b20e9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b20e9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36708/0 (socket says 172.21.15.94:36708) 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b37fe640 1 -- 172.21.15.94:0/2645716305 learned_addr learned my addr 172.21.15.94:0/2645716305 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b27fc640 1 --2- 172.21.15.94:0/2645716305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0ea0 0x55e0b21056d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/2645716305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b37fe640 1 -- 172.21.15.94:0/2645716305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0ea0 msgr2=0x55e0b21056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b37fe640 1 --2- 172.21.15.94:0/2645716305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0ea0 0x55e0b21056d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b37fe640 1 -- 172.21.15.94:0/2645716305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 msgr2=0x55e0b20f9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b37fe640 1 --2- 172.21.15.94:0/2645716305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b37fe640 1 -- 172.21.15.94:0/2645716305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0b20f0cd0 con 0x55e0b20f9a80 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.132+0000 7f27b37fe640 1 --2- 172.21.15.94:0/2645716305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b20e9bf0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f27ac003d10 tx=0x7f27ac016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c018d4833befcc66 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f2798ff9640 1 -- 172.21.15.94:0/2645716305 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac02c070 con 0x55e0b20f9a80 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f2798ff9640 1 -- 172.21.15.94:0/2645716305 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27ac00f480 con 0x55e0b20f9a80 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f2798ff9640 1 -- 172.21.15.94:0/2645716305 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac027410 con 0x55e0b20f9a80 2024-04-21T08:40:06.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f27bc3bf740 1 -- 172.21.15.94:0/2645716305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 msgr2=0x55e0b20e9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/2645716305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b20e9bf0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f27ac003d10 tx=0x7f27ac016000 comp rx=0 tx=0).stop 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f27bc3bf740 1 -- 172.21.15.94:0/2645716305 shutdown_connections 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/2645716305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0ea0 0x55e0b21056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/2645716305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b20e9bf0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/2645716305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f27bc3bf740 1 -- 172.21.15.94:0/2645716305 >> 172.21.15.94:0/2645716305 conn(0x55e0b20eacb0 msgr2=0x55e0b20e94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f27bc3bf740 1 -- 172.21.15.94:0/2645716305 shutdown_connections 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f27bc3bf740 1 -- 172.21.15.94:0/2645716305 wait complete. 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.133+0000 7f27bc3bf740 1 -- start start 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27bc3bf740 1 Processor -- start 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27bc3bf740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b216ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41ada20 con 0x55e0b20f9a80 2024-04-21T08:40:06.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41aee90 con 0x55e0b20f0580 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27bc3bf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41ae0f0 con 0x55e0b20f0ea0 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27bc3bf740 10 client.0 _ll_register_callbacks cb 0x7f27a4009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b216ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b216ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36718/0 (socket says 172.21.15.94:36718) 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b37fe640 1 -- 172.21.15.94:0/3216293322 learned_addr learned my addr 172.21.15.94:0/3216293322 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b27fc640 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b37fe640 1 -- 172.21.15.94:0/3216293322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0ea0 msgr2=0x55e0b20f4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b37fe640 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b37fe640 1 -- 172.21.15.94:0/3216293322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 msgr2=0x55e0b20f3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b37fe640 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b37fe640 1 -- 172.21.15.94:0/3216293322 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0b216f3e0 con 0x55e0b20f9a80 2024-04-21T08:40:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f27b37fe640 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b216ee50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00ff40 tx=0x7f27ac005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7748c2ceb793b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.134+0000 7f2796ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f2796ff5640 1 -- 172.21.15.94:0/3216293322 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac02c070 con 0x55e0b20f9a80 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f2796ff5640 1 -- 172.21.15.94:0/3216293322 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f27ac011260 con 0x55e0b20f9a80 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f27bc3bf740 1 -- 172.21.15.94:0/3216293322 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27ac00f480 con 0x55e0b20f9a80 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f2796ff5640 1 -- 172.21.15.94:0/3216293322 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f27ac00f630 con 0x55e0b20f9a80 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f27bc3bf740 10 client.15702 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f27bc3bf740 1 -- 172.21.15.94:0/3216293322 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e0b21706f0 con 0x55e0b20f9a80 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f2796ff5640 1 -- 172.21.15.94:0/3216293322 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27ac00f480 con 0x55e0b20f9a80 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f2796ff5640 1 -- 172.21.15.94:0/3216293322 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f27ac031bc0 con 0x55e0b20f9a80 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f27bc3bf740 10 client.15702 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f27bc3bf740 10 client.15702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f27bc3bf740 20 client.15702 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e0b2170de0 0x55e0b2173200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f27bc3bf740 4 client.15702 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f27bc3bf740 1 -- 172.21.15.94:0/3216293322 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e0b205ca70 con 0x55e0b2170de0 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.135+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e0b2170de0 0x55e0b2173200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27b2ffd640 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e0b2170de0 0x55e0b2173200 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d00de50d4051c69 server_cookie=c1aec4426af5d83c in_seq=0 out_seq=0 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f2796ff5640 10 client.15702 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f2796ff5640 1 -- 172.21.15.94:0/3216293322 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e0b205ca70 con 0x55e0b2170de0 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f2796ff5640 10 client.15702 handle_command_reply: tid=0 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 client.15702 shutdown 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 -- 172.21.15.94:0/3216293322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 msgr2=0x55e0b216ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b216ee50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00ff40 tx=0x7f27ac005a50 comp rx=0 tx=0).stop 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 -- 172.21.15.94:0/3216293322 shutdown_connections 2024-04-21T08:40:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e0b2170de0 0x55e0b2173200 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0b20f0ea0 0x55e0b20f4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e0b20f9a80 0x55e0b216ee50 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 --2- 172.21.15.94:0/3216293322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e0b20f0580 0x55e0b20f3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 -- 172.21.15.94:0/3216293322 >> 172.21.15.94:0/3216293322 conn(0x55e0b20eacb0 msgr2=0x55e0b20e9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f2796ff5640 0 client.15702 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 -- 172.21.15.94:0/3216293322 shutdown_connections 2024-04-21T08:40:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 1 -- 172.21.15.94:0/3216293322 wait complete. 2024-04-21T08:40:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.136+0000 7f27bc3bf740 20 client.15702 trim_cache size 0 max 16384 2024-04-21T08:40:06.144 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:40:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:40:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:40:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "3fcafc27-00c9-46df-b375-68348f334901": { 2024-04-21T08:40:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:40:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "3fcafc27-00c9-46df-b375-68348f334901", 2024-04-21T08:40:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:40:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:40:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:40:06.145 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:40:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 -- 172.21.15.94:0/622233557 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f279c0a3e40 msgr2=0x7f279c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f279c0a3e40 0x7f279c0a6300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f27ac005fd0 tx=0x7f27ac009500 comp rx=0 tx=0).stop 2024-04-21T08:40:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 -- 172.21.15.94:0/622233557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b4103ae0 msgr2=0x7f27b4198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b4103ae0 0x7f27b4198c50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f27a400e9f0 tx=0x7f27a400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 -- 172.21.15.94:0/622233557 shutdown_connections 2024-04-21T08:40:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f279c0a3e40 0x7f279c0a6300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27b410c500 0x7f27b419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27b4103ae0 0x7f27b4198c50 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 --2- 172.21.15.94:0/622233557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27b41031c0 0x7f27b4198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 -- 172.21.15.94:0/622233557 >> 172.21.15.94:0/622233557 conn(0x7f27b4078000 msgr2=0x7f27b40fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 -- 172.21.15.94:0/622233557 shutdown_connections 2024-04-21T08:40:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.137+0000 7f27b97ce640 1 -- 172.21.15.94:0/622233557 wait complete. 2024-04-21T08:40:06.148 INFO:tasks.cephfs.filesystem:scrub status for tag:3fcafc27-00c9-46df-b375-68348f334901 - {'path': '/', 'tag': '3fcafc27-00c9-46df-b375-68348f334901', 'options': 'recursive,force'} 2024-04-21T08:40:06.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.179+0000 7f61a3e25640 1 -- 172.21.15.94:0/2178886823 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f619c062390 con 0x7f61880a3ee0 2024-04-21T08:40:06.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.245+0000 7f6192ffd640 1 -- 172.21.15.94:0/2178886823 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f619c062390 con 0x7f61880a3ee0 2024-04-21T08:40:06.245 INFO:teuthology.orchestra.run.smithi094.stdout:Schedule set for path / 2024-04-21T08:40:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.247+0000 7f61a3e25640 1 -- 172.21.15.94:0/2178886823 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61880a3ee0 msgr2=0x7f61880a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.247+0000 7f61a3e25640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61880a3ee0 0x7f61880a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6194005fd0 tx=0x7f6194005850 comp rx=0 tx=0).stop 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.247+0000 7f61a3e25640 1 -- 172.21.15.94:0/2178886823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f619c103b00 msgr2=0x7f619c198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.247+0000 7f61a3e25640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f619c103b00 0x7f619c198b50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f618c00d950 tx=0x7f618c00de10 comp rx=0 tx=0).stop 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.247+0000 7f61a3e25640 1 -- 172.21.15.94:0/2178886823 shutdown_connections 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.247+0000 7f61a3e25640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f61880a3ee0 0x7f61880a63a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.247+0000 7f61a3e25640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f619c10c520 0x7f619c19ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.247+0000 7f61a3e25640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f619c103b00 0x7f619c198b50 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.247+0000 7f61a3e25640 1 --2- 172.21.15.94:0/2178886823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f619c1031e0 0x7f619c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.247+0000 7f61a3e25640 1 -- 172.21.15.94:0/2178886823 >> 172.21.15.94:0/2178886823 conn(0x7f619c077fd0 msgr2=0x7f619c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.248+0000 7f61a3e25640 1 -- 172.21.15.94:0/2178886823 shutdown_connections 2024-04-21T08:40:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.248+0000 7f61a3e25640 1 -- 172.21.15.94:0/2178886823 wait complete. 2024-04-21T08:40:06.258 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-21T08:40:06.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.400+0000 7f02c3904640 1 -- 172.21.15.94:0/2803787871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc106180 msgr2=0x7f02bc10dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.400+0000 7f02c3904640 1 --2- 172.21.15.94:0/2803787871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc106180 0x7f02bc10dd50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f02b4009ca0 tx=0x7f02b402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.400+0000 7f02c3904640 1 -- 172.21.15.94:0/2803787871 shutdown_connections 2024-04-21T08:40:06.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.400+0000 7f02c3904640 1 --2- 172.21.15.94:0/2803787871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc106180 0x7f02bc10dd50 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.400+0000 7f02c3904640 1 --2- 172.21.15.94:0/2803787871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc101a60 0x7f02bc105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.400+0000 7f02c3904640 1 --2- 172.21.15.94:0/2803787871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02bc1010b0 0x7f02bc101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.400+0000 7f02c3904640 1 -- 172.21.15.94:0/2803787871 >> 172.21.15.94:0/2803787871 conn(0x7f02bc078000 msgr2=0x7f02bc078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c3904640 1 -- 172.21.15.94:0/2803787871 shutdown_connections 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c3904640 1 -- 172.21.15.94:0/2803787871 wait complete. 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c3904640 1 Processor -- start 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c3904640 1 -- start start 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c3904640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02bc1010b0 0x7f02bc0febc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c3904640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc106180 0x7f02bc0ff100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c3904640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0ff640 0x7f02bc1a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c3904640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f02bc1110d0 con 0x7f02bc106180 2024-04-21T08:40:06.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c3904640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f02bc110f50 con 0x7f02bc0ff640 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c3904640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f02bc111250 con 0x7f02bc1010b0 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c0e9f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc106180 0x7f02bc0ff100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c0e9f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc106180 0x7f02bc0ff100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36726/0 (socket says 172.21.15.94:36726) 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c0e9f640 1 -- 172.21.15.94:0/118126098 learned_addr learned my addr 172.21.15.94:0/118126098 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.401+0000 7f02c1ea1640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0ff640 0x7f02bc1a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02c16a0640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02bc1010b0 0x7f02bc0febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02c0e9f640 1 -- 172.21.15.94:0/118126098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02bc1010b0 msgr2=0x7f02bc0febc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02c0e9f640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02bc1010b0 0x7f02bc0febc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02c0e9f640 1 -- 172.21.15.94:0/118126098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0ff640 msgr2=0x7f02bc1a67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02c0e9f640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0ff640 0x7f02bc1a67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02c0e9f640 1 -- 172.21.15.94:0/118126098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02bc1a6eb0 con 0x7f02bc106180 2024-04-21T08:40:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02c0e9f640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc106180 0x7f02bc0ff100 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00c970 tx=0x7f02ac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02b27fc640 1 -- 172.21.15.94:0/118126098 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac014070 con 0x7f02bc106180 2024-04-21T08:40:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02c3904640 1 -- 172.21.15.94:0/118126098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02bc1a71a0 con 0x7f02bc106180 2024-04-21T08:40:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02b27fc640 1 -- 172.21.15.94:0/118126098 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02ac004440 con 0x7f02bc106180 2024-04-21T08:40:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02b27fc640 1 -- 172.21.15.94:0/118126098 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac003e50 con 0x7f02bc106180 2024-04-21T08:40:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.402+0000 7f02c3904640 1 -- 172.21.15.94:0/118126098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02bc1a76f0 con 0x7f02bc106180 2024-04-21T08:40:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.403+0000 7f02c3904640 1 -- 172.21.15.94:0/118126098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02bc04fce0 con 0x7f02bc106180 2024-04-21T08:40:06.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.404+0000 7f02b27fc640 1 -- 172.21.15.94:0/118126098 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f02ac017070 con 0x7f02bc106180 2024-04-21T08:40:06.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.404+0000 7f02b27fc640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f02a80a40b0 0x7f02a80a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.405+0000 7f02c16a0640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f02a80a40b0 0x7f02a80a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.405+0000 7f02b27fc640 1 -- 172.21.15.94:0/118126098 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f02ac0d0a00 con 0x7f02bc106180 2024-04-21T08:40:06.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.405+0000 7f02c16a0640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f02a80a40b0 0x7f02a80a6570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f02b8005e10 tx=0x7f02b8005da0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:06.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.409+0000 7f02b27fc640 1 -- 172.21.15.94:0/118126098 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f02ac01e030 con 0x7f02bc106180 2024-04-21T08:40:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:06 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1433594137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:06 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2818554077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:06.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:06 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1433594137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:06.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:06 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2818554077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:06.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.610+0000 7f02c3904640 1 -- 172.21.15.94:0/118126098 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f02bc101a60 con 0x7f02a80a40b0 2024-04-21T08:40:06.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.674+0000 7f02b27fc640 1 -- 172.21.15.94:0/118126098 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f02bc101a60 con 0x7f02a80a40b0 2024-04-21T08:40:06.674 INFO:teuthology.orchestra.run.smithi094.stdout:Retention added to path / 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 -- 172.21.15.94:0/118126098 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f02a80a40b0 msgr2=0x7f02a80a6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f02a80a40b0 0x7f02a80a6570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f02b8005e10 tx=0x7f02b8005da0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 -- 172.21.15.94:0/118126098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc106180 msgr2=0x7f02bc0ff100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc106180 0x7f02bc0ff100 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00c970 tx=0x7f02ac00ce30 comp rx=0 tx=0).stop 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 -- 172.21.15.94:0/118126098 shutdown_connections 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f02a80a40b0 0x7f02a80a6570 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0ff640 0x7f02bc1a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc106180 0x7f02bc0ff100 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 --2- 172.21.15.94:0/118126098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02bc1010b0 0x7f02bc0febc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 -- 172.21.15.94:0/118126098 >> 172.21.15.94:0/118126098 conn(0x7f02bc078000 msgr2=0x7f02bc06aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 -- 172.21.15.94:0/118126098 shutdown_connections 2024-04-21T08:40:06.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.677+0000 7f02c3904640 1 -- 172.21.15.94:0/118126098 wait complete. 2024-04-21T08:40:06.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:06 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1433594137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:06.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:06 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2818554077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:06.688 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T08:40:06.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.787+0000 7fc051f6d640 1 -- 172.21.15.94:0/1655103617 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc04c107310 msgr2=0x7fc04c10ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.787+0000 7fc051f6d640 1 --2- 172.21.15.94:0/1655103617 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc04c107310 0x7fc04c10ee90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc044009ca0 tx=0x7fc04402c120 comp rx=0 tx=0).stop 2024-04-21T08:40:06.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.787+0000 7fc051f6d640 1 -- 172.21.15.94:0/1655103617 shutdown_connections 2024-04-21T08:40:06.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.787+0000 7fc051f6d640 1 --2- 172.21.15.94:0/1655103617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc04c1010b0 0x7fc04c106be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.787+0000 7fc051f6d640 1 --2- 172.21.15.94:0/1655103617 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc04c107310 0x7fc04c10ee90 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.787+0000 7fc051f6d640 1 --2- 172.21.15.94:0/1655103617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c100700 0x7fc04c100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.787+0000 7fc051f6d640 1 -- 172.21.15.94:0/1655103617 >> 172.21.15.94:0/1655103617 conn(0x7fc04c077390 msgr2=0x7fc04c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:06.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.787+0000 7fc051f6d640 1 -- 172.21.15.94:0/1655103617 shutdown_connections 2024-04-21T08:40:06.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.787+0000 7fc051f6d640 1 -- 172.21.15.94:0/1655103617 wait complete. 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.787+0000 7fc051f6d640 1 Processor -- start 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc051f6d640 1 -- start start 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc051f6d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c100700 0x7fc04c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc051f6d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc04c107310 0x7fc04c198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc051f6d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc04c19cfa0 0x7fc04c19d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc051f6d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc04c112260 con 0x7fc04c19cfa0 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc051f6d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc04c1120e0 con 0x7fc04c100700 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc051f6d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc04c1123e0 con 0x7fc04c107310 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc04c19cfa0 0x7fc04c19d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc04c19cfa0 0x7fc04c19d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36752/0 (socket says 172.21.15.94:36752) 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04bfff640 1 -- 172.21.15.94:0/2744997848 learned_addr learned my addr 172.21.15.94:0/2744997848 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04b7fe640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c100700 0x7fc04c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04bfff640 1 -- 172.21.15.94:0/2744997848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc04c107310 msgr2=0x7fc04c198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04affd640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc04c107310 0x7fc04c198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04bfff640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc04c107310 0x7fc04c198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04bfff640 1 -- 172.21.15.94:0/2744997848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c100700 msgr2=0x7fc04c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04bfff640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c100700 0x7fc04c1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04bfff640 1 -- 172.21.15.94:0/2744997848 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04c19da90 con 0x7fc04c19cfa0 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04b7fe640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c100700 0x7fc04c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04affd640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc04c107310 0x7fc04c198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.788+0000 7fc04bfff640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc04c19cfa0 0x7fc04c19d450 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc04402c730 tx=0x7fc044035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.789+0000 7fc048ff9640 1 -- 172.21.15.94:0/2744997848 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc044039040 con 0x7fc04c19cfa0 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.789+0000 7fc048ff9640 1 -- 172.21.15.94:0/2744997848 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc044044070 con 0x7fc04c19cfa0 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.789+0000 7fc051f6d640 1 -- 172.21.15.94:0/2744997848 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc04c19dd20 con 0x7fc04c19cfa0 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.789+0000 7fc048ff9640 1 -- 172.21.15.94:0/2744997848 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc044004380 con 0x7fc04c19cfa0 2024-04-21T08:40:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.789+0000 7fc051f6d640 1 -- 172.21.15.94:0/2744997848 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc04c1a5540 con 0x7fc04c19cfa0 2024-04-21T08:40:06.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.790+0000 7fc051f6d640 1 -- 172.21.15.94:0/2744997848 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc04c04fce0 con 0x7fc04c19cfa0 2024-04-21T08:40:06.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.791+0000 7fc048ff9640 1 -- 172.21.15.94:0/2744997848 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fc044004520 con 0x7fc04c19cfa0 2024-04-21T08:40:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.791+0000 7fc048ff9640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc0340a3f60 0x7fc0340a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.791+0000 7fc04b7fe640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc0340a3f60 0x7fc0340a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.791+0000 7fc048ff9640 1 -- 172.21.15.94:0/2744997848 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc0440f33e0 con 0x7fc04c19cfa0 2024-04-21T08:40:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.792+0000 7fc04b7fe640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc0340a3f60 0x7fc0340a6420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0058c0 tx=0x7fc03c005850 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.796+0000 7fc048ff9640 1 -- 172.21.15.94:0/2744997848 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0440bf560 con 0x7fc04c19cfa0 2024-04-21T08:40:06.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:06.996+0000 7fc051f6d640 1 -- 172.21.15.94:0/2744997848 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fc04c1010b0 con 0x7fc0340a3f60 2024-04-21T08:40:07.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.004+0000 7fc048ff9640 1 -- 172.21.15.94:0/2744997848 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fc04c1010b0 con 0x7fc0340a3f60 2024-04-21T08:40:07.003 INFO:teuthology.orchestra.run.smithi094.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T08:40:06", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-21T08:40:07.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 -- 172.21.15.94:0/2744997848 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc0340a3f60 msgr2=0x7fc0340a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc0340a3f60 0x7fc0340a6420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0058c0 tx=0x7fc03c005850 comp rx=0 tx=0).stop 2024-04-21T08:40:07.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 -- 172.21.15.94:0/2744997848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc04c19cfa0 msgr2=0x7fc04c19d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc04c19cfa0 0x7fc04c19d450 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc04402c730 tx=0x7fc044035d60 comp rx=0 tx=0).stop 2024-04-21T08:40:07.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 -- 172.21.15.94:0/2744997848 shutdown_connections 2024-04-21T08:40:07.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc0340a3f60 0x7fc0340a6420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc04c107310 0x7fc04c198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc04c19cfa0 0x7fc04c19d450 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 --2- 172.21.15.94:0/2744997848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c100700 0x7fc04c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 -- 172.21.15.94:0/2744997848 >> 172.21.15.94:0/2744997848 conn(0x7fc04c077390 msgr2=0x7fc04c1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:07.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 -- 172.21.15.94:0/2744997848 shutdown_connections 2024-04-21T08:40:07.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.006+0000 7fc051f6d640 1 -- 172.21.15.94:0/2744997848 wait complete. 2024-04-21T08:40:07.017 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T08:40:07.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 -- 172.21.15.94:0/2089086572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527c1072e0 msgr2=0x7f527c10ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 --2- 172.21.15.94:0/2089086572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527c1072e0 0x7f527c10ee80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5274009ca0 tx=0x7f527402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 -- 172.21.15.94:0/2089086572 shutdown_connections 2024-04-21T08:40:07.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 --2- 172.21.15.94:0/2089086572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527c1072e0 0x7f527c10ee80 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 --2- 172.21.15.94:0/2089086572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f527c06b3e0 0x7f527c106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 --2- 172.21.15.94:0/2089086572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f527c06aa30 0x7f527c06ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 -- 172.21.15.94:0/2089086572 >> 172.21.15.94:0/2089086572 conn(0x7f527c077ba0 msgr2=0x7f527c077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:07.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 -- 172.21.15.94:0/2089086572 shutdown_connections 2024-04-21T08:40:07.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 -- 172.21.15.94:0/2089086572 wait complete. 2024-04-21T08:40:07.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 Processor -- start 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.118+0000 7f5284632640 1 -- start start 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5284632640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f527c06aa30 0x7f527c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5284632640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f527c1072e0 0x7f527c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5284632640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527c19cf40 0x7f527c19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5284632640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f527c112200 con 0x7f527c19cf40 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5284632640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f527c112080 con 0x7f527c06aa30 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5284632640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f527c112380 con 0x7f527c1072e0 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5282bcf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527c19cf40 0x7f527c19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5282bcf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527c19cf40 0x7f527c19d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36774/0 (socket says 172.21.15.94:36774) 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5282bcf640 1 -- 172.21.15.94:0/1756999911 learned_addr learned my addr 172.21.15.94:0/1756999911 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f52823ce640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f527c06aa30 0x7f527c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5281bcd640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f527c1072e0 0x7f527c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5282bcf640 1 -- 172.21.15.94:0/1756999911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f527c1072e0 msgr2=0x7f527c198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5282bcf640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f527c1072e0 0x7f527c198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5282bcf640 1 -- 172.21.15.94:0/1756999911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f527c06aa30 msgr2=0x7f527c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5282bcf640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f527c06aa30 0x7f527c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5282bcf640 1 -- 172.21.15.94:0/1756999911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f527c19da30 con 0x7f527c19cf40 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f52823ce640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f527c06aa30 0x7f527c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5282bcf640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527c19cf40 0x7f527c19d3f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5274005e30 tx=0x7f5274032420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f52737fe640 1 -- 172.21.15.94:0/1756999911 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274041020 con 0x7f527c19cf40 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f52737fe640 1 -- 172.21.15.94:0/1756999911 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5274032970 con 0x7f527c19cf40 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5284632640 1 -- 172.21.15.94:0/1756999911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f527c19dcc0 con 0x7f527c19cf40 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f52737fe640 1 -- 172.21.15.94:0/1756999911 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274039040 con 0x7f527c19cf40 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.119+0000 7f5284632640 1 -- 172.21.15.94:0/1756999911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f527c1a54e0 con 0x7f527c19cf40 2024-04-21T08:40:07.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.120+0000 7f5284632640 1 -- 172.21.15.94:0/1756999911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f527c103690 con 0x7f527c19cf40 2024-04-21T08:40:07.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.121+0000 7f52737fe640 1 -- 172.21.15.94:0/1756999911 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f5274044020 con 0x7f527c19cf40 2024-04-21T08:40:07.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.122+0000 7f52737fe640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f52680a3f60 0x7f52680a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.122+0000 7f52823ce640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f52680a3f60 0x7f52680a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.122+0000 7f52737fe640 1 -- 172.21.15.94:0/1756999911 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f52740f2f40 con 0x7f527c19cf40 2024-04-21T08:40:07.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.122+0000 7f52823ce640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f52680a3f60 0x7f52680a6420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5278005d00 tx=0x7f5278009550 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:07.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.125+0000 7f52737fe640 1 -- 172.21.15.94:0/1756999911 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f527404b020 con 0x7f527c19cf40 2024-04-21T08:40:07.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.327+0000 7f5284632640 1 -- 172.21.15.94:0/1756999911 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f527c06b3e0 con 0x7f52680a3f60 2024-04-21T08:40:07.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.334+0000 7f52737fe640 1 -- 172.21.15.94:0/1756999911 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f527c06b3e0 con 0x7f52680a3f60 2024-04-21T08:40:07.334 INFO:teuthology.orchestra.run.smithi094.stdout:/ 1m 6m3h 2024-04-21T08:40:07.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 -- 172.21.15.94:0/1756999911 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f52680a3f60 msgr2=0x7f52680a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f52680a3f60 0x7f52680a6420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5278005d00 tx=0x7f5278009550 comp rx=0 tx=0).stop 2024-04-21T08:40:07.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 -- 172.21.15.94:0/1756999911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527c19cf40 msgr2=0x7f527c19d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527c19cf40 0x7f527c19d3f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5274005e30 tx=0x7f5274032420 comp rx=0 tx=0).stop 2024-04-21T08:40:07.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 -- 172.21.15.94:0/1756999911 shutdown_connections 2024-04-21T08:40:07.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f52680a3f60 0x7f52680a6420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f527c1072e0 0x7f527c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527c19cf40 0x7f527c19d3f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 --2- 172.21.15.94:0/1756999911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f527c06aa30 0x7f527c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 -- 172.21.15.94:0/1756999911 >> 172.21.15.94:0/1756999911 conn(0x7f527c077ba0 msgr2=0x7f527c0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:07.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 -- 172.21.15.94:0/1756999911 shutdown_connections 2024-04-21T08:40:07.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.336+0000 7f5284632640 1 -- 172.21.15.94:0/1756999911 wait complete. 2024-04-21T08:40:07.346 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:07 smithi094 ceph-mon[27988]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.3 KiB/s wr, 23 op/s 2024-04-21T08:40:07.346 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:07 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:40:07.346 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:07 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:07.347 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:07 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:40:07.347 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:07 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:07.347 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:07 smithi094 ceph-mon[27988]: from='client.15669 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:07.347 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:07 smithi094 ceph-mon[27988]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:07.348 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-21T08:40:07.372 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-21T08:40:07.385 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-21T08:40:07.420 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:40:07.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-21T08:40:07.423 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-21T08:40:07.459 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:40:07.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:07 smithi173 ceph-mon[31428]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.3 KiB/s wr, 23 op/s 2024-04-21T08:40:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:07 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:40:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:07 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:07 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:40:07.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:07 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:07.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:07 smithi173 ceph-mon[31428]: from='client.15669 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:07.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:07 smithi173 ceph-mon[31428]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:07.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.558+0000 7f23f8626640 1 -- 172.21.15.94:0/4148608703 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23f0107240 msgr2=0x7f23f010ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.558+0000 7f23f8626640 1 --2- 172.21.15.94:0/4148608703 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23f0107240 0x7f23f010ee10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009ca0 tx=0x7f23e802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.558+0000 7f23f8626640 1 -- 172.21.15.94:0/4148608703 shutdown_connections 2024-04-21T08:40:07.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.558+0000 7f23f8626640 1 --2- 172.21.15.94:0/4148608703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f0102b20 0x7f23f0106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.558+0000 7f23f8626640 1 --2- 172.21.15.94:0/4148608703 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23f0107240 0x7f23f010ee10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.558+0000 7f23f8626640 1 --2- 172.21.15.94:0/4148608703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23f0102170 0x7f23f0102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.558+0000 7f23f8626640 1 -- 172.21.15.94:0/4148608703 >> 172.21.15.94:0/4148608703 conn(0x7f23f00793a0 msgr2=0x7f23f00797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:07.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.558+0000 7f23f8626640 1 -- 172.21.15.94:0/4148608703 shutdown_connections 2024-04-21T08:40:07.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.558+0000 7f23f8626640 1 -- 172.21.15.94:0/4148608703 wait complete. 2024-04-21T08:40:07.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.558+0000 7f23f8626640 1 Processor -- start 2024-04-21T08:40:07.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f8626640 1 -- start start 2024-04-21T08:40:07.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f8626640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23f0102170 0x7f23f0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f8626640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23f0107240 0x7f23f0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f8626640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f019cfe0 0x7f23f019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f8626640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f23f0112220 con 0x7f23f0107240 2024-04-21T08:40:07.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f8626640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f23f01120a0 con 0x7f23f0102170 2024-04-21T08:40:07.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f8626640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f23f01123a0 con 0x7f23f019cfe0 2024-04-21T08:40:07.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f5bc1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23f0107240 0x7f23f0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f5bc1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23f0107240 0x7f23f0198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36794/0 (socket says 172.21.15.94:36794) 2024-04-21T08:40:07.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f5bc1640 1 -- 172.21.15.94:0/3439729273 learned_addr learned my addr 172.21.15.94:0/3439729273 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:07.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f6bc3640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f019cfe0 0x7f23f019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f5bc1640 1 -- 172.21.15.94:0/3439729273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f019cfe0 msgr2=0x7f23f019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f5bc1640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f019cfe0 0x7f23f019d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f5bc1640 1 -- 172.21.15.94:0/3439729273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23f0102170 msgr2=0x7f23f0198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f5bc1640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23f0102170 0x7f23f0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f5bc1640 1 -- 172.21.15.94:0/3439729273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f019dad0 con 0x7f23f0107240 2024-04-21T08:40:07.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f6bc3640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f019cfe0 0x7f23f019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:07.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f5bc1640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23f0107240 0x7f23f0198c60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00d740 tx=0x7f23ec00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:07.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23e77fe640 1 -- 172.21.15.94:0/3439729273 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec017070 con 0x7f23f0107240 2024-04-21T08:40:07.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23e77fe640 1 -- 172.21.15.94:0/3439729273 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23ec00de50 con 0x7f23f0107240 2024-04-21T08:40:07.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f8626640 1 -- 172.21.15.94:0/3439729273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f019ddc0 con 0x7f23f0107240 2024-04-21T08:40:07.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23e77fe640 1 -- 172.21.15.94:0/3439729273 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec004ce0 con 0x7f23f0107240 2024-04-21T08:40:07.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.559+0000 7f23f8626640 1 -- 172.21.15.94:0/3439729273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f01a5560 con 0x7f23f0107240 2024-04-21T08:40:07.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.560+0000 7f23f8626640 1 -- 172.21.15.94:0/3439729273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f0069230 con 0x7f23f0107240 2024-04-21T08:40:07.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.561+0000 7f23e77fe640 1 -- 172.21.15.94:0/3439729273 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f23ec004e80 con 0x7f23f0107240 2024-04-21T08:40:07.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.562+0000 7f23e77fe640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f23dc0a3e40 0x7f23dc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.562+0000 7f23e77fe640 1 -- 172.21.15.94:0/3439729273 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f23ec0d0930 con 0x7f23f0107240 2024-04-21T08:40:07.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.562+0000 7f23f63c2640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f23dc0a3e40 0x7f23dc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.562+0000 7f23f63c2640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f23dc0a3e40 0x7f23dc0a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009710 tx=0x7f23e0006c90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:07.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.566+0000 7f23e77fe640 1 -- 172.21.15.94:0/3439729273 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f23ec09c9a0 con 0x7f23f0107240 2024-04-21T08:40:07.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:07 smithi111 ceph-mon[31493]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.3 KiB/s wr, 23 op/s 2024-04-21T08:40:07.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:07 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:40:07.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:07 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:07.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:07 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:40:07.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:07 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:07.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:07 smithi111 ceph-mon[31493]: from='client.15669 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:07.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:07 smithi111 ceph-mon[31493]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:07.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.749+0000 7f23f8626640 1 -- 172.21.15.94:0/3439729273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23f0102b20 con 0x7f23f0107240 2024-04-21T08:40:07.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.750+0000 7f23e77fe640 1 -- 172.21.15.94:0/3439729273 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25266 (secure 0 0 0) 0x7f23ec09c2a0 con 0x7f23f0107240 2024-04-21T08:40:07.750 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:07.750 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:57.898777+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:40:07.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.752+0000 7f23f8626640 1 -- 172.21.15.94:0/3439729273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f23dc0a3e40 msgr2=0x7f23dc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.752+0000 7f23f8626640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f23dc0a3e40 0x7f23dc0a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009710 tx=0x7f23e0006c90 comp rx=0 tx=0).stop 2024-04-21T08:40:07.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.752+0000 7f23f8626640 1 -- 172.21.15.94:0/3439729273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23f0107240 msgr2=0x7f23f0198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.752+0000 7f23f8626640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23f0107240 0x7f23f0198c60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00d740 tx=0x7f23ec00dc00 comp rx=0 tx=0).stop 2024-04-21T08:40:07.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.752+0000 7f23f8626640 1 -- 172.21.15.94:0/3439729273 shutdown_connections 2024-04-21T08:40:07.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.752+0000 7f23f8626640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f23dc0a3e40 0x7f23dc0a6300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.752+0000 7f23f8626640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f019cfe0 0x7f23f019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.752+0000 7f23f8626640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23f0107240 0x7f23f0198c60 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.752+0000 7f23f8626640 1 --2- 172.21.15.94:0/3439729273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23f0102170 0x7f23f0198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.752+0000 7f23f8626640 1 -- 172.21.15.94:0/3439729273 >> 172.21.15.94:0/3439729273 conn(0x7f23f00793a0 msgr2=0x7f23f00ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:07.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.753+0000 7f23f8626640 1 -- 172.21.15.94:0/3439729273 shutdown_connections 2024-04-21T08:40:07.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.753+0000 7f23f8626640 1 -- 172.21.15.94:0/3439729273 wait complete. 2024-04-21T08:40:07.764 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:40:07.764 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:40:07.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.865+0000 7ff531cf4640 1 -- 172.21.15.94:0/1692858715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff52c106f90 msgr2=0x7ff52c107370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.865+0000 7ff531cf4640 1 --2- 172.21.15.94:0/1692858715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff52c106f90 0x7ff52c107370 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009930 tx=0x7ff51c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:07.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.865+0000 7ff531cf4640 1 -- 172.21.15.94:0/1692858715 shutdown_connections 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.865+0000 7ff531cf4640 1 --2- 172.21.15.94:0/1692858715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c06a570 0x7ff52c06a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.865+0000 7ff531cf4640 1 --2- 172.21.15.94:0/1692858715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c06af30 0x7ff52c10ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.865+0000 7ff531cf4640 1 --2- 172.21.15.94:0/1692858715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff52c106f90 0x7ff52c107370 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.865+0000 7ff531cf4640 1 -- 172.21.15.94:0/1692858715 >> 172.21.15.94:0/1692858715 conn(0x7ff52c077e30 msgr2=0x7ff52c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.865+0000 7ff531cf4640 1 -- 172.21.15.94:0/1692858715 shutdown_connections 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.865+0000 7ff531cf4640 1 -- 172.21.15.94:0/1692858715 wait complete. 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff531cf4640 1 Processor -- start 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff531cf4640 1 -- start start 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff531cf4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c06a570 0x7ff52c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff531cf4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c06af30 0x7ff52c198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff531cf4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff52c106f90 0x7ff52c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff531cf4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1121e0 con 0x7ff52c106f90 2024-04-21T08:40:07.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff531cf4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c112060 con 0x7ff52c06af30 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff531cf4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c112360 con 0x7ff52c06a570 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff52bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff52c106f90 0x7ff52c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff52bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff52c106f90 0x7ff52c19cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36812/0 (socket says 172.21.15.94:36812) 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff52bfff640 1 -- 172.21.15.94:0/1919632461 learned_addr learned my addr 172.21.15.94:0/1919632461 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff52affd640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c06af30 0x7ff52c198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.866+0000 7ff52b7fe640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c06a570 0x7ff52c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff52bfff640 1 -- 172.21.15.94:0/1919632461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c06a570 msgr2=0x7ff52c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff52bfff640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c06a570 0x7ff52c198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff52bfff640 1 -- 172.21.15.94:0/1919632461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c06af30 msgr2=0x7ff52c198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff52bfff640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c06af30 0x7ff52c198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff52bfff640 1 -- 172.21.15.94:0/1919632461 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c19d650 con 0x7ff52c106f90 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff52bfff640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff52c106f90 0x7ff52c19cf20 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff52400cce0 tx=0x7ff52400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff528ff9640 1 -- 172.21.15.94:0/1919632461 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52400f450 con 0x7ff52c106f90 2024-04-21T08:40:07.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff531cf4640 1 -- 172.21.15.94:0/1919632461 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52c19d940 con 0x7ff52c106f90 2024-04-21T08:40:07.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff528ff9640 1 -- 172.21.15.94:0/1919632461 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff52400faa0 con 0x7ff52c106f90 2024-04-21T08:40:07.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff528ff9640 1 -- 172.21.15.94:0/1919632461 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff524018a80 con 0x7ff52c106f90 2024-04-21T08:40:07.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.867+0000 7ff531cf4640 1 -- 172.21.15.94:0/1919632461 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52c1a51d0 con 0x7ff52c106f90 2024-04-21T08:40:07.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.868+0000 7ff531cf4640 1 -- 172.21.15.94:0/1919632461 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff52c0691c0 con 0x7ff52c106f90 2024-04-21T08:40:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.869+0000 7ff528ff9640 1 -- 172.21.15.94:0/1919632461 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7ff52400fc50 con 0x7ff52c106f90 2024-04-21T08:40:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.869+0000 7ff528ff9640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff5140a3e40 0x7ff5140a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.870+0000 7ff52b7fe640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff5140a3e40 0x7ff5140a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.870+0000 7ff528ff9640 1 -- 172.21.15.94:0/1919632461 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff5240d1530 con 0x7ff52c106f90 2024-04-21T08:40:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.870+0000 7ff52b7fe640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff5140a3e40 0x7ff5140a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff51c006010 tx=0x7ff51c004120 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:07.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:07.874+0000 7ff528ff9640 1 -- 172.21.15.94:0/1919632461 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff524014030 con 0x7ff52c106f90 2024-04-21T08:40:08.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.054+0000 7ff531cf4640 1 -- 172.21.15.94:0/1919632461 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff52c10a560 con 0x7ff52c106f90 2024-04-21T08:40:08.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.054+0000 7ff528ff9640 1 -- 172.21.15.94:0/1919632461 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7ff52409d5a0 con 0x7ff52c106f90 2024-04-21T08:40:08.054 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:40:08.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.056+0000 7ff531cf4640 1 -- 172.21.15.94:0/1919632461 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff5140a3e40 msgr2=0x7ff5140a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.056+0000 7ff531cf4640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff5140a3e40 0x7ff5140a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff51c006010 tx=0x7ff51c004120 comp rx=0 tx=0).stop 2024-04-21T08:40:08.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.056+0000 7ff531cf4640 1 -- 172.21.15.94:0/1919632461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff52c106f90 msgr2=0x7ff52c19cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.056+0000 7ff531cf4640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff52c106f90 0x7ff52c19cf20 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff52400cce0 tx=0x7ff52400a620 comp rx=0 tx=0).stop 2024-04-21T08:40:08.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.056+0000 7ff531cf4640 1 -- 172.21.15.94:0/1919632461 shutdown_connections 2024-04-21T08:40:08.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.056+0000 7ff531cf4640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff5140a3e40 0x7ff5140a6300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.056+0000 7ff531cf4640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff52c106f90 0x7ff52c19cf20 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.056+0000 7ff531cf4640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c06af30 0x7ff52c198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.056+0000 7ff531cf4640 1 --2- 172.21.15.94:0/1919632461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c06a570 0x7ff52c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.056+0000 7ff531cf4640 1 -- 172.21.15.94:0/1919632461 >> 172.21.15.94:0/1919632461 conn(0x7ff52c077e30 msgr2=0x7ff52c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:08.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.057+0000 7ff531cf4640 1 -- 172.21.15.94:0/1919632461 shutdown_connections 2024-04-21T08:40:08.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.057+0000 7ff531cf4640 1 -- 172.21.15.94:0/1919632461 wait complete. 2024-04-21T08:40:08.068 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:40:08.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.170+0000 7f5edce43640 1 -- 172.21.15.94:0/1093012088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ed8101a50 msgr2=0x7f5ed8105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.170+0000 7f5edce43640 1 --2- 172.21.15.94:0/1093012088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ed8101a50 0x7f5ed8105a90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0009900 tx=0x7f5ec002bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:08.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.170+0000 7f5edce43640 1 -- 172.21.15.94:0/1093012088 shutdown_connections 2024-04-21T08:40:08.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.170+0000 7f5edce43640 1 --2- 172.21.15.94:0/1093012088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed8106170 0x7f5ed810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.170+0000 7f5edce43640 1 --2- 172.21.15.94:0/1093012088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ed8101a50 0x7f5ed8105a90 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.170+0000 7f5edce43640 1 --2- 172.21.15.94:0/1093012088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ed81010a0 0x7f5ed8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.170+0000 7f5edce43640 1 -- 172.21.15.94:0/1093012088 >> 172.21.15.94:0/1093012088 conn(0x7f5ed8077eb0 msgr2=0x7f5ed80782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:08.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.170+0000 7f5edce43640 1 -- 172.21.15.94:0/1093012088 shutdown_connections 2024-04-21T08:40:08.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.170+0000 7f5edce43640 1 -- 172.21.15.94:0/1093012088 wait complete. 2024-04-21T08:40:08.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5edce43640 1 Processor -- start 2024-04-21T08:40:08.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5edce43640 1 -- start start 2024-04-21T08:40:08.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5edce43640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ed81010a0 0x7f5ed810e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5edce43640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed8106170 0x7f5ed810ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5edce43640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ed810f150 0x7f5ed81a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5edce43640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed8112220 con 0x7f5ed810f150 2024-04-21T08:40:08.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5edce43640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed81120a0 con 0x7f5ed8106170 2024-04-21T08:40:08.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5edce43640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed81123a0 con 0x7f5ed81010a0 2024-04-21T08:40:08.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ed6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ed810f150 0x7f5ed81a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ed6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ed810f150 0x7f5ed81a68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36836/0 (socket says 172.21.15.94:36836) 2024-04-21T08:40:08.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ed6d9d640 1 -- 172.21.15.94:0/821786330 learned_addr learned my addr 172.21.15.94:0/821786330 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:08.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ecffff640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed8106170 0x7f5ed810ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ed659c640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ed81010a0 0x7f5ed810e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ed6d9d640 1 -- 172.21.15.94:0/821786330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ed81010a0 msgr2=0x7f5ed810e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ed6d9d640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ed81010a0 0x7f5ed810e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ed6d9d640 1 -- 172.21.15.94:0/821786330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed8106170 msgr2=0x7f5ed810ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ed6d9d640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed8106170 0x7f5ed810ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ed6d9d640 1 -- 172.21.15.94:0/821786330 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed81a6ff0 con 0x7f5ed810f150 2024-04-21T08:40:08.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.171+0000 7f5ed659c640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ed81010a0 0x7f5ed810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:08.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.172+0000 7f5ed6d9d640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ed810f150 0x7f5ed81a68f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000eaa0 tx=0x7f5ed000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:08.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.172+0000 7f5ecf7fe640 1 -- 172.21.15.94:0/821786330 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed000cdb0 con 0x7f5ed810f150 2024-04-21T08:40:08.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.172+0000 7f5ecf7fe640 1 -- 172.21.15.94:0/821786330 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed0005020 con 0x7f5ed810f150 2024-04-21T08:40:08.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.172+0000 7f5edce43640 1 -- 172.21.15.94:0/821786330 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed81a72e0 con 0x7f5ed810f150 2024-04-21T08:40:08.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.172+0000 7f5ecf7fe640 1 -- 172.21.15.94:0/821786330 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed00116d0 con 0x7f5ed810f150 2024-04-21T08:40:08.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.172+0000 7f5edce43640 1 -- 172.21.15.94:0/821786330 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed81a77b0 con 0x7f5ed810f150 2024-04-21T08:40:08.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.173+0000 7f5edce43640 1 -- 172.21.15.94:0/821786330 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed80691c0 con 0x7f5ed810f150 2024-04-21T08:40:08.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.174+0000 7f5ecf7fe640 1 -- 172.21.15.94:0/821786330 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f5ed00051d0 con 0x7f5ed810f150 2024-04-21T08:40:08.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.174+0000 7f5ecf7fe640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5ec40a3e70 0x7f5ec40a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.175+0000 7f5ed659c640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5ec40a3e70 0x7f5ec40a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.175+0000 7f5ecf7fe640 1 -- 172.21.15.94:0/821786330 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5ed00d0fb0 con 0x7f5ed810f150 2024-04-21T08:40:08.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.175+0000 7f5ed659c640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5ec40a3e70 0x7f5ec40a6330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8009b50 tx=0x7f5ec8005f90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:08.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.179+0000 7f5ecf7fe640 1 -- 172.21.15.94:0/821786330 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ed0016030 con 0x7f5ed810f150 2024-04-21T08:40:08.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.356+0000 7f5edce43640 1 -- 172.21.15.94:0/821786330 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5ed8101a50 con 0x7f5ed810f150 2024-04-21T08:40:08.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.357+0000 7f5ecf7fe640 1 -- 172.21.15.94:0/821786330 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f5ed009d020 con 0x7f5ed810f150 2024-04-21T08:40:08.357 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 -- 172.21.15.94:0/821786330 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5ec40a3e70 msgr2=0x7f5ec40a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5ec40a3e70 0x7f5ec40a6330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8009b50 tx=0x7f5ec8005f90 comp rx=0 tx=0).stop 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 -- 172.21.15.94:0/821786330 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ed810f150 msgr2=0x7f5ed81a68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ed810f150 0x7f5ed81a68f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000eaa0 tx=0x7f5ed000ef60 comp rx=0 tx=0).stop 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 -- 172.21.15.94:0/821786330 shutdown_connections 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5ec40a3e70 0x7f5ec40a6330 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ed810f150 0x7f5ed81a68f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed8106170 0x7f5ed810ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 --2- 172.21.15.94:0/821786330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ed81010a0 0x7f5ed810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 -- 172.21.15.94:0/821786330 >> 172.21.15.94:0/821786330 conn(0x7f5ed8077eb0 msgr2=0x7f5ed8100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 -- 172.21.15.94:0/821786330 shutdown_connections 2024-04-21T08:40:08.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.359+0000 7f5edce43640 1 -- 172.21.15.94:0/821786330 wait complete. 2024-04-21T08:40:08.369 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:08 smithi094 ceph-mon[27988]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:08.369 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:08 smithi094 ceph-mon[27988]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:08.369 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:08 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3439729273' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:08.369 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:08 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1919632461' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:08.370 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:40:08.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.472+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2398878816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3f4101a50 msgr2=0x7fc3f4105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.472+0000 7fc3fc01f640 1 --2- 172.21.15.94:0/2398878816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3f4101a50 0x7fc3f4105a90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009900 tx=0x7fc3e402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.472+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2398878816 shutdown_connections 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.472+0000 7fc3fc01f640 1 --2- 172.21.15.94:0/2398878816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3f4106170 0x7fc3f410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.472+0000 7fc3fc01f640 1 --2- 172.21.15.94:0/2398878816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3f4101a50 0x7fc3f4105a90 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.472+0000 7fc3fc01f640 1 --2- 172.21.15.94:0/2398878816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f41010a0 0x7fc3f4101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.472+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2398878816 >> 172.21.15.94:0/2398878816 conn(0x7fc3f4077eb0 msgr2=0x7fc3f40782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.472+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2398878816 shutdown_connections 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.472+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2398878816 wait complete. 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fc01f640 1 Processor -- start 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fc01f640 1 -- start start 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fc01f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f41010a0 0x7fc3f419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fc01f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3f4106170 0x7fc3f419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fc01f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3f4198640 0x7fc3f4198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fc01f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f4111110 con 0x7fc3f4198640 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fc01f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f4110f90 con 0x7fc3f41010a0 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fc01f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f4111290 con 0x7fc3f4106170 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fa5bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3f4198640 0x7fc3f4198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fa5bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3f4198640 0x7fc3f4198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36860/0 (socket says 172.21.15.94:36860) 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fa5bc640 1 -- 172.21.15.94:0/2489597522 learned_addr learned my addr 172.21.15.94:0/2489597522 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3f9dbb640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f41010a0 0x7fc3f419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3f95ba640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3f4106170 0x7fc3f419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fa5bc640 1 -- 172.21.15.94:0/2489597522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3f4106170 msgr2=0x7fc3f419eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fa5bc640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3f4106170 0x7fc3f419eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fa5bc640 1 -- 172.21.15.94:0/2489597522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f41010a0 msgr2=0x7fc3f419e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fa5bc640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f41010a0 0x7fc3f419e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3fa5bc640 1 -- 172.21.15.94:0/2489597522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f4199240 con 0x7fc3f4198640 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.473+0000 7fc3f9dbb640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f41010a0 0x7fc3f419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:08.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.474+0000 7fc3fa5bc640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3f4198640 0x7fc3f4198ac0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00b970 tx=0x7fc3ec00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:08.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.474+0000 7fc3eaffd640 1 -- 172.21.15.94:0/2489597522 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec019070 con 0x7fc3f4198640 2024-04-21T08:40:08.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.474+0000 7fc3eaffd640 1 -- 172.21.15.94:0/2489597522 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3ec014440 con 0x7fc3f4198640 2024-04-21T08:40:08.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.474+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2489597522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f41a5320 con 0x7fc3f4198640 2024-04-21T08:40:08.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.474+0000 7fc3eaffd640 1 -- 172.21.15.94:0/2489597522 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec0075d0 con 0x7fc3f4198640 2024-04-21T08:40:08.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.474+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2489597522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f41a56d0 con 0x7fc3f4198640 2024-04-21T08:40:08.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.475+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2489597522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f4102570 con 0x7fc3f4198640 2024-04-21T08:40:08.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.476+0000 7fc3eaffd640 1 -- 172.21.15.94:0/2489597522 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fc3ec01c070 con 0x7fc3f4198640 2024-04-21T08:40:08.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.476+0000 7fc3eaffd640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3e00a3e40 0x7fc3e00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.477+0000 7fc3f9dbb640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3e00a3e40 0x7fc3e00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.477+0000 7fc3eaffd640 1 -- 172.21.15.94:0/2489597522 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc3ec0d0400 con 0x7fc3f4198640 2024-04-21T08:40:08.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.477+0000 7fc3f9dbb640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3e00a3e40 0x7fc3e00a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00058c0 tx=0x7fc3f0005ec0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:08.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.481+0000 7fc3eaffd640 1 -- 172.21.15.94:0/2489597522 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3ec09c580 con 0x7fc3f4198640 2024-04-21T08:40:08.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:08 smithi173 ceph-mon[31428]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:08.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:08 smithi173 ceph-mon[31428]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:08.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:08 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3439729273' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:08.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:08 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1919632461' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:08.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.658+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2489597522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc3f410a3d0 con 0x7fc3f4198640 2024-04-21T08:40:08.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.658+0000 7fc3eaffd640 1 -- 172.21.15.94:0/2489597522 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fc3ec09be80 con 0x7fc3f4198640 2024-04-21T08:40:08.658 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:40:08.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2489597522 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3e00a3e40 msgr2=0x7fc3e00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3e00a3e40 0x7fc3e00a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00058c0 tx=0x7fc3f0005ec0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2489597522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3f4198640 msgr2=0x7fc3f4198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3f4198640 0x7fc3f4198ac0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00b970 tx=0x7fc3ec00be30 comp rx=0 tx=0).stop 2024-04-21T08:40:08.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2489597522 shutdown_connections 2024-04-21T08:40:08.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3e00a3e40 0x7fc3e00a6300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3f4106170 0x7fc3f419eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3f4198640 0x7fc3f4198ac0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 --2- 172.21.15.94:0/2489597522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f41010a0 0x7fc3f419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2489597522 >> 172.21.15.94:0/2489597522 conn(0x7fc3f4077eb0 msgr2=0x7fc3f406a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:08.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2489597522 shutdown_connections 2024-04-21T08:40:08.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.660+0000 7fc3fc01f640 1 -- 172.21.15.94:0/2489597522 wait complete. 2024-04-21T08:40:08.671 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:40:08.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:08 smithi111 ceph-mon[31493]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:08.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:08 smithi111 ceph-mon[31493]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:08.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:08 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3439729273' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:08.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:08 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1919632461' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:08.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 -- 172.21.15.94:0/3881691795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b34102b40 msgr2=0x7f3b34106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 --2- 172.21.15.94:0/3881691795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b34102b40 0x7f3b34106b80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009900 tx=0x7f3b2002bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:08.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 -- 172.21.15.94:0/3881691795 shutdown_connections 2024-04-21T08:40:08.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 --2- 172.21.15.94:0/3881691795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b34107260 0x7f3b3410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 --2- 172.21.15.94:0/3881691795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b34102b40 0x7f3b34106b80 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 --2- 172.21.15.94:0/3881691795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b34102190 0x7f3b34102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 -- 172.21.15.94:0/3881691795 >> 172.21.15.94:0/3881691795 conn(0x7f3b340793a0 msgr2=0x7f3b340797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:08.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 -- 172.21.15.94:0/3881691795 shutdown_connections 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 -- 172.21.15.94:0/3881691795 wait complete. 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 Processor -- start 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.773+0000 7f3b3ac50640 1 -- start start 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b3ac50640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b34102190 0x7f3b3419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b3ac50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b34107260 0x7f3b3419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b3ac50640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b341985d0 0x7f3b34198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b3ac50640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3b341121f0 con 0x7f3b34107260 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b3ac50640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b34112070 con 0x7f3b341985d0 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b3ac50640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b34112370 con 0x7f3b34102190 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b33fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b34107260 0x7f3b3419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b33fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b34107260 0x7f3b3419e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36878/0 (socket says 172.21.15.94:36878) 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b33fff640 1 -- 172.21.15.94:0/763490571 learned_addr learned my addr 172.21.15.94:0/763490571 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b391ed640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b341985d0 0x7f3b34198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b389ec640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b34102190 0x7f3b3419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b33fff640 1 -- 172.21.15.94:0/763490571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b34102190 msgr2=0x7f3b3419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b33fff640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b34102190 0x7f3b3419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b33fff640 1 -- 172.21.15.94:0/763490571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b341985d0 msgr2=0x7f3b34198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b33fff640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b341985d0 0x7f3b34198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b33fff640 1 -- 172.21.15.94:0/763490571 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b341990f0 con 0x7f3b34107260 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b389ec640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b34102190 0x7f3b3419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b391ed640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b341985d0 0x7f3b34198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b33fff640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b34107260 0x7f3b3419e960 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3b20000c00 tx=0x7f3b2002c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b31ffb640 1 -- 172.21.15.94:0/763490571 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20043020 con 0x7f3b34107260 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b31ffb640 1 -- 172.21.15.94:0/763490571 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b2002cc60 con 0x7f3b34107260 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b3ac50640 1 -- 172.21.15.94:0/763490571 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b34199380 con 0x7f3b34107260 2024-04-21T08:40:08.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.774+0000 7f3b31ffb640 1 -- 172.21.15.94:0/763490571 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20003ca0 con 0x7f3b34107260 2024-04-21T08:40:08.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.775+0000 7f3b3ac50640 1 -- 172.21.15.94:0/763490571 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b341a55d0 con 0x7f3b34107260 2024-04-21T08:40:08.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.776+0000 7f3b3ac50640 1 -- 172.21.15.94:0/763490571 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b340691c0 con 0x7f3b34107260 2024-04-21T08:40:08.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.777+0000 7f3b31ffb640 1 -- 172.21.15.94:0/763490571 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f3b20046020 con 0x7f3b34107260 2024-04-21T08:40:08.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.778+0000 7f3b31ffb640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b240a3e40 0x7f3b240a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:08.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.778+0000 7f3b389ec640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b240a3e40 0x7f3b240a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:08.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.778+0000 7f3b31ffb640 1 -- 172.21.15.94:0/763490571 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3b200f3410 con 0x7f3b34107260 2024-04-21T08:40:08.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.778+0000 7f3b389ec640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b240a3e40 0x7f3b240a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009b40 tx=0x7f3b28005e50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:08.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.782+0000 7f3b31ffb640 1 -- 172.21.15.94:0/763490571 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b2004d020 con 0x7f3b34107260 2024-04-21T08:40:08.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.960+0000 7f3b3ac50640 1 -- 172.21.15.94:0/763490571 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3b34102b40 con 0x7f3b34107260 2024-04-21T08:40:08.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.961+0000 7f3b31ffb640 1 -- 172.21.15.94:0/763490571 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f3b200473b0 con 0x7f3b34107260 2024-04-21T08:40:08.960 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 -- 172.21.15.94:0/763490571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b240a3e40 msgr2=0x7f3b240a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b240a3e40 0x7f3b240a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009b40 tx=0x7f3b28005e50 comp rx=0 tx=0).stop 2024-04-21T08:40:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 -- 172.21.15.94:0/763490571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b34107260 msgr2=0x7f3b3419e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b34107260 0x7f3b3419e960 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3b20000c00 tx=0x7f3b2002c860 comp rx=0 tx=0).stop 2024-04-21T08:40:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 -- 172.21.15.94:0/763490571 shutdown_connections 2024-04-21T08:40:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b240a3e40 0x7f3b240a6300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b341985d0 0x7f3b34198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b34107260 0x7f3b3419e960 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 --2- 172.21.15.94:0/763490571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b34102190 0x7f3b3419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 -- 172.21.15.94:0/763490571 >> 172.21.15.94:0/763490571 conn(0x7f3b340793a0 msgr2=0x7f3b34100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:08.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 -- 172.21.15.94:0/763490571 shutdown_connections 2024-04-21T08:40:08.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:08.963+0000 7f3b3ac50640 1 -- 172.21.15.94:0/763490571 wait complete. 2024-04-21T08:40:08.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:40:09.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 -- 172.21.15.94:0/2917235807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d9c069aa0 msgr2=0x7f8d9c10ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 --2- 172.21.15.94:0/2917235807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d9c069aa0 0x7f8d9c10ede0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009ca0 tx=0x7f8d9402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 -- 172.21.15.94:0/2917235807 shutdown_connections 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 --2- 172.21.15.94:0/2917235807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d9c069aa0 0x7f8d9c10ede0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 --2- 172.21.15.94:0/2917235807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d9c0690e0 0x7f8d9c069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 --2- 172.21.15.94:0/2917235807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d9c106f50 0x7f8d9c107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 -- 172.21.15.94:0/2917235807 >> 172.21.15.94:0/2917235807 conn(0x7f8d9c077e30 msgr2=0x7f8d9c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 -- 172.21.15.94:0/2917235807 shutdown_connections 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 -- 172.21.15.94:0/2917235807 wait complete. 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 Processor -- start 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.073+0000 7f8da0e93640 1 -- start start 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8da0e93640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d9c0690e0 0x7f8d9c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8da0e93640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d9c069aa0 0x7f8d9c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8da0e93640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d9c106f50 0x7f8d9c19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8da0e93640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c112170 con 0x7f8d9c0690e0 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8da0e93640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c111ff0 con 0x7f8d9c106f50 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8da0e93640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c1122f0 con 0x7f8d9c069aa0 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d9a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d9c0690e0 0x7f8d9c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d9a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d9c0690e0 0x7f8d9c198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36896/0 (socket says 172.21.15.94:36896) 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d9a59c640 1 -- 172.21.15.94:0/668016777 learned_addr learned my addr 172.21.15.94:0/668016777 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d93fff640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d9c069aa0 0x7f8d9c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d9a59c640 1 -- 172.21.15.94:0/668016777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d9c069aa0 msgr2=0x7f8d9c198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d9a59c640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d9c069aa0 0x7f8d9c198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d9a59c640 1 -- 172.21.15.94:0/668016777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d9c106f50 msgr2=0x7f8d9c19cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d9a59c640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d9c106f50 0x7f8d9c19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d9a59c640 1 -- 172.21.15.94:0/668016777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c19d5f0 con 0x7f8d9c0690e0 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d9a59c640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d9c0690e0 0x7f8d9c198650 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00d770 tx=0x7f8d8c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:09.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d937fe640 1 -- 172.21.15.94:0/668016777 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c017070 con 0x7f8d9c0690e0 2024-04-21T08:40:09.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8da0e93640 1 -- 172.21.15.94:0/668016777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9c19d8e0 con 0x7f8d9c0690e0 2024-04-21T08:40:09.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d937fe640 1 -- 172.21.15.94:0/668016777 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d8c0054d0 con 0x7f8d9c0690e0 2024-04-21T08:40:09.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8d937fe640 1 -- 172.21.15.94:0/668016777 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c004c30 con 0x7f8d9c0690e0 2024-04-21T08:40:09.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.074+0000 7f8da0e93640 1 -- 172.21.15.94:0/668016777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d9c1a5170 con 0x7f8d9c0690e0 2024-04-21T08:40:09.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.075+0000 7f8da0e93640 1 -- 172.21.15.94:0/668016777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d9c068de0 con 0x7f8d9c0690e0 2024-04-21T08:40:09.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.076+0000 7f8d937fe640 1 -- 172.21.15.94:0/668016777 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f8d8c004dd0 con 0x7f8d9c0690e0 2024-04-21T08:40:09.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.077+0000 7f8d937fe640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8d880a3e40 0x7f8d880a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.077+0000 7f8d93fff640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8d880a3e40 0x7f8d880a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.077+0000 7f8d937fe640 1 -- 172.21.15.94:0/668016777 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f8d8c0d0db0 con 0x7f8d9c0690e0 2024-04-21T08:40:09.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.077+0000 7f8d93fff640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8d880a3e40 0x7f8d880a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8d84005d00 tx=0x7f8d84005c50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:09.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.081+0000 7f8d937fe640 1 -- 172.21.15.94:0/668016777 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d8c09ce20 con 0x7f8d9c0690e0 2024-04-21T08:40:09.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.261+0000 7f8da0e93640 1 -- 172.21.15.94:0/668016777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d9c109ea0 con 0x7f8d9c0690e0 2024-04-21T08:40:09.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.263+0000 7f8d937fe640 1 -- 172.21.15.94:0/668016777 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25266 (secure 0 0 0) 0x7f8d8c09c720 con 0x7f8d9c0690e0 2024-04-21T08:40:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:09.263 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:57.898777+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:40:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 -- 172.21.15.94:0/668016777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8d880a3e40 msgr2=0x7f8d880a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8d880a3e40 0x7f8d880a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8d84005d00 tx=0x7f8d84005c50 comp rx=0 tx=0).stop 2024-04-21T08:40:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 -- 172.21.15.94:0/668016777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d9c0690e0 msgr2=0x7f8d9c198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d9c0690e0 0x7f8d9c198650 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00d770 tx=0x7f8d8c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:40:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 -- 172.21.15.94:0/668016777 shutdown_connections 2024-04-21T08:40:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8d880a3e40 0x7f8d880a6300 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d9c069aa0 0x7f8d9c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d9c106f50 0x7f8d9c19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 --2- 172.21.15.94:0/668016777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d9c0690e0 0x7f8d9c198650 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 -- 172.21.15.94:0/668016777 >> 172.21.15.94:0/668016777 conn(0x7f8d9c077e30 msgr2=0x7f8d9c0fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 -- 172.21.15.94:0/668016777 shutdown_connections 2024-04-21T08:40:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.265+0000 7f8da0e93640 1 -- 172.21.15.94:0/668016777 wait complete. 2024-04-21T08:40:09.277 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:40:09.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:40:09.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.379+0000 7f10b7c32640 1 -- 172.21.15.94:0/1812871892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10b0107280 msgr2=0x7f10b010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.379+0000 7f10b7c32640 1 --2- 172.21.15.94:0/1812871892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10b0107280 0x7f10b010ee30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009ca0 tx=0x7f10a802c120 comp rx=0 tx=0).stop 2024-04-21T08:40:09.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.379+0000 7f10b7c32640 1 -- 172.21.15.94:0/1812871892 shutdown_connections 2024-04-21T08:40:09.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.379+0000 7f10b7c32640 1 --2- 172.21.15.94:0/1812871892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b006b170 0x7f10b0106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.379+0000 7f10b7c32640 1 --2- 172.21.15.94:0/1812871892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10b0107280 0x7f10b010ee30 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.379+0000 7f10b7c32640 1 --2- 172.21.15.94:0/1812871892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b006a7c0 0x7f10b006aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.379+0000 7f10b7c32640 1 -- 172.21.15.94:0/1812871892 >> 172.21.15.94:0/1812871892 conn(0x7f10b0076c50 msgr2=0x7f10b0077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.379+0000 7f10b7c32640 1 -- 172.21.15.94:0/1812871892 shutdown_connections 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.379+0000 7f10b7c32640 1 -- 172.21.15.94:0/1812871892 wait complete. 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b7c32640 1 Processor -- start 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b7c32640 1 -- start start 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b7c32640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b006a7c0 0x7f10b0198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b7c32640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10b0107280 0x7f10b0198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b7c32640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b019cfc0 0x7f10b019d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b7c32640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f10b0112220 con 0x7f10b0107280 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b7c32640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01120a0 con 0x7f10b006a7c0 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b7c32640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01123a0 con 0x7f10b019cfc0 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b51cd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10b0107280 0x7f10b0198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b51cd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10b0107280 0x7f10b0198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36910/0 (socket says 172.21.15.94:36910) 2024-04-21T08:40:09.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b51cd640 1 -- 172.21.15.94:0/2737940439 learned_addr learned my addr 172.21.15.94:0/2737940439 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b59ce640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b006a7c0 0x7f10b0198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b51cd640 1 -- 172.21.15.94:0/2737940439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b019cfc0 msgr2=0x7f10b019d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b51cd640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b019cfc0 0x7f10b019d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b51cd640 1 -- 172.21.15.94:0/2737940439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b006a7c0 msgr2=0x7f10b0198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b51cd640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b006a7c0 0x7f10b0198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b51cd640 1 -- 172.21.15.94:0/2737940439 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b019dab0 con 0x7f10b0107280 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.380+0000 7f10b51cd640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10b0107280 0x7f10b0198c40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f10a0006fd0 tx=0x7f10a000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.381+0000 7f10a6ffd640 1 -- 172.21.15.94:0/2737940439 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0011830 con 0x7f10b0107280 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.381+0000 7f10a6ffd640 1 -- 172.21.15.94:0/2737940439 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a0016070 con 0x7f10b0107280 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.381+0000 7f10b7c32640 1 -- 172.21.15.94:0/2737940439 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b019dda0 con 0x7f10b0107280 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.381+0000 7f10a6ffd640 1 -- 172.21.15.94:0/2737940439 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0010e00 con 0x7f10b0107280 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.381+0000 7f10b7c32640 1 -- 172.21.15.94:0/2737940439 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b01a5660 con 0x7f10b0107280 2024-04-21T08:40:09.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.381+0000 7f10b7c32640 1 -- 172.21.15.94:0/2737940439 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10b0103680 con 0x7f10b0107280 2024-04-21T08:40:09.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.383+0000 7f10a6ffd640 1 -- 172.21.15.94:0/2737940439 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f10a001b460 con 0x7f10b0107280 2024-04-21T08:40:09.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.383+0000 7f10a6ffd640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f109c0a3ec0 0x7f109c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.383+0000 7f10b59ce640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f109c0a3ec0 0x7f109c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.383+0000 7f10a6ffd640 1 -- 172.21.15.94:0/2737940439 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f10a00d0300 con 0x7f10b0107280 2024-04-21T08:40:09.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.384+0000 7f10b59ce640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f109c0a3ec0 0x7f109c0a6380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00bbe0 tx=0x7f10ac00b390 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:09.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.388+0000 7f10a6ffd640 1 -- 172.21.15.94:0/2737940439 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10a009c480 con 0x7f10b0107280 2024-04-21T08:40:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:09 smithi094 ceph-mon[27988]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.0 KiB/s wr, 18 op/s 2024-04-21T08:40:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:09 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/821786330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:09 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2489597522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:09 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/763490571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:09.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:09 smithi173 ceph-mon[31428]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.0 KiB/s wr, 18 op/s 2024-04-21T08:40:09.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:09 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/821786330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:09.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:09 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2489597522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:09.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:09 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/763490571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.567+0000 7f10b7c32640 1 -- 172.21.15.94:0/2737940439 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f10b0199a50 con 0x7f10b0107280 2024-04-21T08:40:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.567+0000 7f10a6ffd640 1 -- 172.21.15.94:0/2737940439 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f10a009bd80 con 0x7f10b0107280 2024-04-21T08:40:09.568 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:40:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 -- 172.21.15.94:0/2737940439 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f109c0a3ec0 msgr2=0x7f109c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f109c0a3ec0 0x7f109c0a6380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00bbe0 tx=0x7f10ac00b390 comp rx=0 tx=0).stop 2024-04-21T08:40:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 -- 172.21.15.94:0/2737940439 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10b0107280 msgr2=0x7f10b0198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10b0107280 0x7f10b0198c40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f10a0006fd0 tx=0x7f10a000b8e0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 -- 172.21.15.94:0/2737940439 shutdown_connections 2024-04-21T08:40:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f109c0a3ec0 0x7f109c0a6380 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b019cfc0 0x7f10b019d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10b0107280 0x7f10b0198c40 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 --2- 172.21.15.94:0/2737940439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b006a7c0 0x7f10b0198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 -- 172.21.15.94:0/2737940439 >> 172.21.15.94:0/2737940439 conn(0x7f10b0076c50 msgr2=0x7f10b00ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 -- 172.21.15.94:0/2737940439 shutdown_connections 2024-04-21T08:40:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.569+0000 7f10b7c32640 1 -- 172.21.15.94:0/2737940439 wait complete. 2024-04-21T08:40:09.580 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:40:09.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 -- 172.21.15.94:0/3305150807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfb8101090 msgr2=0x7fcfb8101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 --2- 172.21.15.94:0/3305150807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfb8101090 0x7fcfb8101470 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009900 tx=0x7fcfa802bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:09.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 -- 172.21.15.94:0/3305150807 shutdown_connections 2024-04-21T08:40:09.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 --2- 172.21.15.94:0/3305150807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfb8106160 0x7fcfb810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 --2- 172.21.15.94:0/3305150807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfb8101a40 0x7fcfb8105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 --2- 172.21.15.94:0/3305150807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfb8101090 0x7fcfb8101470 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 -- 172.21.15.94:0/3305150807 >> 172.21.15.94:0/3305150807 conn(0x7fcfb8077f50 msgr2=0x7fcfb8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 -- 172.21.15.94:0/3305150807 shutdown_connections 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 -- 172.21.15.94:0/3305150807 wait complete. 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 Processor -- start 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 -- start start 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfb8101090 0x7fcfb8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfb8106160 0x7fcfb8198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfb819d020 0x7fcfb819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81121e0 con 0x7fcfb8101090 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb8112060 con 0x7fcfb8106160 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.683+0000 7fcfbf89a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb8112360 con 0x7fcfb819d020 2024-04-21T08:40:09.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbd636640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfb8101090 0x7fcfb8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbd636640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfb8101090 0x7fcfb8198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36924/0 (socket says 172.21.15.94:36924) 2024-04-21T08:40:09.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbd636640 1 -- 172.21.15.94:0/888744256 learned_addr learned my addr 172.21.15.94:0/888744256 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:09.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbce35640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfb8106160 0x7fcfb8198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbd636640 1 -- 172.21.15.94:0/888744256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfb819d020 msgr2=0x7fcfb819d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbde37640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfb819d020 0x7fcfb819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbd636640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfb819d020 0x7fcfb819d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbd636640 1 -- 172.21.15.94:0/888744256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfb8106160 msgr2=0x7fcfb8198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbd636640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfb8106160 0x7fcfb8198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbd636640 1 -- 172.21.15.94:0/888744256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb819db10 con 0x7fcfb8101090 2024-04-21T08:40:09.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbce35640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfb8106160 0x7fcfb8198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:09.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbde37640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfb819d020 0x7fcfb819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:09.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbd636640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfb8101090 0x7fcfb8198760 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8005940 tx=0x7fcfa802c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:09.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfae7fc640 1 -- 172.21.15.94:0/888744256 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8043020 con 0x7fcfb8101090 2024-04-21T08:40:09.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfae7fc640 1 -- 172.21.15.94:0/888744256 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfa802cc60 con 0x7fcfb8101090 2024-04-21T08:40:09.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbf89a640 1 -- 172.21.15.94:0/888744256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb819dda0 con 0x7fcfb8101090 2024-04-21T08:40:09.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfae7fc640 1 -- 172.21.15.94:0/888744256 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8003c30 con 0x7fcfb8101090 2024-04-21T08:40:09.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.684+0000 7fcfbf89a640 1 -- 172.21.15.94:0/888744256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb81a55c0 con 0x7fcfb8101090 2024-04-21T08:40:09.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.685+0000 7fcfbf89a640 1 -- 172.21.15.94:0/888744256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfb8068d70 con 0x7fcfb8101090 2024-04-21T08:40:09.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:09 smithi111 ceph-mon[31493]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.0 KiB/s wr, 18 op/s 2024-04-21T08:40:09.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:09 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/821786330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:09.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:09 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2489597522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:09.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:09 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/763490571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:09.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.686+0000 7fcfae7fc640 1 -- 172.21.15.94:0/888744256 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fcfa8046020 con 0x7fcfb8101090 2024-04-21T08:40:09.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.687+0000 7fcfae7fc640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfa40a3e40 0x7fcfa40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.687+0000 7fcfae7fc640 1 -- 172.21.15.94:0/888744256 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fcfa80f3850 con 0x7fcfb8101090 2024-04-21T08:40:09.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.687+0000 7fcfbce35640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfa40a3e40 0x7fcfa40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.687+0000 7fcfbce35640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfa40a3e40 0x7fcfa40a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8100a50 tx=0x7fcfb4005d30 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:09.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.690+0000 7fcfae7fc640 1 -- 172.21.15.94:0/888744256 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcfa804d020 con 0x7fcfb8101090 2024-04-21T08:40:09.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.867+0000 7fcfbf89a640 1 -- 172.21.15.94:0/888744256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcfb8109390 con 0x7fcfb8101090 2024-04-21T08:40:09.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.867+0000 7fcfae7fc640 1 -- 172.21.15.94:0/888744256 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fcfa803e450 con 0x7fcfb8101090 2024-04-21T08:40:09.867 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:09.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.869+0000 7fcfbf89a640 1 -- 172.21.15.94:0/888744256 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfa40a3e40 msgr2=0x7fcfa40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.869+0000 7fcfbf89a640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfa40a3e40 0x7fcfa40a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8100a50 tx=0x7fcfb4005d30 comp rx=0 tx=0).stop 2024-04-21T08:40:09.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.869+0000 7fcfbf89a640 1 -- 172.21.15.94:0/888744256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfb8101090 msgr2=0x7fcfb8198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.869+0000 7fcfbf89a640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfb8101090 0x7fcfb8198760 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8005940 tx=0x7fcfa802c860 comp rx=0 tx=0).stop 2024-04-21T08:40:09.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.870+0000 7fcfbf89a640 1 -- 172.21.15.94:0/888744256 shutdown_connections 2024-04-21T08:40:09.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.870+0000 7fcfbf89a640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfa40a3e40 0x7fcfa40a6300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.870+0000 7fcfbf89a640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfb819d020 0x7fcfb819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.870+0000 7fcfbf89a640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfb8106160 0x7fcfb8198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.870+0000 7fcfbf89a640 1 --2- 172.21.15.94:0/888744256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfb8101090 0x7fcfb8198760 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.870+0000 7fcfbf89a640 1 -- 172.21.15.94:0/888744256 >> 172.21.15.94:0/888744256 conn(0x7fcfb8077f50 msgr2=0x7fcfb80ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:09.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.870+0000 7fcfbf89a640 1 -- 172.21.15.94:0/888744256 shutdown_connections 2024-04-21T08:40:09.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.870+0000 7fcfbf89a640 1 -- 172.21.15.94:0/888744256 wait complete. 2024-04-21T08:40:09.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:40:09.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.983+0000 7f4bd81f4640 1 -- 172.21.15.94:0/2636768033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd006a550 msgr2=0x7f4bd006a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.983+0000 7f4bd81f4640 1 --2- 172.21.15.94:0/2636768033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd006a550 0x7f4bd006a9d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009930 tx=0x7f4bc002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.983+0000 7f4bd81f4640 1 -- 172.21.15.94:0/2636768033 shutdown_connections 2024-04-21T08:40:09.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.983+0000 7f4bd81f4640 1 --2- 172.21.15.94:0/2636768033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bd006af10 0x7f4bd010edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.983+0000 7f4bd81f4640 1 --2- 172.21.15.94:0/2636768033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd006a550 0x7f4bd006a9d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.983+0000 7f4bd81f4640 1 --2- 172.21.15.94:0/2636768033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bd0108150 0x7f4bd0108530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.983+0000 7f4bd81f4640 1 -- 172.21.15.94:0/2636768033 >> 172.21.15.94:0/2636768033 conn(0x7f4bd0076c50 msgr2=0x7f4bd0077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.983+0000 7f4bd81f4640 1 -- 172.21.15.94:0/2636768033 shutdown_connections 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.983+0000 7f4bd81f4640 1 -- 172.21.15.94:0/2636768033 wait complete. 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd81f4640 1 Processor -- start 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd81f4640 1 -- start start 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd81f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd006a550 0x7f4bd01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd81f4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bd006af10 0x7f4bd0198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd81f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bd0108150 0x7f4bd019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd81f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd0112180 con 0x7f4bd006a550 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd81f4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd0112000 con 0x7f4bd006af10 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd81f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd0112300 con 0x7f4bd0108150 2024-04-21T08:40:09.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd5f90640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd006a550 0x7f4bd01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd5f90640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd006a550 0x7f4bd01986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36944/0 (socket says 172.21.15.94:36944) 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd5f90640 1 -- 172.21.15.94:0/1229382387 learned_addr learned my addr 172.21.15.94:0/1229382387 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd578f640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bd006af10 0x7f4bd0198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd6791640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bd0108150 0x7f4bd019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd5f90640 1 -- 172.21.15.94:0/1229382387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bd0108150 msgr2=0x7f4bd019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd5f90640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bd0108150 0x7f4bd019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd5f90640 1 -- 172.21.15.94:0/1229382387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bd006af10 msgr2=0x7f4bd0198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd5f90640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bd006af10 0x7f4bd0198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd5f90640 1 -- 172.21.15.94:0/1229382387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd019d6d0 con 0x7f4bd006a550 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.984+0000 7f4bd578f640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bd006af10 0x7f4bd0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.985+0000 7f4bd5f90640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd006a550 0x7f4bd01986e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00d770 tx=0x7f4bcc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.985+0000 7f4bc6ffd640 1 -- 172.21.15.94:0/1229382387 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc017070 con 0x7f4bd006a550 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.985+0000 7f4bd81f4640 1 -- 172.21.15.94:0/1229382387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd019d9c0 con 0x7f4bd006a550 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.985+0000 7f4bc6ffd640 1 -- 172.21.15.94:0/1229382387 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bcc009c30 con 0x7f4bd006a550 2024-04-21T08:40:09.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.985+0000 7f4bc6ffd640 1 -- 172.21.15.94:0/1229382387 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc003cc0 con 0x7f4bd006a550 2024-04-21T08:40:09.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.985+0000 7f4bd81f4640 1 -- 172.21.15.94:0/1229382387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd01a5250 con 0x7f4bd006a550 2024-04-21T08:40:09.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.986+0000 7f4bd81f4640 1 -- 172.21.15.94:0/1229382387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd0103680 con 0x7f4bd006a550 2024-04-21T08:40:09.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.987+0000 7f4bc6ffd640 1 -- 172.21.15.94:0/1229382387 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f4bcc003e60 con 0x7f4bd006a550 2024-04-21T08:40:09.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.987+0000 7f4bc6ffd640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4bbc0a3e40 0x7f4bbc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:09.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.987+0000 7f4bd578f640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4bbc0a3e40 0x7f4bbc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:09.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.987+0000 7f4bc6ffd640 1 -- 172.21.15.94:0/1229382387 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4bcc0d0900 con 0x7f4bd006a550 2024-04-21T08:40:09.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.988+0000 7f4bd578f640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4bbc0a3e40 0x7f4bbc0a6300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0006010 tx=0x7f4bc0009610 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:09.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:09.992+0000 7f4bc6ffd640 1 -- 172.21.15.94:0/1229382387 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4bcc09c970 con 0x7f4bd006a550 2024-04-21T08:40:10.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.169+0000 7f4bd81f4640 1 -- 172.21.15.94:0/1229382387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4bd0199a30 con 0x7f4bd006a550 2024-04-21T08:40:10.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.169+0000 7f4bc6ffd640 1 -- 172.21.15.94:0/1229382387 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f4bcc009de0 con 0x7f4bd006a550 2024-04-21T08:40:10.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:40:10.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 -- 172.21.15.94:0/1229382387 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4bbc0a3e40 msgr2=0x7f4bbc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4bbc0a3e40 0x7f4bbc0a6300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0006010 tx=0x7f4bc0009610 comp rx=0 tx=0).stop 2024-04-21T08:40:10.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 -- 172.21.15.94:0/1229382387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd006a550 msgr2=0x7f4bd01986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd006a550 0x7f4bd01986e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00d770 tx=0x7f4bcc00dc30 comp rx=0 tx=0).stop 2024-04-21T08:40:10.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 -- 172.21.15.94:0/1229382387 shutdown_connections 2024-04-21T08:40:10.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4bbc0a3e40 0x7f4bbc0a6300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bd0108150 0x7f4bd019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bd006af10 0x7f4bd0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 --2- 172.21.15.94:0/1229382387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd006a550 0x7f4bd01986e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 -- 172.21.15.94:0/1229382387 >> 172.21.15.94:0/1229382387 conn(0x7f4bd0076c50 msgr2=0x7f4bd00ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:10.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 -- 172.21.15.94:0/1229382387 shutdown_connections 2024-04-21T08:40:10.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.171+0000 7f4bd81f4640 1 -- 172.21.15.94:0/1229382387 wait complete. 2024-04-21T08:40:10.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:40:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.285+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/396073036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9f8107030 msgr2=0x7fb9f8107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.285+0000 7fb9ffdd5640 1 --2- 172.21.15.94:0/396073036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9f8107030 0x7fb9f8107410 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009930 tx=0x7fb9f402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.285+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/396073036 shutdown_connections 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.285+0000 7fb9ffdd5640 1 --2- 172.21.15.94:0/396073036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f81011a0 0x7fb9f8101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.285+0000 7fb9ffdd5640 1 --2- 172.21.15.94:0/396073036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9f8101b40 0x7fb9f810dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.285+0000 7fb9ffdd5640 1 --2- 172.21.15.94:0/396073036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9f8107030 0x7fb9f8107410 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.285+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/396073036 >> 172.21.15.94:0/396073036 conn(0x7fb9f8076c50 msgr2=0x7fb9f8077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.285+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/396073036 shutdown_connections 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.285+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/396073036 wait complete. 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9ffdd5640 1 Processor -- start 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9ffdd5640 1 -- start start 2024-04-21T08:40:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9ffdd5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9f81011a0 0x7fb9f8199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9ffdd5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8101b40 0x7fb9f8199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9ffdd5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9f8107030 0x7fb9f81a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9ffdd5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f81110e0 con 0x7fb9f81011a0 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9ffdd5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f8110f60 con 0x7fb9f8107030 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9ffdd5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f8111260 con 0x7fb9f8101b40 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fdb71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9f81011a0 0x7fb9f8199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fdb71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9f81011a0 0x7fb9f8199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36962/0 (socket says 172.21.15.94:36962) 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fdb71640 1 -- 172.21.15.94:0/2883492950 learned_addr learned my addr 172.21.15.94:0/2883492950 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fd370640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8101b40 0x7fb9f8199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fdb71640 1 -- 172.21.15.94:0/2883492950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8101b40 msgr2=0x7fb9f8199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fdb71640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8101b40 0x7fb9f8199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fdb71640 1 -- 172.21.15.94:0/2883492950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9f8107030 msgr2=0x7fb9f81a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fe372640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9f8107030 0x7fb9f81a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fdb71640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9f8107030 0x7fb9f81a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fdb71640 1 -- 172.21.15.94:0/2883492950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f81a4c00 con 0x7fb9f81011a0 2024-04-21T08:40:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.286+0000 7fb9fd370640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8101b40 0x7fb9f8199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.287+0000 7fb9fdb71640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9f81011a0 0x7fb9f8199640 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009230 tx=0x7fb9f40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.287+0000 7fb9eeffd640 1 -- 172.21.15.94:0/2883492950 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4043020 con 0x7fb9f81011a0 2024-04-21T08:40:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.287+0000 7fb9eeffd640 1 -- 172.21.15.94:0/2883492950 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f402cb70 con 0x7fb9f81011a0 2024-04-21T08:40:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.287+0000 7fb9eeffd640 1 -- 172.21.15.94:0/2883492950 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4003b50 con 0x7fb9f81011a0 2024-04-21T08:40:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.287+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/2883492950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f81a4dd0 con 0x7fb9f81011a0 2024-04-21T08:40:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.287+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/2883492950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f81a5230 con 0x7fb9f81011a0 2024-04-21T08:40:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.288+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/2883492950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9f8102590 con 0x7fb9f81011a0 2024-04-21T08:40:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.289+0000 7fb9eeffd640 1 -- 172.21.15.94:0/2883492950 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fb9f4046020 con 0x7fb9f81011a0 2024-04-21T08:40:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.289+0000 7fb9eeffd640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb9e40a3e40 0x7fb9e40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.290+0000 7fb9fd370640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb9e40a3e40 0x7fb9e40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.290+0000 7fb9eeffd640 1 -- 172.21.15.94:0/2883492950 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb9f40f3020 con 0x7fb9f81011a0 2024-04-21T08:40:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.290+0000 7fb9fd370640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb9e40a3e40 0x7fb9e40a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80058c0 tx=0x7fb9e8005850 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.294+0000 7fb9eeffd640 1 -- 172.21.15.94:0/2883492950 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9f404d020 con 0x7fb9f81011a0 2024-04-21T08:40:10.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.473+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/2883492950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb9f806ab40 con 0x7fb9f81011a0 2024-04-21T08:40:10.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.473+0000 7fb9eeffd640 1 -- 172.21.15.94:0/2883492950 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fb9f403c350 con 0x7fb9f81011a0 2024-04-21T08:40:10.473 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:10.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/2883492950 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb9e40a3e40 msgr2=0x7fb9e40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb9e40a3e40 0x7fb9e40a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80058c0 tx=0x7fb9e8005850 comp rx=0 tx=0).stop 2024-04-21T08:40:10.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/2883492950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9f81011a0 msgr2=0x7fb9f8199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9f81011a0 0x7fb9f8199640 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009230 tx=0x7fb9f40045e0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/2883492950 shutdown_connections 2024-04-21T08:40:10.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb9e40a3e40 0x7fb9e40a6300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8101b40 0x7fb9f8199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9f8107030 0x7fb9f81a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 --2- 172.21.15.94:0/2883492950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9f81011a0 0x7fb9f8199640 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/2883492950 >> 172.21.15.94:0/2883492950 conn(0x7fb9f8076c50 msgr2=0x7fb9f8100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:10.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.475+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/2883492950 shutdown_connections 2024-04-21T08:40:10.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.476+0000 7fb9ffdd5640 1 -- 172.21.15.94:0/2883492950 wait complete. 2024-04-21T08:40:10.485 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:10 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/668016777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:10.485 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:10 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2737940439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:10.486 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:10 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/888744256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:10.486 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:10 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1229382387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:10.487 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-21T08:40:10.487 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:40:10.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:10 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/668016777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:10.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:10 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2737940439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:10.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:10 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/888744256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:10.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:10 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1229382387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:10.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.585+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2919963279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc8406a8b0 msgr2=0x7fbc8406ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.585+0000 7fbc8afa4640 1 --2- 172.21.15.94:0/2919963279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc8406a8b0 0x7fbc8406ac90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009ca0 tx=0x7fbc7c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.585+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2919963279 shutdown_connections 2024-04-21T08:40:10.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.585+0000 7fbc8afa4640 1 --2- 172.21.15.94:0/2919963279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84107340 0x7fbc8410eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.585+0000 7fbc8afa4640 1 --2- 172.21.15.94:0/2919963279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8406b260 0x7fbc84106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.585+0000 7fbc8afa4640 1 --2- 172.21.15.94:0/2919963279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc8406a8b0 0x7fbc8406ac90 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.585+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2919963279 >> 172.21.15.94:0/2919963279 conn(0x7fbc84077e30 msgr2=0x7fbc84078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.585+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2919963279 shutdown_connections 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.585+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2919963279 wait complete. 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc8afa4640 1 Processor -- start 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc8afa4640 1 -- start start 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc8afa4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8406a8b0 0x7fbc8419e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc8afa4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84107340 0x7fbc8419e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc8afa4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84198600 0x7fbc84198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc8afa4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbc84112260 con 0x7fbc84107340 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc8afa4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbc841120e0 con 0x7fbc8406a8b0 2024-04-21T08:40:10.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc8afa4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc841123e0 con 0x7fbc84198600 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc83fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84107340 0x7fbc8419e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc83fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84107340 0x7fbc8419e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36980/0 (socket says 172.21.15.94:36980) 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc83fff640 1 -- 172.21.15.94:0/2653583729 learned_addr learned my addr 172.21.15.94:0/2653583729 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc88d40640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8406a8b0 0x7fbc8419e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc89541640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84198600 0x7fbc84198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc83fff640 1 -- 172.21.15.94:0/2653583729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84198600 msgr2=0x7fbc84198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc83fff640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84198600 0x7fbc84198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc83fff640 1 -- 172.21.15.94:0/2653583729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8406a8b0 msgr2=0x7fbc8419e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc83fff640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8406a8b0 0x7fbc8419e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc83fff640 1 -- 172.21.15.94:0/2653583729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc84199120 con 0x7fbc84107340 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.586+0000 7fbc83fff640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84107340 0x7fbc8419e990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d950 tx=0x7fbc6c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.587+0000 7fbc81ffb640 1 -- 172.21.15.94:0/2653583729 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c017070 con 0x7fbc84107340 2024-04-21T08:40:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.587+0000 7fbc81ffb640 1 -- 172.21.15.94:0/2653583729 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c009c30 con 0x7fbc84107340 2024-04-21T08:40:10.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.587+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2653583729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc84199410 con 0x7fbc84107340 2024-04-21T08:40:10.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.587+0000 7fbc81ffb640 1 -- 172.21.15.94:0/2653583729 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0047a0 con 0x7fbc84107340 2024-04-21T08:40:10.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.587+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2653583729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc841a56b0 con 0x7fbc84107340 2024-04-21T08:40:10.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.588+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2653583729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc84069230 con 0x7fbc84107340 2024-04-21T08:40:10.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.589+0000 7fbc81ffb640 1 -- 172.21.15.94:0/2653583729 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fbc6c004020 con 0x7fbc84107340 2024-04-21T08:40:10.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.589+0000 7fbc81ffb640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc700a3ec0 0x7fbc700a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.590+0000 7fbc88d40640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc700a3ec0 0x7fbc700a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.590+0000 7fbc81ffb640 1 -- 172.21.15.94:0/2653583729 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fbc6c0d09d0 con 0x7fbc84107340 2024-04-21T08:40:10.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.590+0000 7fbc88d40640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc700a3ec0 0x7fbc700a6380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c004af0 tx=0x7fbc7c005c50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:10.594 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:10 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/668016777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:10.594 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:10 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2737940439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:10.594 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:10 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/888744256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:10.594 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:10 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1229382387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.594+0000 7fbc81ffb640 1 -- 172.21.15.94:0/2653583729 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc6c020020 con 0x7fbc84107340 2024-04-21T08:40:10.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.770+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2653583729 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbc8410a520 con 0x7fbc700a3ec0 2024-04-21T08:40:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.780+0000 7fbc81ffb640 1 -- 172.21.15.94:0/2653583729 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339720 (secure 0 0 0) 0x7fbc8410a520 con 0x7fbc700a3ec0 2024-04-21T08:40:10.784 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:10.786 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-21T08:40:10.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.783+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2653583729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc700a3ec0 msgr2=0x7fbc700a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.783+0000 7fbc8afa4640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc700a3ec0 0x7fbc700a6380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c004af0 tx=0x7fbc7c005c50 comp rx=0 tx=0).stop 2024-04-21T08:40:10.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.783+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2653583729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84107340 msgr2=0x7fbc8419e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.783+0000 7fbc8afa4640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84107340 0x7fbc8419e990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d950 tx=0x7fbc6c00de10 comp rx=0 tx=0).stop 2024-04-21T08:40:10.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.783+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2653583729 shutdown_connections 2024-04-21T08:40:10.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.783+0000 7fbc8afa4640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc700a3ec0 0x7fbc700a6380 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.783+0000 7fbc8afa4640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84198600 0x7fbc84198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.783+0000 7fbc8afa4640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc84107340 0x7fbc8419e990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.783+0000 7fbc8afa4640 1 --2- 172.21.15.94:0/2653583729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8406a8b0 0x7fbc8419e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.783+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2653583729 >> 172.21.15.94:0/2653583729 conn(0x7fbc84077e30 msgr2=0x7fbc840ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:10.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.784+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2653583729 shutdown_connections 2024-04-21T08:40:10.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.784+0000 7fbc8afa4640 1 -- 172.21.15.94:0/2653583729 wait complete. 2024-04-21T08:40:10.795 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-21T08:40:09.917339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1391043,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1420,"num_read_kb":1143,"num_write":627,"num_write_kb":5306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378652,"kb_used_data":12812,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563172,"statfs":{"total":1151940427776,"available":1151552688128,"internally_reserved":0,"allocated":13119488,"data_stored":10506743,"data_compressed":573078,"data_compressed_allocated":774144,"data_compressed_original":4878336,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":200325,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":420,"num_read_kb":479,"num_write":155,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.988315"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318569+0000","last_change":"2024-04-21T08:37:31.448345+0000","last_active":"2024-04-21T08:39:58.318569+0000","last_peered":"2024-04-21T08:39:58.318569+0000","last_clean":"2024-04-21T08:39:58.318569+0000","last_became_active":"2024-04-21T08:37:31.447878+0000","last_became_peered":"2024-04-21T08:37:31.447878+0000","last_unstale":"2024-04-21T08:39:58.318569+0000","last_undegraded":"2024-04-21T08:39:58.318569+0000","last_fullsized":"2024-04-21T08:39:58.318569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:45.589165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319782+0000","last_change":"2024-04-21T08:37:31.452895+0000","last_active":"2024-04-21T08:39:58.319782+0000","last_peered":"2024-04-21T08:39:58.319782+0000","last_clean":"2024-04-21T08:39:58.319782+0000","last_became_active":"2024-04-21T08:37:31.452550+0000","last_became_peered":"2024-04-21T08:37:31.452550+0000","last_unstale":"2024-04-21T08:39:58.319782+0000","last_undegraded":"2024-04-21T08:39:58.319782+0000","last_fullsized":"2024-04-21T08:39:58.319782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:02.497802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319443+0000","last_change":"2024-04-21T08:37:31.447552+0000","last_active":"2024-04-21T08:39:58.319443+0000","last_peered":"2024-04-21T08:39:58.319443+0000","last_clean":"2024-04-21T08:39:58.319443+0000","last_became_active":"2024-04-21T08:37:31.446044+0000","last_became_peered":"2024-04-21T08:37:31.446044+0000","last_unstale":"2024-04-21T08:39:58.319443+0000","last_undegraded":"2024-04-21T08:39:58.319443+0000","last_fullsized":"2024-04-21T08:39:58.319443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:22.801977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151531+0000","last_change":"2024-04-21T08:37:31.452400+0000","last_active":"2024-04-21T08:39:58.151531+0000","last_peered":"2024-04-21T08:39:58.151531+0000","last_clean":"2024-04-21T08:39:58.151531+0000","last_became_active":"2024-04-21T08:37:31.451462+0000","last_became_peered":"2024-04-21T08:37:31.451462+0000","last_unstale":"2024-04-21T08:39:58.151531+0000","last_undegraded":"2024-04-21T08:39:58.151531+0000","last_fullsized":"2024-04-21T08:39:58.151531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:22.463481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009087+0000","last_change":"2024-04-21T08:37:31.448942+0000","last_active":"2024-04-21T08:39:58.009087+0000","last_peered":"2024-04-21T08:39:58.009087+0000","last_clean":"2024-04-21T08:39:58.009087+0000","last_became_active":"2024-04-21T08:37:31.448526+0000","last_became_peered":"2024-04-21T08:37:31.448526+0000","last_unstale":"2024-04-21T08:39:58.009087+0000","last_undegraded":"2024-04-21T08:39:58.009087+0000","last_fullsized":"2024-04-21T08:39:58.009087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:29.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017166+0000","last_change":"2024-04-21T08:37:31.452165+0000","last_active":"2024-04-21T08:39:58.017166+0000","last_peered":"2024-04-21T08:39:58.017166+0000","last_clean":"2024-04-21T08:39:58.017166+0000","last_became_active":"2024-04-21T08:37:31.451802+0000","last_became_peered":"2024-04-21T08:37:31.451802+0000","last_unstale":"2024-04-21T08:39:58.017166+0000","last_undegraded":"2024-04-21T08:39:58.017166+0000","last_fullsized":"2024-04-21T08:39:58.017166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:42.210544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905010+0000","last_change":"2024-04-21T08:39:31.681660+0000","last_active":"2024-04-21T08:39:57.905010+0000","last_peered":"2024-04-21T08:39:57.905010+0000","last_clean":"2024-04-21T08:39:57.905010+0000","last_became_active":"2024-04-21T08:39:31.681203+0000","last_became_peered":"2024-04-21T08:39:31.681203+0000","last_unstale":"2024-04-21T08:39:57.905010+0000","last_undegraded":"2024-04-21T08:39:57.905010+0000","last_fullsized":"2024-04-21T08:39:57.905010+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:54.120862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903472+0000","last_change":"2024-04-21T08:37:31.448257+0000","last_active":"2024-04-21T08:39:57.903472+0000","last_peered":"2024-04-21T08:39:57.903472+0000","last_clean":"2024-04-21T08:39:57.903472+0000","last_became_active":"2024-04-21T08:37:31.447845+0000","last_became_peered":"2024-04-21T08:37:31.447845+0000","last_unstale":"2024-04-21T08:39:57.903472+0000","last_undegraded":"2024-04-21T08:39:57.903472+0000","last_fullsized":"2024-04-21T08:39:57.903472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:19.956699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154785+0000","last_change":"2024-04-21T08:37:31.443022+0000","last_active":"2024-04-21T08:39:58.154785+0000","last_peered":"2024-04-21T08:39:58.154785+0000","last_clean":"2024-04-21T08:39:58.154785+0000","last_became_active":"2024-04-21T08:37:31.442645+0000","last_became_peered":"2024-04-21T08:37:31.442645+0000","last_unstale":"2024-04-21T08:39:58.154785+0000","last_undegraded":"2024-04-21T08:39:58.154785+0000","last_fullsized":"2024-04-21T08:39:58.154785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:18:10.861281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017434+0000","last_change":"2024-04-21T08:37:31.453427+0000","last_active":"2024-04-21T08:39:58.017434+0000","last_peered":"2024-04-21T08:39:58.017434+0000","last_clean":"2024-04-21T08:39:58.017434+0000","last_became_active":"2024-04-21T08:37:31.453096+0000","last_became_peered":"2024-04-21T08:37:31.453096+0000","last_unstale":"2024-04-21T08:39:58.017434+0000","last_undegraded":"2024-04-21T08:39:58.017434+0000","last_fullsized":"2024-04-21T08:39:58.017434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:49.450351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905929+0000","last_change":"2024-04-21T08:37:31.449908+0000","last_active":"2024-04-21T08:39:57.905929+0000","last_peered":"2024-04-21T08:39:57.905929+0000","last_clean":"2024-04-21T08:39:57.905929+0000","last_became_active":"2024-04-21T08:37:31.449558+0000","last_became_peered":"2024-04-21T08:37:31.449558+0000","last_unstale":"2024-04-21T08:39:57.905929+0000","last_undegraded":"2024-04-21T08:39:57.905929+0000","last_fullsized":"2024-04-21T08:39:57.905929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:21.538689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151539+0000","last_change":"2024-04-21T08:37:31.452973+0000","last_active":"2024-04-21T08:39:58.151539+0000","last_peered":"2024-04-21T08:39:58.151539+0000","last_clean":"2024-04-21T08:39:58.151539+0000","last_became_active":"2024-04-21T08:37:31.452586+0000","last_became_peered":"2024-04-21T08:37:31.452586+0000","last_unstale":"2024-04-21T08:39:58.151539+0000","last_undegraded":"2024-04-21T08:39:58.151539+0000","last_fullsized":"2024-04-21T08:39:58.151539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:09:11.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319144+0000","last_change":"2024-04-21T08:37:31.445801+0000","last_active":"2024-04-21T08:39:58.319144+0000","last_peered":"2024-04-21T08:39:58.319144+0000","last_clean":"2024-04-21T08:39:58.319144+0000","last_became_active":"2024-04-21T08:37:31.445478+0000","last_became_peered":"2024-04-21T08:37:31.445478+0000","last_unstale":"2024-04-21T08:39:58.319144+0000","last_undegraded":"2024-04-21T08:39:58.319144+0000","last_fullsized":"2024-04-21T08:39:58.319144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:33.352739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319201+0000","last_change":"2024-04-21T08:37:31.450735+0000","last_active":"2024-04-21T08:39:58.319201+0000","last_peered":"2024-04-21T08:39:58.319201+0000","last_clean":"2024-04-21T08:39:58.319201+0000","last_became_active":"2024-04-21T08:37:31.450384+0000","last_became_peered":"2024-04-21T08:37:31.450384+0000","last_unstale":"2024-04-21T08:39:58.319201+0000","last_undegraded":"2024-04-21T08:39:58.319201+0000","last_fullsized":"2024-04-21T08:39:58.319201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:45.950489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319354+0000","last_change":"2024-04-21T08:37:31.454690+0000","last_active":"2024-04-21T08:39:58.319354+0000","last_peered":"2024-04-21T08:39:58.319354+0000","last_clean":"2024-04-21T08:39:58.319354+0000","last_became_active":"2024-04-21T08:37:31.454330+0000","last_became_peered":"2024-04-21T08:37:31.454330+0000","last_unstale":"2024-04-21T08:39:58.319354+0000","last_undegraded":"2024-04-21T08:39:58.319354+0000","last_fullsized":"2024-04-21T08:39:58.319354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:52.644317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153871+0000","last_change":"2024-04-21T08:37:31.441389+0000","last_active":"2024-04-21T08:39:58.153871+0000","last_peered":"2024-04-21T08:39:58.153871+0000","last_clean":"2024-04-21T08:39:58.153871+0000","last_became_active":"2024-04-21T08:37:31.441065+0000","last_became_peered":"2024-04-21T08:37:31.441065+0000","last_unstale":"2024-04-21T08:39:58.153871+0000","last_undegraded":"2024-04-21T08:39:58.153871+0000","last_fullsized":"2024-04-21T08:39:58.153871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:56.528154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318575+0000","last_change":"2024-04-21T08:37:31.446722+0000","last_active":"2024-04-21T08:39:58.318575+0000","last_peered":"2024-04-21T08:39:58.318575+0000","last_clean":"2024-04-21T08:39:58.318575+0000","last_became_active":"2024-04-21T08:37:31.446396+0000","last_became_peered":"2024-04-21T08:37:31.446396+0000","last_unstale":"2024-04-21T08:39:58.318575+0000","last_undegraded":"2024-04-21T08:39:58.318575+0000","last_fullsized":"2024-04-21T08:39:58.318575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:17.488633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007275+0000","last_change":"2024-04-21T08:37:31.444388+0000","last_active":"2024-04-21T08:39:58.007275+0000","last_peered":"2024-04-21T08:39:58.007275+0000","last_clean":"2024-04-21T08:39:58.007275+0000","last_became_active":"2024-04-21T08:37:31.444023+0000","last_became_peered":"2024-04-21T08:37:31.444023+0000","last_unstale":"2024-04-21T08:39:58.007275+0000","last_undegraded":"2024-04-21T08:39:58.007275+0000","last_fullsized":"2024-04-21T08:39:58.007275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:07:46.238941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905278+0000","last_change":"2024-04-21T08:37:31.447850+0000","last_active":"2024-04-21T08:39:57.905278+0000","last_peered":"2024-04-21T08:39:57.905278+0000","last_clean":"2024-04-21T08:39:57.905278+0000","last_became_active":"2024-04-21T08:37:31.447505+0000","last_became_peered":"2024-04-21T08:37:31.447505+0000","last_unstale":"2024-04-21T08:39:57.905278+0000","last_undegraded":"2024-04-21T08:39:57.905278+0000","last_fullsized":"2024-04-21T08:39:57.905278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:55.399410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905076+0000","last_change":"2024-04-21T08:37:35.608747+0000","last_active":"2024-04-21T08:39:57.905076+0000","last_peered":"2024-04-21T08:39:57.905076+0000","last_clean":"2024-04-21T08:39:57.905076+0000","last_became_active":"2024-04-21T08:37:35.608157+0000","last_became_peered":"2024-04-21T08:37:35.608157+0000","last_unstale":"2024-04-21T08:39:57.905076+0000","last_undegraded":"2024-04-21T08:39:57.905076+0000","last_fullsized":"2024-04-21T08:39:57.905076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:53.384190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007094+0000","last_change":"2024-04-21T08:38:30.679843+0000","last_active":"2024-04-21T08:39:58.007094+0000","last_peered":"2024-04-21T08:39:58.007094+0000","last_clean":"2024-04-21T08:39:58.007094+0000","last_became_active":"2024-04-21T08:38:30.670964+0000","last_became_peered":"2024-04-21T08:38:30.670964+0000","last_unstale":"2024-04-21T08:39:58.007094+0000","last_undegraded":"2024-04-21T08:39:58.007094+0000","last_fullsized":"2024-04-21T08:39:58.007094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:49.652571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904100+0000","last_change":"2024-04-21T08:37:31.451813+0000","last_active":"2024-04-21T08:39:57.904100+0000","last_peered":"2024-04-21T08:39:57.904100+0000","last_clean":"2024-04-21T08:39:57.904100+0000","last_became_active":"2024-04-21T08:37:31.451447+0000","last_became_peered":"2024-04-21T08:37:31.451447+0000","last_unstale":"2024-04-21T08:39:57.904100+0000","last_undegraded":"2024-04-21T08:39:57.904100+0000","last_fullsized":"2024-04-21T08:39:57.904100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:46:45.235326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151276+0000","last_change":"2024-04-21T08:37:35.603518+0000","last_active":"2024-04-21T08:39:58.151276+0000","last_peered":"2024-04-21T08:39:58.151276+0000","last_clean":"2024-04-21T08:39:58.151276+0000","last_became_active":"2024-04-21T08:37:35.603038+0000","last_became_peered":"2024-04-21T08:37:35.603038+0000","last_unstale":"2024-04-21T08:39:58.151276+0000","last_undegraded":"2024-04-21T08:39:58.151276+0000","last_fullsized":"2024-04-21T08:39:58.151276+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:06:29.224025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904228+0000","last_change":"2024-04-21T08:37:30.424608+0000","last_active":"2024-04-21T08:39:57.904228+0000","last_peered":"2024-04-21T08:39:57.904228+0000","last_clean":"2024-04-21T08:39:57.904228+0000","last_became_active":"2024-04-21T08:37:30.424259+0000","last_became_peered":"2024-04-21T08:37:30.424259+0000","last_unstale":"2024-04-21T08:39:57.904228+0000","last_undegraded":"2024-04-21T08:39:57.904228+0000","last_fullsized":"2024-04-21T08:39:57.904228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:38.007095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319688+0000","last_change":"2024-04-21T08:37:31.444656+0000","last_active":"2024-04-21T08:39:58.319688+0000","last_peered":"2024-04-21T08:39:58.319688+0000","last_clean":"2024-04-21T08:39:58.319688+0000","last_became_active":"2024-04-21T08:37:31.444300+0000","last_became_peered":"2024-04-21T08:37:31.444300+0000","last_unstale":"2024-04-21T08:39:58.319688+0000","last_undegraded":"2024-04-21T08:39:58.319688+0000","last_fullsized":"2024-04-21T08:39:58.319688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:40.901674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151439+0000","last_change":"2024-04-21T08:37:35.611732+0000","last_active":"2024-04-21T08:39:58.151439+0000","last_peered":"2024-04-21T08:39:58.151439+0000","last_clean":"2024-04-21T08:39:58.151439+0000","last_became_active":"2024-04-21T08:37:35.611060+0000","last_became_peered":"2024-04-21T08:37:35.611060+0000","last_unstale":"2024-04-21T08:39:58.151439+0000","last_undegraded":"2024-04-21T08:39:58.151439+0000","last_fullsized":"2024-04-21T08:39:58.151439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:36:01.491949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318678+0000","last_change":"2024-04-21T08:38:30.689722+0000","last_active":"2024-04-21T08:39:58.318678+0000","last_peered":"2024-04-21T08:39:58.318678+0000","last_clean":"2024-04-21T08:39:58.318678+0000","last_became_active":"2024-04-21T08:38:30.676521+0000","last_became_peered":"2024-04-21T08:38:30.676521+0000","last_unstale":"2024-04-21T08:39:58.318678+0000","last_undegraded":"2024-04-21T08:39:58.318678+0000","last_fullsized":"2024-04-21T08:39:58.318678+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:24:15.799308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154378+0000","last_change":"2024-04-21T08:37:31.443896+0000","last_active":"2024-04-21T08:39:58.154378+0000","last_peered":"2024-04-21T08:39:58.154378+0000","last_clean":"2024-04-21T08:39:58.154378+0000","last_became_active":"2024-04-21T08:37:31.442684+0000","last_became_peered":"2024-04-21T08:37:31.442684+0000","last_unstale":"2024-04-21T08:39:58.154378+0000","last_undegraded":"2024-04-21T08:39:58.154378+0000","last_fullsized":"2024-04-21T08:39:58.154378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:14:18.649565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151277+0000","last_change":"2024-04-21T08:37:35.600468+0000","last_active":"2024-04-21T08:39:58.151277+0000","last_peered":"2024-04-21T08:39:58.151277+0000","last_clean":"2024-04-21T08:39:58.151277+0000","last_became_active":"2024-04-21T08:37:35.599940+0000","last_became_peered":"2024-04-21T08:37:35.599940+0000","last_unstale":"2024-04-21T08:39:58.151277+0000","last_undegraded":"2024-04-21T08:39:58.151277+0000","last_fullsized":"2024-04-21T08:39:58.151277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:48.125434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009395+0000","last_change":"2024-04-21T08:37:30.416480+0000","last_active":"2024-04-21T08:39:58.009395+0000","last_peered":"2024-04-21T08:39:58.009395+0000","last_clean":"2024-04-21T08:39:58.009395+0000","last_became_active":"2024-04-21T08:37:30.416072+0000","last_became_peered":"2024-04-21T08:37:30.416072+0000","last_unstale":"2024-04-21T08:39:58.009395+0000","last_undegraded":"2024-04-21T08:39:58.009395+0000","last_fullsized":"2024-04-21T08:39:58.009395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:28:22.400926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014201+0000","last_change":"2024-04-21T08:37:31.457216+0000","last_active":"2024-04-21T08:39:58.014201+0000","last_peered":"2024-04-21T08:39:58.014201+0000","last_clean":"2024-04-21T08:39:58.014201+0000","last_became_active":"2024-04-21T08:37:31.456856+0000","last_became_peered":"2024-04-21T08:37:31.456856+0000","last_unstale":"2024-04-21T08:39:58.014201+0000","last_undegraded":"2024-04-21T08:39:58.014201+0000","last_fullsized":"2024-04-21T08:39:58.014201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:10.272886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905222+0000","last_change":"2024-04-21T08:37:35.601715+0000","last_active":"2024-04-21T08:39:57.905222+0000","last_peered":"2024-04-21T08:39:57.905222+0000","last_clean":"2024-04-21T08:39:57.905222+0000","last_became_active":"2024-04-21T08:37:35.601269+0000","last_became_peered":"2024-04-21T08:37:35.601269+0000","last_unstale":"2024-04-21T08:39:57.905222+0000","last_undegraded":"2024-04-21T08:39:57.905222+0000","last_fullsized":"2024-04-21T08:39:57.905222+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:57.192918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":176,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.903763+0000","last_change":"2024-04-21T08:37:30.415510+0000","last_active":"2024-04-21T08:40:07.903763+0000","last_peered":"2024-04-21T08:40:07.903763+0000","last_clean":"2024-04-21T08:40:07.903763+0000","last_became_active":"2024-04-21T08:37:30.415199+0000","last_became_peered":"2024-04-21T08:37:30.415199+0000","last_unstale":"2024-04-21T08:40:07.903763+0000","last_undegraded":"2024-04-21T08:40:07.903763+0000","last_fullsized":"2024-04-21T08:40:07.903763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:32.336247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009124+0000","last_change":"2024-04-21T08:39:31.681460+0000","last_active":"2024-04-21T08:39:58.009124+0000","last_peered":"2024-04-21T08:39:58.009124+0000","last_clean":"2024-04-21T08:39:58.009124+0000","last_became_active":"2024-04-21T08:39:31.681057+0000","last_became_peered":"2024-04-21T08:39:31.681057+0000","last_unstale":"2024-04-21T08:39:58.009124+0000","last_undegraded":"2024-04-21T08:39:58.009124+0000","last_fullsized":"2024-04-21T08:39:58.009124+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:43.407369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905274+0000","last_change":"2024-04-21T08:38:30.714455+0000","last_active":"2024-04-21T08:39:57.905274+0000","last_peered":"2024-04-21T08:39:57.905274+0000","last_clean":"2024-04-21T08:39:57.905274+0000","last_became_active":"2024-04-21T08:38:30.713910+0000","last_became_peered":"2024-04-21T08:38:30.713910+0000","last_unstale":"2024-04-21T08:39:57.905274+0000","last_undegraded":"2024-04-21T08:39:57.905274+0000","last_fullsized":"2024-04-21T08:39:57.905274+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.448652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,10],"acting":[2,6,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017063+0000","last_change":"2024-04-21T08:37:30.419877+0000","last_active":"2024-04-21T08:39:58.017063+0000","last_peered":"2024-04-21T08:39:58.017063+0000","last_clean":"2024-04-21T08:39:58.017063+0000","last_became_active":"2024-04-21T08:37:30.419572+0000","last_became_peered":"2024-04-21T08:37:30.419572+0000","last_unstale":"2024-04-21T08:39:58.017063+0000","last_undegraded":"2024-04-21T08:39:58.017063+0000","last_fullsized":"2024-04-21T08:39:58.017063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:32:35.524089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014094+0000","last_change":"2024-04-21T08:37:31.458590+0000","last_active":"2024-04-21T08:39:58.014094+0000","last_peered":"2024-04-21T08:39:58.014094+0000","last_clean":"2024-04-21T08:39:58.014094+0000","last_became_active":"2024-04-21T08:37:31.458196+0000","last_became_peered":"2024-04-21T08:37:31.458196+0000","last_unstale":"2024-04-21T08:39:58.014094+0000","last_undegraded":"2024-04-21T08:39:58.014094+0000","last_fullsized":"2024-04-21T08:39:58.014094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:00.890859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318442+0000","last_change":"2024-04-21T08:38:30.803663+0000","last_active":"2024-04-21T08:39:58.318442+0000","last_peered":"2024-04-21T08:39:58.318442+0000","last_clean":"2024-04-21T08:39:58.318442+0000","last_became_active":"2024-04-21T08:38:30.803203+0000","last_became_peered":"2024-04-21T08:38:30.803203+0000","last_unstale":"2024-04-21T08:39:58.318442+0000","last_undegraded":"2024-04-21T08:39:58.318442+0000","last_fullsized":"2024-04-21T08:39:58.318442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:06.085380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318695+0000","last_change":"2024-04-21T08:37:30.415786+0000","last_active":"2024-04-21T08:39:58.318695+0000","last_peered":"2024-04-21T08:39:58.318695+0000","last_clean":"2024-04-21T08:39:58.318695+0000","last_became_active":"2024-04-21T08:37:30.415472+0000","last_became_peered":"2024-04-21T08:37:30.415472+0000","last_unstale":"2024-04-21T08:39:58.318695+0000","last_undegraded":"2024-04-21T08:39:58.318695+0000","last_fullsized":"2024-04-21T08:39:58.318695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.417132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903648+0000","last_change":"2024-04-21T08:37:31.455720+0000","last_active":"2024-04-21T08:39:57.903648+0000","last_peered":"2024-04-21T08:39:57.903648+0000","last_clean":"2024-04-21T08:39:57.903648+0000","last_became_active":"2024-04-21T08:37:31.455334+0000","last_became_peered":"2024-04-21T08:37:31.455334+0000","last_unstale":"2024-04-21T08:39:57.903648+0000","last_undegraded":"2024-04-21T08:39:57.903648+0000","last_fullsized":"2024-04-21T08:39:57.903648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:06:55.220656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319675+0000","last_change":"2024-04-21T08:37:35.603305+0000","last_active":"2024-04-21T08:39:58.319675+0000","last_peered":"2024-04-21T08:39:58.319675+0000","last_clean":"2024-04-21T08:39:58.319675+0000","last_became_active":"2024-04-21T08:37:35.602854+0000","last_became_peered":"2024-04-21T08:37:35.602854+0000","last_unstale":"2024-04-21T08:39:58.319675+0000","last_undegraded":"2024-04-21T08:39:58.319675+0000","last_fullsized":"2024-04-21T08:39:58.319675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:10.993864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013890+0000","last_change":"2024-04-21T08:37:30.420797+0000","last_active":"2024-04-21T08:39:58.013890+0000","last_peered":"2024-04-21T08:39:58.013890+0000","last_clean":"2024-04-21T08:39:58.013890+0000","last_became_active":"2024-04-21T08:37:30.420485+0000","last_became_peered":"2024-04-21T08:37:30.420485+0000","last_unstale":"2024-04-21T08:39:58.013890+0000","last_undegraded":"2024-04-21T08:39:58.013890+0000","last_fullsized":"2024-04-21T08:39:58.013890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:03.275677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006844+0000","last_change":"2024-04-21T08:37:31.448406+0000","last_active":"2024-04-21T08:39:58.006844+0000","last_peered":"2024-04-21T08:39:58.006844+0000","last_clean":"2024-04-21T08:39:58.006844+0000","last_became_active":"2024-04-21T08:37:31.448085+0000","last_became_peered":"2024-04-21T08:37:31.448085+0000","last_unstale":"2024-04-21T08:39:58.006844+0000","last_undegraded":"2024-04-21T08:39:58.006844+0000","last_fullsized":"2024-04-21T08:39:58.006844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:49.574064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903948+0000","last_change":"2024-04-21T08:37:35.601338+0000","last_active":"2024-04-21T08:39:57.903948+0000","last_peered":"2024-04-21T08:39:57.903948+0000","last_clean":"2024-04-21T08:39:57.903948+0000","last_became_active":"2024-04-21T08:37:35.600890+0000","last_became_peered":"2024-04-21T08:37:35.600890+0000","last_unstale":"2024-04-21T08:39:57.903948+0000","last_undegraded":"2024-04-21T08:39:57.903948+0000","last_fullsized":"2024-04-21T08:39:57.903948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:30:21.755850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'4","reported_seq":171,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.903166+0000","last_change":"2024-04-21T08:37:30.423320+0000","last_active":"2024-04-21T08:40:07.903166+0000","last_peered":"2024-04-21T08:40:07.903166+0000","last_clean":"2024-04-21T08:40:07.903166+0000","last_became_active":"2024-04-21T08:37:30.423020+0000","last_became_peered":"2024-04-21T08:37:30.423020+0000","last_unstale":"2024-04-21T08:40:07.903166+0000","last_undegraded":"2024-04-21T08:40:07.903166+0000","last_fullsized":"2024-04-21T08:40:07.903166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:08:27.313196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":133,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154669+0000","last_change":"2024-04-21T08:37:31.442425+0000","last_active":"2024-04-21T08:39:58.154669+0000","last_peered":"2024-04-21T08:39:58.154669+0000","last_clean":"2024-04-21T08:39:58.154669+0000","last_became_active":"2024-04-21T08:37:31.442028+0000","last_became_peered":"2024-04-21T08:37:31.442028+0000","last_unstale":"2024-04-21T08:39:58.154669+0000","last_undegraded":"2024-04-21T08:39:58.154669+0000","last_fullsized":"2024-04-21T08:39:58.154669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:45.498656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006697+0000","last_change":"2024-04-21T08:37:35.607829+0000","last_active":"2024-04-21T08:39:58.006697+0000","last_peered":"2024-04-21T08:39:58.006697+0000","last_clean":"2024-04-21T08:39:58.006697+0000","last_became_active":"2024-04-21T08:37:35.607150+0000","last_became_peered":"2024-04-21T08:37:35.607150+0000","last_unstale":"2024-04-21T08:39:58.006697+0000","last_undegraded":"2024-04-21T08:39:58.006697+0000","last_fullsized":"2024-04-21T08:39:58.006697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:01:34.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905384+0000","last_change":"2024-04-21T08:37:30.418613+0000","last_active":"2024-04-21T08:39:57.905384+0000","last_peered":"2024-04-21T08:39:57.905384+0000","last_clean":"2024-04-21T08:39:57.905384+0000","last_became_active":"2024-04-21T08:37:30.418300+0000","last_became_peered":"2024-04-21T08:37:30.418300+0000","last_unstale":"2024-04-21T08:39:57.905384+0000","last_undegraded":"2024-04-21T08:39:57.905384+0000","last_fullsized":"2024-04-21T08:39:57.905384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:17:35.370753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319739+0000","last_change":"2024-04-21T08:39:31.682628+0000","last_active":"2024-04-21T08:39:58.319739+0000","last_peered":"2024-04-21T08:39:58.319739+0000","last_clean":"2024-04-21T08:39:58.319739+0000","last_became_active":"2024-04-21T08:39:31.682383+0000","last_became_peered":"2024-04-21T08:39:31.682383+0000","last_unstale":"2024-04-21T08:39:58.319739+0000","last_undegraded":"2024-04-21T08:39:58.319739+0000","last_fullsized":"2024-04-21T08:39:58.319739+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:38.831288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151734+0000","last_change":"2024-04-21T08:37:35.608280+0000","last_active":"2024-04-21T08:39:58.151734+0000","last_peered":"2024-04-21T08:39:58.151734+0000","last_clean":"2024-04-21T08:39:58.151734+0000","last_became_active":"2024-04-21T08:37:35.607898+0000","last_became_peered":"2024-04-21T08:37:35.607898+0000","last_unstale":"2024-04-21T08:39:58.151734+0000","last_undegraded":"2024-04-21T08:39:58.151734+0000","last_fullsized":"2024-04-21T08:39:58.151734+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:55.689519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014140+0000","last_change":"2024-04-21T08:37:30.417352+0000","last_active":"2024-04-21T08:39:58.014140+0000","last_peered":"2024-04-21T08:39:58.014140+0000","last_clean":"2024-04-21T08:39:58.014140+0000","last_became_active":"2024-04-21T08:37:30.417000+0000","last_became_peered":"2024-04-21T08:37:30.417000+0000","last_unstale":"2024-04-21T08:39:58.014140+0000","last_undegraded":"2024-04-21T08:39:58.014140+0000","last_fullsized":"2024-04-21T08:39:58.014140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:56.087334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006804+0000","last_change":"2024-04-21T08:39:31.679434+0000","last_active":"2024-04-21T08:39:58.006804+0000","last_peered":"2024-04-21T08:39:58.006804+0000","last_clean":"2024-04-21T08:39:58.006804+0000","last_became_active":"2024-04-21T08:39:31.679041+0000","last_became_peered":"2024-04-21T08:39:31.679041+0000","last_unstale":"2024-04-21T08:39:58.006804+0000","last_undegraded":"2024-04-21T08:39:58.006804+0000","last_fullsized":"2024-04-21T08:39:58.006804+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:27:11.011449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006731+0000","last_change":"2024-04-21T08:37:35.601077+0000","last_active":"2024-04-21T08:39:58.006731+0000","last_peered":"2024-04-21T08:39:58.006731+0000","last_clean":"2024-04-21T08:39:58.006731+0000","last_became_active":"2024-04-21T08:37:35.600673+0000","last_became_peered":"2024-04-21T08:37:35.600673+0000","last_unstale":"2024-04-21T08:39:58.006731+0000","last_undegraded":"2024-04-21T08:39:58.006731+0000","last_fullsized":"2024-04-21T08:39:58.006731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:04.554208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'43","reported_seq":358,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:08.904470+0000","last_change":"2024-04-21T08:37:30.421440+0000","last_active":"2024-04-21T08:40:08.904470+0000","last_peered":"2024-04-21T08:40:08.904470+0000","last_clean":"2024-04-21T08:40:08.904470+0000","last_became_active":"2024-04-21T08:37:30.420988+0000","last_became_peered":"2024-04-21T08:37:30.420988+0000","last_unstale":"2024-04-21T08:40:08.904470+0000","last_undegraded":"2024-04-21T08:40:08.904470+0000","last_fullsized":"2024-04-21T08:40:08.904470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:38:03.850705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72979,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":281,"num_read_kb":213,"num_write":44,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009331+0000","last_change":"2024-04-21T08:37:31.447208+0000","last_active":"2024-04-21T08:39:58.009331+0000","last_peered":"2024-04-21T08:39:58.009331+0000","last_clean":"2024-04-21T08:39:58.009331+0000","last_became_active":"2024-04-21T08:37:31.444464+0000","last_became_peered":"2024-04-21T08:37:31.444464+0000","last_unstale":"2024-04-21T08:39:58.009331+0000","last_undegraded":"2024-04-21T08:39:58.009331+0000","last_fullsized":"2024-04-21T08:39:58.009331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:33.805676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014409+0000","last_change":"2024-04-21T08:37:35.603937+0000","last_active":"2024-04-21T08:39:58.014409+0000","last_peered":"2024-04-21T08:39:58.014409+0000","last_clean":"2024-04-21T08:39:58.014409+0000","last_became_active":"2024-04-21T08:37:35.603477+0000","last_became_peered":"2024-04-21T08:37:35.603477+0000","last_unstale":"2024-04-21T08:39:58.014409+0000","last_undegraded":"2024-04-21T08:39:58.014409+0000","last_fullsized":"2024-04-21T08:39:58.014409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:40:33.901570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905391+0000","last_change":"2024-04-21T08:37:30.416941+0000","last_active":"2024-04-21T08:39:57.905391+0000","last_peered":"2024-04-21T08:39:57.905391+0000","last_clean":"2024-04-21T08:39:57.905391+0000","last_became_active":"2024-04-21T08:37:30.416625+0000","last_became_peered":"2024-04-21T08:37:30.416625+0000","last_unstale":"2024-04-21T08:39:57.905391+0000","last_undegraded":"2024-04-21T08:39:57.905391+0000","last_fullsized":"2024-04-21T08:39:57.905391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:07.782779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318733+0000","last_change":"2024-04-21T08:37:31.441707+0000","last_active":"2024-04-21T08:39:58.318733+0000","last_peered":"2024-04-21T08:39:58.318733+0000","last_clean":"2024-04-21T08:39:58.318733+0000","last_became_active":"2024-04-21T08:37:31.441366+0000","last_became_peered":"2024-04-21T08:37:31.441366+0000","last_unstale":"2024-04-21T08:39:58.318733+0000","last_undegraded":"2024-04-21T08:39:58.318733+0000","last_fullsized":"2024-04-21T08:39:58.318733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:16.573804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318906+0000","last_change":"2024-04-21T08:37:35.596214+0000","last_active":"2024-04-21T08:39:58.318906+0000","last_peered":"2024-04-21T08:39:58.318906+0000","last_clean":"2024-04-21T08:39:58.318906+0000","last_became_active":"2024-04-21T08:37:35.595773+0000","last_became_peered":"2024-04-21T08:37:35.595773+0000","last_unstale":"2024-04-21T08:39:58.318906+0000","last_undegraded":"2024-04-21T08:39:58.318906+0000","last_fullsized":"2024-04-21T08:39:58.318906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:07.538190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318273+0000","last_change":"2024-04-21T08:38:30.684828+0000","last_active":"2024-04-21T08:39:58.318273+0000","last_peered":"2024-04-21T08:39:58.318273+0000","last_clean":"2024-04-21T08:39:58.318273+0000","last_became_active":"2024-04-21T08:38:30.673923+0000","last_became_peered":"2024-04-21T08:38:30.673923+0000","last_unstale":"2024-04-21T08:39:58.318273+0000","last_undegraded":"2024-04-21T08:39:58.318273+0000","last_fullsized":"2024-04-21T08:39:58.318273+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:51:37.906150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905472+0000","last_change":"2024-04-21T08:37:31.453680+0000","last_active":"2024-04-21T08:39:57.905472+0000","last_peered":"2024-04-21T08:39:57.905472+0000","last_clean":"2024-04-21T08:39:57.905472+0000","last_became_active":"2024-04-21T08:37:31.453319+0000","last_became_peered":"2024-04-21T08:37:31.453319+0000","last_unstale":"2024-04-21T08:39:57.905472+0000","last_undegraded":"2024-04-21T08:39:57.905472+0000","last_fullsized":"2024-04-21T08:39:57.905472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:43.139789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007023+0000","last_change":"2024-04-21T08:37:35.608906+0000","last_active":"2024-04-21T08:39:58.007023+0000","last_peered":"2024-04-21T08:39:58.007023+0000","last_clean":"2024-04-21T08:39:58.007023+0000","last_became_active":"2024-04-21T08:37:35.608535+0000","last_became_peered":"2024-04-21T08:37:35.608535+0000","last_unstale":"2024-04-21T08:39:58.007023+0000","last_undegraded":"2024-04-21T08:39:58.007023+0000","last_fullsized":"2024-04-21T08:39:58.007023+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:58:01.422308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'9","reported_seq":183,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.902877+0000","last_change":"2024-04-21T08:37:30.414865+0000","last_active":"2024-04-21T08:40:07.902877+0000","last_peered":"2024-04-21T08:40:07.902877+0000","last_clean":"2024-04-21T08:40:07.902877+0000","last_became_active":"2024-04-21T08:37:30.414471+0000","last_became_peered":"2024-04-21T08:37:30.414471+0000","last_unstale":"2024-04-21T08:40:07.902877+0000","last_undegraded":"2024-04-21T08:40:07.902877+0000","last_fullsized":"2024-04-21T08:40:07.902877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:46.943574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2601,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":8,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905326+0000","last_change":"2024-04-21T08:37:31.456634+0000","last_active":"2024-04-21T08:39:57.905326+0000","last_peered":"2024-04-21T08:39:57.905326+0000","last_clean":"2024-04-21T08:39:57.905326+0000","last_became_active":"2024-04-21T08:37:31.456278+0000","last_became_peered":"2024-04-21T08:37:31.456278+0000","last_unstale":"2024-04-21T08:39:57.905326+0000","last_undegraded":"2024-04-21T08:39:57.905326+0000","last_fullsized":"2024-04-21T08:39:57.905326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:04.870024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319167+0000","last_change":"2024-04-21T08:37:35.601760+0000","last_active":"2024-04-21T08:39:58.319167+0000","last_peered":"2024-04-21T08:39:58.319167+0000","last_clean":"2024-04-21T08:39:58.319167+0000","last_became_active":"2024-04-21T08:37:35.601332+0000","last_became_peered":"2024-04-21T08:37:35.601332+0000","last_unstale":"2024-04-21T08:39:58.319167+0000","last_undegraded":"2024-04-21T08:39:58.319167+0000","last_fullsized":"2024-04-21T08:39:58.319167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:04:03.694581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008961+0000","last_change":"2024-04-21T08:37:30.422419+0000","last_active":"2024-04-21T08:39:58.008961+0000","last_peered":"2024-04-21T08:39:58.008961+0000","last_clean":"2024-04-21T08:39:58.008961+0000","last_became_active":"2024-04-21T08:37:30.422123+0000","last_became_peered":"2024-04-21T08:37:30.422123+0000","last_unstale":"2024-04-21T08:39:58.008961+0000","last_undegraded":"2024-04-21T08:39:58.008961+0000","last_fullsized":"2024-04-21T08:39:58.008961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:32:54.029041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013848+0000","last_change":"2024-04-21T08:37:31.458992+0000","last_active":"2024-04-21T08:39:58.013848+0000","last_peered":"2024-04-21T08:39:58.013848+0000","last_clean":"2024-04-21T08:39:58.013848+0000","last_became_active":"2024-04-21T08:37:31.458635+0000","last_became_peered":"2024-04-21T08:37:31.458635+0000","last_unstale":"2024-04-21T08:39:58.013848+0000","last_undegraded":"2024-04-21T08:39:58.013848+0000","last_fullsized":"2024-04-21T08:39:58.013848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:48:14.325571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013750+0000","last_change":"2024-04-21T08:37:35.603739+0000","last_active":"2024-04-21T08:39:58.013750+0000","last_peered":"2024-04-21T08:39:58.013750+0000","last_clean":"2024-04-21T08:39:58.013750+0000","last_became_active":"2024-04-21T08:37:35.603310+0000","last_became_peered":"2024-04-21T08:37:35.603310+0000","last_unstale":"2024-04-21T08:39:58.013750+0000","last_undegraded":"2024-04-21T08:39:58.013750+0000","last_fullsized":"2024-04-21T08:39:58.013750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:19.162520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009465+0000","last_change":"2024-04-21T08:37:30.418104+0000","last_active":"2024-04-21T08:39:58.009465+0000","last_peered":"2024-04-21T08:39:58.009465+0000","last_clean":"2024-04-21T08:39:58.009465+0000","last_became_active":"2024-04-21T08:37:30.417814+0000","last_became_peered":"2024-04-21T08:37:30.417814+0000","last_unstale":"2024-04-21T08:39:58.009465+0000","last_undegraded":"2024-04-21T08:39:58.009465+0000","last_fullsized":"2024-04-21T08:39:58.009465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:50:56.696513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318582+0000","last_change":"2024-04-21T08:37:31.451952+0000","last_active":"2024-04-21T08:39:58.318582+0000","last_peered":"2024-04-21T08:39:58.318582+0000","last_clean":"2024-04-21T08:39:58.318582+0000","last_became_active":"2024-04-21T08:37:31.451588+0000","last_became_peered":"2024-04-21T08:37:31.451588+0000","last_unstale":"2024-04-21T08:39:58.318582+0000","last_undegraded":"2024-04-21T08:39:58.318582+0000","last_fullsized":"2024-04-21T08:39:58.318582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:06.506769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009071+0000","last_change":"2024-04-21T08:37:35.760661+0000","last_active":"2024-04-21T08:39:58.009071+0000","last_peered":"2024-04-21T08:39:58.009071+0000","last_clean":"2024-04-21T08:39:58.009071+0000","last_became_active":"2024-04-21T08:37:35.759925+0000","last_became_peered":"2024-04-21T08:37:35.759925+0000","last_unstale":"2024-04-21T08:39:58.009071+0000","last_undegraded":"2024-04-21T08:39:58.009071+0000","last_fullsized":"2024-04-21T08:39:58.009071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:11:53.062251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007012+0000","last_change":"2024-04-21T08:37:30.416487+0000","last_active":"2024-04-21T08:39:58.007012+0000","last_peered":"2024-04-21T08:39:58.007012+0000","last_clean":"2024-04-21T08:39:58.007012+0000","last_became_active":"2024-04-21T08:37:30.416076+0000","last_became_peered":"2024-04-21T08:37:30.416076+0000","last_unstale":"2024-04-21T08:39:58.007012+0000","last_undegraded":"2024-04-21T08:39:58.007012+0000","last_fullsized":"2024-04-21T08:39:58.007012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:18:33.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017205+0000","last_change":"2024-04-21T08:37:31.448824+0000","last_active":"2024-04-21T08:39:58.017205+0000","last_peered":"2024-04-21T08:39:58.017205+0000","last_clean":"2024-04-21T08:39:58.017205+0000","last_became_active":"2024-04-21T08:37:31.448432+0000","last_became_peered":"2024-04-21T08:37:31.448432+0000","last_unstale":"2024-04-21T08:39:58.017205+0000","last_undegraded":"2024-04-21T08:39:58.017205+0000","last_fullsized":"2024-04-21T08:39:58.017205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:41.111272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318815+0000","last_change":"2024-04-21T08:37:35.604539+0000","last_active":"2024-04-21T08:39:58.318815+0000","last_peered":"2024-04-21T08:39:58.318815+0000","last_clean":"2024-04-21T08:39:58.318815+0000","last_became_active":"2024-04-21T08:37:35.604132+0000","last_became_peered":"2024-04-21T08:37:35.604132+0000","last_unstale":"2024-04-21T08:39:58.318815+0000","last_undegraded":"2024-04-21T08:39:58.318815+0000","last_fullsized":"2024-04-21T08:39:58.318815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:12.180730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905085+0000","last_change":"2024-04-21T08:37:30.418854+0000","last_active":"2024-04-21T08:39:57.905085+0000","last_peered":"2024-04-21T08:39:57.905085+0000","last_clean":"2024-04-21T08:39:57.905085+0000","last_became_active":"2024-04-21T08:37:30.418511+0000","last_became_peered":"2024-04-21T08:37:30.418511+0000","last_unstale":"2024-04-21T08:39:57.905085+0000","last_undegraded":"2024-04-21T08:39:57.905085+0000","last_fullsized":"2024-04-21T08:39:57.905085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:26.751924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.016918+0000","last_change":"2024-04-21T08:37:31.453365+0000","last_active":"2024-04-21T08:39:58.016918+0000","last_peered":"2024-04-21T08:39:58.016918+0000","last_clean":"2024-04-21T08:39:58.016918+0000","last_became_active":"2024-04-21T08:37:31.453042+0000","last_became_peered":"2024-04-21T08:37:31.453042+0000","last_unstale":"2024-04-21T08:39:58.016918+0000","last_undegraded":"2024-04-21T08:39:58.016918+0000","last_fullsized":"2024-04-21T08:39:58.016918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:49:46.648754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318806+0000","last_change":"2024-04-21T08:37:35.760626+0000","last_active":"2024-04-21T08:39:58.318806+0000","last_peered":"2024-04-21T08:39:58.318806+0000","last_clean":"2024-04-21T08:39:58.318806+0000","last_became_active":"2024-04-21T08:37:35.760125+0000","last_became_peered":"2024-04-21T08:37:35.760125+0000","last_unstale":"2024-04-21T08:39:58.318806+0000","last_undegraded":"2024-04-21T08:39:58.318806+0000","last_fullsized":"2024-04-21T08:39:58.318806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:58.366557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013932+0000","last_change":"2024-04-21T08:37:31.457654+0000","last_active":"2024-04-21T08:39:58.013932+0000","last_peered":"2024-04-21T08:39:58.013932+0000","last_clean":"2024-04-21T08:39:58.013932+0000","last_became_active":"2024-04-21T08:37:31.456950+0000","last_became_peered":"2024-04-21T08:37:31.456950+0000","last_unstale":"2024-04-21T08:39:58.013932+0000","last_undegraded":"2024-04-21T08:39:58.013932+0000","last_fullsized":"2024-04-21T08:39:58.013932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:02.863000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":173,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.902250+0000","last_change":"2024-04-21T08:37:30.422021+0000","last_active":"2024-04-21T08:40:07.902250+0000","last_peered":"2024-04-21T08:40:07.902250+0000","last_clean":"2024-04-21T08:40:07.902250+0000","last_became_active":"2024-04-21T08:37:30.421711+0000","last_became_peered":"2024-04-21T08:37:30.421711+0000","last_unstale":"2024-04-21T08:40:07.902250+0000","last_undegraded":"2024-04-21T08:40:07.902250+0000","last_fullsized":"2024-04-21T08:40:07.902250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:01.612205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":133,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905863+0000","last_change":"2024-04-21T08:37:35.598041+0000","last_active":"2024-04-21T08:39:57.905863+0000","last_peered":"2024-04-21T08:39:57.905863+0000","last_clean":"2024-04-21T08:39:57.905863+0000","last_became_active":"2024-04-21T08:37:35.597635+0000","last_became_peered":"2024-04-21T08:37:35.597635+0000","last_unstale":"2024-04-21T08:39:57.905863+0000","last_undegraded":"2024-04-21T08:39:57.905863+0000","last_fullsized":"2024-04-21T08:39:57.905863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:35:27.452340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903604+0000","last_change":"2024-04-21T08:37:31.450423+0000","last_active":"2024-04-21T08:39:57.903604+0000","last_peered":"2024-04-21T08:39:57.903604+0000","last_clean":"2024-04-21T08:39:57.903604+0000","last_became_active":"2024-04-21T08:37:31.450112+0000","last_became_peered":"2024-04-21T08:37:31.450112+0000","last_unstale":"2024-04-21T08:39:57.903604+0000","last_undegraded":"2024-04-21T08:39:57.903604+0000","last_fullsized":"2024-04-21T08:39:57.903604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:10:47.006769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151568+0000","last_change":"2024-04-21T08:37:30.416519+0000","last_active":"2024-04-21T08:39:58.151568+0000","last_peered":"2024-04-21T08:39:58.151568+0000","last_clean":"2024-04-21T08:39:58.151568+0000","last_became_active":"2024-04-21T08:37:30.416131+0000","last_became_peered":"2024-04-21T08:37:30.416131+0000","last_unstale":"2024-04-21T08:39:58.151568+0000","last_undegraded":"2024-04-21T08:39:58.151568+0000","last_fullsized":"2024-04-21T08:39:58.151568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:12.280847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":412,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:00.021477+0000","last_change":"2024-04-21T08:36:20.257759+0000","last_active":"2024-04-21T08:40:00.021477+0000","last_peered":"2024-04-21T08:40:00.021477+0000","last_clean":"2024-04-21T08:40:00.021477+0000","last_became_active":"2024-04-21T08:36:20.257313+0000","last_became_peered":"2024-04-21T08:36:20.257313+0000","last_unstale":"2024-04-21T08:40:00.021477+0000","last_undegraded":"2024-04-21T08:40:00.021477+0000","last_fullsized":"2024-04-21T08:40:00.021477+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_clean_scrub_stamp":"2024-04-21T08:32:56.546295+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:42:44.393447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014207+0000","last_change":"2024-04-21T08:37:35.761562+0000","last_active":"2024-04-21T08:39:58.014207+0000","last_peered":"2024-04-21T08:39:58.014207+0000","last_clean":"2024-04-21T08:39:58.014207+0000","last_became_active":"2024-04-21T08:37:35.760703+0000","last_became_peered":"2024-04-21T08:37:35.760703+0000","last_unstale":"2024-04-21T08:39:58.014207+0000","last_undegraded":"2024-04-21T08:39:58.014207+0000","last_fullsized":"2024-04-21T08:39:58.014207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:12:47.511117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318506+0000","last_change":"2024-04-21T08:39:31.678868+0000","last_active":"2024-04-21T08:39:58.318506+0000","last_peered":"2024-04-21T08:39:58.318506+0000","last_clean":"2024-04-21T08:39:58.318506+0000","last_became_active":"2024-04-21T08:39:31.678457+0000","last_became_peered":"2024-04-21T08:39:31.678457+0000","last_unstale":"2024-04-21T08:39:58.318506+0000","last_undegraded":"2024-04-21T08:39:58.318506+0000","last_fullsized":"2024-04-21T08:39:58.318506+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:49.918106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905042+0000","last_change":"2024-04-21T08:37:30.420560+0000","last_active":"2024-04-21T08:39:57.905042+0000","last_peered":"2024-04-21T08:39:57.905042+0000","last_clean":"2024-04-21T08:39:57.905042+0000","last_became_active":"2024-04-21T08:37:30.420223+0000","last_became_peered":"2024-04-21T08:37:30.420223+0000","last_unstale":"2024-04-21T08:39:57.905042+0000","last_undegraded":"2024-04-21T08:39:57.905042+0000","last_fullsized":"2024-04-21T08:39:57.905042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:43:55.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318381+0000","last_change":"2024-04-21T08:37:35.598294+0000","last_active":"2024-04-21T08:39:58.318381+0000","last_peered":"2024-04-21T08:39:58.318381+0000","last_clean":"2024-04-21T08:39:58.318381+0000","last_became_active":"2024-04-21T08:37:35.597883+0000","last_became_peered":"2024-04-21T08:37:35.597883+0000","last_unstale":"2024-04-21T08:39:58.318381+0000","last_undegraded":"2024-04-21T08:39:58.318381+0000","last_fullsized":"2024-04-21T08:39:58.318381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:24:05.144763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009321+0000","last_change":"2024-04-21T08:37:30.423182+0000","last_active":"2024-04-21T08:39:58.009321+0000","last_peered":"2024-04-21T08:39:58.009321+0000","last_clean":"2024-04-21T08:39:58.009321+0000","last_became_active":"2024-04-21T08:37:30.422792+0000","last_became_peered":"2024-04-21T08:37:30.422792+0000","last_unstale":"2024-04-21T08:39:58.009321+0000","last_undegraded":"2024-04-21T08:39:58.009321+0000","last_fullsized":"2024-04-21T08:39:58.009321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:44:00.727269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318924+0000","last_change":"2024-04-21T08:37:31.444013+0000","last_active":"2024-04-21T08:39:58.318924+0000","last_peered":"2024-04-21T08:39:58.318924+0000","last_clean":"2024-04-21T08:39:58.318924+0000","last_became_active":"2024-04-21T08:37:31.443684+0000","last_became_peered":"2024-04-21T08:37:31.443684+0000","last_unstale":"2024-04-21T08:39:58.318924+0000","last_undegraded":"2024-04-21T08:39:58.318924+0000","last_fullsized":"2024-04-21T08:39:58.318924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:00:35.400818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903525+0000","last_change":"2024-04-21T08:37:35.759817+0000","last_active":"2024-04-21T08:39:57.903525+0000","last_peered":"2024-04-21T08:39:57.903525+0000","last_clean":"2024-04-21T08:39:57.903525+0000","last_became_active":"2024-04-21T08:37:35.759396+0000","last_became_peered":"2024-04-21T08:37:35.759396+0000","last_unstale":"2024-04-21T08:39:57.903525+0000","last_undegraded":"2024-04-21T08:39:57.903525+0000","last_fullsized":"2024-04-21T08:39:57.903525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:20:36.170880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'98","reported_seq":224,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.333603+0000","last_change":"2024-04-21T08:37:30.414102+0000","last_active":"2024-04-21T08:40:07.333603+0000","last_peered":"2024-04-21T08:40:07.333603+0000","last_clean":"2024-04-21T08:40:07.333603+0000","last_became_active":"2024-04-21T08:37:30.413670+0000","last_became_peered":"2024-04-21T08:37:30.413670+0000","last_unstale":"2024-04-21T08:40:07.333603+0000","last_undegraded":"2024-04-21T08:40:07.333603+0000","last_fullsized":"2024-04-21T08:40:07.333603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:12:49.696516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":179,"num_write_kb":1285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904320+0000","last_change":"2024-04-21T08:37:31.457451+0000","last_active":"2024-04-21T08:39:57.904320+0000","last_peered":"2024-04-21T08:39:57.904320+0000","last_clean":"2024-04-21T08:39:57.904320+0000","last_became_active":"2024-04-21T08:37:31.457120+0000","last_became_peered":"2024-04-21T08:37:31.457120+0000","last_unstale":"2024-04-21T08:39:57.904320+0000","last_undegraded":"2024-04-21T08:39:57.904320+0000","last_fullsized":"2024-04-21T08:39:57.904320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:07:48.025843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904850+0000","last_change":"2024-04-21T08:37:35.600815+0000","last_active":"2024-04-21T08:39:57.904850+0000","last_peered":"2024-04-21T08:39:57.904850+0000","last_clean":"2024-04-21T08:39:57.904850+0000","last_became_active":"2024-04-21T08:37:35.600380+0000","last_became_peered":"2024-04-21T08:37:35.600380+0000","last_unstale":"2024-04-21T08:39:57.904850+0000","last_undegraded":"2024-04-21T08:39:57.904850+0000","last_fullsized":"2024-04-21T08:39:57.904850+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:19:33.710431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151729+0000","last_change":"2024-04-21T08:37:30.425592+0000","last_active":"2024-04-21T08:39:58.151729+0000","last_peered":"2024-04-21T08:39:58.151729+0000","last_clean":"2024-04-21T08:39:58.151729+0000","last_became_active":"2024-04-21T08:37:30.425274+0000","last_became_peered":"2024-04-21T08:37:30.425274+0000","last_unstale":"2024-04-21T08:39:58.151729+0000","last_undegraded":"2024-04-21T08:39:58.151729+0000","last_fullsized":"2024-04-21T08:39:58.151729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:27:29.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319672+0000","last_change":"2024-04-21T08:39:31.680028+0000","last_active":"2024-04-21T08:39:58.319672+0000","last_peered":"2024-04-21T08:39:58.319672+0000","last_clean":"2024-04-21T08:39:58.319672+0000","last_became_active":"2024-04-21T08:39:31.679162+0000","last_became_peered":"2024-04-21T08:39:31.679162+0000","last_unstale":"2024-04-21T08:39:58.319672+0000","last_undegraded":"2024-04-21T08:39:58.319672+0000","last_fullsized":"2024-04-21T08:39:58.319672+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:32.824411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151564+0000","last_change":"2024-04-21T08:38:30.713521+0000","last_active":"2024-04-21T08:39:58.151564+0000","last_peered":"2024-04-21T08:39:58.151564+0000","last_clean":"2024-04-21T08:39:58.151564+0000","last_became_active":"2024-04-21T08:38:30.713096+0000","last_became_peered":"2024-04-21T08:38:30.713096+0000","last_unstale":"2024-04-21T08:39:58.151564+0000","last_undegraded":"2024-04-21T08:39:58.151564+0000","last_fullsized":"2024-04-21T08:39:58.151564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:54:59.078776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153818+0000","last_change":"2024-04-21T08:37:30.416696+0000","last_active":"2024-04-21T08:39:58.153818+0000","last_peered":"2024-04-21T08:39:58.153818+0000","last_clean":"2024-04-21T08:39:58.153818+0000","last_became_active":"2024-04-21T08:37:30.415868+0000","last_became_peered":"2024-04-21T08:37:30.415868+0000","last_unstale":"2024-04-21T08:39:58.153818+0000","last_undegraded":"2024-04-21T08:39:58.153818+0000","last_fullsized":"2024-04-21T08:39:58.153818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:48:08.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008945+0000","last_change":"2024-04-21T08:37:31.453861+0000","last_active":"2024-04-21T08:39:58.008945+0000","last_peered":"2024-04-21T08:39:58.008945+0000","last_clean":"2024-04-21T08:39:58.008945+0000","last_became_active":"2024-04-21T08:37:31.453562+0000","last_became_peered":"2024-04-21T08:37:31.453562+0000","last_unstale":"2024-04-21T08:39:58.008945+0000","last_undegraded":"2024-04-21T08:39:58.008945+0000","last_fullsized":"2024-04-21T08:39:58.008945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:17:50.513917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006977+0000","last_change":"2024-04-21T08:37:35.609217+0000","last_active":"2024-04-21T08:39:58.006977+0000","last_peered":"2024-04-21T08:39:58.006977+0000","last_clean":"2024-04-21T08:39:58.006977+0000","last_became_active":"2024-04-21T08:37:35.608786+0000","last_became_peered":"2024-04-21T08:37:35.608786+0000","last_unstale":"2024-04-21T08:39:58.006977+0000","last_undegraded":"2024-04-21T08:39:58.006977+0000","last_fullsized":"2024-04-21T08:39:58.006977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:45:50.945845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903818+0000","last_change":"2024-04-21T08:37:30.417156+0000","last_active":"2024-04-21T08:39:57.903818+0000","last_peered":"2024-04-21T08:39:57.903818+0000","last_clean":"2024-04-21T08:39:57.903818+0000","last_became_active":"2024-04-21T08:37:30.415717+0000","last_became_peered":"2024-04-21T08:37:30.415717+0000","last_unstale":"2024-04-21T08:39:57.903818+0000","last_undegraded":"2024-04-21T08:39:57.903818+0000","last_fullsized":"2024-04-21T08:39:57.903818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:37.819082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318260+0000","last_change":"2024-04-21T08:37:31.444948+0000","last_active":"2024-04-21T08:39:58.318260+0000","last_peered":"2024-04-21T08:39:58.318260+0000","last_clean":"2024-04-21T08:39:58.318260+0000","last_became_active":"2024-04-21T08:37:31.444604+0000","last_became_peered":"2024-04-21T08:37:31.444604+0000","last_unstale":"2024-04-21T08:39:58.318260+0000","last_undegraded":"2024-04-21T08:39:58.318260+0000","last_fullsized":"2024-04-21T08:39:58.318260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.820503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151386+0000","last_change":"2024-04-21T08:37:35.611541+0000","last_active":"2024-04-21T08:39:58.151386+0000","last_peered":"2024-04-21T08:39:58.151386+0000","last_clean":"2024-04-21T08:39:58.151386+0000","last_became_active":"2024-04-21T08:37:35.611136+0000","last_became_peered":"2024-04-21T08:37:35.611136+0000","last_unstale":"2024-04-21T08:39:58.151386+0000","last_undegraded":"2024-04-21T08:39:58.151386+0000","last_fullsized":"2024-04-21T08:39:58.151386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:28.254529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905028+0000","last_change":"2024-04-21T08:37:30.414216+0000","last_active":"2024-04-21T08:39:57.905028+0000","last_peered":"2024-04-21T08:39:57.905028+0000","last_clean":"2024-04-21T08:39:57.905028+0000","last_became_active":"2024-04-21T08:37:30.413890+0000","last_became_peered":"2024-04-21T08:37:30.413890+0000","last_unstale":"2024-04-21T08:39:57.905028+0000","last_undegraded":"2024-04-21T08:39:57.905028+0000","last_fullsized":"2024-04-21T08:39:57.905028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:16.972618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903800+0000","last_change":"2024-04-21T08:37:31.449307+0000","last_active":"2024-04-21T08:39:57.903800+0000","last_peered":"2024-04-21T08:39:57.903800+0000","last_clean":"2024-04-21T08:39:57.903800+0000","last_became_active":"2024-04-21T08:37:31.448980+0000","last_became_peered":"2024-04-21T08:37:31.448980+0000","last_unstale":"2024-04-21T08:39:57.903800+0000","last_undegraded":"2024-04-21T08:39:57.903800+0000","last_fullsized":"2024-04-21T08:39:57.903800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:30:31.939274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151476+0000","last_change":"2024-04-21T08:37:35.760616+0000","last_active":"2024-04-21T08:39:58.151476+0000","last_peered":"2024-04-21T08:39:58.151476+0000","last_clean":"2024-04-21T08:39:58.151476+0000","last_became_active":"2024-04-21T08:37:35.760131+0000","last_became_peered":"2024-04-21T08:37:35.760131+0000","last_unstale":"2024-04-21T08:39:58.151476+0000","last_undegraded":"2024-04-21T08:39:58.151476+0000","last_fullsized":"2024-04-21T08:39:58.151476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:51.510259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009114+0000","last_change":"2024-04-21T08:38:30.682791+0000","last_active":"2024-04-21T08:39:58.009114+0000","last_peered":"2024-04-21T08:39:58.009114+0000","last_clean":"2024-04-21T08:39:58.009114+0000","last_became_active":"2024-04-21T08:38:30.671977+0000","last_became_peered":"2024-04-21T08:38:30.671977+0000","last_unstale":"2024-04-21T08:39:58.009114+0000","last_undegraded":"2024-04-21T08:39:58.009114+0000","last_fullsized":"2024-04-21T08:39:58.009114+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:55:15.899399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905569+0000","last_change":"2024-04-21T08:37:31.447234+0000","last_active":"2024-04-21T08:39:57.905569+0000","last_peered":"2024-04-21T08:39:57.905569+0000","last_clean":"2024-04-21T08:39:57.905569+0000","last_became_active":"2024-04-21T08:37:31.446876+0000","last_became_peered":"2024-04-21T08:37:31.446876+0000","last_unstale":"2024-04-21T08:39:57.905569+0000","last_undegraded":"2024-04-21T08:39:57.905569+0000","last_fullsized":"2024-04-21T08:39:57.905569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:51.784126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153981+0000","last_change":"2024-04-21T08:37:35.598577+0000","last_active":"2024-04-21T08:39:58.153981+0000","last_peered":"2024-04-21T08:39:58.153981+0000","last_clean":"2024-04-21T08:39:58.153981+0000","last_became_active":"2024-04-21T08:37:35.598119+0000","last_became_peered":"2024-04-21T08:37:35.598119+0000","last_unstale":"2024-04-21T08:39:58.153981+0000","last_undegraded":"2024-04-21T08:39:58.153981+0000","last_fullsized":"2024-04-21T08:39:58.153981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:41.986674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905332+0000","last_change":"2024-04-21T08:37:30.425737+0000","last_active":"2024-04-21T08:39:57.905332+0000","last_peered":"2024-04-21T08:39:57.905332+0000","last_clean":"2024-04-21T08:39:57.905332+0000","last_became_active":"2024-04-21T08:37:30.425180+0000","last_became_peered":"2024-04-21T08:37:30.425180+0000","last_unstale":"2024-04-21T08:39:57.905332+0000","last_undegraded":"2024-04-21T08:39:57.905332+0000","last_fullsized":"2024-04-21T08:39:57.905332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:08.646106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.906011+0000","last_change":"2024-04-21T08:37:31.451505+0000","last_active":"2024-04-21T08:39:57.906011+0000","last_peered":"2024-04-21T08:39:57.906011+0000","last_clean":"2024-04-21T08:39:57.906011+0000","last_became_active":"2024-04-21T08:37:31.451146+0000","last_became_peered":"2024-04-21T08:37:31.451146+0000","last_unstale":"2024-04-21T08:39:57.906011+0000","last_undegraded":"2024-04-21T08:39:57.906011+0000","last_fullsized":"2024-04-21T08:39:57.906011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:27:01.793910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319583+0000","last_change":"2024-04-21T08:37:35.608995+0000","last_active":"2024-04-21T08:39:58.319583+0000","last_peered":"2024-04-21T08:39:58.319583+0000","last_clean":"2024-04-21T08:39:58.319583+0000","last_became_active":"2024-04-21T08:37:35.608587+0000","last_became_peered":"2024-04-21T08:37:35.608587+0000","last_unstale":"2024-04-21T08:39:58.319583+0000","last_undegraded":"2024-04-21T08:39:58.319583+0000","last_fullsized":"2024-04-21T08:39:58.319583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.829418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'65","reported_seq":130,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.001710+0000","last_change":"2024-04-21T08:37:30.419026+0000","last_active":"2024-04-21T08:40:07.001710+0000","last_peered":"2024-04-21T08:40:07.001710+0000","last_clean":"2024-04-21T08:40:07.001710+0000","last_became_active":"2024-04-21T08:37:30.418706+0000","last_became_peered":"2024-04-21T08:37:30.418706+0000","last_unstale":"2024-04-21T08:40:07.001710+0000","last_undegraded":"2024-04-21T08:40:07.001710+0000","last_fullsized":"2024-04-21T08:40:07.001710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:24.441918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905192+0000","last_change":"2024-04-21T08:37:31.446561+0000","last_active":"2024-04-21T08:39:57.905192+0000","last_peered":"2024-04-21T08:39:57.905192+0000","last_clean":"2024-04-21T08:39:57.905192+0000","last_became_active":"2024-04-21T08:37:31.446162+0000","last_became_peered":"2024-04-21T08:37:31.446162+0000","last_unstale":"2024-04-21T08:39:57.905192+0000","last_undegraded":"2024-04-21T08:39:57.905192+0000","last_fullsized":"2024-04-21T08:39:57.905192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:38:24.684537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013588+0000","last_change":"2024-04-21T08:37:35.598748+0000","last_active":"2024-04-21T08:39:58.013588+0000","last_peered":"2024-04-21T08:39:58.013588+0000","last_clean":"2024-04-21T08:39:58.013588+0000","last_became_active":"2024-04-21T08:37:35.598353+0000","last_became_peered":"2024-04-21T08:37:35.598353+0000","last_unstale":"2024-04-21T08:39:58.013588+0000","last_undegraded":"2024-04-21T08:39:58.013588+0000","last_fullsized":"2024-04-21T08:39:58.013588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:47:36.519694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319054+0000","last_change":"2024-04-21T08:38:30.691292+0000","last_active":"2024-04-21T08:39:58.319054+0000","last_peered":"2024-04-21T08:39:58.319054+0000","last_clean":"2024-04-21T08:39:58.319054+0000","last_became_active":"2024-04-21T08:38:30.676580+0000","last_became_peered":"2024-04-21T08:38:30.676580+0000","last_unstale":"2024-04-21T08:39:58.319054+0000","last_undegraded":"2024-04-21T08:39:58.319054+0000","last_fullsized":"2024-04-21T08:39:58.319054+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:43:53.150369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905478+0000","last_change":"2024-04-21T08:37:31.452367+0000","last_active":"2024-04-21T08:39:57.905478+0000","last_peered":"2024-04-21T08:39:57.905478+0000","last_clean":"2024-04-21T08:39:57.905478+0000","last_became_active":"2024-04-21T08:37:31.452031+0000","last_became_peered":"2024-04-21T08:37:31.452031+0000","last_unstale":"2024-04-21T08:39:57.905478+0000","last_undegraded":"2024-04-21T08:39:57.905478+0000","last_fullsized":"2024-04-21T08:39:57.905478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:58.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318378+0000","last_change":"2024-04-21T08:37:31.451222+0000","last_active":"2024-04-21T08:39:58.318378+0000","last_peered":"2024-04-21T08:39:58.318378+0000","last_clean":"2024-04-21T08:39:58.318378+0000","last_became_active":"2024-04-21T08:37:31.450869+0000","last_became_peered":"2024-04-21T08:37:31.450869+0000","last_unstale":"2024-04-21T08:39:58.318378+0000","last_undegraded":"2024-04-21T08:39:58.318378+0000","last_fullsized":"2024-04-21T08:39:58.318378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:17.051330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905137+0000","last_change":"2024-04-21T08:37:31.453415+0000","last_active":"2024-04-21T08:39:57.905137+0000","last_peered":"2024-04-21T08:39:57.905137+0000","last_clean":"2024-04-21T08:39:57.905137+0000","last_became_active":"2024-04-21T08:37:31.453089+0000","last_became_peered":"2024-04-21T08:37:31.453089+0000","last_unstale":"2024-04-21T08:39:57.905137+0000","last_undegraded":"2024-04-21T08:39:57.905137+0000","last_fullsized":"2024-04-21T08:39:57.905137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:25:44.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904904+0000","last_change":"2024-04-21T08:37:31.446707+0000","last_active":"2024-04-21T08:39:57.904904+0000","last_peered":"2024-04-21T08:39:57.904904+0000","last_clean":"2024-04-21T08:39:57.904904+0000","last_became_active":"2024-04-21T08:37:31.446340+0000","last_became_peered":"2024-04-21T08:37:31.446340+0000","last_unstale":"2024-04-21T08:39:57.904904+0000","last_undegraded":"2024-04-21T08:39:57.904904+0000","last_fullsized":"2024-04-21T08:39:57.904904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:13.649607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319500+0000","last_change":"2024-04-21T08:37:31.455532+0000","last_active":"2024-04-21T08:39:58.319500+0000","last_peered":"2024-04-21T08:39:58.319500+0000","last_clean":"2024-04-21T08:39:58.319500+0000","last_became_active":"2024-04-21T08:37:31.453547+0000","last_became_peered":"2024-04-21T08:37:31.453547+0000","last_unstale":"2024-04-21T08:39:58.319500+0000","last_undegraded":"2024-04-21T08:39:58.319500+0000","last_fullsized":"2024-04-21T08:39:58.319500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:39:47.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318944+0000","last_change":"2024-04-21T08:39:31.679882+0000","last_active":"2024-04-21T08:39:58.318944+0000","last_peered":"2024-04-21T08:39:58.318944+0000","last_clean":"2024-04-21T08:39:58.318944+0000","last_became_active":"2024-04-21T08:39:31.679440+0000","last_became_peered":"2024-04-21T08:39:31.679440+0000","last_unstale":"2024-04-21T08:39:58.318944+0000","last_undegraded":"2024-04-21T08:39:58.318944+0000","last_fullsized":"2024-04-21T08:39:58.318944+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:57.679038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318685+0000","last_change":"2024-04-21T08:37:31.451806+0000","last_active":"2024-04-21T08:39:58.318685+0000","last_peered":"2024-04-21T08:39:58.318685+0000","last_clean":"2024-04-21T08:39:58.318685+0000","last_became_active":"2024-04-21T08:37:31.451442+0000","last_became_peered":"2024-04-21T08:37:31.451442+0000","last_unstale":"2024-04-21T08:39:58.318685+0000","last_undegraded":"2024-04-21T08:39:58.318685+0000","last_fullsized":"2024-04-21T08:39:58.318685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:07.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905544+0000","last_change":"2024-04-21T08:37:31.444440+0000","last_active":"2024-04-21T08:39:57.905544+0000","last_peered":"2024-04-21T08:39:57.905544+0000","last_clean":"2024-04-21T08:39:57.905544+0000","last_became_active":"2024-04-21T08:37:31.444079+0000","last_became_peered":"2024-04-21T08:37:31.444079+0000","last_unstale":"2024-04-21T08:39:57.905544+0000","last_undegraded":"2024-04-21T08:39:57.905544+0000","last_fullsized":"2024-04-21T08:39:57.905544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:13.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904413+0000","last_change":"2024-04-21T08:37:31.453698+0000","last_active":"2024-04-21T08:39:57.904413+0000","last_peered":"2024-04-21T08:39:57.904413+0000","last_clean":"2024-04-21T08:39:57.904413+0000","last_became_active":"2024-04-21T08:37:31.453379+0000","last_became_peered":"2024-04-21T08:37:31.453379+0000","last_unstale":"2024-04-21T08:39:57.904413+0000","last_undegraded":"2024-04-21T08:39:57.904413+0000","last_fullsized":"2024-04-21T08:39:57.904413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:46:29.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013777+0000","last_change":"2024-04-21T08:37:31.444480+0000","last_active":"2024-04-21T08:39:58.013777+0000","last_peered":"2024-04-21T08:39:58.013777+0000","last_clean":"2024-04-21T08:39:58.013777+0000","last_became_active":"2024-04-21T08:37:31.444030+0000","last_became_peered":"2024-04-21T08:37:31.444030+0000","last_unstale":"2024-04-21T08:39:58.013777+0000","last_undegraded":"2024-04-21T08:39:58.013777+0000","last_fullsized":"2024-04-21T08:39:58.013777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:15:05.668885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903935+0000","last_change":"2024-04-21T08:37:31.456602+0000","last_active":"2024-04-21T08:39:57.903935+0000","last_peered":"2024-04-21T08:39:57.903935+0000","last_clean":"2024-04-21T08:39:57.903935+0000","last_became_active":"2024-04-21T08:37:31.456286+0000","last_became_peered":"2024-04-21T08:37:31.456286+0000","last_unstale":"2024-04-21T08:39:57.903935+0000","last_undegraded":"2024-04-21T08:39:57.903935+0000","last_fullsized":"2024-04-21T08:39:57.903935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:48:12.462037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154176+0000","last_change":"2024-04-21T08:37:31.447008+0000","last_active":"2024-04-21T08:39:58.154176+0000","last_peered":"2024-04-21T08:39:58.154176+0000","last_clean":"2024-04-21T08:39:58.154176+0000","last_became_active":"2024-04-21T08:37:31.445192+0000","last_became_peered":"2024-04-21T08:37:31.445192+0000","last_unstale":"2024-04-21T08:39:58.154176+0000","last_undegraded":"2024-04-21T08:39:58.154176+0000","last_fullsized":"2024-04-21T08:39:58.154176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:08.025647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318698+0000","last_change":"2024-04-21T08:37:31.443288+0000","last_active":"2024-04-21T08:39:58.318698+0000","last_peered":"2024-04-21T08:39:58.318698+0000","last_clean":"2024-04-21T08:39:58.318698+0000","last_became_active":"2024-04-21T08:37:31.442586+0000","last_became_peered":"2024-04-21T08:37:31.442586+0000","last_unstale":"2024-04-21T08:39:58.318698+0000","last_undegraded":"2024-04-21T08:39:58.318698+0000","last_fullsized":"2024-04-21T08:39:58.318698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:01.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318501+0000","last_change":"2024-04-21T08:37:31.453899+0000","last_active":"2024-04-21T08:39:58.318501+0000","last_peered":"2024-04-21T08:39:58.318501+0000","last_clean":"2024-04-21T08:39:58.318501+0000","last_became_active":"2024-04-21T08:37:31.453527+0000","last_became_peered":"2024-04-21T08:37:31.453527+0000","last_unstale":"2024-04-21T08:39:58.318501+0000","last_undegraded":"2024-04-21T08:39:58.318501+0000","last_fullsized":"2024-04-21T08:39:58.318501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:23:18.429618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108246,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":800441,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1183,"num_read_kb":813,"num_write":424,"num_write_kb":2366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2401323,"data_compressed":108456,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383724,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":529807,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32384,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712768,"statfs":{"total":95995035648,"available":95961874432,"internally_reserved":0,"allocated":1228800,"data_stored":1520299,"data_compressed":82643,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":927355,"data_compressed":48987,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":922597,"data_compressed":48987,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1127885,"data_compressed":65780,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1028096,"data_stored":525219,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":598690,"data_compressed":31359,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1052672,"data_stored":526607,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1126400,"data_stored":599849,"data_compressed":31359,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1246861,"data_compressed":64189,"data_compressed_allocated":86016,"data_compressed_original":782336,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":531882,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1150976,"data_stored":1449692,"data_compressed":80982,"data_compressed_allocated":106496,"data_compressed_original":978944,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":73561,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":74720,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330099,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393372,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":404802,"data_compressed":16863,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:40:10.801 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:40:10.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.933+0000 7fbb3603a640 1 -- 172.21.15.94:0/2827695013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb30101090 msgr2=0x7fbb30101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.933+0000 7fbb3603a640 1 --2- 172.21.15.94:0/2827695013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb30101090 0x7fbb30101470 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009900 tx=0x7fbb2002bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:10.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.933+0000 7fbb3603a640 1 -- 172.21.15.94:0/2827695013 shutdown_connections 2024-04-21T08:40:10.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.933+0000 7fbb3603a640 1 --2- 172.21.15.94:0/2827695013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb30106160 0x7fbb3010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.933+0000 7fbb3603a640 1 --2- 172.21.15.94:0/2827695013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb30101a40 0x7fbb30105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.933+0000 7fbb3603a640 1 --2- 172.21.15.94:0/2827695013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb30101090 0x7fbb30101470 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.933+0000 7fbb3603a640 1 -- 172.21.15.94:0/2827695013 >> 172.21.15.94:0/2827695013 conn(0x7fbb300fc910 msgr2=0x7fbb300fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.933+0000 7fbb3603a640 1 -- 172.21.15.94:0/2827695013 shutdown_connections 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.933+0000 7fbb3603a640 1 -- 172.21.15.94:0/2827695013 wait complete. 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.933+0000 7fbb3603a640 1 Processor -- start 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb3603a640 1 -- start start 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb3603a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb30101090 0x7fbb30199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb3603a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb30106160 0x7fbb30199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb3603a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb30199f80 0x7fbb301a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb3603a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301110a0 con 0x7fbb30199f80 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb3603a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30110f20 con 0x7fbb30106160 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb3603a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30111220 con 0x7fbb30101090 2024-04-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb30199f80 0x7fbb301a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb30199f80 0x7fbb301a4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36998/0 (socket says 172.21.15.94:36998) 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2ffff640 1 -- 172.21.15.94:0/3841159255 learned_addr learned my addr 172.21.15.94:0/3841159255 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2effd640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb30106160 0x7fbb30199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2f7fe640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb30101090 0x7fbb30199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2ffff640 1 -- 172.21.15.94:0/3841159255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb30101090 msgr2=0x7fbb30199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2ffff640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb30101090 0x7fbb30199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2ffff640 1 -- 172.21.15.94:0/3841159255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb30106160 msgr2=0x7fbb30199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2ffff640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb30106160 0x7fbb30199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2ffff640 1 -- 172.21.15.94:0/3841159255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb301a4d20 con 0x7fbb30199f80 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.934+0000 7fbb2ffff640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb30199f80 0x7fbb301a4580 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800cce0 tx=0x7fbb2800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.935+0000 7fbb2cff9640 1 -- 172.21.15.94:0/3841159255 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800f450 con 0x7fbb30199f80 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.935+0000 7fbb2cff9640 1 -- 172.21.15.94:0/3841159255 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb2800faa0 con 0x7fbb30199f80 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.935+0000 7fbb3603a640 1 -- 172.21.15.94:0/3841159255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb301a5010 con 0x7fbb30199f80 2024-04-21T08:40:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.935+0000 7fbb2cff9640 1 -- 172.21.15.94:0/3841159255 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28018b70 con 0x7fbb30199f80 2024-04-21T08:40:10.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.935+0000 7fbb3603a640 1 -- 172.21.15.94:0/3841159255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb301a54e0 con 0x7fbb30199f80 2024-04-21T08:40:10.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.936+0000 7fbb3603a640 1 -- 172.21.15.94:0/3841159255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb30069230 con 0x7fbb30199f80 2024-04-21T08:40:10.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.937+0000 7fbb2cff9640 1 -- 172.21.15.94:0/3841159255 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fbb28018d10 con 0x7fbb30199f80 2024-04-21T08:40:10.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.937+0000 7fbb2cff9640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbb180a3e70 0x7fbb180a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:10.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.937+0000 7fbb2f7fe640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbb180a3e70 0x7fbb180a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:10.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.937+0000 7fbb2cff9640 1 -- 172.21.15.94:0/3841159255 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fbb280d1280 con 0x7fbb30199f80 2024-04-21T08:40:10.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.938+0000 7fbb2f7fe640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbb180a3e70 0x7fbb180a6330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbb20036040 tx=0x7fbb20000bc0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:10.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:10.942+0000 7fbb2cff9640 1 -- 172.21.15.94:0/3841159255 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb28014030 con 0x7fbb30199f80 2024-04-21T08:40:11.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.118+0000 7fbb3603a640 1 -- 172.21.15.94:0/3841159255 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb30101a40 con 0x7fbb180a3e70 2024-04-21T08:40:11.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.127+0000 7fbb2cff9640 1 -- 172.21.15.94:0/3841159255 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339720 (secure 0 0 0) 0x7fbb30101a40 con 0x7fbb180a3e70 2024-04-21T08:40:11.130 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:11.132 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-21T08:40:11.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 -- 172.21.15.94:0/3841159255 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbb180a3e70 msgr2=0x7fbb180a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbb180a3e70 0x7fbb180a6330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbb20036040 tx=0x7fbb20000bc0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 -- 172.21.15.94:0/3841159255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb30199f80 msgr2=0x7fbb301a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb30199f80 0x7fbb301a4580 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800cce0 tx=0x7fbb2800a620 comp rx=0 tx=0).stop 2024-04-21T08:40:11.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 -- 172.21.15.94:0/3841159255 shutdown_connections 2024-04-21T08:40:11.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbb180a3e70 0x7fbb180a6330 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb30199f80 0x7fbb301a4580 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb30106160 0x7fbb30199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 --2- 172.21.15.94:0/3841159255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb30101090 0x7fbb30199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 -- 172.21.15.94:0/3841159255 >> 172.21.15.94:0/3841159255 conn(0x7fbb300fc910 msgr2=0x7fbb300fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:11.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 -- 172.21.15.94:0/3841159255 shutdown_connections 2024-04-21T08:40:11.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.130+0000 7fbb3603a640 1 -- 172.21.15.94:0/3841159255 wait complete. 2024-04-21T08:40:11.141 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-21T08:40:09.917339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1391043,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1420,"num_read_kb":1143,"num_write":627,"num_write_kb":5306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378652,"kb_used_data":12812,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563172,"statfs":{"total":1151940427776,"available":1151552688128,"internally_reserved":0,"allocated":13119488,"data_stored":10506743,"data_compressed":573078,"data_compressed_allocated":774144,"data_compressed_original":4878336,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":200325,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":420,"num_read_kb":479,"num_write":155,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.988315"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318569+0000","last_change":"2024-04-21T08:37:31.448345+0000","last_active":"2024-04-21T08:39:58.318569+0000","last_peered":"2024-04-21T08:39:58.318569+0000","last_clean":"2024-04-21T08:39:58.318569+0000","last_became_active":"2024-04-21T08:37:31.447878+0000","last_became_peered":"2024-04-21T08:37:31.447878+0000","last_unstale":"2024-04-21T08:39:58.318569+0000","last_undegraded":"2024-04-21T08:39:58.318569+0000","last_fullsized":"2024-04-21T08:39:58.318569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:45.589165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319782+0000","last_change":"2024-04-21T08:37:31.452895+0000","last_active":"2024-04-21T08:39:58.319782+0000","last_peered":"2024-04-21T08:39:58.319782+0000","last_clean":"2024-04-21T08:39:58.319782+0000","last_became_active":"2024-04-21T08:37:31.452550+0000","last_became_peered":"2024-04-21T08:37:31.452550+0000","last_unstale":"2024-04-21T08:39:58.319782+0000","last_undegraded":"2024-04-21T08:39:58.319782+0000","last_fullsized":"2024-04-21T08:39:58.319782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:02.497802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319443+0000","last_change":"2024-04-21T08:37:31.447552+0000","last_active":"2024-04-21T08:39:58.319443+0000","last_peered":"2024-04-21T08:39:58.319443+0000","last_clean":"2024-04-21T08:39:58.319443+0000","last_became_active":"2024-04-21T08:37:31.446044+0000","last_became_peered":"2024-04-21T08:37:31.446044+0000","last_unstale":"2024-04-21T08:39:58.319443+0000","last_undegraded":"2024-04-21T08:39:58.319443+0000","last_fullsized":"2024-04-21T08:39:58.319443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:22.801977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151531+0000","last_change":"2024-04-21T08:37:31.452400+0000","last_active":"2024-04-21T08:39:58.151531+0000","last_peered":"2024-04-21T08:39:58.151531+0000","last_clean":"2024-04-21T08:39:58.151531+0000","last_became_active":"2024-04-21T08:37:31.451462+0000","last_became_peered":"2024-04-21T08:37:31.451462+0000","last_unstale":"2024-04-21T08:39:58.151531+0000","last_undegraded":"2024-04-21T08:39:58.151531+0000","last_fullsized":"2024-04-21T08:39:58.151531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:22.463481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009087+0000","last_change":"2024-04-21T08:37:31.448942+0000","last_active":"2024-04-21T08:39:58.009087+0000","last_peered":"2024-04-21T08:39:58.009087+0000","last_clean":"2024-04-21T08:39:58.009087+0000","last_became_active":"2024-04-21T08:37:31.448526+0000","last_became_peered":"2024-04-21T08:37:31.448526+0000","last_unstale":"2024-04-21T08:39:58.009087+0000","last_undegraded":"2024-04-21T08:39:58.009087+0000","last_fullsized":"2024-04-21T08:39:58.009087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:29.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017166+0000","last_change":"2024-04-21T08:37:31.452165+0000","last_active":"2024-04-21T08:39:58.017166+0000","last_peered":"2024-04-21T08:39:58.017166+0000","last_clean":"2024-04-21T08:39:58.017166+0000","last_became_active":"2024-04-21T08:37:31.451802+0000","last_became_peered":"2024-04-21T08:37:31.451802+0000","last_unstale":"2024-04-21T08:39:58.017166+0000","last_undegraded":"2024-04-21T08:39:58.017166+0000","last_fullsized":"2024-04-21T08:39:58.017166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:42.210544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905010+0000","last_change":"2024-04-21T08:39:31.681660+0000","last_active":"2024-04-21T08:39:57.905010+0000","last_peered":"2024-04-21T08:39:57.905010+0000","last_clean":"2024-04-21T08:39:57.905010+0000","last_became_active":"2024-04-21T08:39:31.681203+0000","last_became_peered":"2024-04-21T08:39:31.681203+0000","last_unstale":"2024-04-21T08:39:57.905010+0000","last_undegraded":"2024-04-21T08:39:57.905010+0000","last_fullsized":"2024-04-21T08:39:57.905010+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:54.120862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903472+0000","last_change":"2024-04-21T08:37:31.448257+0000","last_active":"2024-04-21T08:39:57.903472+0000","last_peered":"2024-04-21T08:39:57.903472+0000","last_clean":"2024-04-21T08:39:57.903472+0000","last_became_active":"2024-04-21T08:37:31.447845+0000","last_became_peered":"2024-04-21T08:37:31.447845+0000","last_unstale":"2024-04-21T08:39:57.903472+0000","last_undegraded":"2024-04-21T08:39:57.903472+0000","last_fullsized":"2024-04-21T08:39:57.903472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:19.956699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154785+0000","last_change":"2024-04-21T08:37:31.443022+0000","last_active":"2024-04-21T08:39:58.154785+0000","last_peered":"2024-04-21T08:39:58.154785+0000","last_clean":"2024-04-21T08:39:58.154785+0000","last_became_active":"2024-04-21T08:37:31.442645+0000","last_became_peered":"2024-04-21T08:37:31.442645+0000","last_unstale":"2024-04-21T08:39:58.154785+0000","last_undegraded":"2024-04-21T08:39:58.154785+0000","last_fullsized":"2024-04-21T08:39:58.154785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:18:10.861281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017434+0000","last_change":"2024-04-21T08:37:31.453427+0000","last_active":"2024-04-21T08:39:58.017434+0000","last_peered":"2024-04-21T08:39:58.017434+0000","last_clean":"2024-04-21T08:39:58.017434+0000","last_became_active":"2024-04-21T08:37:31.453096+0000","last_became_peered":"2024-04-21T08:37:31.453096+0000","last_unstale":"2024-04-21T08:39:58.017434+0000","last_undegraded":"2024-04-21T08:39:58.017434+0000","last_fullsized":"2024-04-21T08:39:58.017434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:49.450351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905929+0000","last_change":"2024-04-21T08:37:31.449908+0000","last_active":"2024-04-21T08:39:57.905929+0000","last_peered":"2024-04-21T08:39:57.905929+0000","last_clean":"2024-04-21T08:39:57.905929+0000","last_became_active":"2024-04-21T08:37:31.449558+0000","last_became_peered":"2024-04-21T08:37:31.449558+0000","last_unstale":"2024-04-21T08:39:57.905929+0000","last_undegraded":"2024-04-21T08:39:57.905929+0000","last_fullsized":"2024-04-21T08:39:57.905929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:21.538689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151539+0000","last_change":"2024-04-21T08:37:31.452973+0000","last_active":"2024-04-21T08:39:58.151539+0000","last_peered":"2024-04-21T08:39:58.151539+0000","last_clean":"2024-04-21T08:39:58.151539+0000","last_became_active":"2024-04-21T08:37:31.452586+0000","last_became_peered":"2024-04-21T08:37:31.452586+0000","last_unstale":"2024-04-21T08:39:58.151539+0000","last_undegraded":"2024-04-21T08:39:58.151539+0000","last_fullsized":"2024-04-21T08:39:58.151539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:09:11.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319144+0000","last_change":"2024-04-21T08:37:31.445801+0000","last_active":"2024-04-21T08:39:58.319144+0000","last_peered":"2024-04-21T08:39:58.319144+0000","last_clean":"2024-04-21T08:39:58.319144+0000","last_became_active":"2024-04-21T08:37:31.445478+0000","last_became_peered":"2024-04-21T08:37:31.445478+0000","last_unstale":"2024-04-21T08:39:58.319144+0000","last_undegraded":"2024-04-21T08:39:58.319144+0000","last_fullsized":"2024-04-21T08:39:58.319144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:33.352739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319201+0000","last_change":"2024-04-21T08:37:31.450735+0000","last_active":"2024-04-21T08:39:58.319201+0000","last_peered":"2024-04-21T08:39:58.319201+0000","last_clean":"2024-04-21T08:39:58.319201+0000","last_became_active":"2024-04-21T08:37:31.450384+0000","last_became_peered":"2024-04-21T08:37:31.450384+0000","last_unstale":"2024-04-21T08:39:58.319201+0000","last_undegraded":"2024-04-21T08:39:58.319201+0000","last_fullsized":"2024-04-21T08:39:58.319201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:45.950489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319354+0000","last_change":"2024-04-21T08:37:31.454690+0000","last_active":"2024-04-21T08:39:58.319354+0000","last_peered":"2024-04-21T08:39:58.319354+0000","last_clean":"2024-04-21T08:39:58.319354+0000","last_became_active":"2024-04-21T08:37:31.454330+0000","last_became_peered":"2024-04-21T08:37:31.454330+0000","last_unstale":"2024-04-21T08:39:58.319354+0000","last_undegraded":"2024-04-21T08:39:58.319354+0000","last_fullsized":"2024-04-21T08:39:58.319354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:52.644317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153871+0000","last_change":"2024-04-21T08:37:31.441389+0000","last_active":"2024-04-21T08:39:58.153871+0000","last_peered":"2024-04-21T08:39:58.153871+0000","last_clean":"2024-04-21T08:39:58.153871+0000","last_became_active":"2024-04-21T08:37:31.441065+0000","last_became_peered":"2024-04-21T08:37:31.441065+0000","last_unstale":"2024-04-21T08:39:58.153871+0000","last_undegraded":"2024-04-21T08:39:58.153871+0000","last_fullsized":"2024-04-21T08:39:58.153871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:56.528154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318575+0000","last_change":"2024-04-21T08:37:31.446722+0000","last_active":"2024-04-21T08:39:58.318575+0000","last_peered":"2024-04-21T08:39:58.318575+0000","last_clean":"2024-04-21T08:39:58.318575+0000","last_became_active":"2024-04-21T08:37:31.446396+0000","last_became_peered":"2024-04-21T08:37:31.446396+0000","last_unstale":"2024-04-21T08:39:58.318575+0000","last_undegraded":"2024-04-21T08:39:58.318575+0000","last_fullsized":"2024-04-21T08:39:58.318575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:17.488633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007275+0000","last_change":"2024-04-21T08:37:31.444388+0000","last_active":"2024-04-21T08:39:58.007275+0000","last_peered":"2024-04-21T08:39:58.007275+0000","last_clean":"2024-04-21T08:39:58.007275+0000","last_became_active":"2024-04-21T08:37:31.444023+0000","last_became_peered":"2024-04-21T08:37:31.444023+0000","last_unstale":"2024-04-21T08:39:58.007275+0000","last_undegraded":"2024-04-21T08:39:58.007275+0000","last_fullsized":"2024-04-21T08:39:58.007275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:07:46.238941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905278+0000","last_change":"2024-04-21T08:37:31.447850+0000","last_active":"2024-04-21T08:39:57.905278+0000","last_peered":"2024-04-21T08:39:57.905278+0000","last_clean":"2024-04-21T08:39:57.905278+0000","last_became_active":"2024-04-21T08:37:31.447505+0000","last_became_peered":"2024-04-21T08:37:31.447505+0000","last_unstale":"2024-04-21T08:39:57.905278+0000","last_undegraded":"2024-04-21T08:39:57.905278+0000","last_fullsized":"2024-04-21T08:39:57.905278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:55.399410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905076+0000","last_change":"2024-04-21T08:37:35.608747+0000","last_active":"2024-04-21T08:39:57.905076+0000","last_peered":"2024-04-21T08:39:57.905076+0000","last_clean":"2024-04-21T08:39:57.905076+0000","last_became_active":"2024-04-21T08:37:35.608157+0000","last_became_peered":"2024-04-21T08:37:35.608157+0000","last_unstale":"2024-04-21T08:39:57.905076+0000","last_undegraded":"2024-04-21T08:39:57.905076+0000","last_fullsized":"2024-04-21T08:39:57.905076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:53.384190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007094+0000","last_change":"2024-04-21T08:38:30.679843+0000","last_active":"2024-04-21T08:39:58.007094+0000","last_peered":"2024-04-21T08:39:58.007094+0000","last_clean":"2024-04-21T08:39:58.007094+0000","last_became_active":"2024-04-21T08:38:30.670964+0000","last_became_peered":"2024-04-21T08:38:30.670964+0000","last_unstale":"2024-04-21T08:39:58.007094+0000","last_undegraded":"2024-04-21T08:39:58.007094+0000","last_fullsized":"2024-04-21T08:39:58.007094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:49.652571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904100+0000","last_change":"2024-04-21T08:37:31.451813+0000","last_active":"2024-04-21T08:39:57.904100+0000","last_peered":"2024-04-21T08:39:57.904100+0000","last_clean":"2024-04-21T08:39:57.904100+0000","last_became_active":"2024-04-21T08:37:31.451447+0000","last_became_peered":"2024-04-21T08:37:31.451447+0000","last_unstale":"2024-04-21T08:39:57.904100+0000","last_undegraded":"2024-04-21T08:39:57.904100+0000","last_fullsized":"2024-04-21T08:39:57.904100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:46:45.235326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151276+0000","last_change":"2024-04-21T08:37:35.603518+0000","last_active":"2024-04-21T08:39:58.151276+0000","last_peered":"2024-04-21T08:39:58.151276+0000","last_clean":"2024-04-21T08:39:58.151276+0000","last_became_active":"2024-04-21T08:37:35.603038+0000","last_became_peered":"2024-04-21T08:37:35.603038+0000","last_unstale":"2024-04-21T08:39:58.151276+0000","last_undegraded":"2024-04-21T08:39:58.151276+0000","last_fullsized":"2024-04-21T08:39:58.151276+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:06:29.224025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904228+0000","last_change":"2024-04-21T08:37:30.424608+0000","last_active":"2024-04-21T08:39:57.904228+0000","last_peered":"2024-04-21T08:39:57.904228+0000","last_clean":"2024-04-21T08:39:57.904228+0000","last_became_active":"2024-04-21T08:37:30.424259+0000","last_became_peered":"2024-04-21T08:37:30.424259+0000","last_unstale":"2024-04-21T08:39:57.904228+0000","last_undegraded":"2024-04-21T08:39:57.904228+0000","last_fullsized":"2024-04-21T08:39:57.904228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:38.007095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319688+0000","last_change":"2024-04-21T08:37:31.444656+0000","last_active":"2024-04-21T08:39:58.319688+0000","last_peered":"2024-04-21T08:39:58.319688+0000","last_clean":"2024-04-21T08:39:58.319688+0000","last_became_active":"2024-04-21T08:37:31.444300+0000","last_became_peered":"2024-04-21T08:37:31.444300+0000","last_unstale":"2024-04-21T08:39:58.319688+0000","last_undegraded":"2024-04-21T08:39:58.319688+0000","last_fullsized":"2024-04-21T08:39:58.319688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:40.901674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151439+0000","last_change":"2024-04-21T08:37:35.611732+0000","last_active":"2024-04-21T08:39:58.151439+0000","last_peered":"2024-04-21T08:39:58.151439+0000","last_clean":"2024-04-21T08:39:58.151439+0000","last_became_active":"2024-04-21T08:37:35.611060+0000","last_became_peered":"2024-04-21T08:37:35.611060+0000","last_unstale":"2024-04-21T08:39:58.151439+0000","last_undegraded":"2024-04-21T08:39:58.151439+0000","last_fullsized":"2024-04-21T08:39:58.151439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:36:01.491949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318678+0000","last_change":"2024-04-21T08:38:30.689722+0000","last_active":"2024-04-21T08:39:58.318678+0000","last_peered":"2024-04-21T08:39:58.318678+0000","last_clean":"2024-04-21T08:39:58.318678+0000","last_became_active":"2024-04-21T08:38:30.676521+0000","last_became_peered":"2024-04-21T08:38:30.676521+0000","last_unstale":"2024-04-21T08:39:58.318678+0000","last_undegraded":"2024-04-21T08:39:58.318678+0000","last_fullsized":"2024-04-21T08:39:58.318678+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:24:15.799308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154378+0000","last_change":"2024-04-21T08:37:31.443896+0000","last_active":"2024-04-21T08:39:58.154378+0000","last_peered":"2024-04-21T08:39:58.154378+0000","last_clean":"2024-04-21T08:39:58.154378+0000","last_became_active":"2024-04-21T08:37:31.442684+0000","last_became_peered":"2024-04-21T08:37:31.442684+0000","last_unstale":"2024-04-21T08:39:58.154378+0000","last_undegraded":"2024-04-21T08:39:58.154378+0000","last_fullsized":"2024-04-21T08:39:58.154378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:14:18.649565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151277+0000","last_change":"2024-04-21T08:37:35.600468+0000","last_active":"2024-04-21T08:39:58.151277+0000","last_peered":"2024-04-21T08:39:58.151277+0000","last_clean":"2024-04-21T08:39:58.151277+0000","last_became_active":"2024-04-21T08:37:35.599940+0000","last_became_peered":"2024-04-21T08:37:35.599940+0000","last_unstale":"2024-04-21T08:39:58.151277+0000","last_undegraded":"2024-04-21T08:39:58.151277+0000","last_fullsized":"2024-04-21T08:39:58.151277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:48.125434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009395+0000","last_change":"2024-04-21T08:37:30.416480+0000","last_active":"2024-04-21T08:39:58.009395+0000","last_peered":"2024-04-21T08:39:58.009395+0000","last_clean":"2024-04-21T08:39:58.009395+0000","last_became_active":"2024-04-21T08:37:30.416072+0000","last_became_peered":"2024-04-21T08:37:30.416072+0000","last_unstale":"2024-04-21T08:39:58.009395+0000","last_undegraded":"2024-04-21T08:39:58.009395+0000","last_fullsized":"2024-04-21T08:39:58.009395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:28:22.400926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014201+0000","last_change":"2024-04-21T08:37:31.457216+0000","last_active":"2024-04-21T08:39:58.014201+0000","last_peered":"2024-04-21T08:39:58.014201+0000","last_clean":"2024-04-21T08:39:58.014201+0000","last_became_active":"2024-04-21T08:37:31.456856+0000","last_became_peered":"2024-04-21T08:37:31.456856+0000","last_unstale":"2024-04-21T08:39:58.014201+0000","last_undegraded":"2024-04-21T08:39:58.014201+0000","last_fullsized":"2024-04-21T08:39:58.014201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:10.272886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905222+0000","last_change":"2024-04-21T08:37:35.601715+0000","last_active":"2024-04-21T08:39:57.905222+0000","last_peered":"2024-04-21T08:39:57.905222+0000","last_clean":"2024-04-21T08:39:57.905222+0000","last_became_active":"2024-04-21T08:37:35.601269+0000","last_became_peered":"2024-04-21T08:37:35.601269+0000","last_unstale":"2024-04-21T08:39:57.905222+0000","last_undegraded":"2024-04-21T08:39:57.905222+0000","last_fullsized":"2024-04-21T08:39:57.905222+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:57.192918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":176,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.903763+0000","last_change":"2024-04-21T08:37:30.415510+0000","last_active":"2024-04-21T08:40:07.903763+0000","last_peered":"2024-04-21T08:40:07.903763+0000","last_clean":"2024-04-21T08:40:07.903763+0000","last_became_active":"2024-04-21T08:37:30.415199+0000","last_became_peered":"2024-04-21T08:37:30.415199+0000","last_unstale":"2024-04-21T08:40:07.903763+0000","last_undegraded":"2024-04-21T08:40:07.903763+0000","last_fullsized":"2024-04-21T08:40:07.903763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:32.336247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009124+0000","last_change":"2024-04-21T08:39:31.681460+0000","last_active":"2024-04-21T08:39:58.009124+0000","last_peered":"2024-04-21T08:39:58.009124+0000","last_clean":"2024-04-21T08:39:58.009124+0000","last_became_active":"2024-04-21T08:39:31.681057+0000","last_became_peered":"2024-04-21T08:39:31.681057+0000","last_unstale":"2024-04-21T08:39:58.009124+0000","last_undegraded":"2024-04-21T08:39:58.009124+0000","last_fullsized":"2024-04-21T08:39:58.009124+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:43.407369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905274+0000","last_change":"2024-04-21T08:38:30.714455+0000","last_active":"2024-04-21T08:39:57.905274+0000","last_peered":"2024-04-21T08:39:57.905274+0000","last_clean":"2024-04-21T08:39:57.905274+0000","last_became_active":"2024-04-21T08:38:30.713910+0000","last_became_peered":"2024-04-21T08:38:30.713910+0000","last_unstale":"2024-04-21T08:39:57.905274+0000","last_undegraded":"2024-04-21T08:39:57.905274+0000","last_fullsized":"2024-04-21T08:39:57.905274+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.448652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,10],"acting":[2,6,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017063+0000","last_change":"2024-04-21T08:37:30.419877+0000","last_active":"2024-04-21T08:39:58.017063+0000","last_peered":"2024-04-21T08:39:58.017063+0000","last_clean":"2024-04-21T08:39:58.017063+0000","last_became_active":"2024-04-21T08:37:30.419572+0000","last_became_peered":"2024-04-21T08:37:30.419572+0000","last_unstale":"2024-04-21T08:39:58.017063+0000","last_undegraded":"2024-04-21T08:39:58.017063+0000","last_fullsized":"2024-04-21T08:39:58.017063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:32:35.524089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014094+0000","last_change":"2024-04-21T08:37:31.458590+0000","last_active":"2024-04-21T08:39:58.014094+0000","last_peered":"2024-04-21T08:39:58.014094+0000","last_clean":"2024-04-21T08:39:58.014094+0000","last_became_active":"2024-04-21T08:37:31.458196+0000","last_became_peered":"2024-04-21T08:37:31.458196+0000","last_unstale":"2024-04-21T08:39:58.014094+0000","last_undegraded":"2024-04-21T08:39:58.014094+0000","last_fullsized":"2024-04-21T08:39:58.014094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:00.890859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318442+0000","last_change":"2024-04-21T08:38:30.803663+0000","last_active":"2024-04-21T08:39:58.318442+0000","last_peered":"2024-04-21T08:39:58.318442+0000","last_clean":"2024-04-21T08:39:58.318442+0000","last_became_active":"2024-04-21T08:38:30.803203+0000","last_became_peered":"2024-04-21T08:38:30.803203+0000","last_unstale":"2024-04-21T08:39:58.318442+0000","last_undegraded":"2024-04-21T08:39:58.318442+0000","last_fullsized":"2024-04-21T08:39:58.318442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:06.085380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318695+0000","last_change":"2024-04-21T08:37:30.415786+0000","last_active":"2024-04-21T08:39:58.318695+0000","last_peered":"2024-04-21T08:39:58.318695+0000","last_clean":"2024-04-21T08:39:58.318695+0000","last_became_active":"2024-04-21T08:37:30.415472+0000","last_became_peered":"2024-04-21T08:37:30.415472+0000","last_unstale":"2024-04-21T08:39:58.318695+0000","last_undegraded":"2024-04-21T08:39:58.318695+0000","last_fullsized":"2024-04-21T08:39:58.318695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.417132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903648+0000","last_change":"2024-04-21T08:37:31.455720+0000","last_active":"2024-04-21T08:39:57.903648+0000","last_peered":"2024-04-21T08:39:57.903648+0000","last_clean":"2024-04-21T08:39:57.903648+0000","last_became_active":"2024-04-21T08:37:31.455334+0000","last_became_peered":"2024-04-21T08:37:31.455334+0000","last_unstale":"2024-04-21T08:39:57.903648+0000","last_undegraded":"2024-04-21T08:39:57.903648+0000","last_fullsized":"2024-04-21T08:39:57.903648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:06:55.220656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319675+0000","last_change":"2024-04-21T08:37:35.603305+0000","last_active":"2024-04-21T08:39:58.319675+0000","last_peered":"2024-04-21T08:39:58.319675+0000","last_clean":"2024-04-21T08:39:58.319675+0000","last_became_active":"2024-04-21T08:37:35.602854+0000","last_became_peered":"2024-04-21T08:37:35.602854+0000","last_unstale":"2024-04-21T08:39:58.319675+0000","last_undegraded":"2024-04-21T08:39:58.319675+0000","last_fullsized":"2024-04-21T08:39:58.319675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:10.993864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013890+0000","last_change":"2024-04-21T08:37:30.420797+0000","last_active":"2024-04-21T08:39:58.013890+0000","last_peered":"2024-04-21T08:39:58.013890+0000","last_clean":"2024-04-21T08:39:58.013890+0000","last_became_active":"2024-04-21T08:37:30.420485+0000","last_became_peered":"2024-04-21T08:37:30.420485+0000","last_unstale":"2024-04-21T08:39:58.013890+0000","last_undegraded":"2024-04-21T08:39:58.013890+0000","last_fullsized":"2024-04-21T08:39:58.013890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:03.275677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006844+0000","last_change":"2024-04-21T08:37:31.448406+0000","last_active":"2024-04-21T08:39:58.006844+0000","last_peered":"2024-04-21T08:39:58.006844+0000","last_clean":"2024-04-21T08:39:58.006844+0000","last_became_active":"2024-04-21T08:37:31.448085+0000","last_became_peered":"2024-04-21T08:37:31.448085+0000","last_unstale":"2024-04-21T08:39:58.006844+0000","last_undegraded":"2024-04-21T08:39:58.006844+0000","last_fullsized":"2024-04-21T08:39:58.006844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:49.574064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903948+0000","last_change":"2024-04-21T08:37:35.601338+0000","last_active":"2024-04-21T08:39:57.903948+0000","last_peered":"2024-04-21T08:39:57.903948+0000","last_clean":"2024-04-21T08:39:57.903948+0000","last_became_active":"2024-04-21T08:37:35.600890+0000","last_became_peered":"2024-04-21T08:37:35.600890+0000","last_unstale":"2024-04-21T08:39:57.903948+0000","last_undegraded":"2024-04-21T08:39:57.903948+0000","last_fullsized":"2024-04-21T08:39:57.903948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:30:21.755850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'4","reported_seq":171,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.903166+0000","last_change":"2024-04-21T08:37:30.423320+0000","last_active":"2024-04-21T08:40:07.903166+0000","last_peered":"2024-04-21T08:40:07.903166+0000","last_clean":"2024-04-21T08:40:07.903166+0000","last_became_active":"2024-04-21T08:37:30.423020+0000","last_became_peered":"2024-04-21T08:37:30.423020+0000","last_unstale":"2024-04-21T08:40:07.903166+0000","last_undegraded":"2024-04-21T08:40:07.903166+0000","last_fullsized":"2024-04-21T08:40:07.903166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:08:27.313196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":133,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154669+0000","last_change":"2024-04-21T08:37:31.442425+0000","last_active":"2024-04-21T08:39:58.154669+0000","last_peered":"2024-04-21T08:39:58.154669+0000","last_clean":"2024-04-21T08:39:58.154669+0000","last_became_active":"2024-04-21T08:37:31.442028+0000","last_became_peered":"2024-04-21T08:37:31.442028+0000","last_unstale":"2024-04-21T08:39:58.154669+0000","last_undegraded":"2024-04-21T08:39:58.154669+0000","last_fullsized":"2024-04-21T08:39:58.154669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:45.498656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006697+0000","last_change":"2024-04-21T08:37:35.607829+0000","last_active":"2024-04-21T08:39:58.006697+0000","last_peered":"2024-04-21T08:39:58.006697+0000","last_clean":"2024-04-21T08:39:58.006697+0000","last_became_active":"2024-04-21T08:37:35.607150+0000","last_became_peered":"2024-04-21T08:37:35.607150+0000","last_unstale":"2024-04-21T08:39:58.006697+0000","last_undegraded":"2024-04-21T08:39:58.006697+0000","last_fullsized":"2024-04-21T08:39:58.006697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:01:34.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905384+0000","last_change":"2024-04-21T08:37:30.418613+0000","last_active":"2024-04-21T08:39:57.905384+0000","last_peered":"2024-04-21T08:39:57.905384+0000","last_clean":"2024-04-21T08:39:57.905384+0000","last_became_active":"2024-04-21T08:37:30.418300+0000","last_became_peered":"2024-04-21T08:37:30.418300+0000","last_unstale":"2024-04-21T08:39:57.905384+0000","last_undegraded":"2024-04-21T08:39:57.905384+0000","last_fullsized":"2024-04-21T08:39:57.905384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:17:35.370753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319739+0000","last_change":"2024-04-21T08:39:31.682628+0000","last_active":"2024-04-21T08:39:58.319739+0000","last_peered":"2024-04-21T08:39:58.319739+0000","last_clean":"2024-04-21T08:39:58.319739+0000","last_became_active":"2024-04-21T08:39:31.682383+0000","last_became_peered":"2024-04-21T08:39:31.682383+0000","last_unstale":"2024-04-21T08:39:58.319739+0000","last_undegraded":"2024-04-21T08:39:58.319739+0000","last_fullsized":"2024-04-21T08:39:58.319739+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:38.831288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151734+0000","last_change":"2024-04-21T08:37:35.608280+0000","last_active":"2024-04-21T08:39:58.151734+0000","last_peered":"2024-04-21T08:39:58.151734+0000","last_clean":"2024-04-21T08:39:58.151734+0000","last_became_active":"2024-04-21T08:37:35.607898+0000","last_became_peered":"2024-04-21T08:37:35.607898+0000","last_unstale":"2024-04-21T08:39:58.151734+0000","last_undegraded":"2024-04-21T08:39:58.151734+0000","last_fullsized":"2024-04-21T08:39:58.151734+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:55.689519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014140+0000","last_change":"2024-04-21T08:37:30.417352+0000","last_active":"2024-04-21T08:39:58.014140+0000","last_peered":"2024-04-21T08:39:58.014140+0000","last_clean":"2024-04-21T08:39:58.014140+0000","last_became_active":"2024-04-21T08:37:30.417000+0000","last_became_peered":"2024-04-21T08:37:30.417000+0000","last_unstale":"2024-04-21T08:39:58.014140+0000","last_undegraded":"2024-04-21T08:39:58.014140+0000","last_fullsized":"2024-04-21T08:39:58.014140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:56.087334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006804+0000","last_change":"2024-04-21T08:39:31.679434+0000","last_active":"2024-04-21T08:39:58.006804+0000","last_peered":"2024-04-21T08:39:58.006804+0000","last_clean":"2024-04-21T08:39:58.006804+0000","last_became_active":"2024-04-21T08:39:31.679041+0000","last_became_peered":"2024-04-21T08:39:31.679041+0000","last_unstale":"2024-04-21T08:39:58.006804+0000","last_undegraded":"2024-04-21T08:39:58.006804+0000","last_fullsized":"2024-04-21T08:39:58.006804+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:27:11.011449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006731+0000","last_change":"2024-04-21T08:37:35.601077+0000","last_active":"2024-04-21T08:39:58.006731+0000","last_peered":"2024-04-21T08:39:58.006731+0000","last_clean":"2024-04-21T08:39:58.006731+0000","last_became_active":"2024-04-21T08:37:35.600673+0000","last_became_peered":"2024-04-21T08:37:35.600673+0000","last_unstale":"2024-04-21T08:39:58.006731+0000","last_undegraded":"2024-04-21T08:39:58.006731+0000","last_fullsized":"2024-04-21T08:39:58.006731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:04.554208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'43","reported_seq":358,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:08.904470+0000","last_change":"2024-04-21T08:37:30.421440+0000","last_active":"2024-04-21T08:40:08.904470+0000","last_peered":"2024-04-21T08:40:08.904470+0000","last_clean":"2024-04-21T08:40:08.904470+0000","last_became_active":"2024-04-21T08:37:30.420988+0000","last_became_peered":"2024-04-21T08:37:30.420988+0000","last_unstale":"2024-04-21T08:40:08.904470+0000","last_undegraded":"2024-04-21T08:40:08.904470+0000","last_fullsized":"2024-04-21T08:40:08.904470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:38:03.850705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72979,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":281,"num_read_kb":213,"num_write":44,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009331+0000","last_change":"2024-04-21T08:37:31.447208+0000","last_active":"2024-04-21T08:39:58.009331+0000","last_peered":"2024-04-21T08:39:58.009331+0000","last_clean":"2024-04-21T08:39:58.009331+0000","last_became_active":"2024-04-21T08:37:31.444464+0000","last_became_peered":"2024-04-21T08:37:31.444464+0000","last_unstale":"2024-04-21T08:39:58.009331+0000","last_undegraded":"2024-04-21T08:39:58.009331+0000","last_fullsized":"2024-04-21T08:39:58.009331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:33.805676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014409+0000","last_change":"2024-04-21T08:37:35.603937+0000","last_active":"2024-04-21T08:39:58.014409+0000","last_peered":"2024-04-21T08:39:58.014409+0000","last_clean":"2024-04-21T08:39:58.014409+0000","last_became_active":"2024-04-21T08:37:35.603477+0000","last_became_peered":"2024-04-21T08:37:35.603477+0000","last_unstale":"2024-04-21T08:39:58.014409+0000","last_undegraded":"2024-04-21T08:39:58.014409+0000","last_fullsized":"2024-04-21T08:39:58.014409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:40:33.901570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905391+0000","last_change":"2024-04-21T08:37:30.416941+0000","last_active":"2024-04-21T08:39:57.905391+0000","last_peered":"2024-04-21T08:39:57.905391+0000","last_clean":"2024-04-21T08:39:57.905391+0000","last_became_active":"2024-04-21T08:37:30.416625+0000","last_became_peered":"2024-04-21T08:37:30.416625+0000","last_unstale":"2024-04-21T08:39:57.905391+0000","last_undegraded":"2024-04-21T08:39:57.905391+0000","last_fullsized":"2024-04-21T08:39:57.905391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:07.782779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318733+0000","last_change":"2024-04-21T08:37:31.441707+0000","last_active":"2024-04-21T08:39:58.318733+0000","last_peered":"2024-04-21T08:39:58.318733+0000","last_clean":"2024-04-21T08:39:58.318733+0000","last_became_active":"2024-04-21T08:37:31.441366+0000","last_became_peered":"2024-04-21T08:37:31.441366+0000","last_unstale":"2024-04-21T08:39:58.318733+0000","last_undegraded":"2024-04-21T08:39:58.318733+0000","last_fullsized":"2024-04-21T08:39:58.318733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:16.573804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318906+0000","last_change":"2024-04-21T08:37:35.596214+0000","last_active":"2024-04-21T08:39:58.318906+0000","last_peered":"2024-04-21T08:39:58.318906+0000","last_clean":"2024-04-21T08:39:58.318906+0000","last_became_active":"2024-04-21T08:37:35.595773+0000","last_became_peered":"2024-04-21T08:37:35.595773+0000","last_unstale":"2024-04-21T08:39:58.318906+0000","last_undegraded":"2024-04-21T08:39:58.318906+0000","last_fullsized":"2024-04-21T08:39:58.318906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:07.538190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318273+0000","last_change":"2024-04-21T08:38:30.684828+0000","last_active":"2024-04-21T08:39:58.318273+0000","last_peered":"2024-04-21T08:39:58.318273+0000","last_clean":"2024-04-21T08:39:58.318273+0000","last_became_active":"2024-04-21T08:38:30.673923+0000","last_became_peered":"2024-04-21T08:38:30.673923+0000","last_unstale":"2024-04-21T08:39:58.318273+0000","last_undegraded":"2024-04-21T08:39:58.318273+0000","last_fullsized":"2024-04-21T08:39:58.318273+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:51:37.906150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905472+0000","last_change":"2024-04-21T08:37:31.453680+0000","last_active":"2024-04-21T08:39:57.905472+0000","last_peered":"2024-04-21T08:39:57.905472+0000","last_clean":"2024-04-21T08:39:57.905472+0000","last_became_active":"2024-04-21T08:37:31.453319+0000","last_became_peered":"2024-04-21T08:37:31.453319+0000","last_unstale":"2024-04-21T08:39:57.905472+0000","last_undegraded":"2024-04-21T08:39:57.905472+0000","last_fullsized":"2024-04-21T08:39:57.905472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:43.139789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007023+0000","last_change":"2024-04-21T08:37:35.608906+0000","last_active":"2024-04-21T08:39:58.007023+0000","last_peered":"2024-04-21T08:39:58.007023+0000","last_clean":"2024-04-21T08:39:58.007023+0000","last_became_active":"2024-04-21T08:37:35.608535+0000","last_became_peered":"2024-04-21T08:37:35.608535+0000","last_unstale":"2024-04-21T08:39:58.007023+0000","last_undegraded":"2024-04-21T08:39:58.007023+0000","last_fullsized":"2024-04-21T08:39:58.007023+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:58:01.422308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'9","reported_seq":183,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.902877+0000","last_change":"2024-04-21T08:37:30.414865+0000","last_active":"2024-04-21T08:40:07.902877+0000","last_peered":"2024-04-21T08:40:07.902877+0000","last_clean":"2024-04-21T08:40:07.902877+0000","last_became_active":"2024-04-21T08:37:30.414471+0000","last_became_peered":"2024-04-21T08:37:30.414471+0000","last_unstale":"2024-04-21T08:40:07.902877+0000","last_undegraded":"2024-04-21T08:40:07.902877+0000","last_fullsized":"2024-04-21T08:40:07.902877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:46.943574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2601,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":8,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905326+0000","last_change":"2024-04-21T08:37:31.456634+0000","last_active":"2024-04-21T08:39:57.905326+0000","last_peered":"2024-04-21T08:39:57.905326+0000","last_clean":"2024-04-21T08:39:57.905326+0000","last_became_active":"2024-04-21T08:37:31.456278+0000","last_became_peered":"2024-04-21T08:37:31.456278+0000","last_unstale":"2024-04-21T08:39:57.905326+0000","last_undegraded":"2024-04-21T08:39:57.905326+0000","last_fullsized":"2024-04-21T08:39:57.905326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:04.870024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319167+0000","last_change":"2024-04-21T08:37:35.601760+0000","last_active":"2024-04-21T08:39:58.319167+0000","last_peered":"2024-04-21T08:39:58.319167+0000","last_clean":"2024-04-21T08:39:58.319167+0000","last_became_active":"2024-04-21T08:37:35.601332+0000","last_became_peered":"2024-04-21T08:37:35.601332+0000","last_unstale":"2024-04-21T08:39:58.319167+0000","last_undegraded":"2024-04-21T08:39:58.319167+0000","last_fullsized":"2024-04-21T08:39:58.319167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:04:03.694581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008961+0000","last_change":"2024-04-21T08:37:30.422419+0000","last_active":"2024-04-21T08:39:58.008961+0000","last_peered":"2024-04-21T08:39:58.008961+0000","last_clean":"2024-04-21T08:39:58.008961+0000","last_became_active":"2024-04-21T08:37:30.422123+0000","last_became_peered":"2024-04-21T08:37:30.422123+0000","last_unstale":"2024-04-21T08:39:58.008961+0000","last_undegraded":"2024-04-21T08:39:58.008961+0000","last_fullsized":"2024-04-21T08:39:58.008961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:32:54.029041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013848+0000","last_change":"2024-04-21T08:37:31.458992+0000","last_active":"2024-04-21T08:39:58.013848+0000","last_peered":"2024-04-21T08:39:58.013848+0000","last_clean":"2024-04-21T08:39:58.013848+0000","last_became_active":"2024-04-21T08:37:31.458635+0000","last_became_peered":"2024-04-21T08:37:31.458635+0000","last_unstale":"2024-04-21T08:39:58.013848+0000","last_undegraded":"2024-04-21T08:39:58.013848+0000","last_fullsized":"2024-04-21T08:39:58.013848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:48:14.325571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013750+0000","last_change":"2024-04-21T08:37:35.603739+0000","last_active":"2024-04-21T08:39:58.013750+0000","last_peered":"2024-04-21T08:39:58.013750+0000","last_clean":"2024-04-21T08:39:58.013750+0000","last_became_active":"2024-04-21T08:37:35.603310+0000","last_became_peered":"2024-04-21T08:37:35.603310+0000","last_unstale":"2024-04-21T08:39:58.013750+0000","last_undegraded":"2024-04-21T08:39:58.013750+0000","last_fullsized":"2024-04-21T08:39:58.013750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:19.162520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009465+0000","last_change":"2024-04-21T08:37:30.418104+0000","last_active":"2024-04-21T08:39:58.009465+0000","last_peered":"2024-04-21T08:39:58.009465+0000","last_clean":"2024-04-21T08:39:58.009465+0000","last_became_active":"2024-04-21T08:37:30.417814+0000","last_became_peered":"2024-04-21T08:37:30.417814+0000","last_unstale":"2024-04-21T08:39:58.009465+0000","last_undegraded":"2024-04-21T08:39:58.009465+0000","last_fullsized":"2024-04-21T08:39:58.009465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:50:56.696513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318582+0000","last_change":"2024-04-21T08:37:31.451952+0000","last_active":"2024-04-21T08:39:58.318582+0000","last_peered":"2024-04-21T08:39:58.318582+0000","last_clean":"2024-04-21T08:39:58.318582+0000","last_became_active":"2024-04-21T08:37:31.451588+0000","last_became_peered":"2024-04-21T08:37:31.451588+0000","last_unstale":"2024-04-21T08:39:58.318582+0000","last_undegraded":"2024-04-21T08:39:58.318582+0000","last_fullsized":"2024-04-21T08:39:58.318582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:06.506769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009071+0000","last_change":"2024-04-21T08:37:35.760661+0000","last_active":"2024-04-21T08:39:58.009071+0000","last_peered":"2024-04-21T08:39:58.009071+0000","last_clean":"2024-04-21T08:39:58.009071+0000","last_became_active":"2024-04-21T08:37:35.759925+0000","last_became_peered":"2024-04-21T08:37:35.759925+0000","last_unstale":"2024-04-21T08:39:58.009071+0000","last_undegraded":"2024-04-21T08:39:58.009071+0000","last_fullsized":"2024-04-21T08:39:58.009071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:11:53.062251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007012+0000","last_change":"2024-04-21T08:37:30.416487+0000","last_active":"2024-04-21T08:39:58.007012+0000","last_peered":"2024-04-21T08:39:58.007012+0000","last_clean":"2024-04-21T08:39:58.007012+0000","last_became_active":"2024-04-21T08:37:30.416076+0000","last_became_peered":"2024-04-21T08:37:30.416076+0000","last_unstale":"2024-04-21T08:39:58.007012+0000","last_undegraded":"2024-04-21T08:39:58.007012+0000","last_fullsized":"2024-04-21T08:39:58.007012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:18:33.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017205+0000","last_change":"2024-04-21T08:37:31.448824+0000","last_active":"2024-04-21T08:39:58.017205+0000","last_peered":"2024-04-21T08:39:58.017205+0000","last_clean":"2024-04-21T08:39:58.017205+0000","last_became_active":"2024-04-21T08:37:31.448432+0000","last_became_peered":"2024-04-21T08:37:31.448432+0000","last_unstale":"2024-04-21T08:39:58.017205+0000","last_undegraded":"2024-04-21T08:39:58.017205+0000","last_fullsized":"2024-04-21T08:39:58.017205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:41.111272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318815+0000","last_change":"2024-04-21T08:37:35.604539+0000","last_active":"2024-04-21T08:39:58.318815+0000","last_peered":"2024-04-21T08:39:58.318815+0000","last_clean":"2024-04-21T08:39:58.318815+0000","last_became_active":"2024-04-21T08:37:35.604132+0000","last_became_peered":"2024-04-21T08:37:35.604132+0000","last_unstale":"2024-04-21T08:39:58.318815+0000","last_undegraded":"2024-04-21T08:39:58.318815+0000","last_fullsized":"2024-04-21T08:39:58.318815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:12.180730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905085+0000","last_change":"2024-04-21T08:37:30.418854+0000","last_active":"2024-04-21T08:39:57.905085+0000","last_peered":"2024-04-21T08:39:57.905085+0000","last_clean":"2024-04-21T08:39:57.905085+0000","last_became_active":"2024-04-21T08:37:30.418511+0000","last_became_peered":"2024-04-21T08:37:30.418511+0000","last_unstale":"2024-04-21T08:39:57.905085+0000","last_undegraded":"2024-04-21T08:39:57.905085+0000","last_fullsized":"2024-04-21T08:39:57.905085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:26.751924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.016918+0000","last_change":"2024-04-21T08:37:31.453365+0000","last_active":"2024-04-21T08:39:58.016918+0000","last_peered":"2024-04-21T08:39:58.016918+0000","last_clean":"2024-04-21T08:39:58.016918+0000","last_became_active":"2024-04-21T08:37:31.453042+0000","last_became_peered":"2024-04-21T08:37:31.453042+0000","last_unstale":"2024-04-21T08:39:58.016918+0000","last_undegraded":"2024-04-21T08:39:58.016918+0000","last_fullsized":"2024-04-21T08:39:58.016918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:49:46.648754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318806+0000","last_change":"2024-04-21T08:37:35.760626+0000","last_active":"2024-04-21T08:39:58.318806+0000","last_peered":"2024-04-21T08:39:58.318806+0000","last_clean":"2024-04-21T08:39:58.318806+0000","last_became_active":"2024-04-21T08:37:35.760125+0000","last_became_peered":"2024-04-21T08:37:35.760125+0000","last_unstale":"2024-04-21T08:39:58.318806+0000","last_undegraded":"2024-04-21T08:39:58.318806+0000","last_fullsized":"2024-04-21T08:39:58.318806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:58.366557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013932+0000","last_change":"2024-04-21T08:37:31.457654+0000","last_active":"2024-04-21T08:39:58.013932+0000","last_peered":"2024-04-21T08:39:58.013932+0000","last_clean":"2024-04-21T08:39:58.013932+0000","last_became_active":"2024-04-21T08:37:31.456950+0000","last_became_peered":"2024-04-21T08:37:31.456950+0000","last_unstale":"2024-04-21T08:39:58.013932+0000","last_undegraded":"2024-04-21T08:39:58.013932+0000","last_fullsized":"2024-04-21T08:39:58.013932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:02.863000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":173,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.902250+0000","last_change":"2024-04-21T08:37:30.422021+0000","last_active":"2024-04-21T08:40:07.902250+0000","last_peered":"2024-04-21T08:40:07.902250+0000","last_clean":"2024-04-21T08:40:07.902250+0000","last_became_active":"2024-04-21T08:37:30.421711+0000","last_became_peered":"2024-04-21T08:37:30.421711+0000","last_unstale":"2024-04-21T08:40:07.902250+0000","last_undegraded":"2024-04-21T08:40:07.902250+0000","last_fullsized":"2024-04-21T08:40:07.902250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:01.612205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":133,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905863+0000","last_change":"2024-04-21T08:37:35.598041+0000","last_active":"2024-04-21T08:39:57.905863+0000","last_peered":"2024-04-21T08:39:57.905863+0000","last_clean":"2024-04-21T08:39:57.905863+0000","last_became_active":"2024-04-21T08:37:35.597635+0000","last_became_peered":"2024-04-21T08:37:35.597635+0000","last_unstale":"2024-04-21T08:39:57.905863+0000","last_undegraded":"2024-04-21T08:39:57.905863+0000","last_fullsized":"2024-04-21T08:39:57.905863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:35:27.452340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903604+0000","last_change":"2024-04-21T08:37:31.450423+0000","last_active":"2024-04-21T08:39:57.903604+0000","last_peered":"2024-04-21T08:39:57.903604+0000","last_clean":"2024-04-21T08:39:57.903604+0000","last_became_active":"2024-04-21T08:37:31.450112+0000","last_became_peered":"2024-04-21T08:37:31.450112+0000","last_unstale":"2024-04-21T08:39:57.903604+0000","last_undegraded":"2024-04-21T08:39:57.903604+0000","last_fullsized":"2024-04-21T08:39:57.903604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:10:47.006769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151568+0000","last_change":"2024-04-21T08:37:30.416519+0000","last_active":"2024-04-21T08:39:58.151568+0000","last_peered":"2024-04-21T08:39:58.151568+0000","last_clean":"2024-04-21T08:39:58.151568+0000","last_became_active":"2024-04-21T08:37:30.416131+0000","last_became_peered":"2024-04-21T08:37:30.416131+0000","last_unstale":"2024-04-21T08:39:58.151568+0000","last_undegraded":"2024-04-21T08:39:58.151568+0000","last_fullsized":"2024-04-21T08:39:58.151568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:12.280847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":412,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:00.021477+0000","last_change":"2024-04-21T08:36:20.257759+0000","last_active":"2024-04-21T08:40:00.021477+0000","last_peered":"2024-04-21T08:40:00.021477+0000","last_clean":"2024-04-21T08:40:00.021477+0000","last_became_active":"2024-04-21T08:36:20.257313+0000","last_became_peered":"2024-04-21T08:36:20.257313+0000","last_unstale":"2024-04-21T08:40:00.021477+0000","last_undegraded":"2024-04-21T08:40:00.021477+0000","last_fullsized":"2024-04-21T08:40:00.021477+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_clean_scrub_stamp":"2024-04-21T08:32:56.546295+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:42:44.393447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014207+0000","last_change":"2024-04-21T08:37:35.761562+0000","last_active":"2024-04-21T08:39:58.014207+0000","last_peered":"2024-04-21T08:39:58.014207+0000","last_clean":"2024-04-21T08:39:58.014207+0000","last_became_active":"2024-04-21T08:37:35.760703+0000","last_became_peered":"2024-04-21T08:37:35.760703+0000","last_unstale":"2024-04-21T08:39:58.014207+0000","last_undegraded":"2024-04-21T08:39:58.014207+0000","last_fullsized":"2024-04-21T08:39:58.014207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:12:47.511117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318506+0000","last_change":"2024-04-21T08:39:31.678868+0000","last_active":"2024-04-21T08:39:58.318506+0000","last_peered":"2024-04-21T08:39:58.318506+0000","last_clean":"2024-04-21T08:39:58.318506+0000","last_became_active":"2024-04-21T08:39:31.678457+0000","last_became_peered":"2024-04-21T08:39:31.678457+0000","last_unstale":"2024-04-21T08:39:58.318506+0000","last_undegraded":"2024-04-21T08:39:58.318506+0000","last_fullsized":"2024-04-21T08:39:58.318506+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:49.918106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905042+0000","last_change":"2024-04-21T08:37:30.420560+0000","last_active":"2024-04-21T08:39:57.905042+0000","last_peered":"2024-04-21T08:39:57.905042+0000","last_clean":"2024-04-21T08:39:57.905042+0000","last_became_active":"2024-04-21T08:37:30.420223+0000","last_became_peered":"2024-04-21T08:37:30.420223+0000","last_unstale":"2024-04-21T08:39:57.905042+0000","last_undegraded":"2024-04-21T08:39:57.905042+0000","last_fullsized":"2024-04-21T08:39:57.905042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:43:55.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318381+0000","last_change":"2024-04-21T08:37:35.598294+0000","last_active":"2024-04-21T08:39:58.318381+0000","last_peered":"2024-04-21T08:39:58.318381+0000","last_clean":"2024-04-21T08:39:58.318381+0000","last_became_active":"2024-04-21T08:37:35.597883+0000","last_became_peered":"2024-04-21T08:37:35.597883+0000","last_unstale":"2024-04-21T08:39:58.318381+0000","last_undegraded":"2024-04-21T08:39:58.318381+0000","last_fullsized":"2024-04-21T08:39:58.318381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:24:05.144763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009321+0000","last_change":"2024-04-21T08:37:30.423182+0000","last_active":"2024-04-21T08:39:58.009321+0000","last_peered":"2024-04-21T08:39:58.009321+0000","last_clean":"2024-04-21T08:39:58.009321+0000","last_became_active":"2024-04-21T08:37:30.422792+0000","last_became_peered":"2024-04-21T08:37:30.422792+0000","last_unstale":"2024-04-21T08:39:58.009321+0000","last_undegraded":"2024-04-21T08:39:58.009321+0000","last_fullsized":"2024-04-21T08:39:58.009321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:44:00.727269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318924+0000","last_change":"2024-04-21T08:37:31.444013+0000","last_active":"2024-04-21T08:39:58.318924+0000","last_peered":"2024-04-21T08:39:58.318924+0000","last_clean":"2024-04-21T08:39:58.318924+0000","last_became_active":"2024-04-21T08:37:31.443684+0000","last_became_peered":"2024-04-21T08:37:31.443684+0000","last_unstale":"2024-04-21T08:39:58.318924+0000","last_undegraded":"2024-04-21T08:39:58.318924+0000","last_fullsized":"2024-04-21T08:39:58.318924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:00:35.400818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903525+0000","last_change":"2024-04-21T08:37:35.759817+0000","last_active":"2024-04-21T08:39:57.903525+0000","last_peered":"2024-04-21T08:39:57.903525+0000","last_clean":"2024-04-21T08:39:57.903525+0000","last_became_active":"2024-04-21T08:37:35.759396+0000","last_became_peered":"2024-04-21T08:37:35.759396+0000","last_unstale":"2024-04-21T08:39:57.903525+0000","last_undegraded":"2024-04-21T08:39:57.903525+0000","last_fullsized":"2024-04-21T08:39:57.903525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:20:36.170880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'98","reported_seq":224,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.333603+0000","last_change":"2024-04-21T08:37:30.414102+0000","last_active":"2024-04-21T08:40:07.333603+0000","last_peered":"2024-04-21T08:40:07.333603+0000","last_clean":"2024-04-21T08:40:07.333603+0000","last_became_active":"2024-04-21T08:37:30.413670+0000","last_became_peered":"2024-04-21T08:37:30.413670+0000","last_unstale":"2024-04-21T08:40:07.333603+0000","last_undegraded":"2024-04-21T08:40:07.333603+0000","last_fullsized":"2024-04-21T08:40:07.333603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:12:49.696516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":179,"num_write_kb":1285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904320+0000","last_change":"2024-04-21T08:37:31.457451+0000","last_active":"2024-04-21T08:39:57.904320+0000","last_peered":"2024-04-21T08:39:57.904320+0000","last_clean":"2024-04-21T08:39:57.904320+0000","last_became_active":"2024-04-21T08:37:31.457120+0000","last_became_peered":"2024-04-21T08:37:31.457120+0000","last_unstale":"2024-04-21T08:39:57.904320+0000","last_undegraded":"2024-04-21T08:39:57.904320+0000","last_fullsized":"2024-04-21T08:39:57.904320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:07:48.025843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904850+0000","last_change":"2024-04-21T08:37:35.600815+0000","last_active":"2024-04-21T08:39:57.904850+0000","last_peered":"2024-04-21T08:39:57.904850+0000","last_clean":"2024-04-21T08:39:57.904850+0000","last_became_active":"2024-04-21T08:37:35.600380+0000","last_became_peered":"2024-04-21T08:37:35.600380+0000","last_unstale":"2024-04-21T08:39:57.904850+0000","last_undegraded":"2024-04-21T08:39:57.904850+0000","last_fullsized":"2024-04-21T08:39:57.904850+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:19:33.710431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151729+0000","last_change":"2024-04-21T08:37:30.425592+0000","last_active":"2024-04-21T08:39:58.151729+0000","last_peered":"2024-04-21T08:39:58.151729+0000","last_clean":"2024-04-21T08:39:58.151729+0000","last_became_active":"2024-04-21T08:37:30.425274+0000","last_became_peered":"2024-04-21T08:37:30.425274+0000","last_unstale":"2024-04-21T08:39:58.151729+0000","last_undegraded":"2024-04-21T08:39:58.151729+0000","last_fullsized":"2024-04-21T08:39:58.151729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:27:29.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319672+0000","last_change":"2024-04-21T08:39:31.680028+0000","last_active":"2024-04-21T08:39:58.319672+0000","last_peered":"2024-04-21T08:39:58.319672+0000","last_clean":"2024-04-21T08:39:58.319672+0000","last_became_active":"2024-04-21T08:39:31.679162+0000","last_became_peered":"2024-04-21T08:39:31.679162+0000","last_unstale":"2024-04-21T08:39:58.319672+0000","last_undegraded":"2024-04-21T08:39:58.319672+0000","last_fullsized":"2024-04-21T08:39:58.319672+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:32.824411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151564+0000","last_change":"2024-04-21T08:38:30.713521+0000","last_active":"2024-04-21T08:39:58.151564+0000","last_peered":"2024-04-21T08:39:58.151564+0000","last_clean":"2024-04-21T08:39:58.151564+0000","last_became_active":"2024-04-21T08:38:30.713096+0000","last_became_peered":"2024-04-21T08:38:30.713096+0000","last_unstale":"2024-04-21T08:39:58.151564+0000","last_undegraded":"2024-04-21T08:39:58.151564+0000","last_fullsized":"2024-04-21T08:39:58.151564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:54:59.078776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153818+0000","last_change":"2024-04-21T08:37:30.416696+0000","last_active":"2024-04-21T08:39:58.153818+0000","last_peered":"2024-04-21T08:39:58.153818+0000","last_clean":"2024-04-21T08:39:58.153818+0000","last_became_active":"2024-04-21T08:37:30.415868+0000","last_became_peered":"2024-04-21T08:37:30.415868+0000","last_unstale":"2024-04-21T08:39:58.153818+0000","last_undegraded":"2024-04-21T08:39:58.153818+0000","last_fullsized":"2024-04-21T08:39:58.153818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:48:08.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008945+0000","last_change":"2024-04-21T08:37:31.453861+0000","last_active":"2024-04-21T08:39:58.008945+0000","last_peered":"2024-04-21T08:39:58.008945+0000","last_clean":"2024-04-21T08:39:58.008945+0000","last_became_active":"2024-04-21T08:37:31.453562+0000","last_became_peered":"2024-04-21T08:37:31.453562+0000","last_unstale":"2024-04-21T08:39:58.008945+0000","last_undegraded":"2024-04-21T08:39:58.008945+0000","last_fullsized":"2024-04-21T08:39:58.008945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:17:50.513917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006977+0000","last_change":"2024-04-21T08:37:35.609217+0000","last_active":"2024-04-21T08:39:58.006977+0000","last_peered":"2024-04-21T08:39:58.006977+0000","last_clean":"2024-04-21T08:39:58.006977+0000","last_became_active":"2024-04-21T08:37:35.608786+0000","last_became_peered":"2024-04-21T08:37:35.608786+0000","last_unstale":"2024-04-21T08:39:58.006977+0000","last_undegraded":"2024-04-21T08:39:58.006977+0000","last_fullsized":"2024-04-21T08:39:58.006977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:45:50.945845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903818+0000","last_change":"2024-04-21T08:37:30.417156+0000","last_active":"2024-04-21T08:39:57.903818+0000","last_peered":"2024-04-21T08:39:57.903818+0000","last_clean":"2024-04-21T08:39:57.903818+0000","last_became_active":"2024-04-21T08:37:30.415717+0000","last_became_peered":"2024-04-21T08:37:30.415717+0000","last_unstale":"2024-04-21T08:39:57.903818+0000","last_undegraded":"2024-04-21T08:39:57.903818+0000","last_fullsized":"2024-04-21T08:39:57.903818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:37.819082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318260+0000","last_change":"2024-04-21T08:37:31.444948+0000","last_active":"2024-04-21T08:39:58.318260+0000","last_peered":"2024-04-21T08:39:58.318260+0000","last_clean":"2024-04-21T08:39:58.318260+0000","last_became_active":"2024-04-21T08:37:31.444604+0000","last_became_peered":"2024-04-21T08:37:31.444604+0000","last_unstale":"2024-04-21T08:39:58.318260+0000","last_undegraded":"2024-04-21T08:39:58.318260+0000","last_fullsized":"2024-04-21T08:39:58.318260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.820503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151386+0000","last_change":"2024-04-21T08:37:35.611541+0000","last_active":"2024-04-21T08:39:58.151386+0000","last_peered":"2024-04-21T08:39:58.151386+0000","last_clean":"2024-04-21T08:39:58.151386+0000","last_became_active":"2024-04-21T08:37:35.611136+0000","last_became_peered":"2024-04-21T08:37:35.611136+0000","last_unstale":"2024-04-21T08:39:58.151386+0000","last_undegraded":"2024-04-21T08:39:58.151386+0000","last_fullsized":"2024-04-21T08:39:58.151386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:28.254529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905028+0000","last_change":"2024-04-21T08:37:30.414216+0000","last_active":"2024-04-21T08:39:57.905028+0000","last_peered":"2024-04-21T08:39:57.905028+0000","last_clean":"2024-04-21T08:39:57.905028+0000","last_became_active":"2024-04-21T08:37:30.413890+0000","last_became_peered":"2024-04-21T08:37:30.413890+0000","last_unstale":"2024-04-21T08:39:57.905028+0000","last_undegraded":"2024-04-21T08:39:57.905028+0000","last_fullsized":"2024-04-21T08:39:57.905028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:16.972618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903800+0000","last_change":"2024-04-21T08:37:31.449307+0000","last_active":"2024-04-21T08:39:57.903800+0000","last_peered":"2024-04-21T08:39:57.903800+0000","last_clean":"2024-04-21T08:39:57.903800+0000","last_became_active":"2024-04-21T08:37:31.448980+0000","last_became_peered":"2024-04-21T08:37:31.448980+0000","last_unstale":"2024-04-21T08:39:57.903800+0000","last_undegraded":"2024-04-21T08:39:57.903800+0000","last_fullsized":"2024-04-21T08:39:57.903800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:30:31.939274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151476+0000","last_change":"2024-04-21T08:37:35.760616+0000","last_active":"2024-04-21T08:39:58.151476+0000","last_peered":"2024-04-21T08:39:58.151476+0000","last_clean":"2024-04-21T08:39:58.151476+0000","last_became_active":"2024-04-21T08:37:35.760131+0000","last_became_peered":"2024-04-21T08:37:35.760131+0000","last_unstale":"2024-04-21T08:39:58.151476+0000","last_undegraded":"2024-04-21T08:39:58.151476+0000","last_fullsized":"2024-04-21T08:39:58.151476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:51.510259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009114+0000","last_change":"2024-04-21T08:38:30.682791+0000","last_active":"2024-04-21T08:39:58.009114+0000","last_peered":"2024-04-21T08:39:58.009114+0000","last_clean":"2024-04-21T08:39:58.009114+0000","last_became_active":"2024-04-21T08:38:30.671977+0000","last_became_peered":"2024-04-21T08:38:30.671977+0000","last_unstale":"2024-04-21T08:39:58.009114+0000","last_undegraded":"2024-04-21T08:39:58.009114+0000","last_fullsized":"2024-04-21T08:39:58.009114+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:55:15.899399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905569+0000","last_change":"2024-04-21T08:37:31.447234+0000","last_active":"2024-04-21T08:39:57.905569+0000","last_peered":"2024-04-21T08:39:57.905569+0000","last_clean":"2024-04-21T08:39:57.905569+0000","last_became_active":"2024-04-21T08:37:31.446876+0000","last_became_peered":"2024-04-21T08:37:31.446876+0000","last_unstale":"2024-04-21T08:39:57.905569+0000","last_undegraded":"2024-04-21T08:39:57.905569+0000","last_fullsized":"2024-04-21T08:39:57.905569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:51.784126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153981+0000","last_change":"2024-04-21T08:37:35.598577+0000","last_active":"2024-04-21T08:39:58.153981+0000","last_peered":"2024-04-21T08:39:58.153981+0000","last_clean":"2024-04-21T08:39:58.153981+0000","last_became_active":"2024-04-21T08:37:35.598119+0000","last_became_peered":"2024-04-21T08:37:35.598119+0000","last_unstale":"2024-04-21T08:39:58.153981+0000","last_undegraded":"2024-04-21T08:39:58.153981+0000","last_fullsized":"2024-04-21T08:39:58.153981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:41.986674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905332+0000","last_change":"2024-04-21T08:37:30.425737+0000","last_active":"2024-04-21T08:39:57.905332+0000","last_peered":"2024-04-21T08:39:57.905332+0000","last_clean":"2024-04-21T08:39:57.905332+0000","last_became_active":"2024-04-21T08:37:30.425180+0000","last_became_peered":"2024-04-21T08:37:30.425180+0000","last_unstale":"2024-04-21T08:39:57.905332+0000","last_undegraded":"2024-04-21T08:39:57.905332+0000","last_fullsized":"2024-04-21T08:39:57.905332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:08.646106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.906011+0000","last_change":"2024-04-21T08:37:31.451505+0000","last_active":"2024-04-21T08:39:57.906011+0000","last_peered":"2024-04-21T08:39:57.906011+0000","last_clean":"2024-04-21T08:39:57.906011+0000","last_became_active":"2024-04-21T08:37:31.451146+0000","last_became_peered":"2024-04-21T08:37:31.451146+0000","last_unstale":"2024-04-21T08:39:57.906011+0000","last_undegraded":"2024-04-21T08:39:57.906011+0000","last_fullsized":"2024-04-21T08:39:57.906011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:27:01.793910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319583+0000","last_change":"2024-04-21T08:37:35.608995+0000","last_active":"2024-04-21T08:39:58.319583+0000","last_peered":"2024-04-21T08:39:58.319583+0000","last_clean":"2024-04-21T08:39:58.319583+0000","last_became_active":"2024-04-21T08:37:35.608587+0000","last_became_peered":"2024-04-21T08:37:35.608587+0000","last_unstale":"2024-04-21T08:39:58.319583+0000","last_undegraded":"2024-04-21T08:39:58.319583+0000","last_fullsized":"2024-04-21T08:39:58.319583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.829418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'65","reported_seq":130,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.001710+0000","last_change":"2024-04-21T08:37:30.419026+0000","last_active":"2024-04-21T08:40:07.001710+0000","last_peered":"2024-04-21T08:40:07.001710+0000","last_clean":"2024-04-21T08:40:07.001710+0000","last_became_active":"2024-04-21T08:37:30.418706+0000","last_became_peered":"2024-04-21T08:37:30.418706+0000","last_unstale":"2024-04-21T08:40:07.001710+0000","last_undegraded":"2024-04-21T08:40:07.001710+0000","last_fullsized":"2024-04-21T08:40:07.001710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:24.441918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905192+0000","last_change":"2024-04-21T08:37:31.446561+0000","last_active":"2024-04-21T08:39:57.905192+0000","last_peered":"2024-04-21T08:39:57.905192+0000","last_clean":"2024-04-21T08:39:57.905192+0000","last_became_active":"2024-04-21T08:37:31.446162+0000","last_became_peered":"2024-04-21T08:37:31.446162+0000","last_unstale":"2024-04-21T08:39:57.905192+0000","last_undegraded":"2024-04-21T08:39:57.905192+0000","last_fullsized":"2024-04-21T08:39:57.905192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:38:24.684537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013588+0000","last_change":"2024-04-21T08:37:35.598748+0000","last_active":"2024-04-21T08:39:58.013588+0000","last_peered":"2024-04-21T08:39:58.013588+0000","last_clean":"2024-04-21T08:39:58.013588+0000","last_became_active":"2024-04-21T08:37:35.598353+0000","last_became_peered":"2024-04-21T08:37:35.598353+0000","last_unstale":"2024-04-21T08:39:58.013588+0000","last_undegraded":"2024-04-21T08:39:58.013588+0000","last_fullsized":"2024-04-21T08:39:58.013588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:47:36.519694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319054+0000","last_change":"2024-04-21T08:38:30.691292+0000","last_active":"2024-04-21T08:39:58.319054+0000","last_peered":"2024-04-21T08:39:58.319054+0000","last_clean":"2024-04-21T08:39:58.319054+0000","last_became_active":"2024-04-21T08:38:30.676580+0000","last_became_peered":"2024-04-21T08:38:30.676580+0000","last_unstale":"2024-04-21T08:39:58.319054+0000","last_undegraded":"2024-04-21T08:39:58.319054+0000","last_fullsized":"2024-04-21T08:39:58.319054+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:43:53.150369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905478+0000","last_change":"2024-04-21T08:37:31.452367+0000","last_active":"2024-04-21T08:39:57.905478+0000","last_peered":"2024-04-21T08:39:57.905478+0000","last_clean":"2024-04-21T08:39:57.905478+0000","last_became_active":"2024-04-21T08:37:31.452031+0000","last_became_peered":"2024-04-21T08:37:31.452031+0000","last_unstale":"2024-04-21T08:39:57.905478+0000","last_undegraded":"2024-04-21T08:39:57.905478+0000","last_fullsized":"2024-04-21T08:39:57.905478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:58.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318378+0000","last_change":"2024-04-21T08:37:31.451222+0000","last_active":"2024-04-21T08:39:58.318378+0000","last_peered":"2024-04-21T08:39:58.318378+0000","last_clean":"2024-04-21T08:39:58.318378+0000","last_became_active":"2024-04-21T08:37:31.450869+0000","last_became_peered":"2024-04-21T08:37:31.450869+0000","last_unstale":"2024-04-21T08:39:58.318378+0000","last_undegraded":"2024-04-21T08:39:58.318378+0000","last_fullsized":"2024-04-21T08:39:58.318378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:17.051330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905137+0000","last_change":"2024-04-21T08:37:31.453415+0000","last_active":"2024-04-21T08:39:57.905137+0000","last_peered":"2024-04-21T08:39:57.905137+0000","last_clean":"2024-04-21T08:39:57.905137+0000","last_became_active":"2024-04-21T08:37:31.453089+0000","last_became_peered":"2024-04-21T08:37:31.453089+0000","last_unstale":"2024-04-21T08:39:57.905137+0000","last_undegraded":"2024-04-21T08:39:57.905137+0000","last_fullsized":"2024-04-21T08:39:57.905137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:25:44.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904904+0000","last_change":"2024-04-21T08:37:31.446707+0000","last_active":"2024-04-21T08:39:57.904904+0000","last_peered":"2024-04-21T08:39:57.904904+0000","last_clean":"2024-04-21T08:39:57.904904+0000","last_became_active":"2024-04-21T08:37:31.446340+0000","last_became_peered":"2024-04-21T08:37:31.446340+0000","last_unstale":"2024-04-21T08:39:57.904904+0000","last_undegraded":"2024-04-21T08:39:57.904904+0000","last_fullsized":"2024-04-21T08:39:57.904904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:13.649607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319500+0000","last_change":"2024-04-21T08:37:31.455532+0000","last_active":"2024-04-21T08:39:58.319500+0000","last_peered":"2024-04-21T08:39:58.319500+0000","last_clean":"2024-04-21T08:39:58.319500+0000","last_became_active":"2024-04-21T08:37:31.453547+0000","last_became_peered":"2024-04-21T08:37:31.453547+0000","last_unstale":"2024-04-21T08:39:58.319500+0000","last_undegraded":"2024-04-21T08:39:58.319500+0000","last_fullsized":"2024-04-21T08:39:58.319500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:39:47.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318944+0000","last_change":"2024-04-21T08:39:31.679882+0000","last_active":"2024-04-21T08:39:58.318944+0000","last_peered":"2024-04-21T08:39:58.318944+0000","last_clean":"2024-04-21T08:39:58.318944+0000","last_became_active":"2024-04-21T08:39:31.679440+0000","last_became_peered":"2024-04-21T08:39:31.679440+0000","last_unstale":"2024-04-21T08:39:58.318944+0000","last_undegraded":"2024-04-21T08:39:58.318944+0000","last_fullsized":"2024-04-21T08:39:58.318944+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:57.679038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318685+0000","last_change":"2024-04-21T08:37:31.451806+0000","last_active":"2024-04-21T08:39:58.318685+0000","last_peered":"2024-04-21T08:39:58.318685+0000","last_clean":"2024-04-21T08:39:58.318685+0000","last_became_active":"2024-04-21T08:37:31.451442+0000","last_became_peered":"2024-04-21T08:37:31.451442+0000","last_unstale":"2024-04-21T08:39:58.318685+0000","last_undegraded":"2024-04-21T08:39:58.318685+0000","last_fullsized":"2024-04-21T08:39:58.318685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:07.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905544+0000","last_change":"2024-04-21T08:37:31.444440+0000","last_active":"2024-04-21T08:39:57.905544+0000","last_peered":"2024-04-21T08:39:57.905544+0000","last_clean":"2024-04-21T08:39:57.905544+0000","last_became_active":"2024-04-21T08:37:31.444079+0000","last_became_peered":"2024-04-21T08:37:31.444079+0000","last_unstale":"2024-04-21T08:39:57.905544+0000","last_undegraded":"2024-04-21T08:39:57.905544+0000","last_fullsized":"2024-04-21T08:39:57.905544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:13.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904413+0000","last_change":"2024-04-21T08:37:31.453698+0000","last_active":"2024-04-21T08:39:57.904413+0000","last_peered":"2024-04-21T08:39:57.904413+0000","last_clean":"2024-04-21T08:39:57.904413+0000","last_became_active":"2024-04-21T08:37:31.453379+0000","last_became_peered":"2024-04-21T08:37:31.453379+0000","last_unstale":"2024-04-21T08:39:57.904413+0000","last_undegraded":"2024-04-21T08:39:57.904413+0000","last_fullsized":"2024-04-21T08:39:57.904413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:46:29.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013777+0000","last_change":"2024-04-21T08:37:31.444480+0000","last_active":"2024-04-21T08:39:58.013777+0000","last_peered":"2024-04-21T08:39:58.013777+0000","last_clean":"2024-04-21T08:39:58.013777+0000","last_became_active":"2024-04-21T08:37:31.444030+0000","last_became_peered":"2024-04-21T08:37:31.444030+0000","last_unstale":"2024-04-21T08:39:58.013777+0000","last_undegraded":"2024-04-21T08:39:58.013777+0000","last_fullsized":"2024-04-21T08:39:58.013777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:15:05.668885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903935+0000","last_change":"2024-04-21T08:37:31.456602+0000","last_active":"2024-04-21T08:39:57.903935+0000","last_peered":"2024-04-21T08:39:57.903935+0000","last_clean":"2024-04-21T08:39:57.903935+0000","last_became_active":"2024-04-21T08:37:31.456286+0000","last_became_peered":"2024-04-21T08:37:31.456286+0000","last_unstale":"2024-04-21T08:39:57.903935+0000","last_undegraded":"2024-04-21T08:39:57.903935+0000","last_fullsized":"2024-04-21T08:39:57.903935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:48:12.462037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154176+0000","last_change":"2024-04-21T08:37:31.447008+0000","last_active":"2024-04-21T08:39:58.154176+0000","last_peered":"2024-04-21T08:39:58.154176+0000","last_clean":"2024-04-21T08:39:58.154176+0000","last_became_active":"2024-04-21T08:37:31.445192+0000","last_became_peered":"2024-04-21T08:37:31.445192+0000","last_unstale":"2024-04-21T08:39:58.154176+0000","last_undegraded":"2024-04-21T08:39:58.154176+0000","last_fullsized":"2024-04-21T08:39:58.154176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:08.025647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318698+0000","last_change":"2024-04-21T08:37:31.443288+0000","last_active":"2024-04-21T08:39:58.318698+0000","last_peered":"2024-04-21T08:39:58.318698+0000","last_clean":"2024-04-21T08:39:58.318698+0000","last_became_active":"2024-04-21T08:37:31.442586+0000","last_became_peered":"2024-04-21T08:37:31.442586+0000","last_unstale":"2024-04-21T08:39:58.318698+0000","last_undegraded":"2024-04-21T08:39:58.318698+0000","last_fullsized":"2024-04-21T08:39:58.318698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:01.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318501+0000","last_change":"2024-04-21T08:37:31.453899+0000","last_active":"2024-04-21T08:39:58.318501+0000","last_peered":"2024-04-21T08:39:58.318501+0000","last_clean":"2024-04-21T08:39:58.318501+0000","last_became_active":"2024-04-21T08:37:31.453527+0000","last_became_peered":"2024-04-21T08:37:31.453527+0000","last_unstale":"2024-04-21T08:39:58.318501+0000","last_undegraded":"2024-04-21T08:39:58.318501+0000","last_fullsized":"2024-04-21T08:39:58.318501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:23:18.429618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108246,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":800441,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1183,"num_read_kb":813,"num_write":424,"num_write_kb":2366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2401323,"data_compressed":108456,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383724,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":529807,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32384,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712768,"statfs":{"total":95995035648,"available":95961874432,"internally_reserved":0,"allocated":1228800,"data_stored":1520299,"data_compressed":82643,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":927355,"data_compressed":48987,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":922597,"data_compressed":48987,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1127885,"data_compressed":65780,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1028096,"data_stored":525219,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":598690,"data_compressed":31359,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1052672,"data_stored":526607,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1126400,"data_stored":599849,"data_compressed":31359,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1246861,"data_compressed":64189,"data_compressed_allocated":86016,"data_compressed_original":782336,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":531882,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1150976,"data_stored":1449692,"data_compressed":80982,"data_compressed_allocated":106496,"data_compressed_original":978944,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":73561,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":74720,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330099,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393372,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":404802,"data_compressed":16863,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:40:11.146 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-21T08:40:11.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.282+0000 7f570ec5b640 1 -- 172.21.15.94:0/3536837416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5708102b40 msgr2=0x7f5708106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.282+0000 7f570ec5b640 1 --2- 172.21.15.94:0/3536837416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5708102b40 0x7f5708106b80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009900 tx=0x7f56f402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.282+0000 7f570ec5b640 1 -- 172.21.15.94:0/3536837416 shutdown_connections 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.282+0000 7f570ec5b640 1 --2- 172.21.15.94:0/3536837416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5708107260 0x7f570810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.282+0000 7f570ec5b640 1 --2- 172.21.15.94:0/3536837416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5708102b40 0x7f5708106b80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.282+0000 7f570ec5b640 1 --2- 172.21.15.94:0/3536837416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5708102190 0x7f5708102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.282+0000 7f570ec5b640 1 -- 172.21.15.94:0/3536837416 >> 172.21.15.94:0/3536837416 conn(0x7f57080793e0 msgr2=0x7f57080797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.282+0000 7f570ec5b640 1 -- 172.21.15.94:0/3536837416 shutdown_connections 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.282+0000 7f570ec5b640 1 -- 172.21.15.94:0/3536837416 wait complete. 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570ec5b640 1 Processor -- start 2024-04-21T08:40:11.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570ec5b640 1 -- start start 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570ec5b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5708102190 0x7f570819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570ec5b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5708107260 0x7f570819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570ec5b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57081985d0 0x7f5708198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570ec5b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5708112250 con 0x7f57081985d0 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570ec5b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f57081120d0 con 0x7f5708102190 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570ec5b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f57081123d0 con 0x7f5708107260 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570d1f8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57081985d0 0x7f5708198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570d1f8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57081985d0 0x7f5708198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37010/0 (socket says 172.21.15.94:37010) 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570d1f8640 1 -- 172.21.15.94:0/2005854818 learned_addr learned my addr 172.21.15.94:0/2005854818 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.283+0000 7f570c9f7640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5708102190 0x7f570819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f5707fff640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5708107260 0x7f570819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f570d1f8640 1 -- 172.21.15.94:0/2005854818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5708107260 msgr2=0x7f570819e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f570d1f8640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5708107260 0x7f570819e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f570d1f8640 1 -- 172.21.15.94:0/2005854818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5708102190 msgr2=0x7f570819e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f570d1f8640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5708102190 0x7f570819e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f570d1f8640 1 -- 172.21.15.94:0/2005854818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57081990f0 con 0x7f57081985d0 2024-04-21T08:40:11.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f570d1f8640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57081985d0 0x7f5708198a80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f570000eaa0 tx=0x7f570000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:11.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f5705ffb640 1 -- 172.21.15.94:0/2005854818 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700004d20 con 0x7f57081985d0 2024-04-21T08:40:11.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f5705ffb640 1 -- 172.21.15.94:0/2005854818 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5700005020 con 0x7f57081985d0 2024-04-21T08:40:11.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f570ec5b640 1 -- 172.21.15.94:0/2005854818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57081993e0 con 0x7f57081985d0 2024-04-21T08:40:11.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f5705ffb640 1 -- 172.21.15.94:0/2005854818 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57000116b0 con 0x7f57081985d0 2024-04-21T08:40:11.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.284+0000 7f570ec5b640 1 -- 172.21.15.94:0/2005854818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57081a55e0 con 0x7f57081985d0 2024-04-21T08:40:11.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.285+0000 7f570ec5b640 1 -- 172.21.15.94:0/2005854818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5708069230 con 0x7f57081985d0 2024-04-21T08:40:11.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.286+0000 7f5705ffb640 1 -- 172.21.15.94:0/2005854818 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f57000051d0 con 0x7f57081985d0 2024-04-21T08:40:11.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.287+0000 7f5705ffb640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f56f80a3e40 0x7f56f80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.287+0000 7f570c9f7640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f56f80a3e40 0x7f56f80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.287+0000 7f5705ffb640 1 -- 172.21.15.94:0/2005854818 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f57000d1120 con 0x7f57081985d0 2024-04-21T08:40:11.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.287+0000 7f570c9f7640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f56f80a3e40 0x7f56f80a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009990 tx=0x7f56fc009110 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:11.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.291+0000 7f5705ffb640 1 -- 172.21.15.94:0/2005854818 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5700016030 con 0x7f57081985d0 2024-04-21T08:40:11.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.469+0000 7f570ec5b640 1 -- 172.21.15.94:0/2005854818 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f570810a500 con 0x7f56f80a3e40 2024-04-21T08:40:11.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.478+0000 7f5705ffb640 1 -- 172.21.15.94:0/2005854818 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339720 (secure 0 0 0) 0x7f56fc00e850 con 0x7f56f80a3e40 2024-04-21T08:40:11.482 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:11.484 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-21T08:40:11.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 -- 172.21.15.94:0/2005854818 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f56f80a3e40 msgr2=0x7f56f80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f56f80a3e40 0x7f56f80a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009990 tx=0x7f56fc009110 comp rx=0 tx=0).stop 2024-04-21T08:40:11.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 -- 172.21.15.94:0/2005854818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57081985d0 msgr2=0x7f5708198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57081985d0 0x7f5708198a80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f570000eaa0 tx=0x7f570000eea0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 -- 172.21.15.94:0/2005854818 shutdown_connections 2024-04-21T08:40:11.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f56f80a3e40 0x7f56f80a6300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5708107260 0x7f570819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57081985d0 0x7f5708198a80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 --2- 172.21.15.94:0/2005854818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5708102190 0x7f570819e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 -- 172.21.15.94:0/2005854818 >> 172.21.15.94:0/2005854818 conn(0x7f57080793e0 msgr2=0x7f57080fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:11.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 -- 172.21.15.94:0/2005854818 shutdown_connections 2024-04-21T08:40:11.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.481+0000 7f570ec5b640 1 -- 172.21.15.94:0/2005854818 wait complete. 2024-04-21T08:40:11.491 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:11 smithi094 ceph-mon[27988]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-21T08:40:11.491 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:11 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2883492950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:11.491 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:11 smithi094 ceph-mon[27988]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:11.492 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-21T08:40:09.917339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1391043,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1420,"num_read_kb":1143,"num_write":627,"num_write_kb":5306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378652,"kb_used_data":12812,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563172,"statfs":{"total":1151940427776,"available":1151552688128,"internally_reserved":0,"allocated":13119488,"data_stored":10506743,"data_compressed":573078,"data_compressed_allocated":774144,"data_compressed_original":4878336,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":200325,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":420,"num_read_kb":479,"num_write":155,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.988315"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318569+0000","last_change":"2024-04-21T08:37:31.448345+0000","last_active":"2024-04-21T08:39:58.318569+0000","last_peered":"2024-04-21T08:39:58.318569+0000","last_clean":"2024-04-21T08:39:58.318569+0000","last_became_active":"2024-04-21T08:37:31.447878+0000","last_became_peered":"2024-04-21T08:37:31.447878+0000","last_unstale":"2024-04-21T08:39:58.318569+0000","last_undegraded":"2024-04-21T08:39:58.318569+0000","last_fullsized":"2024-04-21T08:39:58.318569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:45.589165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319782+0000","last_change":"2024-04-21T08:37:31.452895+0000","last_active":"2024-04-21T08:39:58.319782+0000","last_peered":"2024-04-21T08:39:58.319782+0000","last_clean":"2024-04-21T08:39:58.319782+0000","last_became_active":"2024-04-21T08:37:31.452550+0000","last_became_peered":"2024-04-21T08:37:31.452550+0000","last_unstale":"2024-04-21T08:39:58.319782+0000","last_undegraded":"2024-04-21T08:39:58.319782+0000","last_fullsized":"2024-04-21T08:39:58.319782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:02.497802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319443+0000","last_change":"2024-04-21T08:37:31.447552+0000","last_active":"2024-04-21T08:39:58.319443+0000","last_peered":"2024-04-21T08:39:58.319443+0000","last_clean":"2024-04-21T08:39:58.319443+0000","last_became_active":"2024-04-21T08:37:31.446044+0000","last_became_peered":"2024-04-21T08:37:31.446044+0000","last_unstale":"2024-04-21T08:39:58.319443+0000","last_undegraded":"2024-04-21T08:39:58.319443+0000","last_fullsized":"2024-04-21T08:39:58.319443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:22.801977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151531+0000","last_change":"2024-04-21T08:37:31.452400+0000","last_active":"2024-04-21T08:39:58.151531+0000","last_peered":"2024-04-21T08:39:58.151531+0000","last_clean":"2024-04-21T08:39:58.151531+0000","last_became_active":"2024-04-21T08:37:31.451462+0000","last_became_peered":"2024-04-21T08:37:31.451462+0000","last_unstale":"2024-04-21T08:39:58.151531+0000","last_undegraded":"2024-04-21T08:39:58.151531+0000","last_fullsized":"2024-04-21T08:39:58.151531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:22.463481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009087+0000","last_change":"2024-04-21T08:37:31.448942+0000","last_active":"2024-04-21T08:39:58.009087+0000","last_peered":"2024-04-21T08:39:58.009087+0000","last_clean":"2024-04-21T08:39:58.009087+0000","last_became_active":"2024-04-21T08:37:31.448526+0000","last_became_peered":"2024-04-21T08:37:31.448526+0000","last_unstale":"2024-04-21T08:39:58.009087+0000","last_undegraded":"2024-04-21T08:39:58.009087+0000","last_fullsized":"2024-04-21T08:39:58.009087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:29.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017166+0000","last_change":"2024-04-21T08:37:31.452165+0000","last_active":"2024-04-21T08:39:58.017166+0000","last_peered":"2024-04-21T08:39:58.017166+0000","last_clean":"2024-04-21T08:39:58.017166+0000","last_became_active":"2024-04-21T08:37:31.451802+0000","last_became_peered":"2024-04-21T08:37:31.451802+0000","last_unstale":"2024-04-21T08:39:58.017166+0000","last_undegraded":"2024-04-21T08:39:58.017166+0000","last_fullsized":"2024-04-21T08:39:58.017166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:42.210544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905010+0000","last_change":"2024-04-21T08:39:31.681660+0000","last_active":"2024-04-21T08:39:57.905010+0000","last_peered":"2024-04-21T08:39:57.905010+0000","last_clean":"2024-04-21T08:39:57.905010+0000","last_became_active":"2024-04-21T08:39:31.681203+0000","last_became_peered":"2024-04-21T08:39:31.681203+0000","last_unstale":"2024-04-21T08:39:57.905010+0000","last_undegraded":"2024-04-21T08:39:57.905010+0000","last_fullsized":"2024-04-21T08:39:57.905010+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:54.120862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903472+0000","last_change":"2024-04-21T08:37:31.448257+0000","last_active":"2024-04-21T08:39:57.903472+0000","last_peered":"2024-04-21T08:39:57.903472+0000","last_clean":"2024-04-21T08:39:57.903472+0000","last_became_active":"2024-04-21T08:37:31.447845+0000","last_became_peered":"2024-04-21T08:37:31.447845+0000","last_unstale":"2024-04-21T08:39:57.903472+0000","last_undegraded":"2024-04-21T08:39:57.903472+0000","last_fullsized":"2024-04-21T08:39:57.903472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:19.956699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154785+0000","last_change":"2024-04-21T08:37:31.443022+0000","last_active":"2024-04-21T08:39:58.154785+0000","last_peered":"2024-04-21T08:39:58.154785+0000","last_clean":"2024-04-21T08:39:58.154785+0000","last_became_active":"2024-04-21T08:37:31.442645+0000","last_became_peered":"2024-04-21T08:37:31.442645+0000","last_unstale":"2024-04-21T08:39:58.154785+0000","last_undegraded":"2024-04-21T08:39:58.154785+0000","last_fullsized":"2024-04-21T08:39:58.154785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:18:10.861281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017434+0000","last_change":"2024-04-21T08:37:31.453427+0000","last_active":"2024-04-21T08:39:58.017434+0000","last_peered":"2024-04-21T08:39:58.017434+0000","last_clean":"2024-04-21T08:39:58.017434+0000","last_became_active":"2024-04-21T08:37:31.453096+0000","last_became_peered":"2024-04-21T08:37:31.453096+0000","last_unstale":"2024-04-21T08:39:58.017434+0000","last_undegraded":"2024-04-21T08:39:58.017434+0000","last_fullsized":"2024-04-21T08:39:58.017434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:49.450351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905929+0000","last_change":"2024-04-21T08:37:31.449908+0000","last_active":"2024-04-21T08:39:57.905929+0000","last_peered":"2024-04-21T08:39:57.905929+0000","last_clean":"2024-04-21T08:39:57.905929+0000","last_became_active":"2024-04-21T08:37:31.449558+0000","last_became_peered":"2024-04-21T08:37:31.449558+0000","last_unstale":"2024-04-21T08:39:57.905929+0000","last_undegraded":"2024-04-21T08:39:57.905929+0000","last_fullsized":"2024-04-21T08:39:57.905929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:21.538689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151539+0000","last_change":"2024-04-21T08:37:31.452973+0000","last_active":"2024-04-21T08:39:58.151539+0000","last_peered":"2024-04-21T08:39:58.151539+0000","last_clean":"2024-04-21T08:39:58.151539+0000","last_became_active":"2024-04-21T08:37:31.452586+0000","last_became_peered":"2024-04-21T08:37:31.452586+0000","last_unstale":"2024-04-21T08:39:58.151539+0000","last_undegraded":"2024-04-21T08:39:58.151539+0000","last_fullsized":"2024-04-21T08:39:58.151539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:09:11.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319144+0000","last_change":"2024-04-21T08:37:31.445801+0000","last_active":"2024-04-21T08:39:58.319144+0000","last_peered":"2024-04-21T08:39:58.319144+0000","last_clean":"2024-04-21T08:39:58.319144+0000","last_became_active":"2024-04-21T08:37:31.445478+0000","last_became_peered":"2024-04-21T08:37:31.445478+0000","last_unstale":"2024-04-21T08:39:58.319144+0000","last_undegraded":"2024-04-21T08:39:58.319144+0000","last_fullsized":"2024-04-21T08:39:58.319144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:33.352739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319201+0000","last_change":"2024-04-21T08:37:31.450735+0000","last_active":"2024-04-21T08:39:58.319201+0000","last_peered":"2024-04-21T08:39:58.319201+0000","last_clean":"2024-04-21T08:39:58.319201+0000","last_became_active":"2024-04-21T08:37:31.450384+0000","last_became_peered":"2024-04-21T08:37:31.450384+0000","last_unstale":"2024-04-21T08:39:58.319201+0000","last_undegraded":"2024-04-21T08:39:58.319201+0000","last_fullsized":"2024-04-21T08:39:58.319201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:45.950489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319354+0000","last_change":"2024-04-21T08:37:31.454690+0000","last_active":"2024-04-21T08:39:58.319354+0000","last_peered":"2024-04-21T08:39:58.319354+0000","last_clean":"2024-04-21T08:39:58.319354+0000","last_became_active":"2024-04-21T08:37:31.454330+0000","last_became_peered":"2024-04-21T08:37:31.454330+0000","last_unstale":"2024-04-21T08:39:58.319354+0000","last_undegraded":"2024-04-21T08:39:58.319354+0000","last_fullsized":"2024-04-21T08:39:58.319354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:52.644317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153871+0000","last_change":"2024-04-21T08:37:31.441389+0000","last_active":"2024-04-21T08:39:58.153871+0000","last_peered":"2024-04-21T08:39:58.153871+0000","last_clean":"2024-04-21T08:39:58.153871+0000","last_became_active":"2024-04-21T08:37:31.441065+0000","last_became_peered":"2024-04-21T08:37:31.441065+0000","last_unstale":"2024-04-21T08:39:58.153871+0000","last_undegraded":"2024-04-21T08:39:58.153871+0000","last_fullsized":"2024-04-21T08:39:58.153871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:56.528154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318575+0000","last_change":"2024-04-21T08:37:31.446722+0000","last_active":"2024-04-21T08:39:58.318575+0000","last_peered":"2024-04-21T08:39:58.318575+0000","last_clean":"2024-04-21T08:39:58.318575+0000","last_became_active":"2024-04-21T08:37:31.446396+0000","last_became_peered":"2024-04-21T08:37:31.446396+0000","last_unstale":"2024-04-21T08:39:58.318575+0000","last_undegraded":"2024-04-21T08:39:58.318575+0000","last_fullsized":"2024-04-21T08:39:58.318575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:17.488633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007275+0000","last_change":"2024-04-21T08:37:31.444388+0000","last_active":"2024-04-21T08:39:58.007275+0000","last_peered":"2024-04-21T08:39:58.007275+0000","last_clean":"2024-04-21T08:39:58.007275+0000","last_became_active":"2024-04-21T08:37:31.444023+0000","last_became_peered":"2024-04-21T08:37:31.444023+0000","last_unstale":"2024-04-21T08:39:58.007275+0000","last_undegraded":"2024-04-21T08:39:58.007275+0000","last_fullsized":"2024-04-21T08:39:58.007275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:07:46.238941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905278+0000","last_change":"2024-04-21T08:37:31.447850+0000","last_active":"2024-04-21T08:39:57.905278+0000","last_peered":"2024-04-21T08:39:57.905278+0000","last_clean":"2024-04-21T08:39:57.905278+0000","last_became_active":"2024-04-21T08:37:31.447505+0000","last_became_peered":"2024-04-21T08:37:31.447505+0000","last_unstale":"2024-04-21T08:39:57.905278+0000","last_undegraded":"2024-04-21T08:39:57.905278+0000","last_fullsized":"2024-04-21T08:39:57.905278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:55.399410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905076+0000","last_change":"2024-04-21T08:37:35.608747+0000","last_active":"2024-04-21T08:39:57.905076+0000","last_peered":"2024-04-21T08:39:57.905076+0000","last_clean":"2024-04-21T08:39:57.905076+0000","last_became_active":"2024-04-21T08:37:35.608157+0000","last_became_peered":"2024-04-21T08:37:35.608157+0000","last_unstale":"2024-04-21T08:39:57.905076+0000","last_undegraded":"2024-04-21T08:39:57.905076+0000","last_fullsized":"2024-04-21T08:39:57.905076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:44:53.384190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007094+0000","last_change":"2024-04-21T08:38:30.679843+0000","last_active":"2024-04-21T08:39:58.007094+0000","last_peered":"2024-04-21T08:39:58.007094+0000","last_clean":"2024-04-21T08:39:58.007094+0000","last_became_active":"2024-04-21T08:38:30.670964+0000","last_became_peered":"2024-04-21T08:38:30.670964+0000","last_unstale":"2024-04-21T08:39:58.007094+0000","last_undegraded":"2024-04-21T08:39:58.007094+0000","last_fullsized":"2024-04-21T08:39:58.007094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:49.652571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904100+0000","last_change":"2024-04-21T08:37:31.451813+0000","last_active":"2024-04-21T08:39:57.904100+0000","last_peered":"2024-04-21T08:39:57.904100+0000","last_clean":"2024-04-21T08:39:57.904100+0000","last_became_active":"2024-04-21T08:37:31.451447+0000","last_became_peered":"2024-04-21T08:37:31.451447+0000","last_unstale":"2024-04-21T08:39:57.904100+0000","last_undegraded":"2024-04-21T08:39:57.904100+0000","last_fullsized":"2024-04-21T08:39:57.904100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:46:45.235326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151276+0000","last_change":"2024-04-21T08:37:35.603518+0000","last_active":"2024-04-21T08:39:58.151276+0000","last_peered":"2024-04-21T08:39:58.151276+0000","last_clean":"2024-04-21T08:39:58.151276+0000","last_became_active":"2024-04-21T08:37:35.603038+0000","last_became_peered":"2024-04-21T08:37:35.603038+0000","last_unstale":"2024-04-21T08:39:58.151276+0000","last_undegraded":"2024-04-21T08:39:58.151276+0000","last_fullsized":"2024-04-21T08:39:58.151276+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:06:29.224025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904228+0000","last_change":"2024-04-21T08:37:30.424608+0000","last_active":"2024-04-21T08:39:57.904228+0000","last_peered":"2024-04-21T08:39:57.904228+0000","last_clean":"2024-04-21T08:39:57.904228+0000","last_became_active":"2024-04-21T08:37:30.424259+0000","last_became_peered":"2024-04-21T08:37:30.424259+0000","last_unstale":"2024-04-21T08:39:57.904228+0000","last_undegraded":"2024-04-21T08:39:57.904228+0000","last_fullsized":"2024-04-21T08:39:57.904228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:38.007095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319688+0000","last_change":"2024-04-21T08:37:31.444656+0000","last_active":"2024-04-21T08:39:58.319688+0000","last_peered":"2024-04-21T08:39:58.319688+0000","last_clean":"2024-04-21T08:39:58.319688+0000","last_became_active":"2024-04-21T08:37:31.444300+0000","last_became_peered":"2024-04-21T08:37:31.444300+0000","last_unstale":"2024-04-21T08:39:58.319688+0000","last_undegraded":"2024-04-21T08:39:58.319688+0000","last_fullsized":"2024-04-21T08:39:58.319688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:40.901674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151439+0000","last_change":"2024-04-21T08:37:35.611732+0000","last_active":"2024-04-21T08:39:58.151439+0000","last_peered":"2024-04-21T08:39:58.151439+0000","last_clean":"2024-04-21T08:39:58.151439+0000","last_became_active":"2024-04-21T08:37:35.611060+0000","last_became_peered":"2024-04-21T08:37:35.611060+0000","last_unstale":"2024-04-21T08:39:58.151439+0000","last_undegraded":"2024-04-21T08:39:58.151439+0000","last_fullsized":"2024-04-21T08:39:58.151439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:36:01.491949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318678+0000","last_change":"2024-04-21T08:38:30.689722+0000","last_active":"2024-04-21T08:39:58.318678+0000","last_peered":"2024-04-21T08:39:58.318678+0000","last_clean":"2024-04-21T08:39:58.318678+0000","last_became_active":"2024-04-21T08:38:30.676521+0000","last_became_peered":"2024-04-21T08:38:30.676521+0000","last_unstale":"2024-04-21T08:39:58.318678+0000","last_undegraded":"2024-04-21T08:39:58.318678+0000","last_fullsized":"2024-04-21T08:39:58.318678+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:24:15.799308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154378+0000","last_change":"2024-04-21T08:37:31.443896+0000","last_active":"2024-04-21T08:39:58.154378+0000","last_peered":"2024-04-21T08:39:58.154378+0000","last_clean":"2024-04-21T08:39:58.154378+0000","last_became_active":"2024-04-21T08:37:31.442684+0000","last_became_peered":"2024-04-21T08:37:31.442684+0000","last_unstale":"2024-04-21T08:39:58.154378+0000","last_undegraded":"2024-04-21T08:39:58.154378+0000","last_fullsized":"2024-04-21T08:39:58.154378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:14:18.649565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151277+0000","last_change":"2024-04-21T08:37:35.600468+0000","last_active":"2024-04-21T08:39:58.151277+0000","last_peered":"2024-04-21T08:39:58.151277+0000","last_clean":"2024-04-21T08:39:58.151277+0000","last_became_active":"2024-04-21T08:37:35.599940+0000","last_became_peered":"2024-04-21T08:37:35.599940+0000","last_unstale":"2024-04-21T08:39:58.151277+0000","last_undegraded":"2024-04-21T08:39:58.151277+0000","last_fullsized":"2024-04-21T08:39:58.151277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:48.125434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009395+0000","last_change":"2024-04-21T08:37:30.416480+0000","last_active":"2024-04-21T08:39:58.009395+0000","last_peered":"2024-04-21T08:39:58.009395+0000","last_clean":"2024-04-21T08:39:58.009395+0000","last_became_active":"2024-04-21T08:37:30.416072+0000","last_became_peered":"2024-04-21T08:37:30.416072+0000","last_unstale":"2024-04-21T08:39:58.009395+0000","last_undegraded":"2024-04-21T08:39:58.009395+0000","last_fullsized":"2024-04-21T08:39:58.009395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:28:22.400926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014201+0000","last_change":"2024-04-21T08:37:31.457216+0000","last_active":"2024-04-21T08:39:58.014201+0000","last_peered":"2024-04-21T08:39:58.014201+0000","last_clean":"2024-04-21T08:39:58.014201+0000","last_became_active":"2024-04-21T08:37:31.456856+0000","last_became_peered":"2024-04-21T08:37:31.456856+0000","last_unstale":"2024-04-21T08:39:58.014201+0000","last_undegraded":"2024-04-21T08:39:58.014201+0000","last_fullsized":"2024-04-21T08:39:58.014201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:10.272886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905222+0000","last_change":"2024-04-21T08:37:35.601715+0000","last_active":"2024-04-21T08:39:57.905222+0000","last_peered":"2024-04-21T08:39:57.905222+0000","last_clean":"2024-04-21T08:39:57.905222+0000","last_became_active":"2024-04-21T08:37:35.601269+0000","last_became_peered":"2024-04-21T08:37:35.601269+0000","last_unstale":"2024-04-21T08:39:57.905222+0000","last_undegraded":"2024-04-21T08:39:57.905222+0000","last_fullsized":"2024-04-21T08:39:57.905222+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:57.192918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":176,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.903763+0000","last_change":"2024-04-21T08:37:30.415510+0000","last_active":"2024-04-21T08:40:07.903763+0000","last_peered":"2024-04-21T08:40:07.903763+0000","last_clean":"2024-04-21T08:40:07.903763+0000","last_became_active":"2024-04-21T08:37:30.415199+0000","last_became_peered":"2024-04-21T08:37:30.415199+0000","last_unstale":"2024-04-21T08:40:07.903763+0000","last_undegraded":"2024-04-21T08:40:07.903763+0000","last_fullsized":"2024-04-21T08:40:07.903763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:32.336247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009124+0000","last_change":"2024-04-21T08:39:31.681460+0000","last_active":"2024-04-21T08:39:58.009124+0000","last_peered":"2024-04-21T08:39:58.009124+0000","last_clean":"2024-04-21T08:39:58.009124+0000","last_became_active":"2024-04-21T08:39:31.681057+0000","last_became_peered":"2024-04-21T08:39:31.681057+0000","last_unstale":"2024-04-21T08:39:58.009124+0000","last_undegraded":"2024-04-21T08:39:58.009124+0000","last_fullsized":"2024-04-21T08:39:58.009124+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:43.407369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905274+0000","last_change":"2024-04-21T08:38:30.714455+0000","last_active":"2024-04-21T08:39:57.905274+0000","last_peered":"2024-04-21T08:39:57.905274+0000","last_clean":"2024-04-21T08:39:57.905274+0000","last_became_active":"2024-04-21T08:38:30.713910+0000","last_became_peered":"2024-04-21T08:38:30.713910+0000","last_unstale":"2024-04-21T08:39:57.905274+0000","last_undegraded":"2024-04-21T08:39:57.905274+0000","last_fullsized":"2024-04-21T08:39:57.905274+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.448652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,10],"acting":[2,6,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017063+0000","last_change":"2024-04-21T08:37:30.419877+0000","last_active":"2024-04-21T08:39:58.017063+0000","last_peered":"2024-04-21T08:39:58.017063+0000","last_clean":"2024-04-21T08:39:58.017063+0000","last_became_active":"2024-04-21T08:37:30.419572+0000","last_became_peered":"2024-04-21T08:37:30.419572+0000","last_unstale":"2024-04-21T08:39:58.017063+0000","last_undegraded":"2024-04-21T08:39:58.017063+0000","last_fullsized":"2024-04-21T08:39:58.017063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:32:35.524089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014094+0000","last_change":"2024-04-21T08:37:31.458590+0000","last_active":"2024-04-21T08:39:58.014094+0000","last_peered":"2024-04-21T08:39:58.014094+0000","last_clean":"2024-04-21T08:39:58.014094+0000","last_became_active":"2024-04-21T08:37:31.458196+0000","last_became_peered":"2024-04-21T08:37:31.458196+0000","last_unstale":"2024-04-21T08:39:58.014094+0000","last_undegraded":"2024-04-21T08:39:58.014094+0000","last_fullsized":"2024-04-21T08:39:58.014094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:00.890859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318442+0000","last_change":"2024-04-21T08:38:30.803663+0000","last_active":"2024-04-21T08:39:58.318442+0000","last_peered":"2024-04-21T08:39:58.318442+0000","last_clean":"2024-04-21T08:39:58.318442+0000","last_became_active":"2024-04-21T08:38:30.803203+0000","last_became_peered":"2024-04-21T08:38:30.803203+0000","last_unstale":"2024-04-21T08:39:58.318442+0000","last_undegraded":"2024-04-21T08:39:58.318442+0000","last_fullsized":"2024-04-21T08:39:58.318442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:06.085380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318695+0000","last_change":"2024-04-21T08:37:30.415786+0000","last_active":"2024-04-21T08:39:58.318695+0000","last_peered":"2024-04-21T08:39:58.318695+0000","last_clean":"2024-04-21T08:39:58.318695+0000","last_became_active":"2024-04-21T08:37:30.415472+0000","last_became_peered":"2024-04-21T08:37:30.415472+0000","last_unstale":"2024-04-21T08:39:58.318695+0000","last_undegraded":"2024-04-21T08:39:58.318695+0000","last_fullsized":"2024-04-21T08:39:58.318695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.417132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903648+0000","last_change":"2024-04-21T08:37:31.455720+0000","last_active":"2024-04-21T08:39:57.903648+0000","last_peered":"2024-04-21T08:39:57.903648+0000","last_clean":"2024-04-21T08:39:57.903648+0000","last_became_active":"2024-04-21T08:37:31.455334+0000","last_became_peered":"2024-04-21T08:37:31.455334+0000","last_unstale":"2024-04-21T08:39:57.903648+0000","last_undegraded":"2024-04-21T08:39:57.903648+0000","last_fullsized":"2024-04-21T08:39:57.903648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:06:55.220656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319675+0000","last_change":"2024-04-21T08:37:35.603305+0000","last_active":"2024-04-21T08:39:58.319675+0000","last_peered":"2024-04-21T08:39:58.319675+0000","last_clean":"2024-04-21T08:39:58.319675+0000","last_became_active":"2024-04-21T08:37:35.602854+0000","last_became_peered":"2024-04-21T08:37:35.602854+0000","last_unstale":"2024-04-21T08:39:58.319675+0000","last_undegraded":"2024-04-21T08:39:58.319675+0000","last_fullsized":"2024-04-21T08:39:58.319675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:10.993864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013890+0000","last_change":"2024-04-21T08:37:30.420797+0000","last_active":"2024-04-21T08:39:58.013890+0000","last_peered":"2024-04-21T08:39:58.013890+0000","last_clean":"2024-04-21T08:39:58.013890+0000","last_became_active":"2024-04-21T08:37:30.420485+0000","last_became_peered":"2024-04-21T08:37:30.420485+0000","last_unstale":"2024-04-21T08:39:58.013890+0000","last_undegraded":"2024-04-21T08:39:58.013890+0000","last_fullsized":"2024-04-21T08:39:58.013890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:03.275677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006844+0000","last_change":"2024-04-21T08:37:31.448406+0000","last_active":"2024-04-21T08:39:58.006844+0000","last_peered":"2024-04-21T08:39:58.006844+0000","last_clean":"2024-04-21T08:39:58.006844+0000","last_became_active":"2024-04-21T08:37:31.448085+0000","last_became_peered":"2024-04-21T08:37:31.448085+0000","last_unstale":"2024-04-21T08:39:58.006844+0000","last_undegraded":"2024-04-21T08:39:58.006844+0000","last_fullsized":"2024-04-21T08:39:58.006844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:49.574064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903948+0000","last_change":"2024-04-21T08:37:35.601338+0000","last_active":"2024-04-21T08:39:57.903948+0000","last_peered":"2024-04-21T08:39:57.903948+0000","last_clean":"2024-04-21T08:39:57.903948+0000","last_became_active":"2024-04-21T08:37:35.600890+0000","last_became_peered":"2024-04-21T08:37:35.600890+0000","last_unstale":"2024-04-21T08:39:57.903948+0000","last_undegraded":"2024-04-21T08:39:57.903948+0000","last_fullsized":"2024-04-21T08:39:57.903948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:30:21.755850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'4","reported_seq":171,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.903166+0000","last_change":"2024-04-21T08:37:30.423320+0000","last_active":"2024-04-21T08:40:07.903166+0000","last_peered":"2024-04-21T08:40:07.903166+0000","last_clean":"2024-04-21T08:40:07.903166+0000","last_became_active":"2024-04-21T08:37:30.423020+0000","last_became_peered":"2024-04-21T08:37:30.423020+0000","last_unstale":"2024-04-21T08:40:07.903166+0000","last_undegraded":"2024-04-21T08:40:07.903166+0000","last_fullsized":"2024-04-21T08:40:07.903166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:08:27.313196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":133,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154669+0000","last_change":"2024-04-21T08:37:31.442425+0000","last_active":"2024-04-21T08:39:58.154669+0000","last_peered":"2024-04-21T08:39:58.154669+0000","last_clean":"2024-04-21T08:39:58.154669+0000","last_became_active":"2024-04-21T08:37:31.442028+0000","last_became_peered":"2024-04-21T08:37:31.442028+0000","last_unstale":"2024-04-21T08:39:58.154669+0000","last_undegraded":"2024-04-21T08:39:58.154669+0000","last_fullsized":"2024-04-21T08:39:58.154669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:45.498656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006697+0000","last_change":"2024-04-21T08:37:35.607829+0000","last_active":"2024-04-21T08:39:58.006697+0000","last_peered":"2024-04-21T08:39:58.006697+0000","last_clean":"2024-04-21T08:39:58.006697+0000","last_became_active":"2024-04-21T08:37:35.607150+0000","last_became_peered":"2024-04-21T08:37:35.607150+0000","last_unstale":"2024-04-21T08:39:58.006697+0000","last_undegraded":"2024-04-21T08:39:58.006697+0000","last_fullsized":"2024-04-21T08:39:58.006697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:01:34.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905384+0000","last_change":"2024-04-21T08:37:30.418613+0000","last_active":"2024-04-21T08:39:57.905384+0000","last_peered":"2024-04-21T08:39:57.905384+0000","last_clean":"2024-04-21T08:39:57.905384+0000","last_became_active":"2024-04-21T08:37:30.418300+0000","last_became_peered":"2024-04-21T08:37:30.418300+0000","last_unstale":"2024-04-21T08:39:57.905384+0000","last_undegraded":"2024-04-21T08:39:57.905384+0000","last_fullsized":"2024-04-21T08:39:57.905384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:17:35.370753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319739+0000","last_change":"2024-04-21T08:39:31.682628+0000","last_active":"2024-04-21T08:39:58.319739+0000","last_peered":"2024-04-21T08:39:58.319739+0000","last_clean":"2024-04-21T08:39:58.319739+0000","last_became_active":"2024-04-21T08:39:31.682383+0000","last_became_peered":"2024-04-21T08:39:31.682383+0000","last_unstale":"2024-04-21T08:39:58.319739+0000","last_undegraded":"2024-04-21T08:39:58.319739+0000","last_fullsized":"2024-04-21T08:39:58.319739+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:38.831288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151734+0000","last_change":"2024-04-21T08:37:35.608280+0000","last_active":"2024-04-21T08:39:58.151734+0000","last_peered":"2024-04-21T08:39:58.151734+0000","last_clean":"2024-04-21T08:39:58.151734+0000","last_became_active":"2024-04-21T08:37:35.607898+0000","last_became_peered":"2024-04-21T08:37:35.607898+0000","last_unstale":"2024-04-21T08:39:58.151734+0000","last_undegraded":"2024-04-21T08:39:58.151734+0000","last_fullsized":"2024-04-21T08:39:58.151734+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:55.689519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014140+0000","last_change":"2024-04-21T08:37:30.417352+0000","last_active":"2024-04-21T08:39:58.014140+0000","last_peered":"2024-04-21T08:39:58.014140+0000","last_clean":"2024-04-21T08:39:58.014140+0000","last_became_active":"2024-04-21T08:37:30.417000+0000","last_became_peered":"2024-04-21T08:37:30.417000+0000","last_unstale":"2024-04-21T08:39:58.014140+0000","last_undegraded":"2024-04-21T08:39:58.014140+0000","last_fullsized":"2024-04-21T08:39:58.014140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:56.087334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006804+0000","last_change":"2024-04-21T08:39:31.679434+0000","last_active":"2024-04-21T08:39:58.006804+0000","last_peered":"2024-04-21T08:39:58.006804+0000","last_clean":"2024-04-21T08:39:58.006804+0000","last_became_active":"2024-04-21T08:39:31.679041+0000","last_became_peered":"2024-04-21T08:39:31.679041+0000","last_unstale":"2024-04-21T08:39:58.006804+0000","last_undegraded":"2024-04-21T08:39:58.006804+0000","last_fullsized":"2024-04-21T08:39:58.006804+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:27:11.011449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006731+0000","last_change":"2024-04-21T08:37:35.601077+0000","last_active":"2024-04-21T08:39:58.006731+0000","last_peered":"2024-04-21T08:39:58.006731+0000","last_clean":"2024-04-21T08:39:58.006731+0000","last_became_active":"2024-04-21T08:37:35.600673+0000","last_became_peered":"2024-04-21T08:37:35.600673+0000","last_unstale":"2024-04-21T08:39:58.006731+0000","last_undegraded":"2024-04-21T08:39:58.006731+0000","last_fullsized":"2024-04-21T08:39:58.006731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:04.554208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'43","reported_seq":358,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:08.904470+0000","last_change":"2024-04-21T08:37:30.421440+0000","last_active":"2024-04-21T08:40:08.904470+0000","last_peered":"2024-04-21T08:40:08.904470+0000","last_clean":"2024-04-21T08:40:08.904470+0000","last_became_active":"2024-04-21T08:37:30.420988+0000","last_became_peered":"2024-04-21T08:37:30.420988+0000","last_unstale":"2024-04-21T08:40:08.904470+0000","last_undegraded":"2024-04-21T08:40:08.904470+0000","last_fullsized":"2024-04-21T08:40:08.904470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:38:03.850705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72979,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":281,"num_read_kb":213,"num_write":44,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009331+0000","last_change":"2024-04-21T08:37:31.447208+0000","last_active":"2024-04-21T08:39:58.009331+0000","last_peered":"2024-04-21T08:39:58.009331+0000","last_clean":"2024-04-21T08:39:58.009331+0000","last_became_active":"2024-04-21T08:37:31.444464+0000","last_became_peered":"2024-04-21T08:37:31.444464+0000","last_unstale":"2024-04-21T08:39:58.009331+0000","last_undegraded":"2024-04-21T08:39:58.009331+0000","last_fullsized":"2024-04-21T08:39:58.009331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:33.805676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014409+0000","last_change":"2024-04-21T08:37:35.603937+0000","last_active":"2024-04-21T08:39:58.014409+0000","last_peered":"2024-04-21T08:39:58.014409+0000","last_clean":"2024-04-21T08:39:58.014409+0000","last_became_active":"2024-04-21T08:37:35.603477+0000","last_became_peered":"2024-04-21T08:37:35.603477+0000","last_unstale":"2024-04-21T08:39:58.014409+0000","last_undegraded":"2024-04-21T08:39:58.014409+0000","last_fullsized":"2024-04-21T08:39:58.014409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:40:33.901570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905391+0000","last_change":"2024-04-21T08:37:30.416941+0000","last_active":"2024-04-21T08:39:57.905391+0000","last_peered":"2024-04-21T08:39:57.905391+0000","last_clean":"2024-04-21T08:39:57.905391+0000","last_became_active":"2024-04-21T08:37:30.416625+0000","last_became_peered":"2024-04-21T08:37:30.416625+0000","last_unstale":"2024-04-21T08:39:57.905391+0000","last_undegraded":"2024-04-21T08:39:57.905391+0000","last_fullsized":"2024-04-21T08:39:57.905391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:07.782779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318733+0000","last_change":"2024-04-21T08:37:31.441707+0000","last_active":"2024-04-21T08:39:58.318733+0000","last_peered":"2024-04-21T08:39:58.318733+0000","last_clean":"2024-04-21T08:39:58.318733+0000","last_became_active":"2024-04-21T08:37:31.441366+0000","last_became_peered":"2024-04-21T08:37:31.441366+0000","last_unstale":"2024-04-21T08:39:58.318733+0000","last_undegraded":"2024-04-21T08:39:58.318733+0000","last_fullsized":"2024-04-21T08:39:58.318733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:16.573804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318906+0000","last_change":"2024-04-21T08:37:35.596214+0000","last_active":"2024-04-21T08:39:58.318906+0000","last_peered":"2024-04-21T08:39:58.318906+0000","last_clean":"2024-04-21T08:39:58.318906+0000","last_became_active":"2024-04-21T08:37:35.595773+0000","last_became_peered":"2024-04-21T08:37:35.595773+0000","last_unstale":"2024-04-21T08:39:58.318906+0000","last_undegraded":"2024-04-21T08:39:58.318906+0000","last_fullsized":"2024-04-21T08:39:58.318906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:07.538190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318273+0000","last_change":"2024-04-21T08:38:30.684828+0000","last_active":"2024-04-21T08:39:58.318273+0000","last_peered":"2024-04-21T08:39:58.318273+0000","last_clean":"2024-04-21T08:39:58.318273+0000","last_became_active":"2024-04-21T08:38:30.673923+0000","last_became_peered":"2024-04-21T08:38:30.673923+0000","last_unstale":"2024-04-21T08:39:58.318273+0000","last_undegraded":"2024-04-21T08:39:58.318273+0000","last_fullsized":"2024-04-21T08:39:58.318273+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:51:37.906150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905472+0000","last_change":"2024-04-21T08:37:31.453680+0000","last_active":"2024-04-21T08:39:57.905472+0000","last_peered":"2024-04-21T08:39:57.905472+0000","last_clean":"2024-04-21T08:39:57.905472+0000","last_became_active":"2024-04-21T08:37:31.453319+0000","last_became_peered":"2024-04-21T08:37:31.453319+0000","last_unstale":"2024-04-21T08:39:57.905472+0000","last_undegraded":"2024-04-21T08:39:57.905472+0000","last_fullsized":"2024-04-21T08:39:57.905472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:43.139789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007023+0000","last_change":"2024-04-21T08:37:35.608906+0000","last_active":"2024-04-21T08:39:58.007023+0000","last_peered":"2024-04-21T08:39:58.007023+0000","last_clean":"2024-04-21T08:39:58.007023+0000","last_became_active":"2024-04-21T08:37:35.608535+0000","last_became_peered":"2024-04-21T08:37:35.608535+0000","last_unstale":"2024-04-21T08:39:58.007023+0000","last_undegraded":"2024-04-21T08:39:58.007023+0000","last_fullsized":"2024-04-21T08:39:58.007023+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:58:01.422308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'9","reported_seq":183,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.902877+0000","last_change":"2024-04-21T08:37:30.414865+0000","last_active":"2024-04-21T08:40:07.902877+0000","last_peered":"2024-04-21T08:40:07.902877+0000","last_clean":"2024-04-21T08:40:07.902877+0000","last_became_active":"2024-04-21T08:37:30.414471+0000","last_became_peered":"2024-04-21T08:37:30.414471+0000","last_unstale":"2024-04-21T08:40:07.902877+0000","last_undegraded":"2024-04-21T08:40:07.902877+0000","last_fullsized":"2024-04-21T08:40:07.902877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:46.943574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2601,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":8,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905326+0000","last_change":"2024-04-21T08:37:31.456634+0000","last_active":"2024-04-21T08:39:57.905326+0000","last_peered":"2024-04-21T08:39:57.905326+0000","last_clean":"2024-04-21T08:39:57.905326+0000","last_became_active":"2024-04-21T08:37:31.456278+0000","last_became_peered":"2024-04-21T08:37:31.456278+0000","last_unstale":"2024-04-21T08:39:57.905326+0000","last_undegraded":"2024-04-21T08:39:57.905326+0000","last_fullsized":"2024-04-21T08:39:57.905326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:04.870024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319167+0000","last_change":"2024-04-21T08:37:35.601760+0000","last_active":"2024-04-21T08:39:58.319167+0000","last_peered":"2024-04-21T08:39:58.319167+0000","last_clean":"2024-04-21T08:39:58.319167+0000","last_became_active":"2024-04-21T08:37:35.601332+0000","last_became_peered":"2024-04-21T08:37:35.601332+0000","last_unstale":"2024-04-21T08:39:58.319167+0000","last_undegraded":"2024-04-21T08:39:58.319167+0000","last_fullsized":"2024-04-21T08:39:58.319167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:04:03.694581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008961+0000","last_change":"2024-04-21T08:37:30.422419+0000","last_active":"2024-04-21T08:39:58.008961+0000","last_peered":"2024-04-21T08:39:58.008961+0000","last_clean":"2024-04-21T08:39:58.008961+0000","last_became_active":"2024-04-21T08:37:30.422123+0000","last_became_peered":"2024-04-21T08:37:30.422123+0000","last_unstale":"2024-04-21T08:39:58.008961+0000","last_undegraded":"2024-04-21T08:39:58.008961+0000","last_fullsized":"2024-04-21T08:39:58.008961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:32:54.029041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013848+0000","last_change":"2024-04-21T08:37:31.458992+0000","last_active":"2024-04-21T08:39:58.013848+0000","last_peered":"2024-04-21T08:39:58.013848+0000","last_clean":"2024-04-21T08:39:58.013848+0000","last_became_active":"2024-04-21T08:37:31.458635+0000","last_became_peered":"2024-04-21T08:37:31.458635+0000","last_unstale":"2024-04-21T08:39:58.013848+0000","last_undegraded":"2024-04-21T08:39:58.013848+0000","last_fullsized":"2024-04-21T08:39:58.013848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:48:14.325571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013750+0000","last_change":"2024-04-21T08:37:35.603739+0000","last_active":"2024-04-21T08:39:58.013750+0000","last_peered":"2024-04-21T08:39:58.013750+0000","last_clean":"2024-04-21T08:39:58.013750+0000","last_became_active":"2024-04-21T08:37:35.603310+0000","last_became_peered":"2024-04-21T08:37:35.603310+0000","last_unstale":"2024-04-21T08:39:58.013750+0000","last_undegraded":"2024-04-21T08:39:58.013750+0000","last_fullsized":"2024-04-21T08:39:58.013750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:19.162520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009465+0000","last_change":"2024-04-21T08:37:30.418104+0000","last_active":"2024-04-21T08:39:58.009465+0000","last_peered":"2024-04-21T08:39:58.009465+0000","last_clean":"2024-04-21T08:39:58.009465+0000","last_became_active":"2024-04-21T08:37:30.417814+0000","last_became_peered":"2024-04-21T08:37:30.417814+0000","last_unstale":"2024-04-21T08:39:58.009465+0000","last_undegraded":"2024-04-21T08:39:58.009465+0000","last_fullsized":"2024-04-21T08:39:58.009465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:50:56.696513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318582+0000","last_change":"2024-04-21T08:37:31.451952+0000","last_active":"2024-04-21T08:39:58.318582+0000","last_peered":"2024-04-21T08:39:58.318582+0000","last_clean":"2024-04-21T08:39:58.318582+0000","last_became_active":"2024-04-21T08:37:31.451588+0000","last_became_peered":"2024-04-21T08:37:31.451588+0000","last_unstale":"2024-04-21T08:39:58.318582+0000","last_undegraded":"2024-04-21T08:39:58.318582+0000","last_fullsized":"2024-04-21T08:39:58.318582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:06.506769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009071+0000","last_change":"2024-04-21T08:37:35.760661+0000","last_active":"2024-04-21T08:39:58.009071+0000","last_peered":"2024-04-21T08:39:58.009071+0000","last_clean":"2024-04-21T08:39:58.009071+0000","last_became_active":"2024-04-21T08:37:35.759925+0000","last_became_peered":"2024-04-21T08:37:35.759925+0000","last_unstale":"2024-04-21T08:39:58.009071+0000","last_undegraded":"2024-04-21T08:39:58.009071+0000","last_fullsized":"2024-04-21T08:39:58.009071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:11:53.062251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.007012+0000","last_change":"2024-04-21T08:37:30.416487+0000","last_active":"2024-04-21T08:39:58.007012+0000","last_peered":"2024-04-21T08:39:58.007012+0000","last_clean":"2024-04-21T08:39:58.007012+0000","last_became_active":"2024-04-21T08:37:30.416076+0000","last_became_peered":"2024-04-21T08:37:30.416076+0000","last_unstale":"2024-04-21T08:39:58.007012+0000","last_undegraded":"2024-04-21T08:39:58.007012+0000","last_fullsized":"2024-04-21T08:39:58.007012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:18:33.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.017205+0000","last_change":"2024-04-21T08:37:31.448824+0000","last_active":"2024-04-21T08:39:58.017205+0000","last_peered":"2024-04-21T08:39:58.017205+0000","last_clean":"2024-04-21T08:39:58.017205+0000","last_became_active":"2024-04-21T08:37:31.448432+0000","last_became_peered":"2024-04-21T08:37:31.448432+0000","last_unstale":"2024-04-21T08:39:58.017205+0000","last_undegraded":"2024-04-21T08:39:58.017205+0000","last_fullsized":"2024-04-21T08:39:58.017205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:41.111272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318815+0000","last_change":"2024-04-21T08:37:35.604539+0000","last_active":"2024-04-21T08:39:58.318815+0000","last_peered":"2024-04-21T08:39:58.318815+0000","last_clean":"2024-04-21T08:39:58.318815+0000","last_became_active":"2024-04-21T08:37:35.604132+0000","last_became_peered":"2024-04-21T08:37:35.604132+0000","last_unstale":"2024-04-21T08:39:58.318815+0000","last_undegraded":"2024-04-21T08:39:58.318815+0000","last_fullsized":"2024-04-21T08:39:58.318815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:12.180730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905085+0000","last_change":"2024-04-21T08:37:30.418854+0000","last_active":"2024-04-21T08:39:57.905085+0000","last_peered":"2024-04-21T08:39:57.905085+0000","last_clean":"2024-04-21T08:39:57.905085+0000","last_became_active":"2024-04-21T08:37:30.418511+0000","last_became_peered":"2024-04-21T08:37:30.418511+0000","last_unstale":"2024-04-21T08:39:57.905085+0000","last_undegraded":"2024-04-21T08:39:57.905085+0000","last_fullsized":"2024-04-21T08:39:57.905085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:26.751924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.016918+0000","last_change":"2024-04-21T08:37:31.453365+0000","last_active":"2024-04-21T08:39:58.016918+0000","last_peered":"2024-04-21T08:39:58.016918+0000","last_clean":"2024-04-21T08:39:58.016918+0000","last_became_active":"2024-04-21T08:37:31.453042+0000","last_became_peered":"2024-04-21T08:37:31.453042+0000","last_unstale":"2024-04-21T08:39:58.016918+0000","last_undegraded":"2024-04-21T08:39:58.016918+0000","last_fullsized":"2024-04-21T08:39:58.016918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:49:46.648754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318806+0000","last_change":"2024-04-21T08:37:35.760626+0000","last_active":"2024-04-21T08:39:58.318806+0000","last_peered":"2024-04-21T08:39:58.318806+0000","last_clean":"2024-04-21T08:39:58.318806+0000","last_became_active":"2024-04-21T08:37:35.760125+0000","last_became_peered":"2024-04-21T08:37:35.760125+0000","last_unstale":"2024-04-21T08:39:58.318806+0000","last_undegraded":"2024-04-21T08:39:58.318806+0000","last_fullsized":"2024-04-21T08:39:58.318806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:58.366557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013932+0000","last_change":"2024-04-21T08:37:31.457654+0000","last_active":"2024-04-21T08:39:58.013932+0000","last_peered":"2024-04-21T08:39:58.013932+0000","last_clean":"2024-04-21T08:39:58.013932+0000","last_became_active":"2024-04-21T08:37:31.456950+0000","last_became_peered":"2024-04-21T08:37:31.456950+0000","last_unstale":"2024-04-21T08:39:58.013932+0000","last_undegraded":"2024-04-21T08:39:58.013932+0000","last_fullsized":"2024-04-21T08:39:58.013932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:02.863000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":173,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.902250+0000","last_change":"2024-04-21T08:37:30.422021+0000","last_active":"2024-04-21T08:40:07.902250+0000","last_peered":"2024-04-21T08:40:07.902250+0000","last_clean":"2024-04-21T08:40:07.902250+0000","last_became_active":"2024-04-21T08:37:30.421711+0000","last_became_peered":"2024-04-21T08:37:30.421711+0000","last_unstale":"2024-04-21T08:40:07.902250+0000","last_undegraded":"2024-04-21T08:40:07.902250+0000","last_fullsized":"2024-04-21T08:40:07.902250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:01.612205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":133,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905863+0000","last_change":"2024-04-21T08:37:35.598041+0000","last_active":"2024-04-21T08:39:57.905863+0000","last_peered":"2024-04-21T08:39:57.905863+0000","last_clean":"2024-04-21T08:39:57.905863+0000","last_became_active":"2024-04-21T08:37:35.597635+0000","last_became_peered":"2024-04-21T08:37:35.597635+0000","last_unstale":"2024-04-21T08:39:57.905863+0000","last_undegraded":"2024-04-21T08:39:57.905863+0000","last_fullsized":"2024-04-21T08:39:57.905863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:35:27.452340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903604+0000","last_change":"2024-04-21T08:37:31.450423+0000","last_active":"2024-04-21T08:39:57.903604+0000","last_peered":"2024-04-21T08:39:57.903604+0000","last_clean":"2024-04-21T08:39:57.903604+0000","last_became_active":"2024-04-21T08:37:31.450112+0000","last_became_peered":"2024-04-21T08:37:31.450112+0000","last_unstale":"2024-04-21T08:39:57.903604+0000","last_undegraded":"2024-04-21T08:39:57.903604+0000","last_fullsized":"2024-04-21T08:39:57.903604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:10:47.006769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151568+0000","last_change":"2024-04-21T08:37:30.416519+0000","last_active":"2024-04-21T08:39:58.151568+0000","last_peered":"2024-04-21T08:39:58.151568+0000","last_clean":"2024-04-21T08:39:58.151568+0000","last_became_active":"2024-04-21T08:37:30.416131+0000","last_became_peered":"2024-04-21T08:37:30.416131+0000","last_unstale":"2024-04-21T08:39:58.151568+0000","last_undegraded":"2024-04-21T08:39:58.151568+0000","last_fullsized":"2024-04-21T08:39:58.151568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:12.280847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":412,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:00.021477+0000","last_change":"2024-04-21T08:36:20.257759+0000","last_active":"2024-04-21T08:40:00.021477+0000","last_peered":"2024-04-21T08:40:00.021477+0000","last_clean":"2024-04-21T08:40:00.021477+0000","last_became_active":"2024-04-21T08:36:20.257313+0000","last_became_peered":"2024-04-21T08:36:20.257313+0000","last_unstale":"2024-04-21T08:40:00.021477+0000","last_undegraded":"2024-04-21T08:40:00.021477+0000","last_fullsized":"2024-04-21T08:40:00.021477+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:32:56.546295+0000","last_clean_scrub_stamp":"2024-04-21T08:32:56.546295+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:42:44.393447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.014207+0000","last_change":"2024-04-21T08:37:35.761562+0000","last_active":"2024-04-21T08:39:58.014207+0000","last_peered":"2024-04-21T08:39:58.014207+0000","last_clean":"2024-04-21T08:39:58.014207+0000","last_became_active":"2024-04-21T08:37:35.760703+0000","last_became_peered":"2024-04-21T08:37:35.760703+0000","last_unstale":"2024-04-21T08:39:58.014207+0000","last_undegraded":"2024-04-21T08:39:58.014207+0000","last_fullsized":"2024-04-21T08:39:58.014207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:12:47.511117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318506+0000","last_change":"2024-04-21T08:39:31.678868+0000","last_active":"2024-04-21T08:39:58.318506+0000","last_peered":"2024-04-21T08:39:58.318506+0000","last_clean":"2024-04-21T08:39:58.318506+0000","last_became_active":"2024-04-21T08:39:31.678457+0000","last_became_peered":"2024-04-21T08:39:31.678457+0000","last_unstale":"2024-04-21T08:39:58.318506+0000","last_undegraded":"2024-04-21T08:39:58.318506+0000","last_fullsized":"2024-04-21T08:39:58.318506+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:49.918106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905042+0000","last_change":"2024-04-21T08:37:30.420560+0000","last_active":"2024-04-21T08:39:57.905042+0000","last_peered":"2024-04-21T08:39:57.905042+0000","last_clean":"2024-04-21T08:39:57.905042+0000","last_became_active":"2024-04-21T08:37:30.420223+0000","last_became_peered":"2024-04-21T08:37:30.420223+0000","last_unstale":"2024-04-21T08:39:57.905042+0000","last_undegraded":"2024-04-21T08:39:57.905042+0000","last_fullsized":"2024-04-21T08:39:57.905042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:43:55.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318381+0000","last_change":"2024-04-21T08:37:35.598294+0000","last_active":"2024-04-21T08:39:58.318381+0000","last_peered":"2024-04-21T08:39:58.318381+0000","last_clean":"2024-04-21T08:39:58.318381+0000","last_became_active":"2024-04-21T08:37:35.597883+0000","last_became_peered":"2024-04-21T08:37:35.597883+0000","last_unstale":"2024-04-21T08:39:58.318381+0000","last_undegraded":"2024-04-21T08:39:58.318381+0000","last_fullsized":"2024-04-21T08:39:58.318381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:24:05.144763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009321+0000","last_change":"2024-04-21T08:37:30.423182+0000","last_active":"2024-04-21T08:39:58.009321+0000","last_peered":"2024-04-21T08:39:58.009321+0000","last_clean":"2024-04-21T08:39:58.009321+0000","last_became_active":"2024-04-21T08:37:30.422792+0000","last_became_peered":"2024-04-21T08:37:30.422792+0000","last_unstale":"2024-04-21T08:39:58.009321+0000","last_undegraded":"2024-04-21T08:39:58.009321+0000","last_fullsized":"2024-04-21T08:39:58.009321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:44:00.727269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318924+0000","last_change":"2024-04-21T08:37:31.444013+0000","last_active":"2024-04-21T08:39:58.318924+0000","last_peered":"2024-04-21T08:39:58.318924+0000","last_clean":"2024-04-21T08:39:58.318924+0000","last_became_active":"2024-04-21T08:37:31.443684+0000","last_became_peered":"2024-04-21T08:37:31.443684+0000","last_unstale":"2024-04-21T08:39:58.318924+0000","last_undegraded":"2024-04-21T08:39:58.318924+0000","last_fullsized":"2024-04-21T08:39:58.318924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:00:35.400818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903525+0000","last_change":"2024-04-21T08:37:35.759817+0000","last_active":"2024-04-21T08:39:57.903525+0000","last_peered":"2024-04-21T08:39:57.903525+0000","last_clean":"2024-04-21T08:39:57.903525+0000","last_became_active":"2024-04-21T08:37:35.759396+0000","last_became_peered":"2024-04-21T08:37:35.759396+0000","last_unstale":"2024-04-21T08:39:57.903525+0000","last_undegraded":"2024-04-21T08:39:57.903525+0000","last_fullsized":"2024-04-21T08:39:57.903525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:20:36.170880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'98","reported_seq":224,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.333603+0000","last_change":"2024-04-21T08:37:30.414102+0000","last_active":"2024-04-21T08:40:07.333603+0000","last_peered":"2024-04-21T08:40:07.333603+0000","last_clean":"2024-04-21T08:40:07.333603+0000","last_became_active":"2024-04-21T08:37:30.413670+0000","last_became_peered":"2024-04-21T08:37:30.413670+0000","last_unstale":"2024-04-21T08:40:07.333603+0000","last_undegraded":"2024-04-21T08:40:07.333603+0000","last_fullsized":"2024-04-21T08:40:07.333603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:12:49.696516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":179,"num_write_kb":1285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904320+0000","last_change":"2024-04-21T08:37:31.457451+0000","last_active":"2024-04-21T08:39:57.904320+0000","last_peered":"2024-04-21T08:39:57.904320+0000","last_clean":"2024-04-21T08:39:57.904320+0000","last_became_active":"2024-04-21T08:37:31.457120+0000","last_became_peered":"2024-04-21T08:37:31.457120+0000","last_unstale":"2024-04-21T08:39:57.904320+0000","last_undegraded":"2024-04-21T08:39:57.904320+0000","last_fullsized":"2024-04-21T08:39:57.904320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:07:48.025843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904850+0000","last_change":"2024-04-21T08:37:35.600815+0000","last_active":"2024-04-21T08:39:57.904850+0000","last_peered":"2024-04-21T08:39:57.904850+0000","last_clean":"2024-04-21T08:39:57.904850+0000","last_became_active":"2024-04-21T08:37:35.600380+0000","last_became_peered":"2024-04-21T08:37:35.600380+0000","last_unstale":"2024-04-21T08:39:57.904850+0000","last_undegraded":"2024-04-21T08:39:57.904850+0000","last_fullsized":"2024-04-21T08:39:57.904850+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:19:33.710431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151729+0000","last_change":"2024-04-21T08:37:30.425592+0000","last_active":"2024-04-21T08:39:58.151729+0000","last_peered":"2024-04-21T08:39:58.151729+0000","last_clean":"2024-04-21T08:39:58.151729+0000","last_became_active":"2024-04-21T08:37:30.425274+0000","last_became_peered":"2024-04-21T08:37:30.425274+0000","last_unstale":"2024-04-21T08:39:58.151729+0000","last_undegraded":"2024-04-21T08:39:58.151729+0000","last_fullsized":"2024-04-21T08:39:58.151729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:27:29.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319672+0000","last_change":"2024-04-21T08:39:31.680028+0000","last_active":"2024-04-21T08:39:58.319672+0000","last_peered":"2024-04-21T08:39:58.319672+0000","last_clean":"2024-04-21T08:39:58.319672+0000","last_became_active":"2024-04-21T08:39:31.679162+0000","last_became_peered":"2024-04-21T08:39:31.679162+0000","last_unstale":"2024-04-21T08:39:58.319672+0000","last_undegraded":"2024-04-21T08:39:58.319672+0000","last_fullsized":"2024-04-21T08:39:58.319672+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:32.824411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151564+0000","last_change":"2024-04-21T08:38:30.713521+0000","last_active":"2024-04-21T08:39:58.151564+0000","last_peered":"2024-04-21T08:39:58.151564+0000","last_clean":"2024-04-21T08:39:58.151564+0000","last_became_active":"2024-04-21T08:38:30.713096+0000","last_became_peered":"2024-04-21T08:38:30.713096+0000","last_unstale":"2024-04-21T08:39:58.151564+0000","last_undegraded":"2024-04-21T08:39:58.151564+0000","last_fullsized":"2024-04-21T08:39:58.151564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:54:59.078776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153818+0000","last_change":"2024-04-21T08:37:30.416696+0000","last_active":"2024-04-21T08:39:58.153818+0000","last_peered":"2024-04-21T08:39:58.153818+0000","last_clean":"2024-04-21T08:39:58.153818+0000","last_became_active":"2024-04-21T08:37:30.415868+0000","last_became_peered":"2024-04-21T08:37:30.415868+0000","last_unstale":"2024-04-21T08:39:58.153818+0000","last_undegraded":"2024-04-21T08:39:58.153818+0000","last_fullsized":"2024-04-21T08:39:58.153818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:48:08.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.008945+0000","last_change":"2024-04-21T08:37:31.453861+0000","last_active":"2024-04-21T08:39:58.008945+0000","last_peered":"2024-04-21T08:39:58.008945+0000","last_clean":"2024-04-21T08:39:58.008945+0000","last_became_active":"2024-04-21T08:37:31.453562+0000","last_became_peered":"2024-04-21T08:37:31.453562+0000","last_unstale":"2024-04-21T08:39:58.008945+0000","last_undegraded":"2024-04-21T08:39:58.008945+0000","last_fullsized":"2024-04-21T08:39:58.008945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:17:50.513917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.006977+0000","last_change":"2024-04-21T08:37:35.609217+0000","last_active":"2024-04-21T08:39:58.006977+0000","last_peered":"2024-04-21T08:39:58.006977+0000","last_clean":"2024-04-21T08:39:58.006977+0000","last_became_active":"2024-04-21T08:37:35.608786+0000","last_became_peered":"2024-04-21T08:37:35.608786+0000","last_unstale":"2024-04-21T08:39:58.006977+0000","last_undegraded":"2024-04-21T08:39:58.006977+0000","last_fullsized":"2024-04-21T08:39:58.006977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:45:50.945845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903818+0000","last_change":"2024-04-21T08:37:30.417156+0000","last_active":"2024-04-21T08:39:57.903818+0000","last_peered":"2024-04-21T08:39:57.903818+0000","last_clean":"2024-04-21T08:39:57.903818+0000","last_became_active":"2024-04-21T08:37:30.415717+0000","last_became_peered":"2024-04-21T08:37:30.415717+0000","last_unstale":"2024-04-21T08:39:57.903818+0000","last_undegraded":"2024-04-21T08:39:57.903818+0000","last_fullsized":"2024-04-21T08:39:57.903818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:37.819082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318260+0000","last_change":"2024-04-21T08:37:31.444948+0000","last_active":"2024-04-21T08:39:58.318260+0000","last_peered":"2024-04-21T08:39:58.318260+0000","last_clean":"2024-04-21T08:39:58.318260+0000","last_became_active":"2024-04-21T08:37:31.444604+0000","last_became_peered":"2024-04-21T08:37:31.444604+0000","last_unstale":"2024-04-21T08:39:58.318260+0000","last_undegraded":"2024-04-21T08:39:58.318260+0000","last_fullsized":"2024-04-21T08:39:58.318260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:11.820503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151386+0000","last_change":"2024-04-21T08:37:35.611541+0000","last_active":"2024-04-21T08:39:58.151386+0000","last_peered":"2024-04-21T08:39:58.151386+0000","last_clean":"2024-04-21T08:39:58.151386+0000","last_became_active":"2024-04-21T08:37:35.611136+0000","last_became_peered":"2024-04-21T08:37:35.611136+0000","last_unstale":"2024-04-21T08:39:58.151386+0000","last_undegraded":"2024-04-21T08:39:58.151386+0000","last_fullsized":"2024-04-21T08:39:58.151386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:28.254529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905028+0000","last_change":"2024-04-21T08:37:30.414216+0000","last_active":"2024-04-21T08:39:57.905028+0000","last_peered":"2024-04-21T08:39:57.905028+0000","last_clean":"2024-04-21T08:39:57.905028+0000","last_became_active":"2024-04-21T08:37:30.413890+0000","last_became_peered":"2024-04-21T08:37:30.413890+0000","last_unstale":"2024-04-21T08:39:57.905028+0000","last_undegraded":"2024-04-21T08:39:57.905028+0000","last_fullsized":"2024-04-21T08:39:57.905028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:16.972618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903800+0000","last_change":"2024-04-21T08:37:31.449307+0000","last_active":"2024-04-21T08:39:57.903800+0000","last_peered":"2024-04-21T08:39:57.903800+0000","last_clean":"2024-04-21T08:39:57.903800+0000","last_became_active":"2024-04-21T08:37:31.448980+0000","last_became_peered":"2024-04-21T08:37:31.448980+0000","last_unstale":"2024-04-21T08:39:57.903800+0000","last_undegraded":"2024-04-21T08:39:57.903800+0000","last_fullsized":"2024-04-21T08:39:57.903800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:30:31.939274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.151476+0000","last_change":"2024-04-21T08:37:35.760616+0000","last_active":"2024-04-21T08:39:58.151476+0000","last_peered":"2024-04-21T08:39:58.151476+0000","last_clean":"2024-04-21T08:39:58.151476+0000","last_became_active":"2024-04-21T08:37:35.760131+0000","last_became_peered":"2024-04-21T08:37:35.760131+0000","last_unstale":"2024-04-21T08:39:58.151476+0000","last_undegraded":"2024-04-21T08:39:58.151476+0000","last_fullsized":"2024-04-21T08:39:58.151476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:51.510259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.009114+0000","last_change":"2024-04-21T08:38:30.682791+0000","last_active":"2024-04-21T08:39:58.009114+0000","last_peered":"2024-04-21T08:39:58.009114+0000","last_clean":"2024-04-21T08:39:58.009114+0000","last_became_active":"2024-04-21T08:38:30.671977+0000","last_became_peered":"2024-04-21T08:38:30.671977+0000","last_unstale":"2024-04-21T08:39:58.009114+0000","last_undegraded":"2024-04-21T08:39:58.009114+0000","last_fullsized":"2024-04-21T08:39:58.009114+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:55:15.899399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905569+0000","last_change":"2024-04-21T08:37:31.447234+0000","last_active":"2024-04-21T08:39:57.905569+0000","last_peered":"2024-04-21T08:39:57.905569+0000","last_clean":"2024-04-21T08:39:57.905569+0000","last_became_active":"2024-04-21T08:37:31.446876+0000","last_became_peered":"2024-04-21T08:37:31.446876+0000","last_unstale":"2024-04-21T08:39:57.905569+0000","last_undegraded":"2024-04-21T08:39:57.905569+0000","last_fullsized":"2024-04-21T08:39:57.905569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:51.784126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.153981+0000","last_change":"2024-04-21T08:37:35.598577+0000","last_active":"2024-04-21T08:39:58.153981+0000","last_peered":"2024-04-21T08:39:58.153981+0000","last_clean":"2024-04-21T08:39:58.153981+0000","last_became_active":"2024-04-21T08:37:35.598119+0000","last_became_peered":"2024-04-21T08:37:35.598119+0000","last_unstale":"2024-04-21T08:39:58.153981+0000","last_undegraded":"2024-04-21T08:39:58.153981+0000","last_fullsized":"2024-04-21T08:39:58.153981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:41.986674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905332+0000","last_change":"2024-04-21T08:37:30.425737+0000","last_active":"2024-04-21T08:39:57.905332+0000","last_peered":"2024-04-21T08:39:57.905332+0000","last_clean":"2024-04-21T08:39:57.905332+0000","last_became_active":"2024-04-21T08:37:30.425180+0000","last_became_peered":"2024-04-21T08:37:30.425180+0000","last_unstale":"2024-04-21T08:39:57.905332+0000","last_undegraded":"2024-04-21T08:39:57.905332+0000","last_fullsized":"2024-04-21T08:39:57.905332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:08.646106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.906011+0000","last_change":"2024-04-21T08:37:31.451505+0000","last_active":"2024-04-21T08:39:57.906011+0000","last_peered":"2024-04-21T08:39:57.906011+0000","last_clean":"2024-04-21T08:39:57.906011+0000","last_became_active":"2024-04-21T08:37:31.451146+0000","last_became_peered":"2024-04-21T08:37:31.451146+0000","last_unstale":"2024-04-21T08:39:57.906011+0000","last_undegraded":"2024-04-21T08:39:57.906011+0000","last_fullsized":"2024-04-21T08:39:57.906011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:27:01.793910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319583+0000","last_change":"2024-04-21T08:37:35.608995+0000","last_active":"2024-04-21T08:39:58.319583+0000","last_peered":"2024-04-21T08:39:58.319583+0000","last_clean":"2024-04-21T08:39:58.319583+0000","last_became_active":"2024-04-21T08:37:35.608587+0000","last_became_peered":"2024-04-21T08:37:35.608587+0000","last_unstale":"2024-04-21T08:39:58.319583+0000","last_undegraded":"2024-04-21T08:39:58.319583+0000","last_fullsized":"2024-04-21T08:39:58.319583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:50:37.829418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'65","reported_seq":130,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:40:07.001710+0000","last_change":"2024-04-21T08:37:30.419026+0000","last_active":"2024-04-21T08:40:07.001710+0000","last_peered":"2024-04-21T08:40:07.001710+0000","last_clean":"2024-04-21T08:40:07.001710+0000","last_became_active":"2024-04-21T08:37:30.418706+0000","last_became_peered":"2024-04-21T08:37:30.418706+0000","last_unstale":"2024-04-21T08:40:07.001710+0000","last_undegraded":"2024-04-21T08:40:07.001710+0000","last_fullsized":"2024-04-21T08:40:07.001710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:24.441918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905192+0000","last_change":"2024-04-21T08:37:31.446561+0000","last_active":"2024-04-21T08:39:57.905192+0000","last_peered":"2024-04-21T08:39:57.905192+0000","last_clean":"2024-04-21T08:39:57.905192+0000","last_became_active":"2024-04-21T08:37:31.446162+0000","last_became_peered":"2024-04-21T08:37:31.446162+0000","last_unstale":"2024-04-21T08:39:57.905192+0000","last_undegraded":"2024-04-21T08:39:57.905192+0000","last_fullsized":"2024-04-21T08:39:57.905192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:38:24.684537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013588+0000","last_change":"2024-04-21T08:37:35.598748+0000","last_active":"2024-04-21T08:39:58.013588+0000","last_peered":"2024-04-21T08:39:58.013588+0000","last_clean":"2024-04-21T08:39:58.013588+0000","last_became_active":"2024-04-21T08:37:35.598353+0000","last_became_peered":"2024-04-21T08:37:35.598353+0000","last_unstale":"2024-04-21T08:39:58.013588+0000","last_undegraded":"2024-04-21T08:39:58.013588+0000","last_fullsized":"2024-04-21T08:39:58.013588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.572010+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.572010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:47:36.519694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319054+0000","last_change":"2024-04-21T08:38:30.691292+0000","last_active":"2024-04-21T08:39:58.319054+0000","last_peered":"2024-04-21T08:39:58.319054+0000","last_clean":"2024-04-21T08:39:58.319054+0000","last_became_active":"2024-04-21T08:38:30.676580+0000","last_became_peered":"2024-04-21T08:38:30.676580+0000","last_unstale":"2024-04-21T08:39:58.319054+0000","last_undegraded":"2024-04-21T08:39:58.319054+0000","last_fullsized":"2024-04-21T08:39:58.319054+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:29.398430+0000","last_clean_scrub_stamp":"2024-04-21T08:37:29.398430+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:43:53.150369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905478+0000","last_change":"2024-04-21T08:37:31.452367+0000","last_active":"2024-04-21T08:39:57.905478+0000","last_peered":"2024-04-21T08:39:57.905478+0000","last_clean":"2024-04-21T08:39:57.905478+0000","last_became_active":"2024-04-21T08:37:31.452031+0000","last_became_peered":"2024-04-21T08:37:31.452031+0000","last_unstale":"2024-04-21T08:39:57.905478+0000","last_undegraded":"2024-04-21T08:39:57.905478+0000","last_fullsized":"2024-04-21T08:39:57.905478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:27:58.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318378+0000","last_change":"2024-04-21T08:37:31.451222+0000","last_active":"2024-04-21T08:39:58.318378+0000","last_peered":"2024-04-21T08:39:58.318378+0000","last_clean":"2024-04-21T08:39:58.318378+0000","last_became_active":"2024-04-21T08:37:31.450869+0000","last_became_peered":"2024-04-21T08:37:31.450869+0000","last_unstale":"2024-04-21T08:39:58.318378+0000","last_undegraded":"2024-04-21T08:39:58.318378+0000","last_fullsized":"2024-04-21T08:39:58.318378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:35:17.051330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905137+0000","last_change":"2024-04-21T08:37:31.453415+0000","last_active":"2024-04-21T08:39:57.905137+0000","last_peered":"2024-04-21T08:39:57.905137+0000","last_clean":"2024-04-21T08:39:57.905137+0000","last_became_active":"2024-04-21T08:37:31.453089+0000","last_became_peered":"2024-04-21T08:37:31.453089+0000","last_unstale":"2024-04-21T08:39:57.905137+0000","last_undegraded":"2024-04-21T08:39:57.905137+0000","last_fullsized":"2024-04-21T08:39:57.905137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:25:44.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904904+0000","last_change":"2024-04-21T08:37:31.446707+0000","last_active":"2024-04-21T08:39:57.904904+0000","last_peered":"2024-04-21T08:39:57.904904+0000","last_clean":"2024-04-21T08:39:57.904904+0000","last_became_active":"2024-04-21T08:37:31.446340+0000","last_became_peered":"2024-04-21T08:37:31.446340+0000","last_unstale":"2024-04-21T08:39:57.904904+0000","last_undegraded":"2024-04-21T08:39:57.904904+0000","last_fullsized":"2024-04-21T08:39:57.904904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:13.649607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.319500+0000","last_change":"2024-04-21T08:37:31.455532+0000","last_active":"2024-04-21T08:39:58.319500+0000","last_peered":"2024-04-21T08:39:58.319500+0000","last_clean":"2024-04-21T08:39:58.319500+0000","last_became_active":"2024-04-21T08:37:31.453547+0000","last_became_peered":"2024-04-21T08:37:31.453547+0000","last_unstale":"2024-04-21T08:39:58.319500+0000","last_undegraded":"2024-04-21T08:39:58.319500+0000","last_fullsized":"2024-04-21T08:39:58.319500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:39:47.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318944+0000","last_change":"2024-04-21T08:39:31.679882+0000","last_active":"2024-04-21T08:39:58.318944+0000","last_peered":"2024-04-21T08:39:58.318944+0000","last_clean":"2024-04-21T08:39:58.318944+0000","last_became_active":"2024-04-21T08:39:31.679440+0000","last_became_peered":"2024-04-21T08:39:31.679440+0000","last_unstale":"2024-04-21T08:39:58.318944+0000","last_undegraded":"2024-04-21T08:39:58.318944+0000","last_fullsized":"2024-04-21T08:39:58.318944+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:57.679038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318685+0000","last_change":"2024-04-21T08:37:31.451806+0000","last_active":"2024-04-21T08:39:58.318685+0000","last_peered":"2024-04-21T08:39:58.318685+0000","last_clean":"2024-04-21T08:39:58.318685+0000","last_became_active":"2024-04-21T08:37:31.451442+0000","last_became_peered":"2024-04-21T08:37:31.451442+0000","last_unstale":"2024-04-21T08:39:58.318685+0000","last_undegraded":"2024-04-21T08:39:58.318685+0000","last_fullsized":"2024-04-21T08:39:58.318685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:07.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.905544+0000","last_change":"2024-04-21T08:37:31.444440+0000","last_active":"2024-04-21T08:39:57.905544+0000","last_peered":"2024-04-21T08:39:57.905544+0000","last_clean":"2024-04-21T08:39:57.905544+0000","last_became_active":"2024-04-21T08:37:31.444079+0000","last_became_peered":"2024-04-21T08:37:31.444079+0000","last_unstale":"2024-04-21T08:39:57.905544+0000","last_undegraded":"2024-04-21T08:39:57.905544+0000","last_fullsized":"2024-04-21T08:39:57.905544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:13.925700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.904413+0000","last_change":"2024-04-21T08:37:31.453698+0000","last_active":"2024-04-21T08:39:57.904413+0000","last_peered":"2024-04-21T08:39:57.904413+0000","last_clean":"2024-04-21T08:39:57.904413+0000","last_became_active":"2024-04-21T08:37:31.453379+0000","last_became_peered":"2024-04-21T08:37:31.453379+0000","last_unstale":"2024-04-21T08:39:57.904413+0000","last_undegraded":"2024-04-21T08:39:57.904413+0000","last_fullsized":"2024-04-21T08:39:57.904413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:46:29.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.013777+0000","last_change":"2024-04-21T08:37:31.444480+0000","last_active":"2024-04-21T08:39:58.013777+0000","last_peered":"2024-04-21T08:39:58.013777+0000","last_clean":"2024-04-21T08:39:58.013777+0000","last_became_active":"2024-04-21T08:37:31.444030+0000","last_became_peered":"2024-04-21T08:37:31.444030+0000","last_unstale":"2024-04-21T08:39:58.013777+0000","last_undegraded":"2024-04-21T08:39:58.013777+0000","last_fullsized":"2024-04-21T08:39:58.013777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:15:05.668885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:57.903935+0000","last_change":"2024-04-21T08:37:31.456602+0000","last_active":"2024-04-21T08:39:57.903935+0000","last_peered":"2024-04-21T08:39:57.903935+0000","last_clean":"2024-04-21T08:39:57.903935+0000","last_became_active":"2024-04-21T08:37:31.456286+0000","last_became_peered":"2024-04-21T08:37:31.456286+0000","last_unstale":"2024-04-21T08:39:57.903935+0000","last_undegraded":"2024-04-21T08:39:57.903935+0000","last_fullsized":"2024-04-21T08:39:57.903935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:48:12.462037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.154176+0000","last_change":"2024-04-21T08:37:31.447008+0000","last_active":"2024-04-21T08:39:58.154176+0000","last_peered":"2024-04-21T08:39:58.154176+0000","last_clean":"2024-04-21T08:39:58.154176+0000","last_became_active":"2024-04-21T08:37:31.445192+0000","last_became_peered":"2024-04-21T08:37:31.445192+0000","last_unstale":"2024-04-21T08:39:58.154176+0000","last_undegraded":"2024-04-21T08:39:58.154176+0000","last_fullsized":"2024-04-21T08:39:58.154176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:08.025647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318698+0000","last_change":"2024-04-21T08:37:31.443288+0000","last_active":"2024-04-21T08:39:58.318698+0000","last_peered":"2024-04-21T08:39:58.318698+0000","last_clean":"2024-04-21T08:39:58.318698+0000","last_became_active":"2024-04-21T08:37:31.442586+0000","last_became_peered":"2024-04-21T08:37:31.442586+0000","last_unstale":"2024-04-21T08:39:58.318698+0000","last_undegraded":"2024-04-21T08:39:58.318698+0000","last_fullsized":"2024-04-21T08:39:58.318698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:01.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:39:58.318501+0000","last_change":"2024-04-21T08:37:31.453899+0000","last_active":"2024-04-21T08:39:58.318501+0000","last_peered":"2024-04-21T08:39:58.318501+0000","last_clean":"2024-04-21T08:39:58.318501+0000","last_became_active":"2024-04-21T08:37:31.453527+0000","last_became_peered":"2024-04-21T08:37:31.453527+0000","last_unstale":"2024-04-21T08:39:58.318501+0000","last_undegraded":"2024-04-21T08:39:58.318501+0000","last_fullsized":"2024-04-21T08:39:58.318501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:30.399867+0000","last_clean_scrub_stamp":"2024-04-21T08:37:30.399867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:23:18.429618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108246,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":800441,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1183,"num_read_kb":813,"num_write":424,"num_write_kb":2366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2401323,"data_compressed":108456,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383724,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":529807,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32384,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712768,"statfs":{"total":95995035648,"available":95961874432,"internally_reserved":0,"allocated":1228800,"data_stored":1520299,"data_compressed":82643,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":927355,"data_compressed":48987,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":922597,"data_compressed":48987,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1127885,"data_compressed":65780,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1028096,"data_stored":525219,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":598690,"data_compressed":31359,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1052672,"data_stored":526607,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1126400,"data_stored":599849,"data_compressed":31359,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1246861,"data_compressed":64189,"data_compressed_allocated":86016,"data_compressed_original":782336,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":531882,"data_compressed":29698,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1150976,"data_stored":1449692,"data_compressed":80982,"data_compressed_allocated":106496,"data_compressed_original":978944,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36082,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":73561,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":74720,"data_compressed":1661,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330099,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393372,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":404802,"data_compressed":16863,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:40:11.497 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:40:11.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:11 smithi173 ceph-mon[31428]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-21T08:40:11.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:11 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2883492950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:11.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:11 smithi173 ceph-mon[31428]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:11.601 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:11 smithi111 ceph-mon[31493]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-21T08:40:11.601 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:11 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2883492950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:11.601 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:11 smithi111 ceph-mon[31493]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:11.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.631+0000 7f0744006640 1 -- 172.21.15.94:0/3668595258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f073c105ea0 msgr2=0x7f073c106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.631+0000 7f0744006640 1 --2- 172.21.15.94:0/3668595258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f073c105ea0 0x7f073c106280 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0738009900 tx=0x7f073802bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:11.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.631+0000 7f0744006640 1 -- 172.21.15.94:0/3668595258 shutdown_connections 2024-04-21T08:40:11.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.631+0000 7f0744006640 1 --2- 172.21.15.94:0/3668595258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073c1011a0 0x7f073c101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.631+0000 7f0744006640 1 --2- 172.21.15.94:0/3668595258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f073c101b40 0x7f073c10dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.631+0000 7f0744006640 1 --2- 172.21.15.94:0/3668595258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f073c105ea0 0x7f073c106280 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.631+0000 7f0744006640 1 -- 172.21.15.94:0/3668595258 >> 172.21.15.94:0/3668595258 conn(0x7f073c077e30 msgr2=0x7f073c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:11.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.631+0000 7f0744006640 1 -- 172.21.15.94:0/3668595258 shutdown_connections 2024-04-21T08:40:11.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.631+0000 7f0744006640 1 -- 172.21.15.94:0/3668595258 wait complete. 2024-04-21T08:40:11.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f0744006640 1 Processor -- start 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f0744006640 1 -- start start 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f0744006640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f073c1011a0 0x7f073c199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f0744006640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f073c101b40 0x7f073c199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f0744006640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073c105ea0 0x7f073c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f0744006640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f073c1110c0 con 0x7f073c101b40 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f0744006640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f073c110f40 con 0x7f073c1011a0 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f0744006640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f073c111240 con 0x7f073c105ea0 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f07415a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f073c101b40 0x7f073c199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f07415a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f073c101b40 0x7f073c199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37036/0 (socket says 172.21.15.94:37036) 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f07415a1640 1 -- 172.21.15.94:0/3414888163 learned_addr learned my addr 172.21.15.94:0/3414888163 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f0741da2640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f073c1011a0 0x7f073c199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f07415a1640 1 -- 172.21.15.94:0/3414888163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073c105ea0 msgr2=0x7f073c19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f07415a1640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073c105ea0 0x7f073c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f07415a1640 1 -- 172.21.15.94:0/3414888163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f073c1011a0 msgr2=0x7f073c199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f07415a1640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f073c1011a0 0x7f073c199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f07415a1640 1 -- 172.21.15.94:0/3414888163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073c1a4940 con 0x7f073c101b40 2024-04-21T08:40:11.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.632+0000 7f07415a1640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f073c101b40 0x7f073c199a80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f072c00c920 tx=0x7f072c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:11.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.633+0000 7f0732ffd640 1 -- 172.21.15.94:0/3414888163 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c007c70 con 0x7f073c101b40 2024-04-21T08:40:11.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.633+0000 7f0732ffd640 1 -- 172.21.15.94:0/3414888163 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f072c007e10 con 0x7f073c101b40 2024-04-21T08:40:11.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.633+0000 7f0732ffd640 1 -- 172.21.15.94:0/3414888163 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c018820 con 0x7f073c101b40 2024-04-21T08:40:11.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.633+0000 7f0744006640 1 -- 172.21.15.94:0/3414888163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073c1a4c30 con 0x7f073c101b40 2024-04-21T08:40:11.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.633+0000 7f0744006640 1 -- 172.21.15.94:0/3414888163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f073c1a51d0 con 0x7f073c101b40 2024-04-21T08:40:11.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.634+0000 7f0744006640 1 -- 172.21.15.94:0/3414888163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f073c102590 con 0x7f073c101b40 2024-04-21T08:40:11.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.635+0000 7f0732ffd640 1 -- 172.21.15.94:0/3414888163 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f072c00f450 con 0x7f073c101b40 2024-04-21T08:40:11.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.635+0000 7f0732ffd640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f07280a3ec0 0x7f07280a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.635+0000 7f0741da2640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f07280a3ec0 0x7f07280a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.635+0000 7f0732ffd640 1 -- 172.21.15.94:0/3414888163 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f072c0d0f50 con 0x7f073c101b40 2024-04-21T08:40:11.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.636+0000 7f0741da2640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f07280a3ec0 0x7f07280a6380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0738006010 tx=0x7f073802c6c0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:11.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.640+0000 7f0732ffd640 1 -- 172.21.15.94:0/3414888163 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f072c014030 con 0x7f073c101b40 2024-04-21T08:40:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.857+0000 7f0744006640 1 -- 172.21.15.94:0/3414888163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f073c069230 con 0x7f073c101b40 2024-04-21T08:40:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.858+0000 7f0732ffd640 1 -- 172.21.15.94:0/3414888163 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f072c09cfc0 con 0x7f073c101b40 2024-04-21T08:40:11.857 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:11.858 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:40:11.858 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 -- 172.21.15.94:0/3414888163 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f07280a3ec0 msgr2=0x7f07280a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f07280a3ec0 0x7f07280a6380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0738006010 tx=0x7f073802c6c0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 -- 172.21.15.94:0/3414888163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f073c101b40 msgr2=0x7f073c199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f073c101b40 0x7f073c199a80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f072c00c920 tx=0x7f072c00cde0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 -- 172.21.15.94:0/3414888163 shutdown_connections 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f07280a3ec0 0x7f07280a6380 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073c105ea0 0x7f073c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f073c101b40 0x7f073c199a80 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 --2- 172.21.15.94:0/3414888163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f073c1011a0 0x7f073c199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 -- 172.21.15.94:0/3414888163 >> 172.21.15.94:0/3414888163 conn(0x7f073c077e30 msgr2=0x7f073c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 -- 172.21.15.94:0/3414888163 shutdown_connections 2024-04-21T08:40:11.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.860+0000 7f0744006640 1 -- 172.21.15.94:0/3414888163 wait complete. 2024-04-21T08:40:11.871 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:40:11.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.970+0000 7fee81874640 1 -- 172.21.15.94:0/1964422609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7c1011a0 msgr2=0x7fee7c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.970+0000 7fee81874640 1 --2- 172.21.15.94:0/1964422609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7c1011a0 0x7fee7c101620 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fee64009930 tx=0x7fee6402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.970+0000 7fee81874640 1 -- 172.21.15.94:0/1964422609 shutdown_connections 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.970+0000 7fee81874640 1 --2- 172.21.15.94:0/1964422609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee7c101b60 0x7fee7c10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.970+0000 7fee81874640 1 --2- 172.21.15.94:0/1964422609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7c1011a0 0x7fee7c101620 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.970+0000 7fee81874640 1 --2- 172.21.15.94:0/1964422609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee7c105e30 0x7fee7c106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.970+0000 7fee81874640 1 -- 172.21.15.94:0/1964422609 >> 172.21.15.94:0/1964422609 conn(0x7fee7c077e30 msgr2=0x7fee7c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee81874640 1 -- 172.21.15.94:0/1964422609 shutdown_connections 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee81874640 1 -- 172.21.15.94:0/1964422609 wait complete. 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee81874640 1 Processor -- start 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee81874640 1 -- start start 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee81874640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7c1011a0 0x7fee7c197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee81874640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee7c101b60 0x7fee7c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee81874640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee7c105e30 0x7fee7c19bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee81874640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c1110e0 con 0x7fee7c1011a0 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee81874640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c110f60 con 0x7fee7c105e30 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee81874640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c111260 con 0x7fee7c101b60 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee7affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7c1011a0 0x7fee7c197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee7affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7c1011a0 0x7fee7c197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37064/0 (socket says 172.21.15.94:37064) 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee7affd640 1 -- 172.21.15.94:0/1388340150 learned_addr learned my addr 172.21.15.94:0/1388340150 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee7b7fe640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee7c105e30 0x7fee7c19bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.971+0000 7fee73fff640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee7c101b60 0x7fee7c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee7affd640 1 -- 172.21.15.94:0/1388340150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee7c101b60 msgr2=0x7fee7c197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee7affd640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee7c101b60 0x7fee7c197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee7affd640 1 -- 172.21.15.94:0/1388340150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee7c105e30 msgr2=0x7fee7c19bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee7affd640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee7c105e30 0x7fee7c19bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:11.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee7affd640 1 -- 172.21.15.94:0/1388340150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7c19c670 con 0x7fee7c1011a0 2024-04-21T08:40:11.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee73fff640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee7c101b60 0x7fee7c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:11.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee7affd640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7c1011a0 0x7fee7c197680 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00d770 tx=0x7fee6c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:11.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee80872640 1 -- 172.21.15.94:0/1388340150 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c017070 con 0x7fee7c1011a0 2024-04-21T08:40:11.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee80872640 1 -- 172.21.15.94:0/1388340150 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee6c0054d0 con 0x7fee7c1011a0 2024-04-21T08:40:11.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee81874640 1 -- 172.21.15.94:0/1388340150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7c19c960 con 0x7fee7c1011a0 2024-04-21T08:40:11.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee80872640 1 -- 172.21.15.94:0/1388340150 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c004c30 con 0x7fee7c1011a0 2024-04-21T08:40:11.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.972+0000 7fee81874640 1 -- 172.21.15.94:0/1388340150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee7c1a41f0 con 0x7fee7c1011a0 2024-04-21T08:40:11.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.973+0000 7fee81874640 1 -- 172.21.15.94:0/1388340150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee7c068de0 con 0x7fee7c1011a0 2024-04-21T08:40:11.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.974+0000 7fee80872640 1 -- 172.21.15.94:0/1388340150 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fee6c004e10 con 0x7fee7c1011a0 2024-04-21T08:40:11.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.974+0000 7fee80872640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fee680a3e40 0x7fee680a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:11.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.975+0000 7fee73fff640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fee680a3e40 0x7fee680a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:11.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.975+0000 7fee80872640 1 -- 172.21.15.94:0/1388340150 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fee6c0d0960 con 0x7fee7c1011a0 2024-04-21T08:40:11.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.975+0000 7fee73fff640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fee680a3e40 0x7fee680a6300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee64006010 tx=0x7fee64009610 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:11.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:11.979+0000 7fee80872640 1 -- 172.21.15.94:0/1388340150 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee6c09c9d0 con 0x7fee7c1011a0 2024-04-21T08:40:12.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.158+0000 7fee81874640 1 -- 172.21.15.94:0/1388340150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee7c1989d0 con 0x7fee7c1011a0 2024-04-21T08:40:12.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.160+0000 7fee80872640 1 -- 172.21.15.94:0/1388340150 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25266 (secure 0 0 0) 0x7fee6c09c2d0 con 0x7fee7c1011a0 2024-04-21T08:40:12.160 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:12.160 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:57.898777+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:40:12.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 -- 172.21.15.94:0/1388340150 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fee680a3e40 msgr2=0x7fee680a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fee680a3e40 0x7fee680a6300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee64006010 tx=0x7fee64009610 comp rx=0 tx=0).stop 2024-04-21T08:40:12.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 -- 172.21.15.94:0/1388340150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7c1011a0 msgr2=0x7fee7c197680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7c1011a0 0x7fee7c197680 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00d770 tx=0x7fee6c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:40:12.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 -- 172.21.15.94:0/1388340150 shutdown_connections 2024-04-21T08:40:12.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fee680a3e40 0x7fee680a6300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee7c101b60 0x7fee7c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee7c105e30 0x7fee7c19bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 --2- 172.21.15.94:0/1388340150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7c1011a0 0x7fee7c197680 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 -- 172.21.15.94:0/1388340150 >> 172.21.15.94:0/1388340150 conn(0x7fee7c077e30 msgr2=0x7fee7c0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:12.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 -- 172.21.15.94:0/1388340150 shutdown_connections 2024-04-21T08:40:12.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.162+0000 7fee81874640 1 -- 172.21.15.94:0/1388340150 wait complete. 2024-04-21T08:40:12.173 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:40:12.174 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:40:12.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.279+0000 7f94b626d640 1 -- 172.21.15.94:0/4235434955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94b0102b50 msgr2=0x7f94b0106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.279+0000 7f94b626d640 1 --2- 172.21.15.94:0/4235434955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94b0102b50 0x7f94b0106b90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009900 tx=0x7f94a402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.279+0000 7f94b626d640 1 -- 172.21.15.94:0/4235434955 shutdown_connections 2024-04-21T08:40:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.279+0000 7f94b626d640 1 --2- 172.21.15.94:0/4235434955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94b0107270 0x7f94b010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.279+0000 7f94b626d640 1 --2- 172.21.15.94:0/4235434955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94b0102b50 0x7f94b0106b90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.279+0000 7f94b626d640 1 --2- 172.21.15.94:0/4235434955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b01021a0 0x7f94b0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.279+0000 7f94b626d640 1 -- 172.21.15.94:0/4235434955 >> 172.21.15.94:0/4235434955 conn(0x7f94b00fd9c0 msgr2=0x7f94b00ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.279+0000 7f94b626d640 1 -- 172.21.15.94:0/4235434955 shutdown_connections 2024-04-21T08:40:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.279+0000 7f94b626d640 1 -- 172.21.15.94:0/4235434955 wait complete. 2024-04-21T08:40:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b626d640 1 Processor -- start 2024-04-21T08:40:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b626d640 1 -- start start 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b626d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94b01021a0 0x7f94b019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b626d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b0107270 0x7f94b019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b626d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94b01985e0 0x7f94b0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b626d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f94b01121b0 con 0x7f94b01985e0 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b626d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f94b0112030 con 0x7f94b0107270 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b626d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f94b0112330 con 0x7f94b01021a0 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b480a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94b01985e0 0x7f94b0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b480a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94b01985e0 0x7f94b0198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37082/0 (socket says 172.21.15.94:37082) 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94b480a640 1 -- 172.21.15.94:0/3836393148 learned_addr learned my addr 172.21.15.94:0/3836393148 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94affff640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94b01021a0 0x7f94b019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.280+0000 7f94af7fe640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b0107270 0x7f94b019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94b480a640 1 -- 172.21.15.94:0/3836393148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94b01021a0 msgr2=0x7f94b019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94b480a640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94b01021a0 0x7f94b019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94b480a640 1 -- 172.21.15.94:0/3836393148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b0107270 msgr2=0x7f94b019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94b480a640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b0107270 0x7f94b019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94b480a640 1 -- 172.21.15.94:0/3836393148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b0199100 con 0x7f94b01985e0 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94b480a640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94b01985e0 0x7f94b0198a90 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f94a800eaa0 tx=0x7f94a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94ad7fa640 1 -- 172.21.15.94:0/3836393148 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a800cdb0 con 0x7f94b01985e0 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94ad7fa640 1 -- 172.21.15.94:0/3836393148 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94a8005020 con 0x7f94b01985e0 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94b626d640 1 -- 172.21.15.94:0/3836393148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b01993f0 con 0x7f94b01985e0 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94ad7fa640 1 -- 172.21.15.94:0/3836393148 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a80116d0 con 0x7f94b01985e0 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.281+0000 7f94b626d640 1 -- 172.21.15.94:0/3836393148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b01a55f0 con 0x7f94b01985e0 2024-04-21T08:40:12.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.282+0000 7f94b626d640 1 -- 172.21.15.94:0/3836393148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b00691c0 con 0x7f94b01985e0 2024-04-21T08:40:12.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.283+0000 7f94ad7fa640 1 -- 172.21.15.94:0/3836393148 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f94a80051d0 con 0x7f94b01985e0 2024-04-21T08:40:12.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.284+0000 7f94ad7fa640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f94980a3e40 0x7f94980a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.284+0000 7f94affff640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f94980a3e40 0x7f94980a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.284+0000 7f94ad7fa640 1 -- 172.21.15.94:0/3836393148 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f94a80d12d0 con 0x7f94b01985e0 2024-04-21T08:40:12.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.284+0000 7f94affff640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f94980a3e40 0x7f94980a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f94a0005d00 tx=0x7f94a000b410 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:12.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.287+0000 7f94ad7fa640 1 -- 172.21.15.94:0/3836393148 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94a8016030 con 0x7f94b01985e0 2024-04-21T08:40:12.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.464+0000 7f94b626d640 1 -- 172.21.15.94:0/3836393148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94b0102b50 con 0x7f94b01985e0 2024-04-21T08:40:12.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.464+0000 7f94ad7fa640 1 -- 172.21.15.94:0/3836393148 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f94a809ccf0 con 0x7f94b01985e0 2024-04-21T08:40:12.464 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:40:12.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.466+0000 7f94b626d640 1 -- 172.21.15.94:0/3836393148 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f94980a3e40 msgr2=0x7f94980a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.466+0000 7f94b626d640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f94980a3e40 0x7f94980a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f94a0005d00 tx=0x7f94a000b410 comp rx=0 tx=0).stop 2024-04-21T08:40:12.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.466+0000 7f94b626d640 1 -- 172.21.15.94:0/3836393148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94b01985e0 msgr2=0x7f94b0198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.466+0000 7f94b626d640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94b01985e0 0x7f94b0198a90 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f94a800eaa0 tx=0x7f94a800ef60 comp rx=0 tx=0).stop 2024-04-21T08:40:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.467+0000 7f94b626d640 1 -- 172.21.15.94:0/3836393148 shutdown_connections 2024-04-21T08:40:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.467+0000 7f94b626d640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f94980a3e40 0x7f94980a6300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.467+0000 7f94b626d640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94b01985e0 0x7f94b0198a90 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.467+0000 7f94b626d640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b0107270 0x7f94b019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.467+0000 7f94b626d640 1 --2- 172.21.15.94:0/3836393148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94b01021a0 0x7f94b019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.467+0000 7f94b626d640 1 -- 172.21.15.94:0/3836393148 >> 172.21.15.94:0/3836393148 conn(0x7f94b00fd9c0 msgr2=0x7f94b00fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.467+0000 7f94b626d640 1 -- 172.21.15.94:0/3836393148 shutdown_connections 2024-04-21T08:40:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.467+0000 7f94b626d640 1 -- 172.21.15.94:0/3836393148 wait complete. 2024-04-21T08:40:12.476 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:12 smithi094 ceph-mon[27988]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:12.476 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:12 smithi094 ceph-mon[27988]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:12.476 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:12 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3414888163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:12.476 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:12 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1388340150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:12.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:40:12.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:12 smithi173 ceph-mon[31428]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:12.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:12 smithi173 ceph-mon[31428]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:12.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:12 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3414888163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:12.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:12 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1388340150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:12.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 -- 172.21.15.94:0/4001166025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4006a570 msgr2=0x7f9f4006a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 --2- 172.21.15.94:0/4001166025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4006a570 0x7f9f4006a9f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c009930 tx=0x7f9f2c02bc50 comp rx=0 tx=0).stop 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 -- 172.21.15.94:0/4001166025 shutdown_connections 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 --2- 172.21.15.94:0/4001166025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f4006af30 0x7f9f4010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 --2- 172.21.15.94:0/4001166025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4006a570 0x7f9f4006a9f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 --2- 172.21.15.94:0/4001166025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f40106f90 0x7f9f40107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 -- 172.21.15.94:0/4001166025 >> 172.21.15.94:0/4001166025 conn(0x7f9f40077e30 msgr2=0x7f9f40078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 -- 172.21.15.94:0/4001166025 shutdown_connections 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 -- 172.21.15.94:0/4001166025 wait complete. 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 Processor -- start 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.580+0000 7f9f46e3f640 1 -- start start 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f46e3f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4006a570 0x7f9f4019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f46e3f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f4006af30 0x7f9f4019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f46e3f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f40106f90 0x7f9f40198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f46e3f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9f40112290 con 0x7f9f4006a570 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f46e3f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9f40112110 con 0x7f9f4006af30 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f46e3f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9f40112410 con 0x7f9f40106f90 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f44bdb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4006a570 0x7f9f4019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f44bdb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4006a570 0x7f9f4019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37086/0 (socket says 172.21.15.94:37086) 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f44bdb640 1 -- 172.21.15.94:0/352803290 learned_addr learned my addr 172.21.15.94:0/352803290 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f3ffff640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f4006af30 0x7f9f4019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f44bdb640 1 -- 172.21.15.94:0/352803290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f40106f90 msgr2=0x7f9f40198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f44bdb640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f40106f90 0x7f9f40198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f44bdb640 1 -- 172.21.15.94:0/352803290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f4006af30 msgr2=0x7f9f4019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f44bdb640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f4006af30 0x7f9f4019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f44bdb640 1 -- 172.21.15.94:0/352803290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f40198db0 con 0x7f9f4006a570 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f44bdb640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4006a570 0x7f9f4019e430 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9f3400e9f0 tx=0x7f9f3400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:12.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f3dffb640 1 -- 172.21.15.94:0/352803290 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3400cd80 con 0x7f9f4006a570 2024-04-21T08:40:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f3dffb640 1 -- 172.21.15.94:0/352803290 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f340108e0 con 0x7f9f4006a570 2024-04-21T08:40:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f46e3f640 1 -- 172.21.15.94:0/352803290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f401990a0 con 0x7f9f4006a570 2024-04-21T08:40:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.581+0000 7f9f3dffb640 1 -- 172.21.15.94:0/352803290 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f34018980 con 0x7f9f4006a570 2024-04-21T08:40:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.582+0000 7f9f46e3f640 1 -- 172.21.15.94:0/352803290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f401a52f0 con 0x7f9f4006a570 2024-04-21T08:40:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.583+0000 7f9f46e3f640 1 -- 172.21.15.94:0/352803290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f400691c0 con 0x7f9f4006a570 2024-04-21T08:40:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.584+0000 7f9f3dffb640 1 -- 172.21.15.94:0/352803290 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f9f34010430 con 0x7f9f4006a570 2024-04-21T08:40:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.584+0000 7f9f3dffb640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9f300a3e40 0x7f9f300a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.585+0000 7f9f3ffff640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9f300a3e40 0x7f9f300a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.585+0000 7f9f3dffb640 1 -- 172.21.15.94:0/352803290 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9f340d1220 con 0x7f9f4006a570 2024-04-21T08:40:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.585+0000 7f9f3ffff640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9f300a3e40 0x7f9f300a6300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c005ab0 tx=0x7f9f2c005fd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.589+0000 7f9f3dffb640 1 -- 172.21.15.94:0/352803290 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9f3409d290 con 0x7f9f4006a570 2024-04-21T08:40:12.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:12 smithi111 ceph-mon[31493]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:12.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:12 smithi111 ceph-mon[31493]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:40:12.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:12 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3414888163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:12.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:12 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1388340150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:12.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.769+0000 7f9f46e3f640 1 -- 172.21.15.94:0/352803290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9f4002d5c0 con 0x7f9f4006a570 2024-04-21T08:40:12.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.769+0000 7f9f3dffb640 1 -- 172.21.15.94:0/352803290 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f9f3409cb90 con 0x7f9f4006a570 2024-04-21T08:40:12.769 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:12.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 -- 172.21.15.94:0/352803290 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9f300a3e40 msgr2=0x7f9f300a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9f300a3e40 0x7f9f300a6300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c005ab0 tx=0x7f9f2c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 -- 172.21.15.94:0/352803290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4006a570 msgr2=0x7f9f4019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4006a570 0x7f9f4019e430 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9f3400e9f0 tx=0x7f9f3400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 -- 172.21.15.94:0/352803290 shutdown_connections 2024-04-21T08:40:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9f300a3e40 0x7f9f300a6300 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f40106f90 0x7f9f40198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f4006af30 0x7f9f4019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 --2- 172.21.15.94:0/352803290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4006a570 0x7f9f4019e430 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 -- 172.21.15.94:0/352803290 >> 172.21.15.94:0/352803290 conn(0x7f9f40077e30 msgr2=0x7f9f400fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 -- 172.21.15.94:0/352803290 shutdown_connections 2024-04-21T08:40:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.771+0000 7f9f46e3f640 1 -- 172.21.15.94:0/352803290 wait complete. 2024-04-21T08:40:12.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:40:12.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.885+0000 7ff858e81640 1 -- 172.21.15.94:0/3452553710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff854100450 msgr2=0x7ff8541008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.885+0000 7ff858e81640 1 --2- 172.21.15.94:0/3452553710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff854100450 0x7ff8541008d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff848009cd0 tx=0x7ff84802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.885+0000 7ff858e81640 1 -- 172.21.15.94:0/3452553710 shutdown_connections 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.885+0000 7ff858e81640 1 --2- 172.21.15.94:0/3452553710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff854100e10 0x7ff85410ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.885+0000 7ff858e81640 1 --2- 172.21.15.94:0/3452553710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff854100450 0x7ff8541008d0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.885+0000 7ff858e81640 1 --2- 172.21.15.94:0/3452553710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff854106f30 0x7ff854107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.885+0000 7ff858e81640 1 -- 172.21.15.94:0/3452553710 >> 172.21.15.94:0/3452553710 conn(0x7ff854077410 msgr2=0x7ff8540fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.885+0000 7ff858e81640 1 -- 172.21.15.94:0/3452553710 shutdown_connections 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.885+0000 7ff858e81640 1 -- 172.21.15.94:0/3452553710 wait complete. 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff858e81640 1 Processor -- start 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff858e81640 1 -- start start 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff858e81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff854100450 0x7ff854101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff858e81640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff854100e10 0x7ff854101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff858e81640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff854106f30 0x7ff8541020b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff858e81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff8541122f0 con 0x7ff854100450 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff858e81640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff854112170 con 0x7ff854106f30 2024-04-21T08:40:12.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff858e81640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff854112470 con 0x7ff854100e10 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff85259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff854100450 0x7ff854101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff85259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff854100450 0x7ff854101630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37110/0 (socket says 172.21.15.94:37110) 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff85259c640 1 -- 172.21.15.94:0/849039309 learned_addr learned my addr 172.21.15.94:0/849039309 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff851d9b640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff854100e10 0x7ff854101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff852d9d640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff854106f30 0x7ff8541020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff85259c640 1 -- 172.21.15.94:0/849039309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff854100e10 msgr2=0x7ff854101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff85259c640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff854100e10 0x7ff854101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff85259c640 1 -- 172.21.15.94:0/849039309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff854106f30 msgr2=0x7ff8541020b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff85259c640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff854106f30 0x7ff8541020b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff85259c640 1 -- 172.21.15.94:0/849039309 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8541a6d90 con 0x7ff854100450 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff852d9d640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff854106f30 0x7ff8541020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff851d9b640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff854100e10 0x7ff854101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:12.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.886+0000 7ff85259c640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff854100450 0x7ff854101630 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00d950 tx=0x7ff84c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:12.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.887+0000 7ff8437fe640 1 -- 172.21.15.94:0/849039309 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c017070 con 0x7ff854100450 2024-04-21T08:40:12.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.887+0000 7ff858e81640 1 -- 172.21.15.94:0/849039309 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8541a7080 con 0x7ff854100450 2024-04-21T08:40:12.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.887+0000 7ff8437fe640 1 -- 172.21.15.94:0/849039309 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff84c009c30 con 0x7ff854100450 2024-04-21T08:40:12.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.887+0000 7ff8437fe640 1 -- 172.21.15.94:0/849039309 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c00f620 con 0x7ff854100450 2024-04-21T08:40:12.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.887+0000 7ff858e81640 1 -- 172.21.15.94:0/849039309 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8541a75a0 con 0x7ff854100450 2024-04-21T08:40:12.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.888+0000 7ff858e81640 1 -- 172.21.15.94:0/849039309 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff854102b60 con 0x7ff854100450 2024-04-21T08:40:12.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.889+0000 7ff8437fe640 1 -- 172.21.15.94:0/849039309 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7ff84c010460 con 0x7ff854100450 2024-04-21T08:40:12.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.889+0000 7ff8437fe640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff83c0a3e40 0x7ff83c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:12.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.889+0000 7ff851d9b640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff83c0a3e40 0x7ff83c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:12.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.889+0000 7ff8437fe640 1 -- 172.21.15.94:0/849039309 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff84c0d0b80 con 0x7ff854100450 2024-04-21T08:40:12.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.890+0000 7ff851d9b640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff83c0a3e40 0x7ff83c0a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff848006010 tx=0x7ff848005c50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:12.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:12.894+0000 7ff8437fe640 1 -- 172.21.15.94:0/849039309 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff84c09cbf0 con 0x7ff854100450 2024-04-21T08:40:13.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.075+0000 7ff858e81640 1 -- 172.21.15.94:0/849039309 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff85410a430 con 0x7ff854100450 2024-04-21T08:40:13.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.075+0000 7ff8437fe640 1 -- 172.21.15.94:0/849039309 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7ff84c009de0 con 0x7ff854100450 2024-04-21T08:40:13.075 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:40:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 -- 172.21.15.94:0/849039309 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff83c0a3e40 msgr2=0x7ff83c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff83c0a3e40 0x7ff83c0a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff848006010 tx=0x7ff848005c50 comp rx=0 tx=0).stop 2024-04-21T08:40:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 -- 172.21.15.94:0/849039309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff854100450 msgr2=0x7ff854101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff854100450 0x7ff854101630 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00d950 tx=0x7ff84c00de10 comp rx=0 tx=0).stop 2024-04-21T08:40:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 -- 172.21.15.94:0/849039309 shutdown_connections 2024-04-21T08:40:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff83c0a3e40 0x7ff83c0a6300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff854100e10 0x7ff854101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff854106f30 0x7ff8541020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 --2- 172.21.15.94:0/849039309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff854100450 0x7ff854101630 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 -- 172.21.15.94:0/849039309 >> 172.21.15.94:0/849039309 conn(0x7ff854077410 msgr2=0x7ff8541051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 -- 172.21.15.94:0/849039309 shutdown_connections 2024-04-21T08:40:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.077+0000 7ff858e81640 1 -- 172.21.15.94:0/849039309 wait complete. 2024-04-21T08:40:13.088 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:40:13.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 -- 172.21.15.94:0/1298355862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88f8112910 msgr2=0x7f88f8112cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 --2- 172.21.15.94:0/1298355862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88f8112910 0x7f88f8112cf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009930 tx=0x7f88e802bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:13.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 -- 172.21.15.94:0/1298355862 shutdown_connections 2024-04-21T08:40:13.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 --2- 172.21.15.94:0/1298355862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f810c900 0x7f88f810cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 --2- 172.21.15.94:0/1298355862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88f810d2c0 0x7f88f81197d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 --2- 172.21.15.94:0/1298355862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88f8112910 0x7f88f8112cf0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 -- 172.21.15.94:0/1298355862 >> 172.21.15.94:0/1298355862 conn(0x7f88f806cf80 msgr2=0x7f88f810af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:13.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 -- 172.21.15.94:0/1298355862 shutdown_connections 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 -- 172.21.15.94:0/1298355862 wait complete. 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 Processor -- start 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 -- start start 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f810c900 0x7f88f81a3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88f810d2c0 0x7f88f81a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.190+0000 7f88fde62640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88f8112910 0x7f88f81a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88fde62640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f88f811cbd0 con 0x7f88f810d2c0 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88fde62640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f88f811ca50 con 0x7f88f8112910 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88fde62640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f88f811cd50 con 0x7f88f810c900 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88effff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88f810d2c0 0x7f88f81a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88effff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88f810d2c0 0x7f88f81a36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37130/0 (socket says 172.21.15.94:37130) 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88effff640 1 -- 172.21.15.94:0/3500123900 learned_addr learned my addr 172.21.15.94:0/3500123900 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88f7fff640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88f8112910 0x7f88f81a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88effff640 1 -- 172.21.15.94:0/3500123900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f810c900 msgr2=0x7f88f81a3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88f77fe640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f810c900 0x7f88f81a3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88effff640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f810c900 0x7f88f81a3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88effff640 1 -- 172.21.15.94:0/3500123900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88f8112910 msgr2=0x7f88f81a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88effff640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88f8112910 0x7f88f81a7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88effff640 1 -- 172.21.15.94:0/3500123900 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f81a8160 con 0x7f88f810d2c0 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88effff640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88f810d2c0 0x7f88f81a36b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f88e000d770 tx=0x7f88e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88f57fa640 1 -- 172.21.15.94:0/3500123900 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0017070 con 0x7f88f810d2c0 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88f57fa640 1 -- 172.21.15.94:0/3500123900 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88e00054d0 con 0x7f88f810d2c0 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88fde62640 1 -- 172.21.15.94:0/3500123900 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f81a8450 con 0x7f88f810d2c0 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88f57fa640 1 -- 172.21.15.94:0/3500123900 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0004c30 con 0x7f88f810d2c0 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.191+0000 7f88fde62640 1 -- 172.21.15.94:0/3500123900 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f81afce0 con 0x7f88f810d2c0 2024-04-21T08:40:13.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.192+0000 7f88fde62640 1 -- 172.21.15.94:0/3500123900 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f8068d70 con 0x7f88f810d2c0 2024-04-21T08:40:13.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.193+0000 7f88f57fa640 1 -- 172.21.15.94:0/3500123900 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f88e0004e10 con 0x7f88f810d2c0 2024-04-21T08:40:13.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.194+0000 7f88f57fa640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f88e40a3ec0 0x7f88e40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.194+0000 7f88f77fe640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f88e40a3ec0 0x7f88e40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.194+0000 7f88f57fa640 1 -- 172.21.15.94:0/3500123900 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f88e00d08e0 con 0x7f88f810d2c0 2024-04-21T08:40:13.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.194+0000 7f88f77fe640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f88e40a3ec0 0x7f88e40a6380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f88e8036040 tx=0x7f88e8000bc0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:13.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.198+0000 7f88f57fa640 1 -- 172.21.15.94:0/3500123900 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88e009c950 con 0x7f88f810d2c0 2024-04-21T08:40:13.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.375+0000 7f88fde62640 1 -- 172.21.15.94:0/3500123900 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f88f802d5c0 con 0x7f88f810d2c0 2024-04-21T08:40:13.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.376+0000 7f88f57fa640 1 -- 172.21.15.94:0/3500123900 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f88e009c250 con 0x7f88f810d2c0 2024-04-21T08:40:13.376 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 -- 172.21.15.94:0/3500123900 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f88e40a3ec0 msgr2=0x7f88e40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f88e40a3ec0 0x7f88e40a6380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f88e8036040 tx=0x7f88e8000bc0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 -- 172.21.15.94:0/3500123900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88f810d2c0 msgr2=0x7f88f81a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88f810d2c0 0x7f88f81a36b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f88e000d770 tx=0x7f88e000dc30 comp rx=0 tx=0).stop 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 -- 172.21.15.94:0/3500123900 shutdown_connections 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f88e40a3ec0 0x7f88e40a6380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88f8112910 0x7f88f81a7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88f810d2c0 0x7f88f81a36b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 --2- 172.21.15.94:0/3500123900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f810c900 0x7f88f81a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 -- 172.21.15.94:0/3500123900 >> 172.21.15.94:0/3500123900 conn(0x7f88f806cf80 msgr2=0x7f88f810fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 -- 172.21.15.94:0/3500123900 shutdown_connections 2024-04-21T08:40:13.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.378+0000 7f88fde62640 1 -- 172.21.15.94:0/3500123900 wait complete. 2024-04-21T08:40:13.388 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:13 smithi094 ceph-mon[27988]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-21T08:40:13.388 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:13 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3836393148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:13.388 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:13 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/352803290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:13.388 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:13 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/849039309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:13.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:40:13.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 -- 172.21.15.94:0/396102863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a01011a0 msgr2=0x7f25a0101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 --2- 172.21.15.94:0/396102863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a01011a0 0x7f25a0101620 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2588009930 tx=0x7f258802bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:13.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 -- 172.21.15.94:0/396102863 shutdown_connections 2024-04-21T08:40:13.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 --2- 172.21.15.94:0/396102863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a0101b60 0x7f25a010dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 --2- 172.21.15.94:0/396102863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a01011a0 0x7f25a0101620 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 --2- 172.21.15.94:0/396102863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a0105e30 0x7f25a0106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 -- 172.21.15.94:0/396102863 >> 172.21.15.94:0/396102863 conn(0x7f25a0077e30 msgr2=0x7f25a0078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 -- 172.21.15.94:0/396102863 shutdown_connections 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 -- 172.21.15.94:0/396102863 wait complete. 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 Processor -- start 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.488+0000 7f25a4ae4640 1 -- start start 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25a4ae4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a01011a0 0x7f25a0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25a4ae4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a0101b60 0x7f25a0198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25a4ae4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a0105e30 0x7f25a019d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25a4ae4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f25a0112240 con 0x7f25a01011a0 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25a4ae4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f25a01120c0 con 0x7f25a0101b60 2024-04-21T08:40:13.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25a4ae4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f25a01123c0 con 0x7f25a0105e30 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a01011a0 0x7f25a0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a01011a0 0x7f25a0198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37138/0 (socket says 172.21.15.94:37138) 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259e59c640 1 -- 172.21.15.94:0/1377532368 learned_addr learned my addr 172.21.15.94:0/1377532368 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259dd9b640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a0101b60 0x7f25a0198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259e59c640 1 -- 172.21.15.94:0/1377532368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a0105e30 msgr2=0x7f25a019d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259ed9d640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a0105e30 0x7f25a019d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259e59c640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a0105e30 0x7f25a019d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259e59c640 1 -- 172.21.15.94:0/1377532368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a0101b60 msgr2=0x7f25a0198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259e59c640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a0101b60 0x7f25a0198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259e59c640 1 -- 172.21.15.94:0/1377532368 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a019d730 con 0x7f25a01011a0 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259dd9b640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a0101b60 0x7f25a0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f259e59c640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a01011a0 0x7f25a0198740 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f259000c970 tx=0x7f259000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:13.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25977fe640 1 -- 172.21.15.94:0/1377532368 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2590007bb0 con 0x7f25a01011a0 2024-04-21T08:40:13.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25977fe640 1 -- 172.21.15.94:0/1377532368 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2590007d50 con 0x7f25a01011a0 2024-04-21T08:40:13.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25a4ae4640 1 -- 172.21.15.94:0/1377532368 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a019da20 con 0x7f25a01011a0 2024-04-21T08:40:13.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25977fe640 1 -- 172.21.15.94:0/1377532368 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259000f450 con 0x7f25a01011a0 2024-04-21T08:40:13.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.489+0000 7f25a4ae4640 1 -- 172.21.15.94:0/1377532368 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a01a52b0 con 0x7f25a01011a0 2024-04-21T08:40:13.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.490+0000 7f25a4ae4640 1 -- 172.21.15.94:0/1377532368 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a0068de0 con 0x7f25a01011a0 2024-04-21T08:40:13.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.492+0000 7f25977fe640 1 -- 172.21.15.94:0/1377532368 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f2590004f20 con 0x7f25a01011a0 2024-04-21T08:40:13.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.492+0000 7f25977fe640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f258c0a3ec0 0x7f258c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.492+0000 7f259dd9b640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f258c0a3ec0 0x7f258c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.493+0000 7f25977fe640 1 -- 172.21.15.94:0/1377532368 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f25900d0be0 con 0x7f25a01011a0 2024-04-21T08:40:13.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.493+0000 7f259dd9b640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f258c0a3ec0 0x7f258c0a6380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2588006010 tx=0x7f2588009560 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:13.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.497+0000 7f25977fe640 1 -- 172.21.15.94:0/1377532368 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2590014030 con 0x7f25a01011a0 2024-04-21T08:40:13.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:13 smithi173 ceph-mon[31428]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-21T08:40:13.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:13 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3836393148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:13.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:13 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/352803290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:13.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:13 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/849039309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:13.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:13 smithi111 ceph-mon[31493]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-21T08:40:13.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:13 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3836393148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:40:13.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:13 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/352803290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:40:13.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:13 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/849039309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:40:13.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.710+0000 7f25a4ae4640 1 -- 172.21.15.94:0/1377532368 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25a0109390 con 0x7f25a01011a0 2024-04-21T08:40:13.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.711+0000 7f25977fe640 1 -- 172.21.15.94:0/1377532368 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f259009cc50 con 0x7f25a01011a0 2024-04-21T08:40:13.711 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:13.711 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:40:13.711 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:40:13.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 -- 172.21.15.94:0/1377532368 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f258c0a3ec0 msgr2=0x7f258c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f258c0a3ec0 0x7f258c0a6380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2588006010 tx=0x7f2588009560 comp rx=0 tx=0).stop 2024-04-21T08:40:13.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 -- 172.21.15.94:0/1377532368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a01011a0 msgr2=0x7f25a0198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a01011a0 0x7f25a0198740 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f259000c970 tx=0x7f259000ce30 comp rx=0 tx=0).stop 2024-04-21T08:40:13.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 -- 172.21.15.94:0/1377532368 shutdown_connections 2024-04-21T08:40:13.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f258c0a3ec0 0x7f258c0a6380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a0105e30 0x7f25a019d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a0101b60 0x7f25a0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 --2- 172.21.15.94:0/1377532368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25a01011a0 0x7f25a0198740 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 -- 172.21.15.94:0/1377532368 >> 172.21.15.94:0/1377532368 conn(0x7f25a0077e30 msgr2=0x7f25a00fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:13.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 -- 172.21.15.94:0/1377532368 shutdown_connections 2024-04-21T08:40:13.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.713+0000 7f25a4ae4640 1 -- 172.21.15.94:0/1377532368 wait complete. 2024-04-21T08:40:13.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:40:13.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/2659802352 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b4106150 msgr2=0x7fc3b410dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 --2- 172.21.15.94:0/2659802352 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b4106150 0x7fc3b410dd20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac009ca0 tx=0x7fc3ac02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/2659802352 shutdown_connections 2024-04-21T08:40:13.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 --2- 172.21.15.94:0/2659802352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3b41021f0 0x7fc3b4105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 --2- 172.21.15.94:0/2659802352 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b4106150 0x7fc3b410dd20 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 --2- 172.21.15.94:0/2659802352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3b4101840 0x7fc3b4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/2659802352 >> 172.21.15.94:0/2659802352 conn(0x7fc3b4077450 msgr2=0x7fc3b40fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/2659802352 shutdown_connections 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/2659802352 wait complete. 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 Processor -- start 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.823+0000 7fc3bbf7e640 1 -- start start 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3bbf7e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3b4101840 0x7fc3b41975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3bbf7e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3b4106150 0x7fc3b4197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3bbf7e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b419be30 0x7fc3b419c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3bbf7e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b41110c0 con 0x7fc3b419be30 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3bbf7e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b4110f40 con 0x7fc3b4106150 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3bbf7e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b4111240 con 0x7fc3b4101840 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3ba51b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b419be30 0x7fc3b419c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3ba51b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b419be30 0x7fc3b419c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58482/0 (socket says 172.21.15.94:58482) 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3ba51b640 1 -- 172.21.15.94:0/1865798058 learned_addr learned my addr 172.21.15.94:0/1865798058 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3b9519640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3b4106150 0x7fc3b4197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3b9d1a640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3b4101840 0x7fc3b41975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3ba51b640 1 -- 172.21.15.94:0/1865798058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3b4101840 msgr2=0x7fc3b41975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3ba51b640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3b4101840 0x7fc3b41975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3ba51b640 1 -- 172.21.15.94:0/1865798058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3b4106150 msgr2=0x7fc3b4197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3ba51b640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3b4106150 0x7fc3b4197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3ba51b640 1 -- 172.21.15.94:0/1865798058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b419c920 con 0x7fc3b419be30 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3b9519640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3b4106150 0x7fc3b4197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3ba51b640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b419be30 0x7fc3b419c2e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0098d0 tx=0x7fc3ac035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:13.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3a6ffd640 1 -- 172.21.15.94:0/1865798058 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac039040 con 0x7fc3b419be30 2024-04-21T08:40:13.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3a6ffd640 1 -- 172.21.15.94:0/1865798058 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3ac044070 con 0x7fc3b419be30 2024-04-21T08:40:13.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/1865798058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3b419cbb0 con 0x7fc3b419be30 2024-04-21T08:40:13.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3a6ffd640 1 -- 172.21.15.94:0/1865798058 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac0042a0 con 0x7fc3b419be30 2024-04-21T08:40:13.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.824+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/1865798058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3b41a43d0 con 0x7fc3b419be30 2024-04-21T08:40:13.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.825+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/1865798058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3b41988c0 con 0x7fc3b419be30 2024-04-21T08:40:13.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.827+0000 7fc3a6ffd640 1 -- 172.21.15.94:0/1865798058 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fc3ac004440 con 0x7fc3b419be30 2024-04-21T08:40:13.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.828+0000 7fc3a6ffd640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3a00a3ec0 0x7fc3a00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:13.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.828+0000 7fc3b9d1a640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3a00a3ec0 0x7fc3a00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:13.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.828+0000 7fc3a6ffd640 1 -- 172.21.15.94:0/1865798058 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc3ac0f3d80 con 0x7fc3b419be30 2024-04-21T08:40:13.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.828+0000 7fc3b9d1a640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3a00a3ec0 0x7fc3a00a6380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800ba40 tx=0x7fc3a8005cc0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:13.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:13.831+0000 7fc3a6ffd640 1 -- 172.21.15.94:0/1865798058 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3ac0bff00 con 0x7fc3b419be30 2024-04-21T08:40:14.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.012+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/1865798058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3b4109380 con 0x7fc3b419be30 2024-04-21T08:40:14.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.014+0000 7fc3a6ffd640 1 -- 172.21.15.94:0/1865798058 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25266 (secure 0 0 0) 0x7fc3b4109380 con 0x7fc3b419be30 2024-04-21T08:40:14.014 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:14.014 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:39:57.898777+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/1865798058 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3a00a3ec0 msgr2=0x7fc3a00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3a00a3ec0 0x7fc3a00a6380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800ba40 tx=0x7fc3a8005cc0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/1865798058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b419be30 msgr2=0x7fc3b419c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b419be30 0x7fc3b419c2e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0098d0 tx=0x7fc3ac035d60 comp rx=0 tx=0).stop 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/1865798058 shutdown_connections 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3a00a3ec0 0x7fc3a00a6380 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b419be30 0x7fc3b419c2e0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3b4106150 0x7fc3b4197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 --2- 172.21.15.94:0/1865798058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3b4101840 0x7fc3b41975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/1865798058 >> 172.21.15.94:0/1865798058 conn(0x7fc3b4077450 msgr2=0x7fc3b41042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/1865798058 shutdown_connections 2024-04-21T08:40:14.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.016+0000 7fc3bbf7e640 1 -- 172.21.15.94:0/1865798058 wait complete. 2024-04-21T08:40:14.028 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 418771 2024-04-21T08:40:14.028 DEBUG:tasks.quiescer:Yielding 2024-04-21T08:40:14.028 INFO:teuthology.run_tasks:Running task workunit... 2024-04-21T08:40:14.044 INFO:tasks.workunit:Pulling workunits from ref 4f6bc5273643aaf5660909e85400647a8951c10b 2024-04-21T08:40:14.044 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-21T08:40:14.044 INFO:tasks.workunit:timeout=3h 2024-04-21T08:40:14.044 INFO:tasks.workunit:cleanup=True 2024-04-21T08:40:14.045 DEBUG:teuthology.orchestra.run.smithi094:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-21T08:40:14.045 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:40:14.060 INFO:teuthology.orchestra.run.smithi094.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-21T08:40:14.060 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-21T08:40:14.060 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-21T08:40:14.061 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:40:14.061 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-21T08:40:14.061 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-21 08:37:59.089076395 +0000 2024-04-21T08:40:14.061 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-21 08:37:59.089076395 +0000 2024-04-21T08:40:14.061 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-21 08:39:55.746114837 +0000 2024-04-21T08:40:14.061 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-21 08:37:59.089076395 +0000 2024-04-21T08:40:14.062 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-21T08:40:14.062 DEBUG:teuthology.orchestra.run.smithi094:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-21T08:40:14.130 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/leonid-s-usov/ceph-ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4f6bc5273643aaf5660909e85400647a8951c10b 2024-04-21T08:40:14.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.145+0000 7fc43a49a640 1 -- 172.21.15.94:0/616324568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc434101a40 msgr2=0x7fc434105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.145+0000 7fc43a49a640 1 --2- 172.21.15.94:0/616324568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc434101a40 0x7fc434105a80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc424009900 tx=0x7fc42402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:14.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.145+0000 7fc43a49a640 1 -- 172.21.15.94:0/616324568 shutdown_connections 2024-04-21T08:40:14.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.145+0000 7fc43a49a640 1 --2- 172.21.15.94:0/616324568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc434106160 0x7fc43410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.145+0000 7fc43a49a640 1 --2- 172.21.15.94:0/616324568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc434101a40 0x7fc434105a80 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.145+0000 7fc43a49a640 1 --2- 172.21.15.94:0/616324568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc434101090 0x7fc434101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.145+0000 7fc43a49a640 1 -- 172.21.15.94:0/616324568 >> 172.21.15.94:0/616324568 conn(0x7fc4340fc910 msgr2=0x7fc4340fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.145+0000 7fc43a49a640 1 -- 172.21.15.94:0/616324568 shutdown_connections 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.145+0000 7fc43a49a640 1 -- 172.21.15.94:0/616324568 wait complete. 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc43a49a640 1 Processor -- start 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc43a49a640 1 -- start start 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc43a49a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc434101090 0x7fc4341995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc43a49a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc434106160 0x7fc434199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc43a49a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc43419a0c0 0x7fc4341a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc43a49a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc434111150 con 0x7fc434106160 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc43a49a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc434110fd0 con 0x7fc43419a0c0 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc43a49a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc4341112d0 con 0x7fc434101090 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc4337fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc434106160 0x7fc434199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc4337fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc434106160 0x7fc434199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58512/0 (socket says 172.21.15.94:58512) 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc4337fe640 1 -- 172.21.15.94:0/433534346 learned_addr learned my addr 172.21.15.94:0/433534346 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc438a37640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc43419a0c0 0x7fc4341a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc433fff640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc434101090 0x7fc4341995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc4337fe640 1 -- 172.21.15.94:0/433534346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc434101090 msgr2=0x7fc4341995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc4337fe640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc434101090 0x7fc4341995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc4337fe640 1 -- 172.21.15.94:0/433534346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc43419a0c0 msgr2=0x7fc4341a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc4337fe640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc43419a0c0 0x7fc4341a4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc4337fe640 1 -- 172.21.15.94:0/433534346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4341a4e40 con 0x7fc434106160 2024-04-21T08:40:14.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.146+0000 7fc438a37640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc43419a0c0 0x7fc4341a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:14.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.147+0000 7fc4337fe640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc434106160 0x7fc434199b30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc424005ec0 tx=0x7fc42402ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.147+0000 7fc4317fa640 1 -- 172.21.15.94:0/433534346 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc424043020 con 0x7fc434106160 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.147+0000 7fc43a49a640 1 -- 172.21.15.94:0/433534346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4341a50d0 con 0x7fc434106160 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.147+0000 7fc4317fa640 1 -- 172.21.15.94:0/433534346 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc42402cc80 con 0x7fc434106160 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.147+0000 7fc4317fa640 1 -- 172.21.15.94:0/433534346 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc424003c90 con 0x7fc434106160 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.147+0000 7fc43a49a640 1 -- 172.21.15.94:0/433534346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4341a5670 con 0x7fc434106160 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.148+0000 7fc43a49a640 1 -- 172.21.15.94:0/433534346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc434069230 con 0x7fc434106160 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.149+0000 7fc4317fa640 1 -- 172.21.15.94:0/433534346 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fc424046020 con 0x7fc434106160 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.150+0000 7fc4317fa640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc41c0a3e40 0x7fc41c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.150+0000 7fc433fff640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc41c0a3e40 0x7fc41c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.150+0000 7fc4317fa640 1 -- 172.21.15.94:0/433534346 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc4240f3500 con 0x7fc434106160 2024-04-21T08:40:14.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.151+0000 7fc433fff640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc41c0a3e40 0x7fc41c0a6300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc4280058c0 tx=0x7fc42800a4c0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.153+0000 7fc4317fa640 1 -- 172.21.15.94:0/433534346 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc42404d020 con 0x7fc434106160 2024-04-21T08:40:14.161 INFO:tasks.workunit.client.0.smithi094.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-21T08:40:14.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.374+0000 7fc43a49a640 1 -- 172.21.15.94:0/433534346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4341093f0 con 0x7fc434106160 2024-04-21T08:40:14.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.375+0000 7fc4317fa640 1 -- 172.21.15.94:0/433534346 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7fc4240bf680 con 0x7fc434106160 2024-04-21T08:40:14.375 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:14.375 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:40:14.375 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:40:14.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 -- 172.21.15.94:0/433534346 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc41c0a3e40 msgr2=0x7fc41c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc41c0a3e40 0x7fc41c0a6300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc4280058c0 tx=0x7fc42800a4c0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 -- 172.21.15.94:0/433534346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc434106160 msgr2=0x7fc434199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc434106160 0x7fc434199b30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc424005ec0 tx=0x7fc42402ca90 comp rx=0 tx=0).stop 2024-04-21T08:40:14.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 -- 172.21.15.94:0/433534346 shutdown_connections 2024-04-21T08:40:14.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc41c0a3e40 0x7fc41c0a6300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc43419a0c0 0x7fc4341a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc434106160 0x7fc434199b30 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 --2- 172.21.15.94:0/433534346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc434101090 0x7fc4341995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 -- 172.21.15.94:0/433534346 >> 172.21.15.94:0/433534346 conn(0x7fc4340fc910 msgr2=0x7fc4340fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 -- 172.21.15.94:0/433534346 shutdown_connections 2024-04-21T08:40:14.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.377+0000 7fc43a49a640 1 -- 172.21.15.94:0/433534346 wait complete. 2024-04-21T08:40:14.387 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:14 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3500123900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:14.387 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:14 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1377532368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:14.387 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:14 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1865798058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:14.389 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:37:30.753966+0000', 'modified': '2024-04-21T08:39:56.419394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24488, 'mds_1': 24454, 'mds_2': 24446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24416': {'gid': 24416, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6835/3967081988', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 3967081988}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 3967081988}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24421': {'gid': 24421, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/3922099450', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 3922099450}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 3922099450}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24433': {'gid': 24433, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6837/1466520447', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 1466520447}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 1466520447}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.111:6837/550365815', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6836', 'nonce': 550365815}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 550365815}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.173:6839/2987518473', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 2987518473}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 2987518473}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3878872423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3878872423}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3878872423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24488, 'qdb_cluster': [24454, 24488, 24446]} 2024-04-21T08:40:14.389 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-21T08:40:14.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-21T08:40:14.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.529+0000 7f5f84e06640 1 -- 172.21.15.94:0/980613286 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f80101a40 msgr2=0x7f5f80105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.529+0000 7f5f84e06640 1 --2- 172.21.15.94:0/980613286 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f80101a40 0x7f5f80105a80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5f70009900 tx=0x7f5f7002bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.529+0000 7f5f84e06640 1 -- 172.21.15.94:0/980613286 shutdown_connections 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.529+0000 7f5f84e06640 1 --2- 172.21.15.94:0/980613286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f80106160 0x7f5f8010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.529+0000 7f5f84e06640 1 --2- 172.21.15.94:0/980613286 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f80101a40 0x7f5f80105a80 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.529+0000 7f5f84e06640 1 --2- 172.21.15.94:0/980613286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f80101090 0x7f5f80101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.529+0000 7f5f84e06640 1 -- 172.21.15.94:0/980613286 >> 172.21.15.94:0/980613286 conn(0x7f5f80077f50 msgr2=0x7f5f80078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.529+0000 7f5f84e06640 1 -- 172.21.15.94:0/980613286 shutdown_connections 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.529+0000 7f5f84e06640 1 -- 172.21.15.94:0/980613286 wait complete. 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f84e06640 1 Processor -- start 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f84e06640 1 -- start start 2024-04-21T08:40:14.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f84e06640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f80101090 0x7f5f801a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f84e06640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f80106160 0x7f5f801a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f84e06640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f8019a8a0 0x7f5f8019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f84e06640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5f80111180 con 0x7f5f80106160 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f84e06640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f80111000 con 0x7f5f8019a8a0 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f84e06640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5f80111300 con 0x7f5f80101090 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f80106160 0x7f5f801a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f80106160 0x7f5f801a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58530/0 (socket says 172.21.15.94:58530) 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2199489320 learned_addr learned my addr 172.21.15.94:0/2199489320 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f8019a8a0 0x7f5f8019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7e59c640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f80101090 0x7f5f801a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2199489320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f80101090 msgr2=0x7f5f801a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7dd9b640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f80101090 0x7f5f801a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2199489320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f8019a8a0 msgr2=0x7f5f8019ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7dd9b640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f8019a8a0 0x7f5f8019ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2199489320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f8019b3c0 con 0x7f5f80106160 2024-04-21T08:40:14.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f7dd9b640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f80106160 0x7f5f801a0c30 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5f70005ec0 tx=0x7f5f7002ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.530+0000 7f5f6f7fe640 1 -- 172.21.15.94:0/2199489320 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70043020 con 0x7f5f80106160 2024-04-21T08:40:14.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.531+0000 7f5f6f7fe640 1 -- 172.21.15.94:0/2199489320 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f700052b0 con 0x7f5f80106160 2024-04-21T08:40:14.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.531+0000 7f5f84e06640 1 -- 172.21.15.94:0/2199489320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f8019b650 con 0x7f5f80106160 2024-04-21T08:40:14.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.531+0000 7f5f6f7fe640 1 -- 172.21.15.94:0/2199489320 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70003c60 con 0x7f5f80106160 2024-04-21T08:40:14.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.531+0000 7f5f84e06640 1 -- 172.21.15.94:0/2199489320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f801a78a0 con 0x7f5f80106160 2024-04-21T08:40:14.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.533+0000 7f5f6f7fe640 1 -- 172.21.15.94:0/2199489320 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f5f70046020 con 0x7f5f80106160 2024-04-21T08:40:14.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.533+0000 7f5f879fb740 1 Processor -- start 2024-04-21T08:40:14.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.533+0000 7f5f879fb740 1 -- start start 2024-04-21T08:40:14.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.533+0000 7f5f6f7fe640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f680a3e40 0x7f5f680a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.533+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e44b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.533+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e4d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.533+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 0x55f242e50130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.533+0000 7f5f879fb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55f242da8b10 con 0x55f242e45160 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.533+0000 7f5f879fb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f242d46240 con 0x55f242e447b0 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.533+0000 7f5f879fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f242da4ad0 con 0x55f242e4dce0 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e4d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f680a3e40 0x7f5f680a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e4d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58538/0 (socket says 172.21.15.94:58538) 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2793873602 learned_addr learned my addr 172.21.15.94:0/2793873602 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f6f7fe640 1 -- 172.21.15.94:0/2199489320 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5f700f3030 con 0x7f5f80106160 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7e59c640 1 --2- 172.21.15.94:0/2793873602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e44b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2793873602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 msgr2=0x55f242e50130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/2793873602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 0x55f242e50130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7dd9b640 1 --2- 172.21.15.94:0/2793873602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 0x55f242e50130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2793873602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 msgr2=0x55f242e44b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7dd9b640 1 --2- 172.21.15.94:0/2793873602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e44b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2793873602 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f242abb2d0 con 0x55f242e45160 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7e59c640 1 --2- 172.21.15.94:0/2793873602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e44b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f680a3e40 0x7f5f680a6300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009900 tx=0x7f5f78009650 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f7dd9b640 1 --2- 172.21.15.94:0/2793873602 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e4d710 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5f700f09a0 tx=0x7f5f70003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae31811c2f6365f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f6cff9640 1 -- 172.21.15.94:0/2793873602 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f700be880 con 0x55f242e45160 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f6cff9640 1 -- 172.21.15.94:0/2793873602 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f700bf1b0 con 0x55f242e45160 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.534+0000 7f5f6cff9640 1 -- 172.21.15.94:0/2793873602 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f700d8230 con 0x55f242e45160 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.535+0000 7f5f879fb740 1 -- 172.21.15.94:0/2793873602 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 msgr2=0x55f242e4d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.535+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2793873602 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e4d710 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5f700f09a0 tx=0x7f5f70003760 comp rx=0 tx=0).stop 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.535+0000 7f5f879fb740 1 -- 172.21.15.94:0/2793873602 shutdown_connections 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.535+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2793873602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 0x55f242e50130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.535+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2793873602 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e4d710 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.535+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2793873602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e44b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.535+0000 7f5f879fb740 1 -- 172.21.15.94:0/2793873602 >> 172.21.15.94:0/2793873602 conn(0x55f242e3ea60 msgr2=0x55f242e3ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.535+0000 7f5f879fb740 1 -- 172.21.15.94:0/2793873602 shutdown_connections 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.535+0000 7f5f879fb740 1 -- 172.21.15.94:0/2793873602 wait complete. 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.535+0000 7f5f879fb740 1 -- start start 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f879fb740 1 Processor -- start 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e4ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e45160 0x55f242e4b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e4b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f879fb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55f242e50a10 con 0x55f242e4dce0 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f879fb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f242d31190 con 0x55f242e447b0 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f879fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f242e50b90 con 0x55f242e45160 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f879fb740 10 client.0 _ll_register_callbacks cb 0x55f242d2c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e4b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e4b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58550/0 (socket says 172.21.15.94:58550) 2024-04-21T08:40:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7ed9d640 1 -- 172.21.15.94:0/2485894982 learned_addr learned my addr 172.21.15.94:0/2485894982 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7dd9b640 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e45160 0x55f242e4b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7e59c640 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e4ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7ed9d640 1 -- 172.21.15.94:0/2485894982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e45160 msgr2=0x55f242e4b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e45160 0x55f242e4b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7ed9d640 1 -- 172.21.15.94:0/2485894982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 msgr2=0x55f242e4ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e4ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7ed9d640 1 -- 172.21.15.94:0/2485894982 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f242b856b0 con 0x55f242e4dce0 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e4b7a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5f78018dd0 tx=0x7f5f78013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f7aa39fdb6b54e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2485894982 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78020620 con 0x55f242e4dce0 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2485894982 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5f780207c0 con 0x55f242e4dce0 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.536+0000 7f5f879fb740 1 -- 172.21.15.94:0/2485894982 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f700bf1b0 con 0x55f242e4dce0 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2485894982 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5f7802e050 con 0x55f242e4dce0 2024-04-21T08:40:14.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f879fb740 10 client.15864 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f879fb740 1 -- 172.21.15.94:0/2485894982 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f242e5cd60 con 0x55f242e4dce0 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2485894982 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f700bf1b0 con 0x55f242e4dce0 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2485894982 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5f78017070 con 0x55f242e4dce0 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f879fb740 10 client.15864 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f879fb740 10 client.15864 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f879fb740 20 client.15864 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55f242e5f3d0 0x55f242e60840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f879fb740 4 client.15864 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f879fb740 1 -- 172.21.15.94:0/2485894982 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f242db6530 con 0x55f242e5f3d0 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.537+0000 7f5f7e59c640 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55f242e5f3d0 0x55f242e60840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.538+0000 7f5f65ffb640 10 client.15864 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.538+0000 7f5f7e59c640 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55f242e5f3d0 0x55f242e60840 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b96dee6fdacd07c5 server_cookie=a31f0902889796bc in_seq=0 out_seq=0 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.538+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2485894982 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f242db6530 con 0x55f242e5f3d0 2024-04-21T08:40:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.538+0000 7f5f65ffb640 10 client.15864 handle_command_reply: tid=0 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.538+0000 7f5f879fb740 1 client.15864 shutdown 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 1 -- 172.21.15.94:0/2485894982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 msgr2=0x55f242e4b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e4b7a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5f78018dd0 tx=0x7f5f78013000 comp rx=0 tx=0).stop 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 1 -- 172.21.15.94:0/2485894982 shutdown_connections 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55f242e5f3d0 0x55f242e60840 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e45160 0x55f242e4b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e4b7a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f65ffb640 0 client.15864 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2485894982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e4ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 1 -- 172.21.15.94:0/2485894982 >> 172.21.15.94:0/2485894982 conn(0x55f242e3ea60 msgr2=0x55f242e40bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 1 -- 172.21.15.94:0/2485894982 shutdown_connections 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 1 -- 172.21.15.94:0/2485894982 wait complete. 2024-04-21T08:40:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.539+0000 7f5f879fb740 20 client.15864 trim_cache size 0 max 16384 2024-04-21T08:40:14.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f879fb740 1 Processor -- start 2024-04-21T08:40:14.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f879fb740 1 -- start start 2024-04-21T08:40:14.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e447b0 0x55f242e46010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e45160 0x55f242e46550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e495f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f879fb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55f242e5eb80 con 0x55f242e4dce0 2024-04-21T08:40:14.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f879fb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f242e5f250 con 0x55f242e45160 2024-04-21T08:40:14.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f879fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f242e5f920 con 0x55f242e447b0 2024-04-21T08:40:14.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f7ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f7ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e495f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58554/0 (socket says 172.21.15.94:58554) 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f7ed9d640 1 -- 172.21.15.94:0/608799575 learned_addr learned my addr 172.21.15.94:0/608799575 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f7ed9d640 1 -- 172.21.15.94:0/608799575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e447b0 msgr2=0x55f242e46010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/608799575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e447b0 0x55f242e46010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f7ed9d640 1 -- 172.21.15.94:0/608799575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e45160 msgr2=0x55f242e46550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/608799575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e45160 0x55f242e46550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f7ed9d640 1 -- 172.21.15.94:0/608799575 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f242e44f00 con 0x55f242e4dce0 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/608799575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e495f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009c80 tx=0x7f5f7801f400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8236557af174b604 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f67fff640 1 -- 172.21.15.94:0/608799575 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78039070 con 0x55f242e4dce0 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f67fff640 1 -- 172.21.15.94:0/608799575 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f78025070 con 0x55f242e4dce0 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.557+0000 7f5f67fff640 1 -- 172.21.15.94:0/608799575 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78034a90 con 0x55f242e4dce0 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 -- 172.21.15.94:0/608799575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 msgr2=0x55f242e495f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 --2- 172.21.15.94:0/608799575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e495f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009c80 tx=0x7f5f7801f400 comp rx=0 tx=0).stop 2024-04-21T08:40:14.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 -- 172.21.15.94:0/608799575 shutdown_connections 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 --2- 172.21.15.94:0/608799575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e4dce0 0x55f242e495f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 --2- 172.21.15.94:0/608799575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e45160 0x55f242e46550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 --2- 172.21.15.94:0/608799575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e447b0 0x55f242e46010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 -- 172.21.15.94:0/608799575 >> 172.21.15.94:0/608799575 conn(0x55f242e3ea60 msgr2=0x55f242e39800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 -- 172.21.15.94:0/608799575 shutdown_connections 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 -- 172.21.15.94:0/608799575 wait complete. 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 -- start start 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 Processor -- start 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e68f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 0x55f242e62820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.558+0000 7f5f879fb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55f242e606c0 con 0x55f242e45160 2024-04-21T08:40:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f879fb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f242e5fff0 con 0x55f242e447b0 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f879fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f242e60d90 con 0x55f242e4dce0 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f879fb740 10 client.0 _ll_register_callbacks cb 0x7f5f78009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e622a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58570/0 (socket says 172.21.15.94:58570) 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2667277099 learned_addr learned my addr 172.21.15.94:0/2667277099 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 0x55f242e62820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7e59c640 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e68f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2667277099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 msgr2=0x55f242e62820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7dd9b640 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 0x55f242e62820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2667277099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 msgr2=0x55f242e68f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7dd9b640 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e68f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7dd9b640 1 -- 172.21.15.94:0/2667277099 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f242e63050 con 0x55f242e45160 2024-04-21T08:40:14.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7e59c640 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e68f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7ed9d640 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 0x55f242e62820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f7dd9b640 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e622a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5f700baf10 tx=0x7f5f700be8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86bbd4058fcc6fd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2667277099 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f700d8410 con 0x55f242e45160 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2667277099 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5f70005460 con 0x55f242e45160 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f879fb740 1 -- 172.21.15.94:0/2667277099 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f78025070 con 0x55f242e45160 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2667277099 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5f700be410 con 0x55f242e45160 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f879fb740 10 client.15870 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f879fb740 1 -- 172.21.15.94:0/2667277099 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f242ec5440 con 0x55f242e45160 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.559+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2667277099 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f78025070 con 0x55f242e45160 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.560+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2667277099 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5f700db3d0 con 0x55f242e45160 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.560+0000 7f5f879fb740 10 client.15870 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.560+0000 7f5f879fb740 10 client.15870 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-21T08:40:14.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.560+0000 7f5f879fb740 20 client.15870 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.560+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55f242ec9d20 0x55f242ec8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.560+0000 7f5f879fb740 4 client.15870 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.560+0000 7f5f879fb740 1 -- 172.21.15.94:0/2667277099 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "status"}) -- 0x55f242db03a0 con 0x55f242ec9d20 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.560+0000 7f5f7e59c640 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55f242ec9d20 0x55f242ec8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.560+0000 7f5f65ffb640 10 client.15870 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.560+0000 7f5f7e59c640 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55f242ec9d20 0x55f242ec8a30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8b6d98b862d6e965 server_cookie=be45dde7c0c2ad6d in_seq=0 out_seq=0 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f65ffb640 1 -- 172.21.15.94:0/2667277099 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55f242db03a0 con 0x55f242ec9d20 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f65ffb640 10 client.15870 handle_command_reply: tid=0 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 client.15870 shutdown 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 -- 172.21.15.94:0/2667277099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 msgr2=0x55f242e622a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e622a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5f700baf10 tx=0x7f5f700be8a0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 -- 172.21.15.94:0/2667277099 shutdown_connections 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55f242ec9d20 0x55f242ec8a30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f242e4dce0 0x55f242e62820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f242e45160 0x55f242e622a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 --2- 172.21.15.94:0/2667277099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f242e447b0 0x55f242e68f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f65ffb640 0 client.15870 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:40:14.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 -- 172.21.15.94:0/2667277099 >> 172.21.15.94:0/2667277099 conn(0x55f242e3ea60 msgr2=0x55f242e46010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 -- 172.21.15.94:0/2667277099 shutdown_connections 2024-04-21T08:40:14.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 1 -- 172.21.15.94:0/2667277099 wait complete. 2024-04-21T08:40:14.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f879fb740 20 client.15870 trim_cache size 0 max 16384 2024-04-21T08:40:14.568 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 2, 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24446, 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 137.905020724, 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 24, 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 94, 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 93, 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 179.602527912 2024-04-21T08:40:14.569 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:40:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 -- 172.21.15.94:0/2199489320 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f680a3e40 msgr2=0x7f5f680a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f680a3e40 0x7f5f680a6300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009900 tx=0x7f5f78009650 comp rx=0 tx=0).stop 2024-04-21T08:40:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 -- 172.21.15.94:0/2199489320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f80106160 msgr2=0x7f5f801a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f80106160 0x7f5f801a0c30 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5f70005ec0 tx=0x7f5f7002ca10 comp rx=0 tx=0).stop 2024-04-21T08:40:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 -- 172.21.15.94:0/2199489320 shutdown_connections 2024-04-21T08:40:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f680a3e40 0x7f5f680a6300 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f8019a8a0 0x7f5f8019ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f80106160 0x7f5f801a0c30 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 --2- 172.21.15.94:0/2199489320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f80101090 0x7f5f801a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 -- 172.21.15.94:0/2199489320 >> 172.21.15.94:0/2199489320 conn(0x7f5f80077f50 msgr2=0x7f5f801000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 -- 172.21.15.94:0/2199489320 shutdown_connections 2024-04-21T08:40:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.561+0000 7f5f84e06640 1 -- 172.21.15.94:0/2199489320 wait complete. 2024-04-21T08:40:14.572 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.672+0000 7f749f72b640 1 -- 172.21.15.94:0/724023714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7498106160 msgr2=0x7f749810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.672+0000 7f749f72b640 1 --2- 172.21.15.94:0/724023714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7498106160 0x7f749810dd30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7494009900 tx=0x7f749402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.672+0000 7f749f72b640 1 -- 172.21.15.94:0/724023714 shutdown_connections 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.672+0000 7f749f72b640 1 --2- 172.21.15.94:0/724023714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7498101a40 0x7f7498105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.672+0000 7f749f72b640 1 --2- 172.21.15.94:0/724023714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7498106160 0x7f749810dd30 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.672+0000 7f749f72b640 1 --2- 172.21.15.94:0/724023714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7498101090 0x7f7498101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.672+0000 7f749f72b640 1 -- 172.21.15.94:0/724023714 >> 172.21.15.94:0/724023714 conn(0x7f7498077f50 msgr2=0x7f7498078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749f72b640 1 -- 172.21.15.94:0/724023714 shutdown_connections 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749f72b640 1 -- 172.21.15.94:0/724023714 wait complete. 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749f72b640 1 Processor -- start 2024-04-21T08:40:14.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749f72b640 1 -- start start 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749f72b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7498101090 0x7f74981986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749f72b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7498106160 0x7f7498198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749f72b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f749819cfa0 0x7f749819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749f72b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f74981121e0 con 0x7f7498101090 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749f72b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7498112060 con 0x7f7498106160 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749f72b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7498112360 con 0x7f749819cfa0 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749d4c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7498101090 0x7f74981986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.673+0000 7f749d4c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7498101090 0x7f74981986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58596/0 (socket says 172.21.15.94:58596) 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749d4c7640 1 -- 172.21.15.94:0/343989636 learned_addr learned my addr 172.21.15.94:0/343989636 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749ccc6640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7498106160 0x7f7498198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749dcc8640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f749819cfa0 0x7f749819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749d4c7640 1 -- 172.21.15.94:0/343989636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f749819cfa0 msgr2=0x7f749819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749d4c7640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f749819cfa0 0x7f749819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749d4c7640 1 -- 172.21.15.94:0/343989636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7498106160 msgr2=0x7f7498198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749d4c7640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7498106160 0x7f7498198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749d4c7640 1 -- 172.21.15.94:0/343989636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749819da90 con 0x7f7498101090 2024-04-21T08:40:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749dcc8640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f749819cfa0 0x7f749819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749d4c7640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7498101090 0x7f74981986e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f749000cce0 tx=0x7f749000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f748e7fc640 1 -- 172.21.15.94:0/343989636 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490004b20 con 0x7f7498101090 2024-04-21T08:40:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749f72b640 1 -- 172.21.15.94:0/343989636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f749819dd80 con 0x7f7498101090 2024-04-21T08:40:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f748e7fc640 1 -- 172.21.15.94:0/343989636 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7490004cc0 con 0x7f7498101090 2024-04-21T08:40:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f748e7fc640 1 -- 172.21.15.94:0/343989636 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749000f6b0 con 0x7f7498101090 2024-04-21T08:40:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.674+0000 7f749f72b640 1 -- 172.21.15.94:0/343989636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74981a5570 con 0x7f7498101090 2024-04-21T08:40:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.676+0000 7f748e7fc640 1 -- 172.21.15.94:0/343989636 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f749000f850 con 0x7f7498101090 2024-04-21T08:40:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.677+0000 7f74a23fb740 1 Processor -- start 2024-04-21T08:40:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.677+0000 7f74a23fb740 1 -- start start 2024-04-21T08:40:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.677+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7e5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.677+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7ee710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.677+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7f1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.677+0000 7f74a23fb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55de6d749b10 con 0x55de6d7eece0 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.677+0000 7f74a23fb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55de6d6e7240 con 0x55de6d7e6160 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.677+0000 7f74a23fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55de6d745ad0 con 0x55de6d7e57b0 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.677+0000 7f748e7fc640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74840a3e40 0x7f74840a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749ccc6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749dcc8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7e5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749dcc8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7e5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:41034/0 (socket says 172.21.15.94:41034) 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749dcc8640 1 -- 172.21.15.94:0/1343915869 learned_addr learned my addr 172.21.15.94:0/1343915869 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f748e7fc640 1 -- 172.21.15.94:0/343989636 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f74900d0f60 con 0x7f7498101090 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749dcc8640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74840a3e40 0x7f74840a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749ccc6640 1 -- 172.21.15.94:0/1343915869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 msgr2=0x55de6d7e5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749ccc6640 1 --2- 172.21.15.94:0/1343915869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7e5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749ccc6640 1 -- 172.21.15.94:0/1343915869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 msgr2=0x55de6d7ee710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749d4c7640 1 --2- 172.21.15.94:0/1343915869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7ee710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749ccc6640 1 --2- 172.21.15.94:0/1343915869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7ee710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749ccc6640 1 -- 172.21.15.94:0/1343915869 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de6d45c2d0 con 0x55de6d7eece0 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749d4c7640 1 --2- 172.21.15.94:0/1343915869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7ee710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749ccc6640 1 --2- 172.21.15.94:0/1343915869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7f1130 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f749806b6e0 tx=0x7f7488011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e97e1fd708e6996 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f749dcc8640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74840a3e40 0x7f74840a6300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7494005ec0 tx=0x7f7494009260 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f74837fe640 1 -- 172.21.15.94:0/1343915869 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7488004050 con 0x55de6d7eece0 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f74837fe640 1 -- 172.21.15.94:0/1343915869 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74880054d0 con 0x55de6d7eece0 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.678+0000 7f74837fe640 1 -- 172.21.15.94:0/1343915869 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7488004ba0 con 0x55de6d7eece0 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 -- 172.21.15.94:0/1343915869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 msgr2=0x55de6d7f1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 --2- 172.21.15.94:0/1343915869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7f1130 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f749806b6e0 tx=0x7f7488011810 comp rx=0 tx=0).stop 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 -- 172.21.15.94:0/1343915869 shutdown_connections 2024-04-21T08:40:14.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 --2- 172.21.15.94:0/1343915869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7f1130 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 --2- 172.21.15.94:0/1343915869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7ee710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 --2- 172.21.15.94:0/1343915869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7e5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 -- 172.21.15.94:0/1343915869 >> 172.21.15.94:0/1343915869 conn(0x55de6d7dfa60 msgr2=0x55de6d7dfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 -- 172.21.15.94:0/1343915869 shutdown_connections 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 -- 172.21.15.94:0/1343915869 wait complete. 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 -- start start 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.679+0000 7f74a23fb740 1 Processor -- start 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7ebcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7ec230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7ec770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f74a23fb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55de6d7f6060 con 0x55de6d7eece0 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f74a23fb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55de6d6d2190 con 0x55de6d7e6160 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f74a23fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55de6d7f61e0 con 0x55de6d7e57b0 2024-04-21T08:40:14.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f74a23fb740 10 client.0 _ll_register_callbacks cb 0x55de6d6cd8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749dcc8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7ec770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749dcc8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7ec770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58620/0 (socket says 172.21.15.94:58620) 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749dcc8640 1 -- 172.21.15.94:0/3527692540 learned_addr learned my addr 172.21.15.94:0/3527692540 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749d4c7640 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7ebcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749ccc6640 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749dcc8640 1 -- 172.21.15.94:0/3527692540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 msgr2=0x55de6d7ebcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749dcc8640 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7ebcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749dcc8640 1 -- 172.21.15.94:0/3527692540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 msgr2=0x55de6d7ec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749dcc8640 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7ec230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749dcc8640 1 -- 172.21.15.94:0/3527692540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de6d5266b0 con 0x55de6d7eece0 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f749dcc8640 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7ec770 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7494036040 tx=0x7f749402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9989e0cbefe5c801 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.680+0000 7f7480ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:14.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f7480ff9640 1 -- 172.21.15.94:0/3527692540 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494052020 con 0x55de6d7eece0 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f7480ff9640 1 -- 172.21.15.94:0/3527692540 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f74940048c0 con 0x55de6d7eece0 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f74a23fb740 1 -- 172.21.15.94:0/3527692540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74880054d0 con 0x55de6d7eece0 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f7480ff9640 1 -- 172.21.15.94:0/3527692540 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f749405d050 con 0x55de6d7eece0 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f74a23fb740 10 client.15882 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f74a23fb740 1 -- 172.21.15.94:0/3527692540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55de6d8022d0 con 0x55de6d7eece0 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f7480ff9640 1 -- 172.21.15.94:0/3527692540 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74880054d0 con 0x55de6d7eece0 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f7480ff9640 1 -- 172.21.15.94:0/3527692540 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7494040040 con 0x55de6d7eece0 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f74a23fb740 10 client.15882 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f74a23fb740 10 client.15882 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f74a23fb740 20 client.15882 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f74a23fb740 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55de6d8049a0 0x55de6d805e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f74a23fb740 4 client.15882 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f74a23fb740 1 -- 172.21.15.94:0/3527692540 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de6d757530 con 0x55de6d8049a0 2024-04-21T08:40:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.681+0000 7f749d4c7640 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55de6d8049a0 0x55de6d805e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.682+0000 7f7480ff9640 10 client.15882 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.682+0000 7f749d4c7640 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55de6d8049a0 0x55de6d805e10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=edb4a6732768bc34 server_cookie=b8fe4fe91c6a60f2 in_seq=0 out_seq=0 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f7480ff9640 1 -- 172.21.15.94:0/3527692540 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55de6d757530 con 0x55de6d8049a0 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f7480ff9640 10 client.15882 handle_command_reply: tid=0 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 client.15882 shutdown 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 -- 172.21.15.94:0/3527692540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 msgr2=0x55de6d7ec770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7ec770 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7494036040 tx=0x7f749402bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 -- 172.21.15.94:0/3527692540 shutdown_connections 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55de6d8049a0 0x55de6d805e10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d7ec770 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7ec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 --2- 172.21.15.94:0/3527692540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7ebcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f7480ff9640 0 client.15882 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:40:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 -- 172.21.15.94:0/3527692540 >> 172.21.15.94:0/3527692540 conn(0x55de6d7dfa60 msgr2=0x55de6d7e1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 -- 172.21.15.94:0/3527692540 shutdown_connections 2024-04-21T08:40:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 1 -- 172.21.15.94:0/3527692540 wait complete. 2024-04-21T08:40:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.683+0000 7f74a23fb740 20 client.15882 trim_cache size 0 max 16384 2024-04-21T08:40:14.691 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:14 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3500123900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:14.691 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:14 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1377532368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:14.691 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:14 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1865798058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:14.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f74a23fb740 1 Processor -- start 2024-04-21T08:40:14.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f74a23fb740 1 -- start start 2024-04-21T08:40:14.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7e57b0 0x55de6d7eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7da800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7eece0 0x55de6d7ed410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f74a23fb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55de6d804150 con 0x55de6d7e57b0 2024-04-21T08:40:14.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f74a23fb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55de6d804820 con 0x55de6d7e6160 2024-04-21T08:40:14.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f74a23fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55de6d804ef0 con 0x55de6d7eece0 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749d4c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7e57b0 0x55de6d7eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749d4c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7e57b0 0x55de6d7eb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58626/0 (socket says 172.21.15.94:58626) 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749d4c7640 1 -- 172.21.15.94:0/246700039 learned_addr learned my addr 172.21.15.94:0/246700039 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749dcc8640 1 --2- 172.21.15.94:0/246700039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7eece0 0x55de6d7ed410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749ccc6640 1 --2- 172.21.15.94:0/246700039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7da800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749d4c7640 1 -- 172.21.15.94:0/246700039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7eece0 msgr2=0x55de6d7ed410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749d4c7640 1 --2- 172.21.15.94:0/246700039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7eece0 0x55de6d7ed410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749d4c7640 1 -- 172.21.15.94:0/246700039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 msgr2=0x55de6d7da800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749d4c7640 1 --2- 172.21.15.94:0/246700039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7da800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749d4c7640 1 -- 172.21.15.94:0/246700039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de6d7e4b00 con 0x55de6d7e57b0 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.701+0000 7f749dcc8640 1 --2- 172.21.15.94:0/246700039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7eece0 0x55de6d7ed410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f749d4c7640 1 --2- 172.21.15.94:0/246700039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7e57b0 0x55de6d7eb5f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7490099280 tx=0x7f74900d33f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9409fa39e61dc9f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f7482ffd640 1 -- 172.21.15.94:0/246700039 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749001d040 con 0x55de6d7e57b0 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f7482ffd640 1 -- 172.21.15.94:0/246700039 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74900a5520 con 0x55de6d7e57b0 2024-04-21T08:40:14.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f7482ffd640 1 -- 172.21.15.94:0/246700039 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74900a5090 con 0x55de6d7e57b0 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f74a23fb740 1 -- 172.21.15.94:0/246700039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7e57b0 msgr2=0x55de6d7eb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f74a23fb740 1 --2- 172.21.15.94:0/246700039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7e57b0 0x55de6d7eb5f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7490099280 tx=0x7f74900d33f0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f74a23fb740 1 -- 172.21.15.94:0/246700039 shutdown_connections 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f74a23fb740 1 --2- 172.21.15.94:0/246700039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7eece0 0x55de6d7ed410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f74a23fb740 1 --2- 172.21.15.94:0/246700039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7da800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f74a23fb740 1 --2- 172.21.15.94:0/246700039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7e57b0 0x55de6d7eb5f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f74a23fb740 1 -- 172.21.15.94:0/246700039 >> 172.21.15.94:0/246700039 conn(0x55de6d7dfa60 msgr2=0x55de6d7dfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f74a23fb740 1 -- 172.21.15.94:0/246700039 shutdown_connections 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.702+0000 7f74a23fb740 1 -- 172.21.15.94:0/246700039 wait complete. 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f74a23fb740 1 -- start start 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f74a23fb740 1 Processor -- start 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7e8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7e8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f74a23fb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d80d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f74a23fb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55de6d805c90 con 0x55de6d7eece0 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f74a23fb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55de6d8055c0 con 0x55de6d7e6160 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f74a23fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55de6d806360 con 0x55de6d7e57b0 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f74a23fb740 10 client.0 _ll_register_callbacks cb 0x7f7494009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f749dcc8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d80d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f749dcc8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d80d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58642/0 (socket says 172.21.15.94:58642) 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f749dcc8640 1 -- 172.21.15.94:0/4076072229 learned_addr learned my addr 172.21.15.94:0/4076072229 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f749ccc6640 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7e8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.703+0000 7f749d4c7640 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7e8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f749dcc8640 1 -- 172.21.15.94:0/4076072229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 msgr2=0x55de6d7e8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f749dcc8640 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7e8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f749dcc8640 1 -- 172.21.15.94:0/4076072229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 msgr2=0x55de6d7e8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f749dcc8640 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7e8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f749dcc8640 1 -- 172.21.15.94:0/4076072229 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de6d80d6a0 con 0x55de6d7eece0 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f749ccc6640 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7e8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f749d4c7640 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7e8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f749dcc8640 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d80d060 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f74940096c0 tx=0x7f7494005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=668d39126908a849 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f7480ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f7480ff9640 1 -- 172.21.15.94:0/4076072229 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494052070 con 0x55de6d7eece0 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f7480ff9640 1 -- 172.21.15.94:0/4076072229 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f74940037f0 con 0x55de6d7eece0 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f74a23fb740 1 -- 172.21.15.94:0/4076072229 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74900a5520 con 0x55de6d7eece0 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f7480ff9640 1 -- 172.21.15.94:0/4076072229 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f749402cca0 con 0x55de6d7eece0 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f74a23fb740 10 client.15888 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f74a23fb740 1 -- 172.21.15.94:0/4076072229 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55de6d80e8d0 con 0x55de6d7eece0 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f7480ff9640 1 -- 172.21.15.94:0/4076072229 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74900a5520 con 0x55de6d7eece0 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f7480ff9640 1 -- 172.21.15.94:0/4076072229 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f749404d500 con 0x55de6d7eece0 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f74a23fb740 10 client.15888 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:14.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f74a23fb740 10 client.15888 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f74a23fb740 20 client.15888 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f74a23fb740 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55de6d86f2f0 0x55de6d86e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f74a23fb740 4 client.15888 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.704+0000 7f74a23fb740 1 -- 172.21.15.94:0/4076072229 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "status"}) -- 0x55de6d7513a0 con 0x55de6d86f2f0 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f749d4c7640 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55de6d86f2f0 0x55de6d86e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f7480ff9640 10 client.15888 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f749d4c7640 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55de6d86f2f0 0x55de6d86e000 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=601e81b16530bb00 server_cookie=2bd403a929293a1b in_seq=0 out_seq=0 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f7480ff9640 1 -- 172.21.15.94:0/4076072229 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55de6d7513a0 con 0x55de6d86f2f0 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f7480ff9640 10 client.15888 handle_command_reply: tid=0 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f74a23fb740 1 client.15888 shutdown 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f74a23fb740 1 -- 172.21.15.94:0/4076072229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 msgr2=0x55de6d80d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f74a23fb740 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d80d060 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f74940096c0 tx=0x7f7494005850 comp rx=0 tx=0).stop 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f74a23fb740 1 -- 172.21.15.94:0/4076072229 shutdown_connections 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f74a23fb740 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55de6d86f2f0 0x55de6d86e000 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f74a23fb740 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55de6d7eece0 0x55de6d80d060 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f7480ff9640 0 client.15888 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:40:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f74a23fb740 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55de6d7e6160 0x55de6d7e8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f74a23fb740 1 --2- 172.21.15.94:0/4076072229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de6d7e57b0 0x55de6d7e8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.705+0000 7f74a23fb740 1 -- 172.21.15.94:0/4076072229 >> 172.21.15.94:0/4076072229 conn(0x55de6d7dfa60 msgr2=0x55de6d7da800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f74a23fb740 1 -- 172.21.15.94:0/4076072229 shutdown_connections 2024-04-21T08:40:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f74a23fb740 1 -- 172.21.15.94:0/4076072229 wait complete. 2024-04-21T08:40:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f74a23fb740 20 client.15888 trim_cache size 0 max 16384 2024-04-21T08:40:14.713 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:40:14.713 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 1, 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24454, 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 139.05514644100001, 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 24, 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 94, 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 93, 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 176.94660239699999 2024-04-21T08:40:14.714 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:40:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 -- 172.21.15.94:0/343989636 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74840a3e40 msgr2=0x7f74840a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74840a3e40 0x7f74840a6300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7494005ec0 tx=0x7f7494009260 comp rx=0 tx=0).stop 2024-04-21T08:40:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 -- 172.21.15.94:0/343989636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7498101090 msgr2=0x7f74981986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7498101090 0x7f74981986e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f749000cce0 tx=0x7f749000a620 comp rx=0 tx=0).stop 2024-04-21T08:40:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 -- 172.21.15.94:0/343989636 shutdown_connections 2024-04-21T08:40:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74840a3e40 0x7f74840a6300 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f749819cfa0 0x7f749819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7498106160 0x7f7498198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 --2- 172.21.15.94:0/343989636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7498101090 0x7f74981986e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 -- 172.21.15.94:0/343989636 >> 172.21.15.94:0/343989636 conn(0x7f7498077f50 msgr2=0x7f74980fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 -- 172.21.15.94:0/343989636 shutdown_connections 2024-04-21T08:40:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.706+0000 7f749f72b640 1 -- 172.21.15.94:0/343989636 wait complete. 2024-04-21T08:40:14.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:40:14.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:14 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3500123900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:40:14.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:14 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1377532368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:14.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:14 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1865798058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:40:14.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.816+0000 7fe359ee0640 1 -- 172.21.15.94:0/872360267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3541031e0 msgr2=0x7fe3541035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.816+0000 7fe359ee0640 1 --2- 172.21.15.94:0/872360267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3541031e0 0x7fe3541035c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe344009900 tx=0x7fe34402bcd0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.816+0000 7fe359ee0640 1 -- 172.21.15.94:0/872360267 shutdown_connections 2024-04-21T08:40:14.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.816+0000 7fe359ee0640 1 --2- 172.21.15.94:0/872360267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe354103b00 0x7fe35410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.816+0000 7fe359ee0640 1 --2- 172.21.15.94:0/872360267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe35410c5b0 0x7fe35410ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.816+0000 7fe359ee0640 1 --2- 172.21.15.94:0/872360267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3541031e0 0x7fe3541035c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.816+0000 7fe359ee0640 1 -- 172.21.15.94:0/872360267 >> 172.21.15.94:0/872360267 conn(0x7fe354077fe0 msgr2=0x7fe3540783f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.816+0000 7fe359ee0640 1 -- 172.21.15.94:0/872360267 shutdown_connections 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.816+0000 7fe359ee0640 1 -- 172.21.15.94:0/872360267 wait complete. 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.816+0000 7fe359ee0640 1 Processor -- start 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe359ee0640 1 -- start start 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe359ee0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3541031e0 0x7fe354198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe359ee0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe354103b00 0x7fe354198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe359ee0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe35410c5b0 0x7fe35419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe359ee0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe3541121f0 con 0x7fe35410c5b0 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe359ee0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe354112070 con 0x7fe3541031e0 2024-04-21T08:40:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe359ee0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe354112370 con 0x7fe354103b00 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe353fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe35410c5b0 0x7fe35419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe353fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe35410c5b0 0x7fe35419cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58662/0 (socket says 172.21.15.94:58662) 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe353fff640 1 -- 172.21.15.94:0/2850662056 learned_addr learned my addr 172.21.15.94:0/2850662056 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe3537fe640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3541031e0 0x7fe354198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe352ffd640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe354103b00 0x7fe354198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe353fff640 1 -- 172.21.15.94:0/2850662056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe354103b00 msgr2=0x7fe354198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe353fff640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe354103b00 0x7fe354198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe353fff640 1 -- 172.21.15.94:0/2850662056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3541031e0 msgr2=0x7fe354198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe353fff640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3541031e0 0x7fe354198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.817+0000 7fe353fff640 1 -- 172.21.15.94:0/2850662056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35419d720 con 0x7fe35410c5b0 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.818+0000 7fe353fff640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe35410c5b0 0x7fe35419cff0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009fd0 tx=0x7fe34c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.818+0000 7fe350ff9640 1 -- 172.21.15.94:0/2850662056 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c017070 con 0x7fe35410c5b0 2024-04-21T08:40:14.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.818+0000 7fe350ff9640 1 -- 172.21.15.94:0/2850662056 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe34c003e20 con 0x7fe35410c5b0 2024-04-21T08:40:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.818+0000 7fe359ee0640 1 -- 172.21.15.94:0/2850662056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe35419da10 con 0x7fe35410c5b0 2024-04-21T08:40:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.818+0000 7fe350ff9640 1 -- 172.21.15.94:0/2850662056 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c010460 con 0x7fe35410c5b0 2024-04-21T08:40:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.818+0000 7fe359ee0640 1 -- 172.21.15.94:0/2850662056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3541a52a0 con 0x7fe35410c5b0 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.820+0000 7fe350ff9640 1 -- 172.21.15.94:0/2850662056 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fe34c003970 con 0x7fe35410c5b0 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.820+0000 7fe35cbad740 1 Processor -- start 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.820+0000 7fe350ff9640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe33c0a3e40 0x7fe33c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.820+0000 7fe35cbad740 1 -- start start 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.820+0000 7fe352ffd640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe33c0a3e40 0x7fe33c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.820+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd932b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.820+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd93fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.820+0000 7fe350ff9640 1 -- 172.21.15.94:0/2850662056 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe34c0d0bb0 con 0x7fe35410c5b0 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.820+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd942500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe35cbad740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd896b10 con 0x55d8bd932750 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe35cbad740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd834240 con 0x55d8bd9400f0 2024-04-21T08:40:14.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe35cbad740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd892ad0 con 0x55d8bd933100 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe3537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd932b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe3537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd932b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58676/0 (socket says 172.21.15.94:58676) 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe3537fe640 1 -- 172.21.15.94:0/520377964 learned_addr learned my addr 172.21.15.94:0/520377964 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe352ffd640 1 --2- 172.21.15.94:0/520377964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd93fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe352ffd640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe33c0a3e40 0x7fe33c0a6300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe348005e10 tx=0x7fe34800af50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe3537fe640 1 -- 172.21.15.94:0/520377964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 msgr2=0x55d8bd93fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe3537fe640 1 --2- 172.21.15.94:0/520377964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd93fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe353fff640 1 --2- 172.21.15.94:0/520377964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd942500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe3537fe640 1 -- 172.21.15.94:0/520377964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 msgr2=0x55d8bd942500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe3537fe640 1 --2- 172.21.15.94:0/520377964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd942500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe3537fe640 1 -- 172.21.15.94:0/520377964 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8bd5a92d0 con 0x55d8bd932750 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe352ffd640 1 --2- 172.21.15.94:0/520377964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd93fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe3537fe640 1 --2- 172.21.15.94:0/520377964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd932b30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe344009900 tx=0x7fe344004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82390c8ad6329f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe339ffb640 1 -- 172.21.15.94:0/520377964 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344004230 con 0x55d8bd932750 2024-04-21T08:40:14.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe339ffb640 1 -- 172.21.15.94:0/520377964 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3440043d0 con 0x55d8bd932750 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.821+0000 7fe339ffb640 1 -- 172.21.15.94:0/520377964 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344004920 con 0x55d8bd932750 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.822+0000 7fe35cbad740 1 -- 172.21.15.94:0/520377964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 msgr2=0x55d8bd932b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.822+0000 7fe35cbad740 1 --2- 172.21.15.94:0/520377964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd932b30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe344009900 tx=0x7fe344004040 comp rx=0 tx=0).stop 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.822+0000 7fe35cbad740 1 -- 172.21.15.94:0/520377964 shutdown_connections 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.822+0000 7fe35cbad740 1 --2- 172.21.15.94:0/520377964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd93fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.822+0000 7fe35cbad740 1 --2- 172.21.15.94:0/520377964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd942500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.822+0000 7fe35cbad740 1 --2- 172.21.15.94:0/520377964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd932b30 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.822+0000 7fe35cbad740 1 -- 172.21.15.94:0/520377964 >> 172.21.15.94:0/520377964 conn(0x55d8bd92ca60 msgr2=0x55d8bd92ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.822+0000 7fe35cbad740 1 -- 172.21.15.94:0/520377964 shutdown_connections 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.822+0000 7fe35cbad740 1 -- 172.21.15.94:0/520377964 wait complete. 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.822+0000 7fe35cbad740 1 -- start start 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe35cbad740 1 Processor -- start 2024-04-21T08:40:14.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd93d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd93dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe35cbad740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd942fc0 con 0x55d8bd932750 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe35cbad740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd81f190 con 0x55d8bd9400f0 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe35cbad740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd943140 con 0x55d8bd933100 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe35cbad740 10 client.0 _ll_register_callbacks cb 0x55d8bd81a8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe3537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe3537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58678/0 (socket says 172.21.15.94:58678) 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe3537fe640 1 -- 172.21.15.94:0/2997479327 learned_addr learned my addr 172.21.15.94:0/2997479327 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe3537fe640 1 -- 172.21.15.94:0/2997479327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 msgr2=0x55d8bd93d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe3537fe640 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd93d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe3537fe640 1 -- 172.21.15.94:0/2997479327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 msgr2=0x55d8bd93dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe3537fe640 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd93dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe3537fe640 1 -- 172.21.15.94:0/2997479327 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8bd6736b0 con 0x55d8bd932750 2024-04-21T08:40:14.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe3537fe640 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93d060 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe3440056f0 tx=0x7fe34402bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b02ba3f541a0ade server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe332ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.823+0000 7fe332ffd640 1 -- 172.21.15.94:0/2997479327 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344034640 con 0x55d8bd932750 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe332ffd640 1 -- 172.21.15.94:0/2997479327 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe3440347e0 con 0x55d8bd932750 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe35cbad740 1 -- 172.21.15.94:0/2997479327 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3440043d0 con 0x55d8bd932750 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe332ffd640 1 -- 172.21.15.94:0/2997479327 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe34402c880 con 0x55d8bd932750 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe35cbad740 10 client.15900 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe35cbad740 1 -- 172.21.15.94:0/2997479327 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d8bd94f2c0 con 0x55d8bd932750 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe332ffd640 1 -- 172.21.15.94:0/2997479327 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3440043d0 con 0x55d8bd932750 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe332ffd640 1 -- 172.21.15.94:0/2997479327 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe34403f8e0 con 0x55d8bd932750 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe35cbad740 10 client.15900 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe35cbad740 10 client.15900 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe35cbad740 20 client.15900 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe35cbad740 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d8bd951630 0x55d8bd952aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe35cbad740 4 client.15900 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe35cbad740 1 -- 172.21.15.94:0/2997479327 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8bd8a4530 con 0x55d8bd951630 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.824+0000 7fe352ffd640 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d8bd951630 0x55d8bd952aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.825+0000 7fe352ffd640 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d8bd951630 0x55d8bd952aa0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c34460ea6f6ae64 server_cookie=1a1e14cc7157050e in_seq=0 out_seq=0 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.825+0000 7fe332ffd640 10 client.15900 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.825+0000 7fe332ffd640 1 -- 172.21.15.94:0/2997479327 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8bd8a4530 con 0x55d8bd951630 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.825+0000 7fe332ffd640 10 client.15900 handle_command_reply: tid=0 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.825+0000 7fe35cbad740 1 client.15900 shutdown 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 1 -- 172.21.15.94:0/2997479327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 msgr2=0x55d8bd93d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93d060 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe3440056f0 tx=0x7fe34402bcd0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 1 -- 172.21.15.94:0/2997479327 shutdown_connections 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d8bd951630 0x55d8bd952aa0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd93d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe332ffd640 0 client.15900 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd93dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 1 --2- 172.21.15.94:0/2997479327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93d060 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 1 -- 172.21.15.94:0/2997479327 >> 172.21.15.94:0/2997479327 conn(0x55d8bd92ca60 msgr2=0x55d8bd92ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 1 -- 172.21.15.94:0/2997479327 shutdown_connections 2024-04-21T08:40:14.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 1 -- 172.21.15.94:0/2997479327 wait complete. 2024-04-21T08:40:14.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.826+0000 7fe35cbad740 20 client.15900 trim_cache size 0 max 16384 2024-04-21T08:40:14.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.843+0000 7fe35cbad740 1 Processor -- start 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe35cbad740 1 -- start start 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd933100 0x55d8bd927800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd9400f0 0x55d8bd93d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe35cbad740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd950de0 con 0x55d8bd932750 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe35cbad740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd9514b0 con 0x55d8bd933100 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe35cbad740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd951b80 con 0x55d8bd9400f0 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58684/0 (socket says 172.21.15.94:58684) 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3537fe640 1 -- 172.21.15.94:0/3256675191 learned_addr learned my addr 172.21.15.94:0/3256675191 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe353fff640 1 --2- 172.21.15.94:0/3256675191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd9400f0 0x55d8bd93d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe352ffd640 1 --2- 172.21.15.94:0/3256675191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd933100 0x55d8bd927800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3537fe640 1 -- 172.21.15.94:0/3256675191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd9400f0 msgr2=0x55d8bd93d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3537fe640 1 --2- 172.21.15.94:0/3256675191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd9400f0 0x55d8bd93d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3537fe640 1 -- 172.21.15.94:0/3256675191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd933100 msgr2=0x55d8bd927800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3537fe640 1 --2- 172.21.15.94:0/3256675191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd933100 0x55d8bd927800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3537fe640 1 -- 172.21.15.94:0/3256675191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8bd92f150 con 0x55d8bd932750 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3537fe640 1 --2- 172.21.15.94:0/3256675191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93f790 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe34402c210 tx=0x7fe3440514a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f06fdbaabe42c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3397fa640 1 -- 172.21.15.94:0/3256675191 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344063070 con 0x55d8bd932750 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3397fa640 1 -- 172.21.15.94:0/3256675191 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe344009520 con 0x55d8bd932750 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.844+0000 7fe3397fa640 1 -- 172.21.15.94:0/3256675191 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34403f690 con 0x55d8bd932750 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.845+0000 7fe35cbad740 1 -- 172.21.15.94:0/3256675191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 msgr2=0x55d8bd93f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.845+0000 7fe35cbad740 1 --2- 172.21.15.94:0/3256675191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93f790 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe34402c210 tx=0x7fe3440514a0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.845+0000 7fe35cbad740 1 -- 172.21.15.94:0/3256675191 shutdown_connections 2024-04-21T08:40:14.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.845+0000 7fe35cbad740 1 --2- 172.21.15.94:0/3256675191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd9400f0 0x55d8bd93d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.845+0000 7fe35cbad740 1 --2- 172.21.15.94:0/3256675191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd933100 0x55d8bd927800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.845+0000 7fe35cbad740 1 --2- 172.21.15.94:0/3256675191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd93f790 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.845+0000 7fe35cbad740 1 -- 172.21.15.94:0/3256675191 >> 172.21.15.94:0/3256675191 conn(0x55d8bd92ca60 msgr2=0x55d8bd950610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.845+0000 7fe35cbad740 1 -- 172.21.15.94:0/3256675191 shutdown_connections 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.845+0000 7fe35cbad740 1 -- 172.21.15.94:0/3256675191 wait complete. 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.845+0000 7fe35cbad740 1 -- start start 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe35cbad740 1 Processor -- start 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd95b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd954550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe35cbad740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd954bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe35cbad740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd952920 con 0x55d8bd932750 2024-04-21T08:40:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe35cbad740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd952250 con 0x55d8bd9400f0 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe35cbad740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d8bd952ff0 con 0x55d8bd933100 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe35cbad740 10 client.0 _ll_register_callbacks cb 0x7fe34402c740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe3537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd95b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe3537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd95b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58692/0 (socket says 172.21.15.94:58692) 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe3537fe640 1 -- 172.21.15.94:0/286780860 learned_addr learned my addr 172.21.15.94:0/286780860 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe353fff640 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd954bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe352ffd640 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd954550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe3537fe640 1 -- 172.21.15.94:0/286780860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 msgr2=0x55d8bd954550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe3537fe640 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd954550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe3537fe640 1 -- 172.21.15.94:0/286780860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 msgr2=0x55d8bd954bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe3537fe640 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd954bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe3537fe640 1 -- 172.21.15.94:0/286780860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8bd958d10 con 0x55d8bd932750 2024-04-21T08:40:14.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe3537fe640 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd95b200 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe3440057c0 tx=0x7fe34402bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d49fc337372f9bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe332ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe332ffd640 1 -- 172.21.15.94:0/286780860 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344063070 con 0x55d8bd932750 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe332ffd640 1 -- 172.21.15.94:0/286780860 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe34403f630 con 0x55d8bd932750 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.846+0000 7fe35cbad740 1 -- 172.21.15.94:0/286780860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe344009520 con 0x55d8bd932750 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe332ffd640 1 -- 172.21.15.94:0/286780860 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe344034860 con 0x55d8bd932750 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe35cbad740 10 client.15906 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe35cbad740 1 -- 172.21.15.94:0/286780860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d8bd9b7710 con 0x55d8bd932750 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe332ffd640 1 -- 172.21.15.94:0/286780860 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe344009520 con 0x55d8bd932750 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe332ffd640 1 -- 172.21.15.94:0/286780860 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe344051ae0 con 0x55d8bd932750 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe35cbad740 10 client.15906 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe35cbad740 10 client.15906 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe35cbad740 20 client.15906 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe35cbad740 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d8bd9bbd70 0x55d8bd9baa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:14.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe35cbad740 4 client.15906 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe35cbad740 1 -- 172.21.15.94:0/286780860 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "status"}) -- 0x55d8bd89e3a0 con 0x55d8bd9bbd70 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.847+0000 7fe352ffd640 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d8bd9bbd70 0x55d8bd9baa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe352ffd640 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d8bd9bbd70 0x55d8bd9baa80 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa4fbc1f88533b7e server_cookie=abc067a01aa7919d in_seq=0 out_seq=0 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe332ffd640 10 client.15906 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe332ffd640 1 -- 172.21.15.94:0/286780860 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55d8bd89e3a0 con 0x55d8bd9bbd70 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe332ffd640 10 client.15906 handle_command_reply: tid=0 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 client.15906 shutdown 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 -- 172.21.15.94:0/286780860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 msgr2=0x55d8bd95b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd95b200 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe3440057c0 tx=0x7fe34402bcd0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 -- 172.21.15.94:0/286780860 shutdown_connections 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d8bd9bbd70 0x55d8bd9baa80 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bd933100 0x55d8bd954550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8bd9400f0 0x55d8bd954bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 --2- 172.21.15.94:0/286780860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d8bd932750 0x55d8bd95b200 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe332ffd640 0 client.15906 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:14.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 -- 172.21.15.94:0/286780860 >> 172.21.15.94:0/286780860 conn(0x55d8bd92ca60 msgr2=0x55d8bd927800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 -- 172.21.15.94:0/286780860 shutdown_connections 2024-04-21T08:40:14.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 1 -- 172.21.15.94:0/286780860 wait complete. 2024-04-21T08:40:14.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.848+0000 7fe35cbad740 20 client.15906 trim_cache size 0 max 16384 2024-04-21T08:40:14.858 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:40:14.858 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "2518ed94-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:40:14.858 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-04-21T08:40:14.858 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24488, 2024-04-21T08:40:14.859 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-21T08:40:14.859 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-21T08:40:14.859 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-21T08:40:14.859 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 163.2791971, 2024-04-21T08:40:14.859 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 24, 2024-04-21T08:40:14.859 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 94, 2024-04-21T08:40:14.859 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 93, 2024-04-21T08:40:14.859 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 168.55101723999999 2024-04-21T08:40:14.859 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:40:14.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 -- 172.21.15.94:0/2850662056 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe33c0a3e40 msgr2=0x7fe33c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe33c0a3e40 0x7fe33c0a6300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe348005e10 tx=0x7fe34800af50 comp rx=0 tx=0).stop 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 -- 172.21.15.94:0/2850662056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe35410c5b0 msgr2=0x7fe35419cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe35410c5b0 0x7fe35419cff0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009fd0 tx=0x7fe34c00b260 comp rx=0 tx=0).stop 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 -- 172.21.15.94:0/2850662056 shutdown_connections 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe33c0a3e40 0x7fe33c0a6300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe354103b00 0x7fe354198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe35410c5b0 0x7fe35419cff0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 --2- 172.21.15.94:0/2850662056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3541031e0 0x7fe354198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 -- 172.21.15.94:0/2850662056 >> 172.21.15.94:0/2850662056 conn(0x7fe354077fe0 msgr2=0x7fe3540fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 -- 172.21.15.94:0/2850662056 shutdown_connections 2024-04-21T08:40:14.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:14.849+0000 7fe359ee0640 1 -- 172.21.15.94:0/2850662056 wait complete. 2024-04-21T08:40:14.862 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 120 sec 2024-04-21T08:40:15.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:15 smithi111 ceph-mon[31493]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-21T08:40:15.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:15 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/433534346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:15.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:15 smithi094 ceph-mon[27988]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-21T08:40:15.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:15 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/433534346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:15.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:15 smithi173 ceph-mon[31428]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-21T08:40:15.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:15 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/433534346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:17.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:17 smithi111 ceph-mon[31493]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-21T08:40:17.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:17 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:40:17.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:17 smithi094 ceph-mon[27988]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-21T08:40:17.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:17 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:40:17.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:17 smithi173 ceph-mon[31428]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-21T08:40:17.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:17 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:40:19.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:19 smithi111 ceph-mon[31493]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-04-21T08:40:19.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:19 smithi094 ceph-mon[27988]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-04-21T08:40:19.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:19 smithi173 ceph-mon[31428]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-04-21T08:40:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.560+0000 7fc3615b6640 1 -- 172.21.15.94:0/2882034068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc35c106170 msgr2=0x7fc35c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.560+0000 7fc3615b6640 1 --2- 172.21.15.94:0/2882034068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc35c106170 0x7fc35c10dd40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc354009ca0 tx=0x7fc35402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.560+0000 7fc3615b6640 1 -- 172.21.15.94:0/2882034068 shutdown_connections 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.560+0000 7fc3615b6640 1 --2- 172.21.15.94:0/2882034068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c101aa0 0x7fc35c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.560+0000 7fc3615b6640 1 --2- 172.21.15.94:0/2882034068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc35c106170 0x7fc35c10dd40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.560+0000 7fc3615b6640 1 --2- 172.21.15.94:0/2882034068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc35c1010f0 0x7fc35c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.560+0000 7fc3615b6640 1 -- 172.21.15.94:0/2882034068 >> 172.21.15.94:0/2882034068 conn(0x7fc35c077fd0 msgr2=0x7fc35c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.560+0000 7fc3615b6640 1 -- 172.21.15.94:0/2882034068 shutdown_connections 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.560+0000 7fc3615b6640 1 -- 172.21.15.94:0/2882034068 wait complete. 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc3615b6640 1 Processor -- start 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc3615b6640 1 -- start start 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc3615b6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c1010f0 0x7fc35c1994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc3615b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc35c106170 0x7fc35c199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc3615b6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc35c199fc0 0x7fc35c1a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc3615b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c111130 con 0x7fc35c106170 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc3615b6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c110fb0 con 0x7fc35c199fc0 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc3615b6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c1112b0 con 0x7fc35c1010f0 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc35c106170 0x7fc35c199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc35c106170 0x7fc35c199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58712/0 (socket says 172.21.15.94:58712) 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35a7fc640 1 -- 172.21.15.94:0/2135823604 learned_addr learned my addr 172.21.15.94:0/2135823604 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35b7fe640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc35c199fc0 0x7fc35c1a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35affd640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c1010f0 0x7fc35c1994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35a7fc640 1 -- 172.21.15.94:0/2135823604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c1010f0 msgr2=0x7fc35c1994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c1010f0 0x7fc35c1994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35a7fc640 1 -- 172.21.15.94:0/2135823604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc35c199fc0 msgr2=0x7fc35c1a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc35c199fc0 0x7fc35c1a4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35a7fc640 1 -- 172.21.15.94:0/2135823604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c1a4d40 con 0x7fc35c106170 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.561+0000 7fc35affd640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c1010f0 0x7fc35c1994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.562+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc35c106170 0x7fc35c199a30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc35000b450 tx=0x7fc35000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.562+0000 7fc343fff640 1 -- 172.21.15.94:0/2135823604 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc350015020 con 0x7fc35c106170 2024-04-21T08:40:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.562+0000 7fc343fff640 1 -- 172.21.15.94:0/2135823604 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc350011d40 con 0x7fc35c106170 2024-04-21T08:40:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.562+0000 7fc3615b6640 1 -- 172.21.15.94:0/2135823604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35c1a4fd0 con 0x7fc35c106170 2024-04-21T08:40:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.562+0000 7fc343fff640 1 -- 172.21.15.94:0/2135823604 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc35000f340 con 0x7fc35c106170 2024-04-21T08:40:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.562+0000 7fc3615b6640 1 -- 172.21.15.94:0/2135823604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc35c1a54a0 con 0x7fc35c106170 2024-04-21T08:40:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.565+0000 7fc343fff640 1 -- 172.21.15.94:0/2135823604 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fc350018020 con 0x7fc35c106170 2024-04-21T08:40:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.565+0000 7fc343fff640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3440a3ee0 0x7fc3440a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.565+0000 7fc3641af740 1 Processor -- start 2024-04-21T08:40:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.565+0000 7fc3641af740 1 -- start start 2024-04-21T08:40:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.565+0000 7fc35affd640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3440a3ee0 0x7fc3440a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc343fff640 1 -- 172.21.15.94:0/2135823604 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc3500d08c0 con 0x7fc35c106170 2024-04-21T08:40:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc3641af740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d48d0 0x5651161d3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc3641af740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161d3b60 0x5651161d3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc3641af740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161dd0e0 0x5651161df4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc3641af740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5651161388d0 con 0x5651161dd0e0 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc3641af740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5651160d5e80 con 0x5651161d3b60 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc3641af740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x565116134890 con 0x5651161d48d0 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161dd0e0 0x5651161df4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161dd0e0 0x5651161df4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58726/0 (socket says 172.21.15.94:58726) 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35b7fe640 1 -- 172.21.15.94:0/1866146702 learned_addr learned my addr 172.21.15.94:0/1866146702 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35affd640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3440a3ee0 0x7fc3440a63a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc34c009b90 tx=0x7fc34c005e50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35affd640 1 --2- 172.21.15.94:0/1866146702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d48d0 0x5651161d3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/1866146702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161d3b60 0x5651161d3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35b7fe640 1 -- 172.21.15.94:0/1866146702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d48d0 msgr2=0x5651161d3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35b7fe640 1 --2- 172.21.15.94:0/1866146702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d48d0 0x5651161d3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35b7fe640 1 -- 172.21.15.94:0/1866146702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161d3b60 msgr2=0x5651161d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35b7fe640 1 --2- 172.21.15.94:0/1866146702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161d3b60 0x5651161d3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35b7fe640 1 -- 172.21.15.94:0/1866146702 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651160f7320 con 0x5651161dd0e0 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/1866146702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161d3b60 0x5651161d3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc35b7fe640 1 --2- 172.21.15.94:0/1866146702 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161dd0e0 0x5651161df4d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc354009ca0 tx=0x7fc354009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eed9b98874ea69b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.566+0000 7fc3417fa640 1 -- 172.21.15.94:0/1866146702 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc354043070 con 0x5651161dd0e0 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3417fa640 1 -- 172.21.15.94:0/1866146702 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc354032970 con 0x5651161dd0e0 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3417fa640 1 -- 172.21.15.94:0/1866146702 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc354039040 con 0x5651161dd0e0 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3641af740 1 -- 172.21.15.94:0/1866146702 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161dd0e0 msgr2=0x5651161df4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3641af740 1 --2- 172.21.15.94:0/1866146702 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161dd0e0 0x5651161df4d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc354009ca0 tx=0x7fc354009900 comp rx=0 tx=0).stop 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3641af740 1 -- 172.21.15.94:0/1866146702 shutdown_connections 2024-04-21T08:40:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3641af740 1 --2- 172.21.15.94:0/1866146702 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161dd0e0 0x5651161df4d0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3641af740 1 --2- 172.21.15.94:0/1866146702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161d3b60 0x5651161d3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3641af740 1 --2- 172.21.15.94:0/1866146702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d48d0 0x5651161d3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3641af740 1 -- 172.21.15.94:0/1866146702 >> 172.21.15.94:0/1866146702 conn(0x56511619ac90 msgr2=0x5651161ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3641af740 1 -- 172.21.15.94:0/1866146702 shutdown_connections 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.567+0000 7fc3641af740 1 -- 172.21.15.94:0/1866146702 wait complete. 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc3641af740 1 -- start start 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc3641af740 1 Processor -- start 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc3641af740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d3b60 0x5651161dca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc3641af740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d48d0 0x5651161d9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc3641af740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161da400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc3641af740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5651161dfdb0 con 0x5651161d3b60 2024-04-21T08:40:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc3641af740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5651160c0eb0 con 0x5651161dd0e0 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc3641af740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5651161dff30 con 0x5651161d48d0 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc3641af740 10 client.0 _ll_register_callbacks cb 0x5651160edb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc35affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d3b60 0x5651161dca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc35affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d3b60 0x5651161dca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58734/0 (socket says 172.21.15.94:58734) 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc35affd640 1 -- 172.21.15.94:0/440318909 learned_addr learned my addr 172.21.15.94:0/440318909 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc35affd640 1 -- 172.21.15.94:0/440318909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d48d0 msgr2=0x5651161d9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc35b7fe640 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161da400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc35affd640 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d48d0 0x5651161d9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc35affd640 1 -- 172.21.15.94:0/440318909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 msgr2=0x5651161da400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc35affd640 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161da400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.568+0000 7fc35affd640 1 -- 172.21.15.94:0/440318909 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565115e4b210 con 0x5651161d3b60 2024-04-21T08:40:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc35affd640 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d3b60 0x5651161dca40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc34c011c10 tx=0x7fc34c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d849b6923f12f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc33eff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc33eff5640 1 -- 172.21.15.94:0/440318909 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c011c40 con 0x5651161d3b60 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc33eff5640 1 -- 172.21.15.94:0/440318909 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc34c011de0 con 0x5651161d3b60 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc3641af740 1 -- 172.21.15.94:0/440318909 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc354032970 con 0x5651161d3b60 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc33eff5640 1 -- 172.21.15.94:0/440318909 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc34c02f050 con 0x5651161d3b60 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc3641af740 10 client.15918 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc3641af740 1 -- 172.21.15.94:0/440318909 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x565115f17620 con 0x5651161d3b60 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc33eff5640 1 -- 172.21.15.94:0/440318909 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc354032970 con 0x5651161d3b60 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc33eff5640 1 -- 172.21.15.94:0/440318909 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc34c00f040 con 0x5651161d3b60 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc3641af740 10 client.15918 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc3641af740 10 client.15918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc3641af740 20 client.15918 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc3641af740 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5651161ed160 0x5651161ef5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc3641af740 4 client.15918 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.569+0000 7fc3641af740 1 -- 172.21.15.94:0/440318909 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56511614c880 con 0x5651161ed160 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.570+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5651161ed160 0x5651161ef5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.570+0000 7fc33eff5640 10 client.15918 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.570+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5651161ed160 0x5651161ef5f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4248baa9608bba70 server_cookie=33bcccc0915a6983 in_seq=0 out_seq=0 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc33eff5640 1 -- 172.21.15.94:0/440318909 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56511614c880 con 0x5651161ed160 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc33eff5640 10 client.15918 handle_command_reply: tid=0 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 client.15918 shutdown 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 -- 172.21.15.94:0/440318909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d3b60 msgr2=0x5651161dca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d3b60 0x5651161dca40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc34c011c10 tx=0x7fc34c004730 comp rx=0 tx=0).stop 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 -- 172.21.15.94:0/440318909 shutdown_connections 2024-04-21T08:40:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5651161ed160 0x5651161ef5f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d48d0 0x5651161d9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161da400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 --2- 172.21.15.94:0/440318909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d3b60 0x5651161dca40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc33eff5640 0 client.15918 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 -- 172.21.15.94:0/440318909 >> 172.21.15.94:0/440318909 conn(0x56511619ac90 msgr2=0x5651161d05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 -- 172.21.15.94:0/440318909 shutdown_connections 2024-04-21T08:40:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 1 -- 172.21.15.94:0/440318909 wait complete. 2024-04-21T08:40:20.579 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.571+0000 7fc3641af740 20 client.15918 trim_cache size 0 max 16384 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc3641af740 1 Processor -- start 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc3641af740 1 -- start start 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc3641af740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d3b60 0x5651161e8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc3641af740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161dc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc3641af740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc3641af740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c1b1bf0 con 0x5651161d48d0 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc3641af740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c1af4c0 con 0x5651161dd0e0 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc3641af740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c1ad310 con 0x5651161d3b60 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc35a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161dc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc35a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161dc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58748/0 (socket says 172.21.15.94:58748) 2024-04-21T08:40:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc35a7fc640 1 -- 172.21.15.94:0/3419437950 learned_addr learned my addr 172.21.15.94:0/3419437950 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc35b7fe640 1 --2- 172.21.15.94:0/3419437950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc35affd640 1 --2- 172.21.15.94:0/3419437950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d3b60 0x5651161e8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc35a7fc640 1 -- 172.21.15.94:0/3419437950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d3b60 msgr2=0x5651161e8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/3419437950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d3b60 0x5651161e8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc35a7fc640 1 -- 172.21.15.94:0/3419437950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 msgr2=0x5651161f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/3419437950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161f4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.586+0000 7fc35a7fc640 1 -- 172.21.15.94:0/3419437950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651161d2e30 con 0x5651161d48d0 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.587+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/3419437950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161dc570 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc35000be10 tx=0x7fc3500047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4ce91af587a337 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.587+0000 7fc340ff9640 1 -- 172.21.15.94:0/3419437950 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3500a4f50 con 0x5651161d48d0 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.587+0000 7fc340ff9640 1 -- 172.21.15.94:0/3419437950 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc350096e50 con 0x5651161d48d0 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.587+0000 7fc340ff9640 1 -- 172.21.15.94:0/3419437950 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3500d0660 con 0x5651161d48d0 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.587+0000 7fc3641af740 1 -- 172.21.15.94:0/3419437950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 msgr2=0x5651161dc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.587+0000 7fc3641af740 1 --2- 172.21.15.94:0/3419437950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161dc570 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc35000be10 tx=0x7fc3500047d0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.587+0000 7fc3641af740 1 -- 172.21.15.94:0/3419437950 shutdown_connections 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.587+0000 7fc3641af740 1 --2- 172.21.15.94:0/3419437950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 --2- 172.21.15.94:0/3419437950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161dc570 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 --2- 172.21.15.94:0/3419437950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d3b60 0x5651161e8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 -- 172.21.15.94:0/3419437950 >> 172.21.15.94:0/3419437950 conn(0x56511619ac90 msgr2=0x5651161ce380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 -- 172.21.15.94:0/3419437950 shutdown_connections 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 -- 172.21.15.94:0/3419437950 wait complete. 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 -- start start 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 Processor -- start 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d3b60 0x5651161e53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161d66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161d6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c1ad980 con 0x5651161d48d0 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c1aedf0 con 0x5651161dd0e0 2024-04-21T08:40:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c1ae050 con 0x5651161d3b60 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc3641af740 10 client.0 _ll_register_callbacks cb 0x7fc34c0118e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.588+0000 7fc35a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161d66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161d66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58752/0 (socket says 172.21.15.94:58752) 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35a7fc640 1 -- 172.21.15.94:0/1291913474 learned_addr learned my addr 172.21.15.94:0/1291913474 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35affd640 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d3b60 0x5651161e53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35b7fe640 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161d6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35a7fc640 1 -- 172.21.15.94:0/1291913474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d3b60 msgr2=0x5651161e53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d3b60 0x5651161e53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35a7fc640 1 -- 172.21.15.94:0/1291913474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 msgr2=0x5651161d6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161d6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35a7fc640 1 -- 172.21.15.94:0/1291913474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651161d71b0 con 0x5651161d48d0 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35b7fe640 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161d6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc35a7fc640 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161d66b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fc35009bd30 tx=0x7fc3500a4fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f127e4b31f937e3e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc33eff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc33eff5640 1 -- 172.21.15.94:0/1291913474 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc350098950 con 0x5651161d48d0 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc33eff5640 1 -- 172.21.15.94:0/1291913474 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc350010500 con 0x5651161d48d0 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc3641af740 1 -- 172.21.15.94:0/1291913474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc350096e50 con 0x5651161d48d0 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc33eff5640 1 -- 172.21.15.94:0/1291913474 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc350098af0 con 0x5651161d48d0 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc3641af740 10 client.15924 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc3641af740 1 -- 172.21.15.94:0/1291913474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x565116257980 con 0x5651161d48d0 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.589+0000 7fc33eff5640 1 -- 172.21.15.94:0/1291913474 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc350096e50 con 0x5651161d48d0 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.590+0000 7fc33eff5640 1 -- 172.21.15.94:0/1291913474 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc35009b730 con 0x5651161d48d0 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.590+0000 7fc3641af740 10 client.15924 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.590+0000 7fc3641af740 10 client.15924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.590+0000 7fc3641af740 20 client.15924 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.590+0000 7fc3641af740 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565116257f90 0x56511625a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.590+0000 7fc3641af740 4 client.15924 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:40:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.590+0000 7fc3641af740 1 -- 172.21.15.94:0/1291913474 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5651160d83d0 con 0x565116257f90 2024-04-21T08:40:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.590+0000 7fc35affd640 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565116257f90 0x56511625a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.591+0000 7fc35affd640 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565116257f90 0x56511625a3b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4fb98d10ff1703 server_cookie=a05e9606c222354 in_seq=0 out_seq=0 2024-04-21T08:40:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.591+0000 7fc33eff5640 10 client.15924 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc33eff5640 1 -- 172.21.15.94:0/1291913474 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5651160d83d0 con 0x565116257f90 2024-04-21T08:40:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc33eff5640 10 client.15924 handle_command_reply: tid=0 2024-04-21T08:40:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 client.15924 shutdown 2024-04-21T08:40:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 -- 172.21.15.94:0/1291913474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 msgr2=0x5651161d66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161d66b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fc35009bd30 tx=0x7fc3500a4fc0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 -- 172.21.15.94:0/1291913474 shutdown_connections 2024-04-21T08:40:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565116257f90 0x56511625a3b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5651161dd0e0 0x5651161d6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc33eff5640 0 client.15924 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651161d48d0 0x5651161d66b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 --2- 172.21.15.94:0/1291913474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651161d3b60 0x5651161e53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 -- 172.21.15.94:0/1291913474 >> 172.21.15.94:0/1291913474 conn(0x56511619ac90 msgr2=0x5651161cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 -- 172.21.15.94:0/1291913474 shutdown_connections 2024-04-21T08:40:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.603+0000 7fc3641af740 1 -- 172.21.15.94:0/1291913474 wait complete. 2024-04-21T08:40:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3641af740 20 client.15924 trim_cache size 0 max 16384 2024-04-21T08:40:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:{ 2024-04-21T08:40:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "message": "", 2024-04-21T08:40:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "return_code": 0 2024-04-21T08:40:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:} 2024-04-21T08:40:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 -- 172.21.15.94:0/2135823604 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3440a3ee0 msgr2=0x7fc3440a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3440a3ee0 0x7fc3440a63a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc34c009b90 tx=0x7fc34c005e50 comp rx=0 tx=0).stop 2024-04-21T08:40:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 -- 172.21.15.94:0/2135823604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc35c106170 msgr2=0x7fc35c199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc35c106170 0x7fc35c199a30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc35000b450 tx=0x7fc35000b910 comp rx=0 tx=0).stop 2024-04-21T08:40:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 -- 172.21.15.94:0/2135823604 shutdown_connections 2024-04-21T08:40:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc3440a3ee0 0x7fc3440a63a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc35c199fc0 0x7fc35c1a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc35c106170 0x7fc35c199a30 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 --2- 172.21.15.94:0/2135823604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc35c1010f0 0x7fc35c1994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 -- 172.21.15.94:0/2135823604 >> 172.21.15.94:0/2135823604 conn(0x7fc35c077fd0 msgr2=0x7fc35c0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 -- 172.21.15.94:0/2135823604 shutdown_connections 2024-04-21T08:40:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:20.604+0000 7fc3615b6640 1 -- 172.21.15.94:0/2135823604 wait complete. 2024-04-21T08:40:21.561 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:21 smithi094 ceph-mon[27988]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-21T08:40:21.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:21 smithi111 ceph-mon[31493]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-21T08:40:21.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:21 smithi173 ceph-mon[31428]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-21T08:40:23.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:23 smithi111 ceph-mon[31493]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-21T08:40:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:23 smithi094 ceph-mon[27988]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-21T08:40:23.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:23 smithi173 ceph-mon[31428]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-21T08:40:25.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:25 smithi111 ceph-mon[31493]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-21T08:40:25.759 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:25 smithi094 ceph-mon[27988]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-21T08:40:25.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:25 smithi173 ceph-mon[31428]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-21T08:40:27.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:27 smithi111 ceph-mon[31493]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-21T08:40:27.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:27 smithi094 ceph-mon[27988]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-21T08:40:27.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:27 smithi173 ceph-mon[31428]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-21T08:40:29.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:29 smithi111 ceph-mon[31493]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-21T08:40:29.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:29 smithi094 ceph-mon[27988]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-21T08:40:29.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:29 smithi173 ceph-mon[31428]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-21T08:40:31.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:31 smithi111 ceph-mon[31493]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-21T08:40:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:31 smithi094 ceph-mon[27988]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-21T08:40:31.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:31 smithi173 ceph-mon[31428]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-21T08:40:33.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:33 smithi111 ceph-mon[31493]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-21T08:40:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.733+0000 7f30ad54b640 1 -- 172.21.15.94:0/626702318 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30a810e910 msgr2=0x7f30a8110d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.733+0000 7f30ad54b640 1 --2- 172.21.15.94:0/626702318 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30a810e910 0x7f30a8110d00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f30a0009ca0 tx=0x7f30a002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.733+0000 7f30ad54b640 1 -- 172.21.15.94:0/626702318 shutdown_connections 2024-04-21T08:40:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.733+0000 7f30ad54b640 1 --2- 172.21.15.94:0/626702318 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30a810e910 0x7f30a8110d00 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.733+0000 7f30ad54b640 1 --2- 172.21.15.94:0/626702318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30a8101680 0x7f30a810e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.733+0000 7f30ad54b640 1 --2- 172.21.15.94:0/626702318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30a8100d60 0x7f30a8101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.733+0000 7f30ad54b640 1 -- 172.21.15.94:0/626702318 >> 172.21.15.94:0/626702318 conn(0x7f30a8077fd0 msgr2=0x7f30a80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.733+0000 7f30ad54b640 1 -- 172.21.15.94:0/626702318 shutdown_connections 2024-04-21T08:40:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.733+0000 7f30ad54b640 1 -- 172.21.15.94:0/626702318 wait complete. 2024-04-21T08:40:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.734+0000 7f30ad54b640 1 Processor -- start 2024-04-21T08:40:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.734+0000 7f30ad54b640 1 -- start start 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.734+0000 7f30ad54b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30a8100d60 0x7f30a819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.734+0000 7f30ad54b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30a8101680 0x7f30a819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.734+0000 7f30ad54b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30a810e910 0x7f30a819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.734+0000 7f30ad54b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f30a81143b0 con 0x7f30a810e910 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.734+0000 7f30ad54b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f30a8114230 con 0x7f30a8101680 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.734+0000 7f30ad54b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f30a8114530 con 0x7f30a8100d60 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30a810e910 0x7f30a819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30a810e910 0x7f30a819eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39384/0 (socket says 172.21.15.94:39384) 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a77fe640 1 -- 172.21.15.94:0/2865365238 learned_addr learned my addr 172.21.15.94:0/2865365238 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a67fc640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30a8101680 0x7f30a819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30a8100d60 0x7f30a819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a77fe640 1 -- 172.21.15.94:0/2865365238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30a8100d60 msgr2=0x7f30a819a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a77fe640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30a8100d60 0x7f30a819a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a77fe640 1 -- 172.21.15.94:0/2865365238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30a8101680 msgr2=0x7f30a819ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a77fe640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30a8101680 0x7f30a819ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a77fe640 1 -- 172.21.15.94:0/2865365238 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a819f5d0 con 0x7f30a810e910 2024-04-21T08:40:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f30a77fe640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30a810e910 0x7f30a819eea0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f30a0004af0 tx=0x7f30a0035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f308ffff640 1 -- 172.21.15.94:0/2865365238 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a0039040 con 0x7f30a810e910 2024-04-21T08:40:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f308ffff640 1 -- 172.21.15.94:0/2865365238 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30a0044070 con 0x7f30a810e910 2024-04-21T08:40:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.735+0000 7f308ffff640 1 -- 172.21.15.94:0/2865365238 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a00042a0 con 0x7f30a810e910 2024-04-21T08:40:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.736+0000 7f30ad54b640 1 -- 172.21.15.94:0/2865365238 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a819f860 con 0x7f30a810e910 2024-04-21T08:40:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.737+0000 7f30ad54b640 1 -- 172.21.15.94:0/2865365238 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a81a7150 con 0x7f30a810e910 2024-04-21T08:40:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.739+0000 7f308ffff640 1 -- 172.21.15.94:0/2865365238 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f30a0004440 con 0x7f30a810e910 2024-04-21T08:40:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.740+0000 7f308ffff640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f30900a3f60 0x7f30900a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.740+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f30900a3f60 0x7f30900a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.740+0000 7f308ffff640 1 -- 172.21.15.94:0/2865365238 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f30a00f2ce0 con 0x7f30a810e910 2024-04-21T08:40:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.740+0000 7f30b0214740 1 Processor -- start 2024-04-21T08:40:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.740+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f30900a3f60 0x7f30900a6420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3098005d00 tx=0x7f309800b410 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.740+0000 7f30b0214740 1 -- start start 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30b0214740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff30620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30b0214740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff30b60 0x55becff30fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30b0214740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff3a0e0 0x55becff3c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30b0214740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55becfe958d0 con 0x55becff318d0 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30b0214740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55becfe32e80 con 0x55becff3a0e0 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30b0214740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55becfe91890 con 0x55becff30b60 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff30620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff30620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39388/0 (socket says 172.21.15.94:39388) 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a6ffd640 1 -- 172.21.15.94:0/4087625477 learned_addr learned my addr 172.21.15.94:0/4087625477 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a77fe640 1 --2- 172.21.15.94:0/4087625477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff3a0e0 0x55becff3c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a67fc640 1 --2- 172.21.15.94:0/4087625477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff30b60 0x55becff30fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a6ffd640 1 -- 172.21.15.94:0/4087625477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff30b60 msgr2=0x55becff30fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/4087625477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff30b60 0x55becff30fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a6ffd640 1 -- 172.21.15.94:0/4087625477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff3a0e0 msgr2=0x55becff3c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/4087625477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff3a0e0 0x55becff3c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a6ffd640 1 -- 172.21.15.94:0/4087625477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55becfe54320 con 0x55becff318d0 2024-04-21T08:40:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.741+0000 7f30a67fc640 1 --2- 172.21.15.94:0/4087625477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff30b60 0x55becff30fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.742+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/4087625477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff30620 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3098011010 tx=0x7f3098013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26cbab4a6f874e21 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.742+0000 7f308d7fa640 1 -- 172.21.15.94:0/4087625477 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309801e620 con 0x55becff318d0 2024-04-21T08:40:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.742+0000 7f308d7fa640 1 -- 172.21.15.94:0/4087625477 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309801ec70 con 0x55becff318d0 2024-04-21T08:40:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.742+0000 7f308d7fa640 1 -- 172.21.15.94:0/4087625477 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3098026c40 con 0x55becff318d0 2024-04-21T08:40:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 -- 172.21.15.94:0/4087625477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 msgr2=0x55becff30620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 --2- 172.21.15.94:0/4087625477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff30620 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3098011010 tx=0x7f3098013000 comp rx=0 tx=0).stop 2024-04-21T08:40:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 -- 172.21.15.94:0/4087625477 shutdown_connections 2024-04-21T08:40:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 --2- 172.21.15.94:0/4087625477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff30b60 0x55becff30fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 --2- 172.21.15.94:0/4087625477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff3a0e0 0x55becff3c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 --2- 172.21.15.94:0/4087625477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff30620 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 -- 172.21.15.94:0/4087625477 >> 172.21.15.94:0/4087625477 conn(0x55becfef7c90 msgr2=0x55becff2b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 -- 172.21.15.94:0/4087625477 shutdown_connections 2024-04-21T08:40:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 -- 172.21.15.94:0/4087625477 wait complete. 2024-04-21T08:40:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 -- start start 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 Processor -- start 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff39b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff36ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.744+0000 7f30b0214740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30b0214740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55becff3ce10 con 0x55becff318d0 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30b0214740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55becfe1deb0 con 0x55becff30b60 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30b0214740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55becff3cf90 con 0x55becff3a0e0 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff36ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30b0214740 10 client.0 _ll_register_callbacks cb 0x55becfe4ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff39b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff39b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:48940/0 (socket says 172.21.15.94:48940) 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a6ffd640 1 -- 172.21.15.94:0/2731543644 learned_addr learned my addr 172.21.15.94:0/2731543644 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a77fe640 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a6ffd640 1 -- 172.21.15.94:0/2731543644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 msgr2=0x55becff375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff375e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a6ffd640 1 -- 172.21.15.94:0/2731543644 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 msgr2=0x55becff36ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff36ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a6ffd640 1 -- 172.21.15.94:0/2731543644 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55becfba8210 con 0x55becff30b60 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff39b10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f30980266c0 tx=0x7f30980266f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58f95ee4d3a5ff0f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.745+0000 7f308aff5640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.746+0000 7f308aff5640 1 -- 172.21.15.94:0/2731543644 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3098027ec0 con 0x55becff30b60 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.746+0000 7f30b0214740 1 -- 172.21.15.94:0/2731543644 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f309801ec70 con 0x55becff30b60 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.746+0000 7f308aff5640 1 -- 172.21.15.94:0/2731543644 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f309801e420 con 0x55becff30b60 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.746+0000 7f308aff5640 1 -- 172.21.15.94:0/2731543644 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f309803f050 con 0x55becff30b60 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.746+0000 7f30b0214740 10 client.25292 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.746+0000 7f30b0214740 1 -- 172.21.15.94:0/2731543644 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55becfc74620 con 0x55becff30b60 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.746+0000 7f308aff5640 1 -- 172.21.15.94:0/2731543644 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309801ec70 con 0x55becff30b60 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.746+0000 7f308aff5640 1 -- 172.21.15.94:0/2731543644 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3098031af0 con 0x55becff30b60 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.746+0000 7f30b0214740 10 client.25292 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.746+0000 7f30b0214740 10 client.25292 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.747+0000 7f30b0214740 20 client.25292 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.747+0000 7f30b0214740 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55becff49fc0 0x55becff4c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.747+0000 7f30b0214740 4 client.25292 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.747+0000 7f30b0214740 1 -- 172.21.15.94:0/2731543644 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55becfea9880 con 0x55becff49fc0 2024-04-21T08:40:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.747+0000 7f30a67fc640 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55becff49fc0 0x55becff4c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.747+0000 7f308aff5640 10 client.25292 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.747+0000 7f30a67fc640 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55becff49fc0 0x55becff4c3b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a38bbd0bc83b0f8 server_cookie=f3b9d03709462109 in_seq=0 out_seq=0 2024-04-21T08:40:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f308aff5640 1 -- 172.21.15.94:0/2731543644 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55becfea9880 con 0x55becff49fc0 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f308aff5640 10 client.25292 handle_command_reply: tid=0 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f30b0214740 1 client.25292 shutdown 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f30b0214740 1 -- 172.21.15.94:0/2731543644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 msgr2=0x55becff39b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f30b0214740 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff39b10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f30980266c0 tx=0x7f30980266f0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f30b0214740 1 -- 172.21.15.94:0/2731543644 shutdown_connections 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f30b0214740 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55becff49fc0 0x55becff4c3b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f30b0214740 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f308aff5640 0 client.25292 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f30b0214740 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff36ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f30b0214740 1 --2- 172.21.15.94:0/2731543644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff39b10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.748+0000 7f30b0214740 1 -- 172.21.15.94:0/2731543644 >> 172.21.15.94:0/2731543644 conn(0x55becfef7c90 msgr2=0x55becff2d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.749+0000 7f30b0214740 1 -- 172.21.15.94:0/2731543644 shutdown_connections 2024-04-21T08:40:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.749+0000 7f30b0214740 1 -- 172.21.15.94:0/2731543644 wait complete. 2024-04-21T08:40:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.749+0000 7f30b0214740 20 client.25292 trim_cache size 0 max 16384 2024-04-21T08:40:33.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:33 smithi094 ceph-mon[27988]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-21T08:40:33.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:33 smithi173 ceph-mon[31428]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-21T08:40:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.763+0000 7f30b0214740 1 Processor -- start 2024-04-21T08:40:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.763+0000 7f30b0214740 1 -- start start 2024-04-21T08:40:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30b0214740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff45d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30b0214740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff51980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30b0214740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff51ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30b0214740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f30a81b3c60 con 0x55becff318d0 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30b0214740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f30a81b1530 con 0x55becff30b60 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30b0214740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f30a81af380 con 0x55becff3a0e0 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff51980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff51980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39410/0 (socket says 172.21.15.94:39410) 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a67fc640 1 -- 172.21.15.94:0/4134288041 learned_addr learned my addr 172.21.15.94:0/4134288041 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a77fe640 1 --2- 172.21.15.94:0/4134288041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff51ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a67fc640 1 -- 172.21.15.94:0/4134288041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 msgr2=0x55becff51ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a67fc640 1 --2- 172.21.15.94:0/4134288041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff51ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/4134288041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff45d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a67fc640 1 -- 172.21.15.94:0/4134288041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 msgr2=0x55becff45d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a67fc640 1 --2- 172.21.15.94:0/4134288041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff45d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a67fc640 1 -- 172.21.15.94:0/4134288041 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55becff353f0 con 0x55becff318d0 2024-04-21T08:40:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/4134288041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff45d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f30a67fc640 1 --2- 172.21.15.94:0/4134288041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff51980 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f309c00bb30 tx=0x7f309c0044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e174a5515c53f5cf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.764+0000 7f308cff9640 1 -- 172.21.15.94:0/4134288041 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c003e70 con 0x55becff318d0 2024-04-21T08:40:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.765+0000 7f308cff9640 1 -- 172.21.15.94:0/4134288041 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309c01e500 con 0x55becff318d0 2024-04-21T08:40:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.765+0000 7f308cff9640 1 -- 172.21.15.94:0/4134288041 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c012a90 con 0x55becff318d0 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.765+0000 7f30b0214740 1 -- 172.21.15.94:0/4134288041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 msgr2=0x55becff51980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.765+0000 7f30b0214740 1 --2- 172.21.15.94:0/4134288041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff51980 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f309c00bb30 tx=0x7f309c0044e0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 -- 172.21.15.94:0/4134288041 shutdown_connections 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 --2- 172.21.15.94:0/4134288041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff51ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 --2- 172.21.15.94:0/4134288041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff318d0 0x55becff51980 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 --2- 172.21.15.94:0/4134288041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff30b60 0x55becff45d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 -- 172.21.15.94:0/4134288041 >> 172.21.15.94:0/4134288041 conn(0x55becfef7c90 msgr2=0x55becff32080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 -- 172.21.15.94:0/4134288041 shutdown_connections 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 -- 172.21.15.94:0/4134288041 wait complete. 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 -- start start 2024-04-21T08:40:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 Processor -- start 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff30b60 0x55becff426f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff318d0 0x55becff344d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff34b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f30a81af9f0 con 0x55becff30b60 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f30a81b0e60 con 0x55becff318d0 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.766+0000 7f30b0214740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f30a81b00c0 con 0x55becff3a0e0 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30b0214740 10 client.0 _ll_register_callbacks cb 0x7f309800bf00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff30b60 0x55becff426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff30b60 0x55becff426f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39412/0 (socket says 172.21.15.94:39412) 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a6ffd640 1 -- 172.21.15.94:0/2528001834 learned_addr learned my addr 172.21.15.94:0/2528001834 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a77fe640 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff34b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a67fc640 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff318d0 0x55becff344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a6ffd640 1 -- 172.21.15.94:0/2528001834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 msgr2=0x55becff34b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff34b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a6ffd640 1 -- 172.21.15.94:0/2528001834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff318d0 msgr2=0x55becff344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff318d0 0x55becff344d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a6ffd640 1 -- 172.21.15.94:0/2528001834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55becff3b130 con 0x55becff30b60 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a67fc640 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff318d0 0x55becff344d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a77fe640 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff34b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30a6ffd640 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff30b60 0x55becff426f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3098010ca0 tx=0x7f3098010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee882af7a15cd8da server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f308aff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f308aff5640 1 -- 172.21.15.94:0/2528001834 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309802fa90 con 0x55becff30b60 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f308aff5640 1 -- 172.21.15.94:0/2528001834 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f309802fc30 con 0x55becff30b60 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.767+0000 7f30b0214740 1 -- 172.21.15.94:0/2528001834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f309c01e500 con 0x55becff30b60 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f308aff5640 1 -- 172.21.15.94:0/2528001834 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f309802fe30 con 0x55becff30b60 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f30b0214740 10 client.15942 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f30b0214740 1 -- 172.21.15.94:0/2528001834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55becffb0630 con 0x55becff30b60 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f308aff5640 1 -- 172.21.15.94:0/2528001834 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309c01e500 con 0x55becff30b60 2024-04-21T08:40:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f308aff5640 1 -- 172.21.15.94:0/2528001834 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f309801e830 con 0x55becff30b60 2024-04-21T08:40:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f30b0214740 10 client.15942 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f30b0214740 10 client.15942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f30b0214740 20 client.15942 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f30b0214740 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55becff49fc0 0x55becffb2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f30b0214740 4 client.15942 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:40:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f30b0214740 1 -- 172.21.15.94:0/2528001834 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55becfe353d0 con 0x55becff49fc0 2024-04-21T08:40:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.768+0000 7f30a67fc640 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55becff49fc0 0x55becffb2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.769+0000 7f308aff5640 10 client.15942 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.769+0000 7f30a67fc640 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55becff49fc0 0x55becffb2de0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c626c5859e4bc4d5 server_cookie=33d2c23c5451d372 in_seq=0 out_seq=0 2024-04-21T08:40:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.772+0000 7f308aff5640 1 -- 172.21.15.94:0/2528001834 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55becfe353d0 con 0x55becff49fc0 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.772+0000 7f308aff5640 10 client.15942 handle_command_reply: tid=0 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.772+0000 7f30b0214740 1 client.15942 shutdown 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.772+0000 7f30b0214740 1 -- 172.21.15.94:0/2528001834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff30b60 msgr2=0x55becff426f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.772+0000 7f30b0214740 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff30b60 0x55becff426f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3098010ca0 tx=0x7f3098010d50 comp rx=0 tx=0).stop 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30b0214740 1 -- 172.21.15.94:0/2528001834 shutdown_connections 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30b0214740 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55becff49fc0 0x55becffb2de0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f308aff5640 0 client.15942 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30b0214740 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55becff3a0e0 0x55becff34b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30b0214740 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55becff318d0 0x55becff344d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30b0214740 1 --2- 172.21.15.94:0/2528001834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55becff30b60 0x55becff426f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30b0214740 1 -- 172.21.15.94:0/2528001834 >> 172.21.15.94:0/2528001834 conn(0x55becfef7c90 msgr2=0x55becff2cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30b0214740 1 -- 172.21.15.94:0/2528001834 shutdown_connections 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30b0214740 1 -- 172.21.15.94:0/2528001834 wait complete. 2024-04-21T08:40:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30b0214740 20 client.15942 trim_cache size 0 max 16384 2024-04-21T08:40:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:{ 2024-04-21T08:40:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "message": "", 2024-04-21T08:40:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "return_code": 0 2024-04-21T08:40:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:} 2024-04-21T08:40:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30ad54b640 1 -- 172.21.15.94:0/2865365238 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f30900a3f60 msgr2=0x7f30900a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30ad54b640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f30900a3f60 0x7f30900a6420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3098005d00 tx=0x7f309800b410 comp rx=0 tx=0).stop 2024-04-21T08:40:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30ad54b640 1 -- 172.21.15.94:0/2865365238 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30a810e910 msgr2=0x7f30a819eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30ad54b640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30a810e910 0x7f30a819eea0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f30a0004af0 tx=0x7f30a0035ce0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30ad54b640 1 -- 172.21.15.94:0/2865365238 shutdown_connections 2024-04-21T08:40:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30ad54b640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f30900a3f60 0x7f30900a6420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30ad54b640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30a810e910 0x7f30a819eea0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30ad54b640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30a8101680 0x7f30a819ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30ad54b640 1 --2- 172.21.15.94:0/2865365238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30a8100d60 0x7f30a819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.773+0000 7f30ad54b640 1 -- 172.21.15.94:0/2865365238 >> 172.21.15.94:0/2865365238 conn(0x7f30a8077fd0 msgr2=0x7f30a810fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.774+0000 7f30ad54b640 1 -- 172.21.15.94:0/2865365238 shutdown_connections 2024-04-21T08:40:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:33.774+0000 7f30ad54b640 1 -- 172.21.15.94:0/2865365238 wait complete. 2024-04-21T08:40:35.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:35 smithi111 ceph-mon[31493]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-21T08:40:35.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:35 smithi094 ceph-mon[27988]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-21T08:40:35.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:35 smithi173 ceph-mon[31428]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-21T08:40:36.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:40:36.150 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:40:36.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.311+0000 7f2c573c7640 1 -- 172.21.15.94:0/2857780434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c50101a30 msgr2=0x7f2c50105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.311+0000 7f2c573c7640 1 --2- 172.21.15.94:0/2857780434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c50101a30 0x7f2c50105a70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009900 tx=0x7f2c4002bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:36.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.311+0000 7f2c573c7640 1 -- 172.21.15.94:0/2857780434 shutdown_connections 2024-04-21T08:40:36.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.311+0000 7f2c573c7640 1 --2- 172.21.15.94:0/2857780434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c50106150 0x7f2c5010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.311+0000 7f2c573c7640 1 --2- 172.21.15.94:0/2857780434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c50101a30 0x7f2c50105a70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.311+0000 7f2c573c7640 1 --2- 172.21.15.94:0/2857780434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c50101080 0x7f2c50101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.311+0000 7f2c573c7640 1 -- 172.21.15.94:0/2857780434 >> 172.21.15.94:0/2857780434 conn(0x7f2c50077f30 msgr2=0x7f2c50078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.311+0000 7f2c573c7640 1 -- 172.21.15.94:0/2857780434 shutdown_connections 2024-04-21T08:40:36.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.311+0000 7f2c573c7640 1 -- 172.21.15.94:0/2857780434 wait complete. 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.312+0000 7f2c573c7640 1 Processor -- start 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.312+0000 7f2c573c7640 1 -- start start 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.312+0000 7f2c573c7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c50101080 0x7f2c50198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.312+0000 7f2c573c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c50106150 0x7f2c50198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.312+0000 7f2c573c7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c5019cfe0 0x7f2c5019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c573c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501110e0 con 0x7f2c50106150 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c573c7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2c50110f60 con 0x7f2c5019cfe0 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c573c7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2c50111260 con 0x7f2c50101080 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c54962640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c50106150 0x7f2c50198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c54962640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c50106150 0x7f2c50198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39430/0 (socket says 172.21.15.94:39430) 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c54962640 1 -- 172.21.15.94:0/198192096 learned_addr learned my addr 172.21.15.94:0/198192096 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c55163640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c50101080 0x7f2c50198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c54962640 1 -- 172.21.15.94:0/198192096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c50101080 msgr2=0x7f2c50198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c55964640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c5019cfe0 0x7f2c5019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c54962640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c50101080 0x7f2c50198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c54962640 1 -- 172.21.15.94:0/198192096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c5019cfe0 msgr2=0x7f2c5019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c54962640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c5019cfe0 0x7f2c5019d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c54962640 1 -- 172.21.15.94:0/198192096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c5019dad0 con 0x7f2c50106150 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c54962640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c50106150 0x7f2c50198c60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2c40000c00 tx=0x7f2c400045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.313+0000 7f2c467fc640 1 -- 172.21.15.94:0/198192096 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40043020 con 0x7f2c50106150 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.314+0000 7f2c467fc640 1 -- 172.21.15.94:0/198192096 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c4002cb70 con 0x7f2c50106150 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.314+0000 7f2c467fc640 1 -- 172.21.15.94:0/198192096 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40003b80 con 0x7f2c50106150 2024-04-21T08:40:36.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.314+0000 7f2c573c7640 1 -- 172.21.15.94:0/198192096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c5019dd60 con 0x7f2c50106150 2024-04-21T08:40:36.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.315+0000 7f2c573c7640 1 -- 172.21.15.94:0/198192096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c501a55b0 con 0x7f2c50106150 2024-04-21T08:40:36.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.317+0000 7f2c467fc640 1 -- 172.21.15.94:0/198192096 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f2c40046020 con 0x7f2c50106150 2024-04-21T08:40:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c467fc640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2c3c0a3ec0 0x7f2c3c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c59fbb740 1 Processor -- start 2024-04-21T08:40:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c59fbb740 1 -- start start 2024-04-21T08:40:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c467fc640 1 -- 172.21.15.94:0/198192096 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f2c400f2ad0 con 0x7f2c50106150 2024-04-21T08:40:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c55163640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2c3c0a3ec0 0x7f2c3c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a95e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a95e990 0x56470a95ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a9701b0 0x56470a9725a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56470a8c31e0 con 0x56470a9701b0 2024-04-21T08:40:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56470a860390 con 0x56470a95e990 2024-04-21T08:40:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.318+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56470a8bf1a0 con 0x56470a95dfe0 2024-04-21T08:40:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c55964640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a9701b0 0x56470a9725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c55964640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a9701b0 0x56470a9725a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39446/0 (socket says 172.21.15.94:39446) 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c55964640 1 -- 172.21.15.94:0/2676503085 learned_addr learned my addr 172.21.15.94:0/2676503085 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c54962640 1 --2- 172.21.15.94:0/2676503085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a95e990 0x56470a95ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c55964640 1 -- 172.21.15.94:0/2676503085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 msgr2=0x56470a95e3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c55964640 1 --2- 172.21.15.94:0/2676503085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a95e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c55964640 1 -- 172.21.15.94:0/2676503085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a95e990 msgr2=0x56470a95ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c55964640 1 --2- 172.21.15.94:0/2676503085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a95e990 0x56470a95ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c55964640 1 -- 172.21.15.94:0/2676503085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56470a69f7d0 con 0x56470a9701b0 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c54962640 1 --2- 172.21.15.94:0/2676503085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a95e990 0x56470a95ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c55964640 1 --2- 172.21.15.94:0/2676503085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a9701b0 0x56470a9725a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2c48009e30 tx=0x7f2c4800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aaf6ec77c667931 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c55163640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2c3c0a3ec0 0x7f2c3c0a6380 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c005d00 tx=0x7f2c4c00b410 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.319+0000 7f2c3b7fe640 1 -- 172.21.15.94:0/2676503085 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c48011640 con 0x56470a9701b0 2024-04-21T08:40:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.320+0000 7f2c3b7fe640 1 -- 172.21.15.94:0/2676503085 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c48011c90 con 0x56470a9701b0 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.320+0000 7f2c3b7fe640 1 -- 172.21.15.94:0/2676503085 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c48010c00 con 0x56470a9701b0 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2676503085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a9701b0 msgr2=0x56470a9725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2676503085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a9701b0 0x56470a9725a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2c48009e30 tx=0x7f2c4800f730 comp rx=0 tx=0).stop 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2676503085 shutdown_connections 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2676503085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a9701b0 0x56470a9725a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2676503085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a95e990 0x56470a95ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2676503085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a95e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2676503085 >> 172.21.15.94:0/2676503085 conn(0x56470a958cb0 msgr2=0x56470a959090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2676503085 shutdown_connections 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2676503085 wait complete. 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 -- start start 2024-04-21T08:40:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.321+0000 7f2c59fbb740 1 Processor -- start 2024-04-21T08:40:36.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a9605b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 0x56470a960af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a9701b0 0x56470a961030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56470a973040 con 0x56470a95e990 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56470a87c2f0 con 0x56470a9701b0 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56470a9731c0 con 0x56470a95dfe0 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c59fbb740 10 client.0 _ll_register_callbacks cb 0x56470a889580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c54962640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 0x56470a960af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c54962640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 0x56470a960af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39458/0 (socket says 172.21.15.94:39458) 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c54962640 1 -- 172.21.15.94:0/3453557765 learned_addr learned my addr 172.21.15.94:0/3453557765 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c55964640 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a9701b0 0x56470a961030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c55163640 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a9605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c55964640 1 -- 172.21.15.94:0/3453557765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 msgr2=0x56470a9605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c55964640 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a9605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c55964640 1 -- 172.21.15.94:0/3453557765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 msgr2=0x56470a960af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.322+0000 7f2c55964640 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 0x56470a960af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.323+0000 7f2c55964640 1 -- 172.21.15.94:0/3453557765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56470a5d53f0 con 0x56470a9701b0 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.323+0000 7f2c55964640 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a9701b0 0x56470a961030 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2c48005c10 tx=0x7f2c48010600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fffa90c836884c2a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.323+0000 7f2c38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.323+0000 7f2c38ff9640 1 -- 172.21.15.94:0/3453557765 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c48004c50 con 0x56470a9701b0 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.323+0000 7f2c59fbb740 1 -- 172.21.15.94:0/3453557765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c48011c90 con 0x56470a9701b0 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.323+0000 7f2c38ff9640 1 -- 172.21.15.94:0/3453557765 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f2c48004df0 con 0x56470a9701b0 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.323+0000 7f2c38ff9640 1 -- 172.21.15.94:0/3453557765 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2c4802d050 con 0x56470a9701b0 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.323+0000 7f2c59fbb740 10 client.25301 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.323+0000 7f2c59fbb740 1 -- 172.21.15.94:0/3453557765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56470a5950f0 con 0x56470a9701b0 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.324+0000 7f2c38ff9640 1 -- 172.21.15.94:0/3453557765 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c48011c90 con 0x56470a9701b0 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.324+0000 7f2c38ff9640 1 -- 172.21.15.94:0/3453557765 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2c48011440 con 0x56470a9701b0 2024-04-21T08:40:36.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.324+0000 7f2c59fbb740 10 client.25301 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.324+0000 7f2c59fbb740 10 client.25301 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.324+0000 7f2c59fbb740 20 client.25301 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.324+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56470a980230 0x56470a982620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.324+0000 7f2c59fbb740 4 client.25301 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.324+0000 7f2c59fbb740 1 -- 172.21.15.94:0/3453557765 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56470a8d0c00 con 0x56470a980230 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.324+0000 7f2c55163640 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56470a980230 0x56470a982620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.325+0000 7f2c38ff9640 10 client.25301 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.325+0000 7f2c55163640 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56470a980230 0x56470a982620 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=445a5fe20131aa0a server_cookie=1946e71c16c27dac in_seq=0 out_seq=0 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.325+0000 7f2c38ff9640 1 -- 172.21.15.94:0/3453557765 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56470a8d0c00 con 0x56470a980230 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.325+0000 7f2c38ff9640 10 client.25301 handle_command_reply: tid=0 2024-04-21T08:40:36.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.325+0000 7f2c59fbb740 1 client.25301 shutdown 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 1 -- 172.21.15.94:0/3453557765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a9701b0 msgr2=0x56470a961030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a9701b0 0x56470a961030 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2c48005c10 tx=0x7f2c48010600 comp rx=0 tx=0).stop 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 1 -- 172.21.15.94:0/3453557765 shutdown_connections 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56470a980230 0x56470a982620 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a9701b0 0x56470a961030 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 0x56470a960af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/3453557765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a9605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 1 -- 172.21.15.94:0/3453557765 >> 172.21.15.94:0/3453557765 conn(0x56470a958cb0 msgr2=0x56470a95aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c38ff9640 0 client.25301 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 1 -- 172.21.15.94:0/3453557765 shutdown_connections 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 1 -- 172.21.15.94:0/3453557765 wait complete. 2024-04-21T08:40:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.326+0000 7f2c59fbb740 20 client.25301 trim_cache size 0 max 16384 2024-04-21T08:40:36.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.360+0000 7f2c59fbb740 1 Processor -- start 2024-04-21T08:40:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.360+0000 7f2c59fbb740 1 -- start start 2024-04-21T08:40:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a958cb0 0x56470a962ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95dfe0 0x56470a97bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a95e990 0x56470a95f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501afa00 con 0x56470a95dfe0 2024-04-21T08:40:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501ad2d0 con 0x56470a95e990 2024-04-21T08:40:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501ab120 con 0x56470a958cb0 2024-04-21T08:40:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c54962640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95dfe0 0x56470a97bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c54962640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95dfe0 0x56470a97bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39462/0 (socket says 172.21.15.94:39462) 2024-04-21T08:40:36.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c54962640 1 -- 172.21.15.94:0/2498617282 learned_addr learned my addr 172.21.15.94:0/2498617282 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c55163640 1 --2- 172.21.15.94:0/2498617282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a958cb0 0x56470a962ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c54962640 1 -- 172.21.15.94:0/2498617282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a958cb0 msgr2=0x56470a962ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c54962640 1 --2- 172.21.15.94:0/2498617282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a958cb0 0x56470a962ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c54962640 1 -- 172.21.15.94:0/2498617282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a95e990 msgr2=0x56470a95f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c54962640 1 --2- 172.21.15.94:0/2498617282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a95e990 0x56470a95f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c54962640 1 -- 172.21.15.94:0/2498617282 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56470a96c600 con 0x56470a95dfe0 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c55163640 1 --2- 172.21.15.94:0/2498617282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a958cb0 0x56470a962ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.361+0000 7f2c54962640 1 --2- 172.21.15.94:0/2498617282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95dfe0 0x56470a97bf70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2c400ba230 tx=0x7f2c40041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30801cd613ca8ce8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.362+0000 7f2c3affd640 1 -- 172.21.15.94:0/2498617282 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c400bdeb0 con 0x56470a95dfe0 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.362+0000 7f2c3affd640 1 -- 172.21.15.94:0/2498617282 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c400d7030 con 0x56470a95dfe0 2024-04-21T08:40:36.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.362+0000 7f2c3affd640 1 -- 172.21.15.94:0/2498617282 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c400babc0 con 0x56470a95dfe0 2024-04-21T08:40:36.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2498617282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95dfe0 msgr2=0x56470a97bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2498617282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95dfe0 0x56470a97bf70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2c400ba230 tx=0x7f2c40041040 comp rx=0 tx=0).stop 2024-04-21T08:40:36.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2498617282 shutdown_connections 2024-04-21T08:40:36.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2498617282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a95e990 0x56470a95f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2498617282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95dfe0 0x56470a97bf70 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2498617282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a958cb0 0x56470a962ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2498617282 >> 172.21.15.94:0/2498617282 conn(0x56470a96fd40 msgr2=0x56470a95ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2498617282 shutdown_connections 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2498617282 wait complete. 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 -- start start 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.363+0000 7f2c59fbb740 1 Processor -- start 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a958cb0 0x56470a978670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a96d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c59fbb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 0x56470a96d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501ab790 con 0x56470a95e990 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501acc00 con 0x56470a958cb0 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c59fbb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501abe60 con 0x56470a95dfe0 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c59fbb740 10 client.0 _ll_register_callbacks cb 0x7f2c48015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:36.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c54962640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a96d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c55163640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a958cb0 0x56470a978670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c54962640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a96d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:51092/0 (socket says 172.21.15.94:51092) 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.364+0000 7f2c54962640 1 -- 172.21.15.94:0/2353705769 learned_addr learned my addr 172.21.15.94:0/2353705769 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c55964640 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 0x56470a96d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c54962640 1 -- 172.21.15.94:0/2353705769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a958cb0 msgr2=0x56470a978670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c54962640 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a958cb0 0x56470a978670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c54962640 1 -- 172.21.15.94:0/2353705769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 msgr2=0x56470a96d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c54962640 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 0x56470a96d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c54962640 1 -- 172.21.15.94:0/2353705769 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56470a96df50 con 0x56470a95dfe0 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c55964640 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 0x56470a96d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c54962640 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a96d4a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c400baf20 tx=0x7f2c40003f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f84e3d26cfec731c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-21T08:40:36.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c38ff9640 1 -- 172.21.15.94:0/2353705769 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c400dd050 con 0x56470a95dfe0 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2353705769 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c400d7030 con 0x56470a95dfe0 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.365+0000 7f2c38ff9640 1 -- 172.21.15.94:0/2353705769 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f2c400bc850 con 0x56470a95dfe0 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c38ff9640 1 -- 172.21.15.94:0/2353705769 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2c400056a0 con 0x56470a95dfe0 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c59fbb740 10 client.25393 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2353705769 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56470a96f290 con 0x56470a95dfe0 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c38ff9640 1 -- 172.21.15.94:0/2353705769 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c400d7030 con 0x56470a95dfe0 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c38ff9640 1 -- 172.21.15.94:0/2353705769 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2c400c6e30 con 0x56470a95dfe0 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c59fbb740 10 client.25393 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c59fbb740 10 client.25393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c59fbb740 20 client.25393 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56470a9701b0 0x56470a9e99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c59fbb740 4 client.25393 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2353705769 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56470a8caa70 con 0x56470a9701b0 2024-04-21T08:40:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.366+0000 7f2c55163640 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56470a9701b0 0x56470a9e99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c38ff9640 10 client.25393 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c55163640 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56470a9701b0 0x56470a9e99b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71c89e828d1de02 server_cookie=48d9fffc1d03442b in_seq=0 out_seq=0 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c38ff9640 1 -- 172.21.15.94:0/2353705769 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56470a8caa70 con 0x56470a9701b0 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c38ff9640 10 client.25393 handle_command_reply: tid=0 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 client.25393 shutdown 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2353705769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 msgr2=0x56470a96d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a96d4a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c400baf20 tx=0x7f2c40003f50 comp rx=0 tx=0).stop 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2353705769 shutdown_connections 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56470a9701b0 0x56470a9e99b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56470a95dfe0 0x56470a96d4a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c38ff9640 0 client.25393 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56470a95e990 0x56470a96d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 --2- 172.21.15.94:0/2353705769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56470a958cb0 0x56470a978670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2353705769 >> 172.21.15.94:0/2353705769 conn(0x56470a96fd40 msgr2=0x56470a960dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2353705769 shutdown_connections 2024-04-21T08:40:36.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.367+0000 7f2c59fbb740 1 -- 172.21.15.94:0/2353705769 wait complete. 2024-04-21T08:40:36.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c59fbb740 20 client.25393 trim_cache size 0 max 16384 2024-04-21T08:40:36.373 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:40:36.373 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:40:36.373 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:40:36.373 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:40:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 -- 172.21.15.94:0/198192096 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2c3c0a3ec0 msgr2=0x7f2c3c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2c3c0a3ec0 0x7f2c3c0a6380 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c005d00 tx=0x7f2c4c00b410 comp rx=0 tx=0).stop 2024-04-21T08:40:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 -- 172.21.15.94:0/198192096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c50106150 msgr2=0x7f2c50198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c50106150 0x7f2c50198c60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2c40000c00 tx=0x7f2c400045e0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 -- 172.21.15.94:0/198192096 shutdown_connections 2024-04-21T08:40:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2c3c0a3ec0 0x7f2c3c0a6380 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c5019cfe0 0x7f2c5019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c50106150 0x7f2c50198c60 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 --2- 172.21.15.94:0/198192096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c50101080 0x7f2c50198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 -- 172.21.15.94:0/198192096 >> 172.21.15.94:0/198192096 conn(0x7f2c50077f30 msgr2=0x7f2c50104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 -- 172.21.15.94:0/198192096 shutdown_connections 2024-04-21T08:40:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.368+0000 7f2c573c7640 1 -- 172.21.15.94:0/198192096 wait complete. 2024-04-21T08:40:36.379 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:40:36.380 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:40:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 -- 172.21.15.94:0/14207857 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5050069aa0 msgr2=0x7f505010ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 --2- 172.21.15.94:0/14207857 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5050069aa0 0x7f505010ede0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5048009ca0 tx=0x7f504802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 -- 172.21.15.94:0/14207857 shutdown_connections 2024-04-21T08:40:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 --2- 172.21.15.94:0/14207857 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5050069aa0 0x7f505010ede0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 --2- 172.21.15.94:0/14207857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50500690e0 0x7f5050069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 --2- 172.21.15.94:0/14207857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5050106f50 0x7f5050107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 -- 172.21.15.94:0/14207857 >> 172.21.15.94:0/14207857 conn(0x7f5050077e30 msgr2=0x7f5050078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 -- 172.21.15.94:0/14207857 shutdown_connections 2024-04-21T08:40:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 -- 172.21.15.94:0/14207857 wait complete. 2024-04-21T08:40:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 Processor -- start 2024-04-21T08:40:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 -- start start 2024-04-21T08:40:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50500690e0 0x7f5050198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.480+0000 7f5057a81640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5050069aa0 0x7f5050198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f5057a81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5050106f50 0x7f505019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f5057a81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5050112150 con 0x7f5050106f50 2024-04-21T08:40:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f5057a81640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5050111fd0 con 0x7f50500690e0 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f5057a81640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f50501122d0 con 0x7f5050069aa0 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505601e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5050106f50 0x7f505019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505601e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5050106f50 0x7f505019ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39488/0 (socket says 172.21.15.94:39488) 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505601e640 1 -- 172.21.15.94:0/3410546230 learned_addr learned my addr 172.21.15.94:0/3410546230 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505601e640 1 -- 172.21.15.94:0/3410546230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5050069aa0 msgr2=0x7f5050198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505601e640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5050069aa0 0x7f5050198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505581d640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50500690e0 0x7f5050198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505601e640 1 -- 172.21.15.94:0/3410546230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50500690e0 msgr2=0x7f5050198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505601e640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50500690e0 0x7f5050198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505601e640 1 -- 172.21.15.94:0/3410546230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505019d5b0 con 0x7f5050106f50 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505581d640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50500690e0 0x7f5050198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f505601e640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5050106f50 0x7f505019ce80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f504802c700 tx=0x7f50480346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f5046ffd640 1 -- 172.21.15.94:0/3410546230 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048041020 con 0x7f5050106f50 2024-04-21T08:40:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f5046ffd640 1 -- 172.21.15.94:0/3410546230 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5048032960 con 0x7f5050106f50 2024-04-21T08:40:36.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f5046ffd640 1 -- 172.21.15.94:0/3410546230 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048039040 con 0x7f5050106f50 2024-04-21T08:40:36.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f5057a81640 1 -- 172.21.15.94:0/3410546230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505019d840 con 0x7f5050106f50 2024-04-21T08:40:36.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.481+0000 7f5057a81640 1 -- 172.21.15.94:0/3410546230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50501a5130 con 0x7f5050106f50 2024-04-21T08:40:36.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.483+0000 7f5057a81640 1 -- 172.21.15.94:0/3410546230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5050068de0 con 0x7f5050106f50 2024-04-21T08:40:36.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.484+0000 7f5046ffd640 1 -- 172.21.15.94:0/3410546230 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f5048044020 con 0x7f5050106f50 2024-04-21T08:40:36.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.484+0000 7f5046ffd640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f503c0a3ec0 0x7f503c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.484+0000 7f505581d640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f503c0a3ec0 0x7f503c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.484+0000 7f5046ffd640 1 -- 172.21.15.94:0/3410546230 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f50480f2b90 con 0x7f5050106f50 2024-04-21T08:40:36.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.485+0000 7f505581d640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f503c0a3ec0 0x7f503c0a6380 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f504c005fd0 tx=0x7f504c0094d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.489+0000 7f5046ffd640 1 -- 172.21.15.94:0/3410546230 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f504804b020 con 0x7f5050106f50 2024-04-21T08:40:36.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.714+0000 7f5057a81640 1 -- 172.21.15.94:0/3410546230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5050199910 con 0x7f5050106f50 2024-04-21T08:40:36.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.714+0000 7f5046ffd640 1 -- 172.21.15.94:0/3410546230 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f50480bed10 con 0x7f5050106f50 2024-04-21T08:40:36.714 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:40:36.714 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:40:36.715 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:40:36.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.716+0000 7f5057a81640 1 -- 172.21.15.94:0/3410546230 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f503c0a3ec0 msgr2=0x7f503c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.716+0000 7f5057a81640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f503c0a3ec0 0x7f503c0a6380 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f504c005fd0 tx=0x7f504c0094d0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.716+0000 7f5057a81640 1 -- 172.21.15.94:0/3410546230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5050106f50 msgr2=0x7f505019ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.716+0000 7f5057a81640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5050106f50 0x7f505019ce80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f504802c700 tx=0x7f50480346d0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.717+0000 7f5057a81640 1 -- 172.21.15.94:0/3410546230 shutdown_connections 2024-04-21T08:40:36.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.717+0000 7f5057a81640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f503c0a3ec0 0x7f503c0a6380 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.717+0000 7f5057a81640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5050069aa0 0x7f5050198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.717+0000 7f5057a81640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5050106f50 0x7f505019ce80 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.717+0000 7f5057a81640 1 --2- 172.21.15.94:0/3410546230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50500690e0 0x7f5050198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.717+0000 7f5057a81640 1 -- 172.21.15.94:0/3410546230 >> 172.21.15.94:0/3410546230 conn(0x7f5050077e30 msgr2=0x7f50500fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.717+0000 7f5057a81640 1 -- 172.21.15.94:0/3410546230 shutdown_connections 2024-04-21T08:40:36.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.717+0000 7f5057a81640 1 -- 172.21.15.94:0/3410546230 wait complete. 2024-04-21T08:40:36.731 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:40:36.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.864+0000 7f15b8831640 1 -- 172.21.15.94:0/2869591701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15b4106160 msgr2=0x7f15b410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.864+0000 7f15b8831640 1 --2- 172.21.15.94:0/2869591701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15b4106160 0x7f15b410dd30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009ca0 tx=0x7f15ac02c120 comp rx=0 tx=0).stop 2024-04-21T08:40:36.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.864+0000 7f15b8831640 1 -- 172.21.15.94:0/2869591701 shutdown_connections 2024-04-21T08:40:36.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.864+0000 7f15b8831640 1 --2- 172.21.15.94:0/2869591701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15b4101a40 0x7f15b4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.864+0000 7f15b8831640 1 --2- 172.21.15.94:0/2869591701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15b4106160 0x7f15b410dd30 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.864+0000 7f15b8831640 1 --2- 172.21.15.94:0/2869591701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b4101090 0x7f15b4101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.864+0000 7f15b8831640 1 -- 172.21.15.94:0/2869591701 >> 172.21.15.94:0/2869591701 conn(0x7f15b4077f50 msgr2=0x7f15b4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.864+0000 7f15b8831640 1 -- 172.21.15.94:0/2869591701 shutdown_connections 2024-04-21T08:40:36.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.864+0000 7f15b8831640 1 -- 172.21.15.94:0/2869591701 wait complete. 2024-04-21T08:40:36.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b8831640 1 Processor -- start 2024-04-21T08:40:36.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b8831640 1 -- start start 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b8831640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15b4101090 0x7f15b419e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b8831640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b4106160 0x7f15b419eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b8831640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15b4198630 0x7f15b4198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b8831640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f15b4111160 con 0x7f15b4198630 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b8831640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f15b4110fe0 con 0x7f15b4106160 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b8831640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41112e0 con 0x7f15b4101090 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15b4198630 0x7f15b4198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15b4198630 0x7f15b4198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39496/0 (socket says 172.21.15.94:39496) 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b2d9d640 1 -- 172.21.15.94:0/1053641192 learned_addr learned my addr 172.21.15.94:0/1053641192 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15abfff640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b4106160 0x7f15b419eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b259c640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15b4101090 0x7f15b419e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b2d9d640 1 -- 172.21.15.94:0/1053641192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15b4101090 msgr2=0x7f15b419e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b2d9d640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15b4101090 0x7f15b419e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b2d9d640 1 -- 172.21.15.94:0/1053641192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b4106160 msgr2=0x7f15b419eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b2d9d640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b4106160 0x7f15b419eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15b2d9d640 1 -- 172.21.15.94:0/1053641192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b4199230 con 0x7f15b4198630 2024-04-21T08:40:36.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.865+0000 7f15abfff640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b4106160 0x7f15b419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:36.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.866+0000 7f15b2d9d640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15b4198630 0x7f15b4198ab0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f15ac02c730 tx=0x7f15ac035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.866+0000 7f15ab7fe640 1 -- 172.21.15.94:0/1053641192 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac039040 con 0x7f15b4198630 2024-04-21T08:40:36.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.866+0000 7f15ab7fe640 1 -- 172.21.15.94:0/1053641192 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15ac044070 con 0x7f15b4198630 2024-04-21T08:40:36.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.866+0000 7f15ab7fe640 1 -- 172.21.15.94:0/1053641192 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac004330 con 0x7f15b4198630 2024-04-21T08:40:36.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.866+0000 7f15b8831640 1 -- 172.21.15.94:0/1053641192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b41994c0 con 0x7f15b4198630 2024-04-21T08:40:36.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.866+0000 7f15b8831640 1 -- 172.21.15.94:0/1053641192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b4102cf0 con 0x7f15b4198630 2024-04-21T08:40:36.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.869+0000 7f15ab7fe640 1 -- 172.21.15.94:0/1053641192 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f15ac0044d0 con 0x7f15b4198630 2024-04-21T08:40:36.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.869+0000 7f15bb41e740 1 Processor -- start 2024-04-21T08:40:36.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15bb41e740 1 -- start start 2024-04-21T08:40:36.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e74d0 0x55d05c1e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1e7e80 0x55d05c1f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1f0a00 0x55d05c1f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15bb41e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d05c14c1f0 con 0x55d05c1f0a00 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15bb41e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d05c0e94b0 con 0x55d05c1e74d0 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15bb41e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d05c1481b0 con 0x55d05c1e7e80 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15abfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1f0a00 0x55d05c1f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15b259c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1e7e80 0x55d05c1f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15abfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1f0a00 0x55d05c1f2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39512/0 (socket says 172.21.15.94:39512) 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15abfff640 1 -- 172.21.15.94:0/2154235587 learned_addr learned my addr 172.21.15.94:0/2154235587 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15ab7fe640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f15a00a3e40 0x7f15a00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15abfff640 1 --2- 172.21.15.94:0/2154235587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e74d0 0x55d05c1e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15abfff640 1 -- 172.21.15.94:0/2154235587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1e7e80 msgr2=0x55d05c1f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15abfff640 1 --2- 172.21.15.94:0/2154235587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1e7e80 0x55d05c1f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15abfff640 1 -- 172.21.15.94:0/2154235587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e74d0 msgr2=0x55d05c1e78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15abfff640 1 --2- 172.21.15.94:0/2154235587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e74d0 0x55d05c1e78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15abfff640 1 -- 172.21.15.94:0/2154235587 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d05c0db1b0 con 0x55d05c1f0a00 2024-04-21T08:40:36.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15ab7fe640 1 -- 172.21.15.94:0/1053641192 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f15ac0f3350 con 0x7f15b4198630 2024-04-21T08:40:36.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.870+0000 7f15b2d9d640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f15a00a3e40 0x7f15a00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.871+0000 7f15abfff640 1 --2- 172.21.15.94:0/2154235587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1f0a00 0x55d05c1f2e50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f159c0125f0 tx=0x7f159c0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5198304e7837e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.871+0000 7f15a8ff9640 1 -- 172.21.15.94:0/2154235587 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c010710 con 0x55d05c1f0a00 2024-04-21T08:40:36.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.871+0000 7f15a8ff9640 1 -- 172.21.15.94:0/2154235587 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f159c010d60 con 0x55d05c1f0a00 2024-04-21T08:40:36.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.871+0000 7f15a8ff9640 1 -- 172.21.15.94:0/2154235587 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c01dd80 con 0x55d05c1f0a00 2024-04-21T08:40:36.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 -- 172.21.15.94:0/2154235587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1f0a00 msgr2=0x55d05c1f2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15b2d9d640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f15a00a3e40 0x7f15a00a6300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0f06b0 tx=0x7f15ac034840 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 --2- 172.21.15.94:0/2154235587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1f0a00 0x55d05c1f2e50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f159c0125f0 tx=0x7f159c0097b0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 -- 172.21.15.94:0/2154235587 shutdown_connections 2024-04-21T08:40:36.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 --2- 172.21.15.94:0/2154235587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1e7e80 0x55d05c1f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 --2- 172.21.15.94:0/2154235587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1f0a00 0x55d05c1f2e50 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 --2- 172.21.15.94:0/2154235587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e74d0 0x55d05c1e78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 -- 172.21.15.94:0/2154235587 >> 172.21.15.94:0/2154235587 conn(0x55d05c1e1ba0 msgr2=0x55d05c1e1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 -- 172.21.15.94:0/2154235587 shutdown_connections 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 -- 172.21.15.94:0/2154235587 wait complete. 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 -- start start 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.872+0000 7f15bb41e740 1 Processor -- start 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1f0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e7e80 0x55d05c1ed710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1f0a00 0x55d05c1edd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15bb41e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d05c1f3970 con 0x55d05c1e74d0 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15bb41e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d05c105500 con 0x55d05c1e7e80 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15bb41e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d05c1f3af0 con 0x55d05c1f0a00 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15abfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15abfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1f0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39526/0 (socket says 172.21.15.94:39526) 2024-04-21T08:40:36.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15abfff640 1 -- 172.21.15.94:0/2765810532 learned_addr learned my addr 172.21.15.94:0/2765810532 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15bb41e740 10 client.0 _ll_register_callbacks cb 0x55d05c0d3b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15b259c640 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e7e80 0x55d05c1ed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15abfff640 1 -- 172.21.15.94:0/2765810532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1f0a00 msgr2=0x55d05c1edd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15abfff640 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1f0a00 0x55d05c1edd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15abfff640 1 -- 172.21.15.94:0/2765810532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e7e80 msgr2=0x55d05c1ed710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15abfff640 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e7e80 0x55d05c1ed710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15abfff640 1 -- 172.21.15.94:0/2765810532 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d05be5e3f0 con 0x55d05c1e74d0 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15abfff640 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1f0380 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f159c01dd30 tx=0x7f159c01dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e30aa440134610d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f1599ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f1599ffb640 1 -- 172.21.15.94:0/2765810532 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c023040 con 0x55d05c1e74d0 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f15bb41e740 1 -- 172.21.15.94:0/2765810532 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f159c010d60 con 0x55d05c1e74d0 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.873+0000 7f1599ffb640 1 -- 172.21.15.94:0/2765810532 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f159c005240 con 0x55d05c1e74d0 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f1599ffb640 1 -- 172.21.15.94:0/2765810532 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f159c035050 con 0x55d05c1e74d0 2024-04-21T08:40:36.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f15bb41e740 10 client.15978 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f15bb41e740 1 -- 172.21.15.94:0/2765810532 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d05bf287d0 con 0x55d05c1e74d0 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f1599ffb640 1 -- 172.21.15.94:0/2765810532 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f159c010d60 con 0x55d05c1e74d0 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f1599ffb640 1 -- 172.21.15.94:0/2765810532 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f159c030030 con 0x55d05c1e74d0 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f15bb41e740 10 client.15978 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f15bb41e740 10 client.15978 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f15bb41e740 20 client.15978 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f15bb41e740 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55d05c200df0 0x55d05c2031e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f15bb41e740 4 client.15978 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.874+0000 7f15bb41e740 1 -- 172.21.15.94:0/2765810532 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d05c0eb930 con 0x55d05c200df0 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.875+0000 7f15b259c640 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55d05c200df0 0x55d05c2031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.875+0000 7f1599ffb640 10 client.15978 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.875+0000 7f15b259c640 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55d05c200df0 0x55d05c2031e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=214c34991ef224ae server_cookie=6dcd1c4eb2c81310 in_seq=0 out_seq=0 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f1599ffb640 1 -- 172.21.15.94:0/2765810532 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d05c0eb930 con 0x55d05c200df0 2024-04-21T08:40:36.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f1599ffb640 10 client.15978 handle_command_reply: tid=0 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 client.15978 shutdown 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 -- 172.21.15.94:0/2765810532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 msgr2=0x55d05c1f0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1f0380 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f159c01dd30 tx=0x7f159c01dd60 comp rx=0 tx=0).stop 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 -- 172.21.15.94:0/2765810532 shutdown_connections 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55d05c200df0 0x55d05c2031e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1f0a00 0x55d05c1edd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e7e80 0x55d05c1ed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 --2- 172.21.15.94:0/2765810532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1f0380 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 -- 172.21.15.94:0/2765810532 >> 172.21.15.94:0/2765810532 conn(0x55d05c1e1ba0 msgr2=0x55d05c1e38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f1599ffb640 0 client.15978 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 -- 172.21.15.94:0/2765810532 shutdown_connections 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 1 -- 172.21.15.94:0/2765810532 wait complete. 2024-04-21T08:40:36.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.876+0000 7f15bb41e740 20 client.15978 trim_cache size 0 max 16384 2024-04-21T08:40:36.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15bb41e740 1 Processor -- start 2024-04-21T08:40:36.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15bb41e740 1 -- start start 2024-04-21T08:40:36.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1fc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e7e80 0x55d05c1efe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1f0a00 0x55d05c1f0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15bb41e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41b3ee0 con 0x55d05c1e74d0 2024-04-21T08:40:36.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15bb41e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41b17b0 con 0x55d05c1e7e80 2024-04-21T08:40:36.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15bb41e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41af600 con 0x55d05c1f0a00 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15abfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1fc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15abfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1fc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39542/0 (socket says 172.21.15.94:39542) 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15abfff640 1 -- 172.21.15.94:0/3974171280 learned_addr learned my addr 172.21.15.94:0/3974171280 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15abfff640 1 --2- 172.21.15.94:0/3974171280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1f0a00 0x55d05c1f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.894+0000 7f15b259c640 1 --2- 172.21.15.94:0/3974171280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e7e80 0x55d05c1efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.895+0000 7f15abfff640 1 -- 172.21.15.94:0/3974171280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1f0a00 msgr2=0x55d05c1f0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.895+0000 7f15abfff640 1 --2- 172.21.15.94:0/3974171280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1f0a00 0x55d05c1f0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.895+0000 7f15abfff640 1 -- 172.21.15.94:0/3974171280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e7e80 msgr2=0x55d05c1efe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.895+0000 7f15abfff640 1 --2- 172.21.15.94:0/3974171280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e7e80 0x55d05c1efe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.895+0000 7f15abfff640 1 -- 172.21.15.94:0/3974171280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d05c1ee090 con 0x55d05c1e74d0 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.895+0000 7f15abfff640 1 --2- 172.21.15.94:0/3974171280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1fc450 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f159c01dad0 tx=0x7f159c015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d750881872f6bdc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.895+0000 7f159bfff640 1 -- 172.21.15.94:0/3974171280 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c010710 con 0x55d05c1e74d0 2024-04-21T08:40:36.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.895+0000 7f159bfff640 1 -- 172.21.15.94:0/3974171280 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f159c009d00 con 0x55d05c1e74d0 2024-04-21T08:40:36.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.895+0000 7f159bfff640 1 -- 172.21.15.94:0/3974171280 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c01e8a0 con 0x55d05c1e74d0 2024-04-21T08:40:36.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.896+0000 7f15bb41e740 1 -- 172.21.15.94:0/3974171280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 msgr2=0x55d05c1fc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.896+0000 7f15bb41e740 1 --2- 172.21.15.94:0/3974171280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1fc450 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f159c01dad0 tx=0x7f159c015040 comp rx=0 tx=0).stop 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.896+0000 7f15bb41e740 1 -- 172.21.15.94:0/3974171280 shutdown_connections 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.896+0000 7f15bb41e740 1 --2- 172.21.15.94:0/3974171280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1f0a00 0x55d05c1f0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.896+0000 7f15bb41e740 1 --2- 172.21.15.94:0/3974171280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1e7e80 0x55d05c1efe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.896+0000 7f15bb41e740 1 --2- 172.21.15.94:0/3974171280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e74d0 0x55d05c1fc450 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.896+0000 7f15bb41e740 1 -- 172.21.15.94:0/3974171280 >> 172.21.15.94:0/3974171280 conn(0x55d05c1e1ba0 msgr2=0x55d05c1fbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.896+0000 7f15bb41e740 1 -- 172.21.15.94:0/3974171280 shutdown_connections 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.896+0000 7f15bb41e740 1 -- 172.21.15.94:0/3974171280 wait complete. 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.896+0000 7f15bb41e740 1 -- start start 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15bb41e740 1 Processor -- start 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1e74d0 0x55d05c202e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e7e80 0x55d05c2033d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15bb41e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1f0a00 0x55d05c264fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15bb41e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41afc70 con 0x55d05c1e7e80 2024-04-21T08:40:36.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15bb41e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41b10e0 con 0x55d05c1f0a00 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15bb41e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41b0340 con 0x55d05c1e74d0 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15bb41e740 10 client.0 _ll_register_callbacks cb 0x7f159c01d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15b259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e7e80 0x55d05c2033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15b259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e7e80 0x55d05c2033d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39548/0 (socket says 172.21.15.94:39548) 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15b259c640 1 -- 172.21.15.94:0/463494927 learned_addr learned my addr 172.21.15.94:0/463494927 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15abfff640 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1e74d0 0x55d05c202e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15abfff640 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1f0a00 0x55d05c264fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15b259c640 1 -- 172.21.15.94:0/463494927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1e74d0 msgr2=0x55d05c202e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15b259c640 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1e74d0 0x55d05c202e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15b259c640 1 -- 172.21.15.94:0/463494927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1f0a00 msgr2=0x55d05c264fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:36.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15b259c640 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1f0a00 0x55d05c264fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15b259c640 1 -- 172.21.15.94:0/463494927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d05c265550 con 0x55d05c1e7e80 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.897+0000 7f15abfff640 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1f0a00 0x55d05c264fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f15b259c640 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e7e80 0x55d05c2033d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f15a4006fd0 tx=0x7f15a401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d263128b8374ef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f1599ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f1599ffb640 1 -- 172.21.15.94:0/463494927 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a400be20 con 0x55d05c1e7e80 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f1599ffb640 1 -- 172.21.15.94:0/463494927 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f15a401cd60 con 0x55d05c1e7e80 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f15bb41e740 1 -- 172.21.15.94:0/463494927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f159c009d00 con 0x55d05c1e7e80 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f1599ffb640 1 -- 172.21.15.94:0/463494927 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f15a4004020 con 0x55d05c1e7e80 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f15bb41e740 10 client.15984 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f15bb41e740 1 -- 172.21.15.94:0/463494927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d05c2668e0 con 0x55d05c1e7e80 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f1599ffb640 1 -- 172.21.15.94:0/463494927 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f159c009d00 con 0x55d05c1e7e80 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f1599ffb640 1 -- 172.21.15.94:0/463494927 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f15a402e420 con 0x55d05c1e7e80 2024-04-21T08:40:36.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f15bb41e740 10 client.15984 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f15bb41e740 10 client.15984 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f15bb41e740 20 client.15984 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f15bb41e740 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55d05c267040 0x55d05c269460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f15bb41e740 4 client.15984 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.898+0000 7f15bb41e740 1 -- 172.21.15.94:0/463494927 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d05c159c10 con 0x55d05c267040 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15abfff640 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55d05c267040 0x55d05c269460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f1599ffb640 10 client.15984 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15abfff640 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55d05c267040 0x55d05c269460 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=918e65b07135175e server_cookie=92c1d1661a301b90 in_seq=0 out_seq=0 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f1599ffb640 1 -- 172.21.15.94:0/463494927 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d05c159c10 con 0x55d05c267040 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f1599ffb640 10 client.15984 handle_command_reply: tid=0 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15bb41e740 1 client.15984 shutdown 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15bb41e740 1 -- 172.21.15.94:0/463494927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e7e80 msgr2=0x55d05c2033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15bb41e740 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e7e80 0x55d05c2033d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f15a4006fd0 tx=0x7f15a401c510 comp rx=0 tx=0).stop 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15bb41e740 1 -- 172.21.15.94:0/463494927 shutdown_connections 2024-04-21T08:40:36.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15bb41e740 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55d05c267040 0x55d05c269460 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15bb41e740 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05c1f0a00 0x55d05c264fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15bb41e740 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d05c1e7e80 0x55d05c2033d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15bb41e740 1 --2- 172.21.15.94:0/463494927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d05c1e74d0 0x55d05c202e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f15bb41e740 1 -- 172.21.15.94:0/463494927 >> 172.21.15.94:0/463494927 conn(0x55d05c1e1ba0 msgr2=0x55d05c1fc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.899+0000 7f1599ffb640 0 client.15984 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:40:36.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15bb41e740 1 -- 172.21.15.94:0/463494927 shutdown_connections 2024-04-21T08:40:36.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15bb41e740 1 -- 172.21.15.94:0/463494927 wait complete. 2024-04-21T08:40:36.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15bb41e740 20 client.15984 trim_cache size 0 max 16384 2024-04-21T08:40:36.905 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:40:36.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 -- 172.21.15.94:0/1053641192 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f15a00a3e40 msgr2=0x7f15a00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f15a00a3e40 0x7f15a00a6300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0f06b0 tx=0x7f15ac034840 comp rx=0 tx=0).stop 2024-04-21T08:40:36.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 -- 172.21.15.94:0/1053641192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15b4198630 msgr2=0x7f15b4198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15b4198630 0x7f15b4198ab0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f15ac02c730 tx=0x7f15ac035d60 comp rx=0 tx=0).stop 2024-04-21T08:40:36.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 -- 172.21.15.94:0/1053641192 shutdown_connections 2024-04-21T08:40:36.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f15a00a3e40 0x7f15a00a6300 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15b4198630 0x7f15b4198ab0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b4106160 0x7f15b419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 --2- 172.21.15.94:0/1053641192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15b4101090 0x7f15b419e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 -- 172.21.15.94:0/1053641192 >> 172.21.15.94:0/1053641192 conn(0x7f15b4077f50 msgr2=0x7f15b40694a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 -- 172.21.15.94:0/1053641192 shutdown_connections 2024-04-21T08:40:36.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:36.900+0000 7f15b8831640 1 -- 172.21.15.94:0/1053641192 wait complete. 2024-04-21T08:40:36.911 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:40:37.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.013+0000 7f4c68c69640 1 -- 172.21.15.94:0/2587413076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c64100e80 msgr2=0x7f4c6410ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.013+0000 7f4c68c69640 1 --2- 172.21.15.94:0/2587413076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c64100e80 0x7f4c6410ede0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009cd0 tx=0x7f4c5c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.014+0000 7f4c68c69640 1 -- 172.21.15.94:0/2587413076 shutdown_connections 2024-04-21T08:40:37.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.014+0000 7f4c68c69640 1 --2- 172.21.15.94:0/2587413076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c64100e80 0x7f4c6410ede0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.014+0000 7f4c68c69640 1 --2- 172.21.15.94:0/2587413076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c641004c0 0x7f4c64100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.014+0000 7f4c68c69640 1 --2- 172.21.15.94:0/2587413076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c64106f50 0x7f4c64107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.014+0000 7f4c68c69640 1 -- 172.21.15.94:0/2587413076 >> 172.21.15.94:0/2587413076 conn(0x7f4c64077450 msgr2=0x7f4c640fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.014+0000 7f4c68c69640 1 -- 172.21.15.94:0/2587413076 shutdown_connections 2024-04-21T08:40:37.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.014+0000 7f4c68c69640 1 -- 172.21.15.94:0/2587413076 wait complete. 2024-04-21T08:40:37.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.014+0000 7f4c68c69640 1 Processor -- start 2024-04-21T08:40:37.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.014+0000 7f4c68c69640 1 -- start start 2024-04-21T08:40:37.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c68c69640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c641004c0 0x7f4c641016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c68c69640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c64100e80 0x7f4c64101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c68c69640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c64106f50 0x7f4c64102120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c68c69640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4c64112260 con 0x7f4c64106f50 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c68c69640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4c641120e0 con 0x7f4c64100e80 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c68c69640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4c641123e0 con 0x7f4c641004c0 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c62d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c64106f50 0x7f4c64102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c62d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c64106f50 0x7f4c64102120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39564/0 (socket says 172.21.15.94:39564) 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c62d9d640 1 -- 172.21.15.94:0/3025783130 learned_addr learned my addr 172.21.15.94:0/3025783130 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c6259c640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c641004c0 0x7f4c641016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c64100e80 0x7f4c64101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c62d9d640 1 -- 172.21.15.94:0/3025783130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c641004c0 msgr2=0x7f4c641016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c62d9d640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c641004c0 0x7f4c641016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c62d9d640 1 -- 172.21.15.94:0/3025783130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c64100e80 msgr2=0x7f4c64101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c62d9d640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c64100e80 0x7f4c64101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c62d9d640 1 -- 172.21.15.94:0/3025783130 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c641a6c80 con 0x7f4c64106f50 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c6259c640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c641004c0 0x7f4c641016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c64100e80 0x7f4c64101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:37.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c62d9d640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c64106f50 0x7f4c64102120 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c004af0 tx=0x7f4c5c034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c5b7fe640 1 -- 172.21.15.94:0/3025783130 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c041020 con 0x7f4c64106f50 2024-04-21T08:40:37.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c5b7fe640 1 -- 172.21.15.94:0/3025783130 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c5c032960 con 0x7f4c64106f50 2024-04-21T08:40:37.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.015+0000 7f4c5b7fe640 1 -- 172.21.15.94:0/3025783130 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c039040 con 0x7f4c64106f50 2024-04-21T08:40:37.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.016+0000 7f4c68c69640 1 -- 172.21.15.94:0/3025783130 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c641a6f10 con 0x7f4c64106f50 2024-04-21T08:40:37.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.016+0000 7f4c68c69640 1 -- 172.21.15.94:0/3025783130 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c641a7380 con 0x7f4c64106f50 2024-04-21T08:40:37.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.018+0000 7f4c6b857740 1 Processor -- start 2024-04-21T08:40:37.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c5b7fe640 1 -- 172.21.15.94:0/3025783130 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f4c5c044020 con 0x7f4c64106f50 2024-04-21T08:40:37.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c6b857740 1 -- start start 2024-04-21T08:40:37.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa04e80 0x5633efa0d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa0da00 0x5633efa0fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c6b857740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5633ef9691f0 con 0x5633efa044d0 2024-04-21T08:40:37.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c6b857740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5633ef9064b0 con 0x5633efa0da00 2024-04-21T08:40:37.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c6b857740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5633ef9651b0 con 0x5633efa04e80 2024-04-21T08:40:37.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa048b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39570/0 (socket says 172.21.15.94:39570) 2024-04-21T08:40:37.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.019+0000 7f4c5bfff640 1 -- 172.21.15.94:0/4145899934 learned_addr learned my addr 172.21.15.94:0/4145899934 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:37.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c6259c640 1 --2- 172.21.15.94:0/4145899934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa04e80 0x5633efa0d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/4145899934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa0da00 0x5633efa0fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4c500a3ec0 0x7f4c500a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c5bfff640 1 -- 172.21.15.94:0/4145899934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa04e80 msgr2=0x5633efa0d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/4145899934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa04e80 0x5633efa0d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c5bfff640 1 -- 172.21.15.94:0/4145899934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa0da00 msgr2=0x5633efa0fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/4145899934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa0da00 0x5633efa0fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c5bfff640 1 -- 172.21.15.94:0/4145899934 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633ef8f81b0 con 0x5633efa044d0 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c62d9d640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4c500a3ec0 0x7f4c500a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c5b7fe640 1 -- 172.21.15.94:0/3025783130 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4c5c0f2ab0 con 0x7f4c64106f50 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/4145899934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa048b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0155e0 tx=0x7f4c4c009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be55f1e07fa09cd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c58ff9640 1 -- 172.21.15.94:0/4145899934 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0134c0 con 0x5633efa044d0 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c58ff9640 1 -- 172.21.15.94:0/4145899934 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c4c01b070 con 0x5633efa044d0 2024-04-21T08:40:37.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.020+0000 7f4c58ff9640 1 -- 172.21.15.94:0/4145899934 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c012680 con 0x5633efa044d0 2024-04-21T08:40:37.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.021+0000 7f4c6b857740 1 -- 172.21.15.94:0/4145899934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 msgr2=0x5633efa048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.021+0000 7f4c6b857740 1 --2- 172.21.15.94:0/4145899934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa048b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0155e0 tx=0x7f4c4c009630 comp rx=0 tx=0).stop 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.021+0000 7f4c62d9d640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4c500a3ec0 0x7f4c500a6380 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0efe10 tx=0x7f4c5c003b50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.021+0000 7f4c6b857740 1 -- 172.21.15.94:0/4145899934 shutdown_connections 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.021+0000 7f4c6b857740 1 --2- 172.21.15.94:0/4145899934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa04e80 0x5633efa0d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.021+0000 7f4c6b857740 1 --2- 172.21.15.94:0/4145899934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa0da00 0x5633efa0fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.021+0000 7f4c6b857740 1 --2- 172.21.15.94:0/4145899934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa048b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.021+0000 7f4c6b857740 1 -- 172.21.15.94:0/4145899934 >> 172.21.15.94:0/4145899934 conn(0x5633ef9feba0 msgr2=0x5633ef9fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.021+0000 7f4c6b857740 1 -- 172.21.15.94:0/4145899934 shutdown_connections 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.021+0000 7f4c6b857740 1 -- 172.21.15.94:0/4145899934 wait complete. 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c6b857740 1 -- start start 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c6b857740 1 Processor -- start 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa0d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 0x5633efa0a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa0da00 0x5633efa0ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c6b857740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5633efa10970 con 0x5633efa044d0 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c6b857740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5633ef922500 con 0x5633efa04e80 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c6b857740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5633efa10af0 con 0x5633efa0da00 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c6b857740 10 client.0 _ll_register_callbacks cb 0x5633ef8f0b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa0d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa0d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39574/0 (socket says 172.21.15.94:39574) 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c5bfff640 1 -- 172.21.15.94:0/229304972 learned_addr learned my addr 172.21.15.94:0/229304972 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.022+0000 7f4c6259c640 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 0x5633efa0a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c5bfff640 1 -- 172.21.15.94:0/229304972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa0da00 msgr2=0x5633efa0ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa0da00 0x5633efa0ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c5bfff640 1 -- 172.21.15.94:0/229304972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 msgr2=0x5633efa0a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 0x5633efa0a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c5bfff640 1 -- 172.21.15.94:0/229304972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633ef67b3f0 con 0x5633efa044d0 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c6259c640 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 0x5633efa0a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa0d380 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c018040 tx=0x7f4c4c015320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8463dc57dea0cb22 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c49ffb640 1 -- 172.21.15.94:0/229304972 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c005450 con 0x5633efa044d0 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c49ffb640 1 -- 172.21.15.94:0/229304972 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4c4c0046c0 con 0x5633efa044d0 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c6b857740 1 -- 172.21.15.94:0/229304972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c4c01b070 con 0x5633efa044d0 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c49ffb640 1 -- 172.21.15.94:0/229304972 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4c4c032050 con 0x5633efa044d0 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c6b857740 10 client.15996 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.023+0000 7f4c6b857740 1 -- 172.21.15.94:0/229304972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5633ef7457d0 con 0x5633efa044d0 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.024+0000 7f4c49ffb640 1 -- 172.21.15.94:0/229304972 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c4c01b070 con 0x5633efa044d0 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.024+0000 7f4c49ffb640 1 -- 172.21.15.94:0/229304972 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4c4c00c070 con 0x5633efa044d0 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.024+0000 7f4c6b857740 10 client.15996 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.024+0000 7f4c6b857740 10 client.15996 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:40:37.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.024+0000 7f4c6b857740 20 client.15996 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.024+0000 7f4c6b857740 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5633efa1dba0 0x5633efa1ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.024+0000 7f4c6b857740 4 client.15996 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.024+0000 7f4c6b857740 1 -- 172.21.15.94:0/229304972 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633ef908930 con 0x5633efa1dba0 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.024+0000 7f4c6259c640 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5633efa1dba0 0x5633efa1ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.025+0000 7f4c49ffb640 10 client.15996 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.025+0000 7f4c6259c640 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5633efa1dba0 0x5633efa1ffe0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b173de47163f58bc server_cookie=18d13a253d3c5a73 in_seq=0 out_seq=0 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.025+0000 7f4c49ffb640 1 -- 172.21.15.94:0/229304972 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5633ef908930 con 0x5633efa1dba0 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.025+0000 7f4c49ffb640 10 client.15996 handle_command_reply: tid=0 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.025+0000 7f4c6b857740 1 client.15996 shutdown 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 1 -- 172.21.15.94:0/229304972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 msgr2=0x5633efa0d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa0d380 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c018040 tx=0x7f4c4c015320 comp rx=0 tx=0).stop 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 1 -- 172.21.15.94:0/229304972 shutdown_connections 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa0da00 0x5633efa0ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5633efa1dba0 0x5633efa1ffe0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 0x5633efa0a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 1 --2- 172.21.15.94:0/229304972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa044d0 0x5633efa0d380 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 1 -- 172.21.15.94:0/229304972 >> 172.21.15.94:0/229304972 conn(0x5633ef9feba0 msgr2=0x5633efa008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c49ffb640 0 client.15996 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:40:37.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 1 -- 172.21.15.94:0/229304972 shutdown_connections 2024-04-21T08:40:37.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 1 -- 172.21.15.94:0/229304972 wait complete. 2024-04-21T08:40:37.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.026+0000 7f4c6b857740 20 client.15996 trim_cache size 0 max 16384 2024-04-21T08:40:37.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.050+0000 7f4c6b857740 1 Processor -- start 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.050+0000 7f4c6b857740 1 -- start start 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa044d0 0x5633ef9fde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 0x5633efa19860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa0ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c6b857740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4c641b3f50 con 0x5633efa0da00 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c6b857740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4c641b1820 con 0x5633efa04e80 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c6b857740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4c641af670 con 0x5633efa044d0 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa0ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa0ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39582/0 (socket says 172.21.15.94:39582) 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c5bfff640 1 -- 172.21.15.94:0/3731837883 learned_addr learned my addr 172.21.15.94:0/3731837883 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c6259c640 1 --2- 172.21.15.94:0/3731837883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 0x5633efa19860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/3731837883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa044d0 0x5633ef9fde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c5bfff640 1 -- 172.21.15.94:0/3731837883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa044d0 msgr2=0x5633ef9fde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/3731837883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa044d0 0x5633ef9fde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c5bfff640 1 -- 172.21.15.94:0/3731837883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 msgr2=0x5633efa19860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/3731837883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 0x5633efa19860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c5bfff640 1 -- 172.21.15.94:0/3731837883 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633efa0aa80 con 0x5633efa0da00 2024-04-21T08:40:37.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c6259c640 1 --2- 172.21.15.94:0/3731837883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 0x5633efa19860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:37.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.051+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/3731837883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa0ce40 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009890 tx=0x7f4c4c014f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b8775d58691879 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.052+0000 7f4c4bfff640 1 -- 172.21.15.94:0/3731837883 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0134c0 con 0x5633efa0da00 2024-04-21T08:40:37.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.052+0000 7f4c4bfff640 1 -- 172.21.15.94:0/3731837883 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c4c01b030 con 0x5633efa0da00 2024-04-21T08:40:37.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.052+0000 7f4c4bfff640 1 -- 172.21.15.94:0/3731837883 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c004d50 con 0x5633efa0da00 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 -- 172.21.15.94:0/3731837883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 msgr2=0x5633efa0ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 --2- 172.21.15.94:0/3731837883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa0ce40 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009890 tx=0x7f4c4c014f30 comp rx=0 tx=0).stop 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 -- 172.21.15.94:0/3731837883 shutdown_connections 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 --2- 172.21.15.94:0/3731837883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa0ce40 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 --2- 172.21.15.94:0/3731837883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa04e80 0x5633efa19860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 --2- 172.21.15.94:0/3731837883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa044d0 0x5633ef9fde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 -- 172.21.15.94:0/3731837883 >> 172.21.15.94:0/3731837883 conn(0x5633ef9feba0 msgr2=0x5633efa09740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 -- 172.21.15.94:0/3731837883 shutdown_connections 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 -- 172.21.15.94:0/3731837883 wait complete. 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 -- start start 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.053+0000 7f4c6b857740 1 Processor -- start 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa044d0 0x5633efa11350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa04e80 0x5633efa11890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c6b857740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa157f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c6b857740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4c641afce0 con 0x5633efa0da00 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c6b857740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4c641b1150 con 0x5633efa044d0 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c6b857740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4c641b03b0 con 0x5633efa04e80 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c6b857740 10 client.0 _ll_register_callbacks cb 0x7f4c4c009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa157f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa157f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39592/0 (socket says 172.21.15.94:39592) 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c5bfff640 1 -- 172.21.15.94:0/1400561287 learned_addr learned my addr 172.21.15.94:0/1400561287 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c6259c640 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa04e80 0x5633efa11890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa044d0 0x5633efa11350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c5bfff640 1 -- 172.21.15.94:0/1400561287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa04e80 msgr2=0x5633efa11890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa04e80 0x5633efa11890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c5bfff640 1 -- 172.21.15.94:0/1400561287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa044d0 msgr2=0x5633efa11350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa044d0 0x5633efa11350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c5bfff640 1 -- 172.21.15.94:0/1400561287 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633efa15e60 con 0x5633efa0da00 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa157f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009cb0 tx=0x7f4c4c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a7d27ff93c087e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c49ffb640 1 -- 172.21.15.94:0/1400561287 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c013b20 con 0x5633efa0da00 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c6b857740 1 -- 172.21.15.94:0/1400561287 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c4c01b030 con 0x5633efa0da00 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.054+0000 7f4c49ffb640 1 -- 172.21.15.94:0/1400561287 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4c4c03c070 con 0x5633efa0da00 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c49ffb640 1 -- 172.21.15.94:0/1400561287 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4c4c049050 con 0x5633efa0da00 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c6b857740 10 client.16002 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c6b857740 1 -- 172.21.15.94:0/1400561287 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5633efa83ca0 con 0x5633efa0da00 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c49ffb640 1 -- 172.21.15.94:0/1400561287 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c4c01b030 con 0x5633efa0da00 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c49ffb640 1 -- 172.21.15.94:0/1400561287 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4c4c03f030 con 0x5633efa0da00 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c6b857740 10 client.16002 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c6b857740 10 client.16002 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c6b857740 20 client.16002 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c6b857740 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5633efa84300 0x5633efa866f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c6b857740 4 client.16002 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c6b857740 1 -- 172.21.15.94:0/1400561287 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5633ef976c10 con 0x5633efa84300 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.055+0000 7f4c6259c640 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5633efa84300 0x5633efa866f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c49ffb640 10 client.16002 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6259c640 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5633efa84300 0x5633efa866f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e2deef72a8fc8972 server_cookie=565a76c9410bc0 in_seq=0 out_seq=0 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c49ffb640 1 -- 172.21.15.94:0/1400561287 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5633ef976c10 con 0x5633efa84300 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c49ffb640 10 client.16002 handle_command_reply: tid=0 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 client.16002 shutdown 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 -- 172.21.15.94:0/1400561287 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 msgr2=0x5633efa157f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa157f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009cb0 tx=0x7f4c4c009f80 comp rx=0 tx=0).stop 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 -- 172.21.15.94:0/1400561287 shutdown_connections 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633efa04e80 0x5633efa11890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5633efa84300 0x5633efa866f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5633efa0da00 0x5633efa157f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 --2- 172.21.15.94:0/1400561287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5633efa044d0 0x5633efa11350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 -- 172.21.15.94:0/1400561287 >> 172.21.15.94:0/1400561287 conn(0x5633ef9feba0 msgr2=0x5633efa052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c49ffb640 0 client.16002 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:40:37.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 -- 172.21.15.94:0/1400561287 shutdown_connections 2024-04-21T08:40:37.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.056+0000 7f4c6b857740 1 -- 172.21.15.94:0/1400561287 wait complete. 2024-04-21T08:40:37.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.057+0000 7f4c6b857740 20 client.16002 trim_cache size 0 max 16384 2024-04-21T08:40:37.062 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:40:37.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.057+0000 7f4c68c69640 1 -- 172.21.15.94:0/3025783130 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4c500a3ec0 msgr2=0x7f4c500a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.057+0000 7f4c68c69640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4c500a3ec0 0x7f4c500a6380 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0efe10 tx=0x7f4c5c003b50 comp rx=0 tx=0).stop 2024-04-21T08:40:37.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.057+0000 7f4c68c69640 1 -- 172.21.15.94:0/3025783130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c64106f50 msgr2=0x7f4c64102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.057+0000 7f4c68c69640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c64106f50 0x7f4c64102120 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c004af0 tx=0x7f4c5c034670 comp rx=0 tx=0).stop 2024-04-21T08:40:37.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.058+0000 7f4c68c69640 1 -- 172.21.15.94:0/3025783130 shutdown_connections 2024-04-21T08:40:37.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.058+0000 7f4c68c69640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4c500a3ec0 0x7f4c500a6380 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.058+0000 7f4c68c69640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c64106f50 0x7f4c64102120 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.058+0000 7f4c68c69640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c64100e80 0x7f4c64101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.058+0000 7f4c68c69640 1 --2- 172.21.15.94:0/3025783130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c641004c0 0x7f4c641016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.058+0000 7f4c68c69640 1 -- 172.21.15.94:0/3025783130 >> 172.21.15.94:0/3025783130 conn(0x7f4c64077450 msgr2=0x7f4c64104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.058+0000 7f4c68c69640 1 -- 172.21.15.94:0/3025783130 shutdown_connections 2024-04-21T08:40:37.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.058+0000 7f4c68c69640 1 -- 172.21.15.94:0/3025783130 wait complete. 2024-04-21T08:40:37.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:40:37.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.204+0000 7f82757b2640 1 -- 172.21.15.94:0/2105241186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8270102f10 msgr2=0x7f8270100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.204+0000 7f82757b2640 1 --2- 172.21.15.94:0/2105241186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8270102f10 0x7f8270100460 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8260009900 tx=0x7f826002bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:37.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.204+0000 7f82757b2640 1 -- 172.21.15.94:0/2105241186 shutdown_connections 2024-04-21T08:40:37.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.204+0000 7f82757b2640 1 --2- 172.21.15.94:0/2105241186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f827010ebe0 0x7f8270110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.204+0000 7f82757b2640 1 --2- 172.21.15.94:0/2105241186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8270103380 0x7f82701009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.204+0000 7f82757b2640 1 --2- 172.21.15.94:0/2105241186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8270102f10 0x7f8270100460 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.204+0000 7f82757b2640 1 -- 172.21.15.94:0/2105241186 >> 172.21.15.94:0/2105241186 conn(0x7f82700773f0 msgr2=0x7f82700fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.204+0000 7f82757b2640 1 -- 172.21.15.94:0/2105241186 shutdown_connections 2024-04-21T08:40:37.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.204+0000 7f82757b2640 1 -- 172.21.15.94:0/2105241186 wait complete. 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.205+0000 7f82757b2640 1 Processor -- start 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.205+0000 7f82757b2640 1 -- start start 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.205+0000 7f82757b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8270102f10 0x7f82701a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.205+0000 7f82757b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8270103380 0x7f82701a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f82757b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f827010ebe0 0x7f827019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f82757b2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8270114460 con 0x7f8270103380 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8270103380 0x7f82701a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f82757b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f82701142e0 con 0x7f827010ebe0 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8270103380 0x7f82701a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39614/0 (socket says 172.21.15.94:39614) 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826e7fc640 1 -- 172.21.15.94:0/2292029415 learned_addr learned my addr 172.21.15.94:0/2292029415 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f82757b2640 1 -- 172.21.15.94:0/2292029415 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f82701145e0 con 0x7f8270102f10 2024-04-21T08:40:37.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826effd640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8270102f10 0x7f82701a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826f7fe640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f827010ebe0 0x7f827019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826e7fc640 1 -- 172.21.15.94:0/2292029415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8270102f10 msgr2=0x7f82701a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826e7fc640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8270102f10 0x7f82701a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826e7fc640 1 -- 172.21.15.94:0/2292029415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f827010ebe0 msgr2=0x7f827019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826e7fc640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f827010ebe0 0x7f827019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826e7fc640 1 -- 172.21.15.94:0/2292029415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827019af70 con 0x7f8270103380 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826f7fe640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f827010ebe0 0x7f827019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f826e7fc640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8270103380 0x7f82701a0b70 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f826400e9c0 tx=0x7f826400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f8257fff640 1 -- 172.21.15.94:0/2292029415 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826400ccc0 con 0x7f8270103380 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.206+0000 7f8257fff640 1 -- 172.21.15.94:0/2292029415 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82640054d0 con 0x7f8270103380 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.207+0000 7f82757b2640 1 -- 172.21.15.94:0/2292029415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f827019b260 con 0x7f8270103380 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.207+0000 7f8257fff640 1 -- 172.21.15.94:0/2292029415 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8264010430 con 0x7f8270103380 2024-04-21T08:40:37.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.207+0000 7f82757b2640 1 -- 172.21.15.94:0/2292029415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82701a7480 con 0x7f8270103380 2024-04-21T08:40:37.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.209+0000 7f82783a2740 1 Processor -- start 2024-04-21T08:40:37.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.209+0000 7f82783a2740 1 -- start start 2024-04-21T08:40:37.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.209+0000 7f82783a2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc04d0 0x56055ffc08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.209+0000 7f82783a2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc0e80 0x56055ffc9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.209+0000 7f82783a2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc9a00 0x56055ffcbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.209+0000 7f82783a2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56055ff251f0 con 0x56055ffc04d0 2024-04-21T08:40:37.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.209+0000 7f82783a2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56055fec24b0 con 0x56055ffc9a00 2024-04-21T08:40:37.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.209+0000 7f82783a2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56055ff211b0 con 0x56055ffc0e80 2024-04-21T08:40:37.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f826f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc04d0 0x56055ffc08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f826f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc04d0 0x56055ffc08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39626/0 (socket says 172.21.15.94:39626) 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f826f7fe640 1 -- 172.21.15.94:0/1340939567 learned_addr learned my addr 172.21.15.94:0/1340939567 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f826effd640 1 --2- 172.21.15.94:0/1340939567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc0e80 0x56055ffc9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f826f7fe640 1 -- 172.21.15.94:0/1340939567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc0e80 msgr2=0x56055ffc9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f826f7fe640 1 --2- 172.21.15.94:0/1340939567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc0e80 0x56055ffc9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f826f7fe640 1 -- 172.21.15.94:0/1340939567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc9a00 msgr2=0x56055ffcbe50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f826f7fe640 1 --2- 172.21.15.94:0/1340939567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc9a00 0x56055ffcbe50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f826f7fe640 1 -- 172.21.15.94:0/1340939567 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055feb41b0 con 0x56055ffc04d0 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f826f7fe640 1 --2- 172.21.15.94:0/1340939567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc04d0 0x56055ffc08b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8268009e40 tx=0x7f826800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c876144b5e87de70 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.210+0000 7f82557fa640 1 -- 172.21.15.94:0/1340939567 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268004d20 con 0x56055ffc04d0 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.211+0000 7f82557fa640 1 -- 172.21.15.94:0/1340939567 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82680054d0 con 0x56055ffc04d0 2024-04-21T08:40:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.211+0000 7f82557fa640 1 -- 172.21.15.94:0/1340939567 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82680117c0 con 0x56055ffc04d0 2024-04-21T08:40:37.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.211+0000 7f8257fff640 1 -- 172.21.15.94:0/2292029415 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f82640046c0 con 0x7f8270103380 2024-04-21T08:40:37.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.211+0000 7f8257fff640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f82580a3e40 0x7f82580a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.211+0000 7f82783a2740 1 -- 172.21.15.94:0/1340939567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc04d0 msgr2=0x56055ffc08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.211+0000 7f82783a2740 1 --2- 172.21.15.94:0/1340939567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc04d0 0x56055ffc08b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8268009e40 tx=0x7f826800f730 comp rx=0 tx=0).stop 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.212+0000 7f8257fff640 1 -- 172.21.15.94:0/2292029415 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f82640d1170 con 0x7f8270103380 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.212+0000 7f826effd640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f82580a3e40 0x7f82580a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.212+0000 7f82783a2740 1 -- 172.21.15.94:0/1340939567 shutdown_connections 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.212+0000 7f82783a2740 1 --2- 172.21.15.94:0/1340939567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc0e80 0x56055ffc9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.212+0000 7f82783a2740 1 --2- 172.21.15.94:0/1340939567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc9a00 0x56055ffcbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.212+0000 7f82783a2740 1 --2- 172.21.15.94:0/1340939567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc04d0 0x56055ffc08b0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.212+0000 7f82783a2740 1 -- 172.21.15.94:0/1340939567 >> 172.21.15.94:0/1340939567 conn(0x56055ffbaba0 msgr2=0x56055ffbaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.212+0000 7f82783a2740 1 -- 172.21.15.94:0/1340939567 shutdown_connections 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.212+0000 7f82783a2740 1 -- 172.21.15.94:0/1340939567 wait complete. 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.212+0000 7f826effd640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f82580a3e40 0x7f82580a6300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8260006010 tx=0x7f826002c6c0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.213+0000 7f82783a2740 1 -- start start 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.213+0000 7f82783a2740 1 Processor -- start 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.213+0000 7f82783a2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 0x56055ffc92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.213+0000 7f82783a2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc0e80 0x56055ffc6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.213+0000 7f82783a2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.213+0000 7f82783a2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56055ffcc970 con 0x56055ffc9a00 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.213+0000 7f82783a2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56055fede500 con 0x56055ffc04d0 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.213+0000 7f82783a2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56055ffccaf0 con 0x56055ffc0e80 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 0x56055ffc92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 0x56055ffc92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:49164/0 (socket says 172.21.15.94:49164) 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826effd640 1 -- 172.21.15.94:0/2267512311 learned_addr learned my addr 172.21.15.94:0/2267512311 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f82783a2740 10 client.0 _ll_register_callbacks cb 0x56055feacb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826effd640 1 -- 172.21.15.94:0/2267512311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc0e80 msgr2=0x56055ffc6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826effd640 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc0e80 0x56055ffc6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826effd640 1 -- 172.21.15.94:0/2267512311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 msgr2=0x56055ffc6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826effd640 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826effd640 1 -- 172.21.15.94:0/2267512311 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055fc373f0 con 0x56055ffc04d0 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826f7fe640 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:37.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f826effd640 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 0x56055ffc92d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8260005c40 tx=0x7f826002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce020e1a19d2c551 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f8252ff5640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f8252ff5640 1 -- 172.21.15.94:0/2267512311 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8260003b40 con 0x56055ffc04d0 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f82783a2740 1 -- 172.21.15.94:0/2267512311 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82680054d0 con 0x56055ffc04d0 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.214+0000 7f8252ff5640 1 -- 172.21.15.94:0/2267512311 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f826003a040 con 0x56055ffc04d0 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f8252ff5640 1 -- 172.21.15.94:0/2267512311 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8260003ce0 con 0x56055ffc04d0 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f82783a2740 10 client.25340 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f82783a2740 1 -- 172.21.15.94:0/2267512311 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56055fd017d0 con 0x56055ffc04d0 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f8252ff5640 1 -- 172.21.15.94:0/2267512311 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82680054d0 con 0x56055ffc04d0 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f8252ff5640 1 -- 172.21.15.94:0/2267512311 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f82600404c0 con 0x56055ffc04d0 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f82783a2740 10 client.25340 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f82783a2740 10 client.25340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f82783a2740 20 client.25340 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f82783a2740 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56055ffd9940 0x56055ffdbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f82783a2740 4 client.25340 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:37.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f82783a2740 1 -- 172.21.15.94:0/2267512311 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56055fec4930 con 0x56055ffd9940 2024-04-21T08:40:37.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.215+0000 7f826e7fc640 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56055ffd9940 0x56055ffdbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.216+0000 7f8252ff5640 10 client.25340 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:37.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.216+0000 7f826e7fc640 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56055ffd9940 0x56055ffdbdd0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c824a6789be2ff7 server_cookie=ae66b053f5ecb45d in_seq=0 out_seq=0 2024-04-21T08:40:37.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.216+0000 7f8252ff5640 1 -- 172.21.15.94:0/2267512311 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56055fec4930 con 0x56055ffd9940 2024-04-21T08:40:37.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.216+0000 7f8252ff5640 10 client.25340 handle_command_reply: tid=0 2024-04-21T08:40:37.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.216+0000 7f82783a2740 1 client.25340 shutdown 2024-04-21T08:40:37.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 1 -- 172.21.15.94:0/2267512311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 msgr2=0x56055ffc92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 0x56055ffc92d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8260005c40 tx=0x7f826002bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:37.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 1 -- 172.21.15.94:0/2267512311 shutdown_connections 2024-04-21T08:40:37.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56055ffd9940 0x56055ffdbdd0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc0e80 0x56055ffc6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 1 --2- 172.21.15.94:0/2267512311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 0x56055ffc92d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 1 -- 172.21.15.94:0/2267512311 >> 172.21.15.94:0/2267512311 conn(0x56055ffbaba0 msgr2=0x56055ffbc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f8252ff5640 0 client.25340 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:37.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 1 -- 172.21.15.94:0/2267512311 shutdown_connections 2024-04-21T08:40:37.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 1 -- 172.21.15.94:0/2267512311 wait complete. 2024-04-21T08:40:37.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.217+0000 7f82783a2740 20 client.25340 trim_cache size 0 max 16384 2024-04-21T08:40:37.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.242+0000 7f82783a2740 1 Processor -- start 2024-04-21T08:40:37.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.242+0000 7f82783a2740 1 -- start start 2024-04-21T08:40:37.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.242+0000 7f82783a2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 0x56055ffd5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.242+0000 7f82783a2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc0e80 0x56055ffc8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.242+0000 7f82783a2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc9a00 0x56055ffc92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.242+0000 7f82783a2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f82701b3f10 con 0x56055ffc0e80 2024-04-21T08:40:37.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.242+0000 7f82783a2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f82701b17e0 con 0x56055ffc04d0 2024-04-21T08:40:37.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.242+0000 7f82783a2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f82701af630 con 0x56055ffc9a00 2024-04-21T08:40:37.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc0e80 0x56055ffc8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc0e80 0x56055ffc8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39642/0 (socket says 172.21.15.94:39642) 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826e7fc640 1 -- 172.21.15.94:0/2292532038 learned_addr learned my addr 172.21.15.94:0/2292532038 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826effd640 1 --2- 172.21.15.94:0/2292532038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 0x56055ffd5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826f7fe640 1 --2- 172.21.15.94:0/2292532038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc9a00 0x56055ffc92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826e7fc640 1 -- 172.21.15.94:0/2292532038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc9a00 msgr2=0x56055ffc92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826e7fc640 1 --2- 172.21.15.94:0/2292532038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc9a00 0x56055ffc92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826e7fc640 1 -- 172.21.15.94:0/2292532038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 msgr2=0x56055ffd5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826e7fc640 1 --2- 172.21.15.94:0/2292532038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 0x56055ffd5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826e7fc640 1 -- 172.21.15.94:0/2292532038 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055ffc6f30 con 0x56055ffc0e80 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f826e7fc640 1 --2- 172.21.15.94:0/2292532038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc0e80 0x56055ffc8d90 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f826409c7e0 tx=0x7f8264009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a766b6a4f329f399 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f8254ff9640 1 -- 172.21.15.94:0/2292532038 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640b5070 con 0x56055ffc0e80 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f8254ff9640 1 -- 172.21.15.94:0/2292532038 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82640a0e90 con 0x56055ffc0e80 2024-04-21T08:40:37.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.243+0000 7f8254ff9640 1 -- 172.21.15.94:0/2292532038 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826409cd40 con 0x56055ffc0e80 2024-04-21T08:40:37.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.244+0000 7f82783a2740 1 -- 172.21.15.94:0/2292532038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc0e80 msgr2=0x56055ffc8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.244+0000 7f82783a2740 1 --2- 172.21.15.94:0/2292532038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc0e80 0x56055ffc8d90 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f826409c7e0 tx=0x7f8264009760 comp rx=0 tx=0).stop 2024-04-21T08:40:37.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.244+0000 7f82783a2740 1 -- 172.21.15.94:0/2292532038 shutdown_connections 2024-04-21T08:40:37.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.244+0000 7f82783a2740 1 --2- 172.21.15.94:0/2292532038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc9a00 0x56055ffc92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.244+0000 7f82783a2740 1 --2- 172.21.15.94:0/2292532038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc0e80 0x56055ffc8d90 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.244+0000 7f82783a2740 1 --2- 172.21.15.94:0/2292532038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc04d0 0x56055ffd5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.244+0000 7f82783a2740 1 -- 172.21.15.94:0/2292532038 >> 172.21.15.94:0/2292532038 conn(0x56055ffbaba0 msgr2=0x56055ffb9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.244+0000 7f82783a2740 1 -- 172.21.15.94:0/2292532038 shutdown_connections 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.244+0000 7f82783a2740 1 -- 172.21.15.94:0/2292532038 wait complete. 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f82783a2740 1 -- start start 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f82783a2740 1 Processor -- start 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f82783a2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc04d0 0x56055ffc3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f82783a2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f82783a2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc3e00 0x56056003ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f82783a2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f82701afca0 con 0x56055ffc9a00 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f82783a2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f82701b1110 con 0x56055ffc3e00 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f82783a2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f82701b0370 con 0x56055ffc04d0 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f82783a2740 10 client.0 _ll_register_callbacks cb 0x7f8260009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f826e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f826e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39654/0 (socket says 172.21.15.94:39654) 2024-04-21T08:40:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f826e7fc640 1 -- 172.21.15.94:0/4184608528 learned_addr learned my addr 172.21.15.94:0/4184608528 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f826effd640 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc04d0 0x56055ffc3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f826f7fe640 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc3e00 0x56056003ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f826e7fc640 1 -- 172.21.15.94:0/4184608528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc04d0 msgr2=0x56055ffc3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f826e7fc640 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc04d0 0x56055ffc3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f826e7fc640 1 -- 172.21.15.94:0/4184608528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc3e00 msgr2=0x56056003ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.245+0000 7f826e7fc640 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc3e00 0x56056003ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f826e7fc640 1 -- 172.21.15.94:0/4184608528 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56056003f470 con 0x56055ffc9a00 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f826e7fc640 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc38c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f82640a5220 tx=0x7f826400efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1126ba3bc67edc97 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f8252ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f8252ff5640 1 -- 172.21.15.94:0/4184608528 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640b5070 con 0x56055ffc9a00 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f8252ff5640 1 -- 172.21.15.94:0/4184608528 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f8264099150 con 0x56055ffc9a00 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f82783a2740 1 -- 172.21.15.94:0/4184608528 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82640a0e90 con 0x56055ffc9a00 2024-04-21T08:40:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f8252ff5640 1 -- 172.21.15.94:0/4184608528 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f826409cd40 con 0x56055ffc9a00 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f82783a2740 10 client.16020 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f82783a2740 1 -- 172.21.15.94:0/4184608528 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5605600406e0 con 0x56055ffc9a00 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f8252ff5640 1 -- 172.21.15.94:0/4184608528 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82640a0e90 con 0x56055ffc9a00 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.246+0000 7f8252ff5640 1 -- 172.21.15.94:0/4184608528 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f82640ce640 con 0x56055ffc9a00 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f82783a2740 10 client.16020 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f82783a2740 10 client.16020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f82783a2740 20 client.16020 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f82783a2740 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560560040ce0 0x560560043100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f82783a2740 4 client.16020 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f82783a2740 1 -- 172.21.15.94:0/4184608528 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56055ff32c10 con 0x560560040ce0 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f826effd640 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560560040ce0 0x560560043100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f826effd640 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560560040ce0 0x560560043100 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d776e97ea1dea791 server_cookie=e635e1259dcd039d in_seq=0 out_seq=0 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f8252ff5640 10 client.16020 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f8252ff5640 1 -- 172.21.15.94:0/4184608528 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56055ff32c10 con 0x560560040ce0 2024-04-21T08:40:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f8252ff5640 10 client.16020 handle_command_reply: tid=0 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.247+0000 7f82783a2740 1 client.16020 shutdown 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 1 -- 172.21.15.94:0/4184608528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 msgr2=0x56055ffc38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc38c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f82640a5220 tx=0x7f826400efb0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 1 -- 172.21.15.94:0/4184608528 shutdown_connections 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560560040ce0 0x560560043100 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56055ffc3e00 0x56056003ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56055ffc9a00 0x56055ffc38c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f8252ff5640 0 client.16020 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 1 --2- 172.21.15.94:0/4184608528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56055ffc04d0 0x56055ffc3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 1 -- 172.21.15.94:0/4184608528 >> 172.21.15.94:0/4184608528 conn(0x56055ffbaba0 msgr2=0x56055ffbbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 1 -- 172.21.15.94:0/4184608528 shutdown_connections 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 1 -- 172.21.15.94:0/4184608528 wait complete. 2024-04-21T08:40:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82783a2740 20 client.16020 trim_cache size 0 max 16384 2024-04-21T08:40:37.253 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:40:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82757b2640 1 -- 172.21.15.94:0/2292029415 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f82580a3e40 msgr2=0x7f82580a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82757b2640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f82580a3e40 0x7f82580a6300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8260006010 tx=0x7f826002c6c0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82757b2640 1 -- 172.21.15.94:0/2292029415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8270103380 msgr2=0x7f82701a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82757b2640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8270103380 0x7f82701a0b70 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f826400e9c0 tx=0x7f826400ee80 comp rx=0 tx=0).stop 2024-04-21T08:40:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82757b2640 1 -- 172.21.15.94:0/2292029415 shutdown_connections 2024-04-21T08:40:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82757b2640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f82580a3e40 0x7f82580a6300 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.248+0000 7f82757b2640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f827010ebe0 0x7f827019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.249+0000 7f82757b2640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8270103380 0x7f82701a0b70 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.249+0000 7f82757b2640 1 --2- 172.21.15.94:0/2292029415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8270102f10 0x7f82701a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.249+0000 7f82757b2640 1 -- 172.21.15.94:0/2292029415 >> 172.21.15.94:0/2292029415 conn(0x7f82700773f0 msgr2=0x7f8270110120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.249+0000 7f82757b2640 1 -- 172.21.15.94:0/2292029415 shutdown_connections 2024-04-21T08:40:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:37.249+0000 7f82757b2640 1 -- 172.21.15.94:0/2292029415 wait complete. 2024-04-21T08:40:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:37 smithi094 ceph-mon[27988]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-21T08:40:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:37 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3410546230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:37.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:37 smithi111 ceph-mon[31493]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-21T08:40:37.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:37 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3410546230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:37.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:37 smithi173 ceph-mon[31428]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-21T08:40:37.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:37 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3410546230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:40:38.264 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:40:38.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:40:38.265 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:40:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.413+0000 7f9bdfec1640 1 -- 172.21.15.94:0/111599002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bd8102b30 msgr2=0x7f9bd8106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.413+0000 7f9bdfec1640 1 --2- 172.21.15.94:0/111599002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bd8102b30 0x7f9bd8106b70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009900 tx=0x7f9bc802bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.414+0000 7f9bdfec1640 1 -- 172.21.15.94:0/111599002 shutdown_connections 2024-04-21T08:40:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.414+0000 7f9bdfec1640 1 --2- 172.21.15.94:0/111599002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bd8107250 0x7f9bd810ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.414+0000 7f9bdfec1640 1 --2- 172.21.15.94:0/111599002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bd8102b30 0x7f9bd8106b70 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.414+0000 7f9bdfec1640 1 --2- 172.21.15.94:0/111599002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bd8102180 0x7f9bd8102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.414+0000 7f9bdfec1640 1 -- 172.21.15.94:0/111599002 >> 172.21.15.94:0/111599002 conn(0x7f9bd80fd9c0 msgr2=0x7f9bd80ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.414+0000 7f9bdfec1640 1 -- 172.21.15.94:0/111599002 shutdown_connections 2024-04-21T08:40:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.414+0000 7f9bdfec1640 1 -- 172.21.15.94:0/111599002 wait complete. 2024-04-21T08:40:38.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.415+0000 7f9bdfec1640 1 Processor -- start 2024-04-21T08:40:38.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.415+0000 7f9bdfec1640 1 -- start start 2024-04-21T08:40:38.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.415+0000 7f9bdfec1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bd8102180 0x7f9bd81987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.415+0000 7f9bdfec1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bd8107250 0x7f9bd8198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.415+0000 7f9bdfec1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bd819d010 0x7f9bd819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bde45e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bd819d010 0x7f9bd819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bde45e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bd819d010 0x7f9bd819d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39676/0 (socket says 172.21.15.94:39676) 2024-04-21T08:40:38.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bddc5d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bd8102180 0x7f9bd81987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bde45e640 1 -- 172.21.15.94:0/2806333653 learned_addr learned my addr 172.21.15.94:0/2806333653 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd8112190 con 0x7f9bd819d010 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bd8107250 0x7f9bd8198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd8112010 con 0x7f9bd8107250 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd8112310 con 0x7f9bd8102180 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bddc5d640 1 -- 172.21.15.94:0/2806333653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bd8107250 msgr2=0x7f9bd8198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bd8107250 0x7f9bd8198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bddc5d640 1 -- 172.21.15.94:0/2806333653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bd819d010 msgr2=0x7f9bd819d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bd819d010 0x7f9bd819d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bddc5d640 1 -- 172.21.15.94:0/2806333653 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd819db00 con 0x7f9bd8102180 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.416+0000 7f9bde45e640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bd819d010 0x7f9bd819d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.417+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bd8102180 0x7f9bd81987a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400e9f0 tx=0x7f9bd400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.417+0000 7f9bceffd640 1 -- 172.21.15.94:0/2806333653 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd400ccc0 con 0x7f9bd8102180 2024-04-21T08:40:38.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.417+0000 7f9bceffd640 1 -- 172.21.15.94:0/2806333653 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bd4005020 con 0x7f9bd8102180 2024-04-21T08:40:38.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.417+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd819ddf0 con 0x7f9bd8102180 2024-04-21T08:40:38.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.417+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd81a55b0 con 0x7f9bd8102180 2024-04-21T08:40:38.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.417+0000 7f9bceffd640 1 -- 172.21.15.94:0/2806333653 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4004a00 con 0x7f9bd8102180 2024-04-21T08:40:38.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.420+0000 7f9bceffd640 1 -- 172.21.15.94:0/2806333653 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f9bd4004ba0 con 0x7f9bd8102180 2024-04-21T08:40:38.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.420+0000 7f9be2b91740 1 Processor -- start 2024-04-21T08:40:38.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9be2b91740 1 -- start start 2024-04-21T08:40:38.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81b0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 0x559bf81b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9be2b91740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x559bf810bf00 con 0x559bf81a8290 2024-04-21T08:40:38.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9be2b91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559bf80a9590 con 0x559bf81a78e0 2024-04-21T08:40:38.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9be2b91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559bf8107ec0 con 0x559bf81b0e10 2024-04-21T08:40:38.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9bde45e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9bde45e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81a7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:49190/0 (socket says 172.21.15.94:49190) 2024-04-21T08:40:38.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9bddc5d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9bde45e640 1 -- 172.21.15.94:0/3361742424 learned_addr learned my addr 172.21.15.94:0/3361742424 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:38.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.421+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/3361742424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 0x559bf81b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.422+0000 7f9bde45e640 1 -- 172.21.15.94:0/3361742424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 msgr2=0x559bf81b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.422+0000 7f9bde45e640 1 --2- 172.21.15.94:0/3361742424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 0x559bf81b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.422+0000 7f9bde45e640 1 -- 172.21.15.94:0/3361742424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 msgr2=0x559bf81b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.422+0000 7f9bceffd640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bc40a3ec0 0x7f9bc40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.422+0000 7f9bde45e640 1 --2- 172.21.15.94:0/3361742424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.422+0000 7f9bde45e640 1 -- 172.21.15.94:0/3361742424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bf7ee67e0 con 0x559bf81a78e0 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.422+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bc40a3ec0 0x7f9bc40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.422+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/3361742424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.422+0000 7f9bde45e640 1 --2- 172.21.15.94:0/3361742424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81a7cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0011780 tx=0x7f9bd0011c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=595c8b4d300a4f2a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.423+0000 7f9bc3fff640 1 -- 172.21.15.94:0/3361742424 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd000fb40 con 0x559bf81a78e0 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.423+0000 7f9bc3fff640 1 -- 172.21.15.94:0/3361742424 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bd0011e90 con 0x559bf81a78e0 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.423+0000 7f9bc3fff640 1 -- 172.21.15.94:0/3361742424 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd001a650 con 0x559bf81a78e0 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.423+0000 7f9bceffd640 1 -- 172.21.15.94:0/2806333653 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9bd40d0db0 con 0x7f9bd8102180 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 -- 172.21.15.94:0/3361742424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 msgr2=0x559bf81a7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bc40a3ec0 0x7f9bc40a6380 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009230 tx=0x7f9bc8005fd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 --2- 172.21.15.94:0/3361742424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81a7cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0011780 tx=0x7f9bd0011c40 comp rx=0 tx=0).stop 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 -- 172.21.15.94:0/3361742424 shutdown_connections 2024-04-21T08:40:38.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 --2- 172.21.15.94:0/3361742424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 0x559bf81b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 --2- 172.21.15.94:0/3361742424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 --2- 172.21.15.94:0/3361742424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81a7cc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 -- 172.21.15.94:0/3361742424 >> 172.21.15.94:0/3361742424 conn(0x559bf81a1fb0 msgr2=0x559bf81a2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 -- 172.21.15.94:0/3361742424 shutdown_connections 2024-04-21T08:40:38.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 -- 172.21.15.94:0/3361742424 wait complete. 2024-04-21T08:40:38.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 -- start start 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.424+0000 7f9be2b91740 1 Processor -- start 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81addf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81ae330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 0x559bf81ae870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9be2b91740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x559bf81b8190 con 0x559bf81a8290 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9be2b91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559bf80c5580 con 0x559bf81a78e0 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9be2b91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559bf81b8310 con 0x559bf81b0e10 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9bddc5d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9bddc5d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:49198/0 (socket says 172.21.15.94:49198) 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9bddc5d640 1 -- 172.21.15.94:0/2483384528 learned_addr learned my addr 172.21.15.94:0/2483384528 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:38.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bde45e640 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 0x559bf81ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.425+0000 7f9be2b91740 10 client.0 _ll_register_callbacks cb 0x559bf80cb650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bddc5d640 1 -- 172.21.15.94:0/2483384528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 msgr2=0x559bf81ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 0x559bf81ae870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bddc5d640 1 -- 172.21.15.94:0/2483384528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 msgr2=0x559bf81ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bddc5d640 1 -- 172.21.15.94:0/2483384528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bf7f46f70 con 0x559bf81a78e0 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bde45e640 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 0x559bf81ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9be2b91740 1 -- 172.21.15.94:0/2483384528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bd0011e90 con 0x559bf81a78e0 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81addf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9bd409c6b0 tx=0x7f9bd40096e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ca3e9f2081ad1c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bc17fa640 10 client.25358 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bc17fa640 1 -- 172.21.15.94:0/2483384528 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd401c070 con 0x559bf81a78e0 2024-04-21T08:40:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.426+0000 7f9bc17fa640 1 -- 172.21.15.94:0/2483384528 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9bd40054f0 con 0x559bf81a78e0 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.427+0000 7f9bc17fa640 1 -- 172.21.15.94:0/2483384528 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9bd409c260 con 0x559bf81a78e0 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.427+0000 7f9be2b91740 10 client.25358 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.427+0000 7f9be2b91740 1 -- 172.21.15.94:0/2483384528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({fsmap=24}) -- 0x559bf7dde170 con 0x559bf81a78e0 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.427+0000 7f9bc17fa640 1 -- 172.21.15.94:0/2483384528 <== mon.2 v2:172.21.15.111:3300/0 4 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9bd40b5070 con 0x559bf81a78e0 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.427+0000 7f9be2b91740 10 client.25358 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.427+0000 7f9be2b91740 10 client.25358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.427+0000 7f9be2b91740 20 client.25358 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.427+0000 7f9be2b91740 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559bf81c5560 0x559bf81c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.427+0000 7f9be2b91740 4 client.25358 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.427+0000 7f9be2b91740 1 -- 172.21.15.94:0/2483384528 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bf8119920 con 0x559bf81c5560 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.428+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559bf81c5560 0x559bf81c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.428+0000 7f9bc17fa640 10 client.25358 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.428+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559bf81c5560 0x559bf81c7990 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b527c83fb8fb6983 server_cookie=ab331f1ba5b4cd3f in_seq=0 out_seq=0 2024-04-21T08:40:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9bc17fa640 1 -- 172.21.15.94:0/2483384528 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559bf8119920 con 0x559bf81c5560 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9bc17fa640 10 client.25358 handle_command_reply: tid=0 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9be2b91740 1 client.25358 shutdown 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9be2b91740 1 -- 172.21.15.94:0/2483384528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 msgr2=0x559bf81addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9be2b91740 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81addf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9bd409c6b0 tx=0x7f9bd40096e0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9be2b91740 1 -- 172.21.15.94:0/2483384528 shutdown_connections 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9be2b91740 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559bf81c5560 0x559bf81c7990 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9be2b91740 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81b0e10 0x559bf81ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9bc17fa640 0 client.25358 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9be2b91740 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9be2b91740 1 --2- 172.21.15.94:0/2483384528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81a78e0 0x559bf81addf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.429+0000 7f9be2b91740 1 -- 172.21.15.94:0/2483384528 >> 172.21.15.94:0/2483384528 conn(0x559bf81a1fb0 msgr2=0x559bf81a3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.430+0000 7f9be2b91740 1 -- 172.21.15.94:0/2483384528 shutdown_connections 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.430+0000 7f9be2b91740 1 -- 172.21.15.94:0/2483384528 wait complete. 2024-04-21T08:40:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.430+0000 7f9be2b91740 20 client.25358 trim_cache size 0 max 16384 2024-04-21T08:40:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9be2b91740 1 Processor -- start 2024-04-21T08:40:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9be2b91740 1 -- start start 2024-04-21T08:40:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81a78e0 0x559bf81b0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81c1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 0x559bf81a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9be2b91740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd81afad0 con 0x559bf81a8290 2024-04-21T08:40:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9be2b91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd81ad3a0 con 0x559bf81b0e10 2024-04-21T08:40:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9be2b91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd81ab1f0 con 0x559bf81a78e0 2024-04-21T08:40:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9bdd45c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81c1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9bdd45c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81c1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39690/0 (socket says 172.21.15.94:39690) 2024-04-21T08:40:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.464+0000 7f9bdd45c640 1 -- 172.21.15.94:0/315217419 learned_addr learned my addr 172.21.15.94:0/315217419 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/315217419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81a78e0 0x559bf81b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bde45e640 1 --2- 172.21.15.94:0/315217419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 0x559bf81a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bdd45c640 1 -- 172.21.15.94:0/315217419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81a78e0 msgr2=0x559bf81b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/315217419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81a78e0 0x559bf81b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bdd45c640 1 -- 172.21.15.94:0/315217419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 msgr2=0x559bf81a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/315217419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 0x559bf81a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bdd45c640 1 -- 172.21.15.94:0/315217419 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bf81a9330 con 0x559bf81a8290 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bde45e640 1 --2- 172.21.15.94:0/315217419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 0x559bf81a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/315217419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81c1050 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9bd806b5f0 tx=0x7f9bc8045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfb69ae27c354ec server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bc37fe640 1 -- 172.21.15.94:0/315217419 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc802ca80 con 0x559bf81a8290 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bc37fe640 1 -- 172.21.15.94:0/315217419 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bc805b070 con 0x559bf81a8290 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.465+0000 7f9bc37fe640 1 -- 172.21.15.94:0/315217419 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8004bc0 con 0x559bf81a8290 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.466+0000 7f9be2b91740 1 -- 172.21.15.94:0/315217419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 msgr2=0x559bf81c1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.466+0000 7f9be2b91740 1 --2- 172.21.15.94:0/315217419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81c1050 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9bd806b5f0 tx=0x7f9bc8045000 comp rx=0 tx=0).stop 2024-04-21T08:40:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.466+0000 7f9be2b91740 1 -- 172.21.15.94:0/315217419 shutdown_connections 2024-04-21T08:40:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.466+0000 7f9be2b91740 1 --2- 172.21.15.94:0/315217419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 0x559bf81a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.466+0000 7f9be2b91740 1 --2- 172.21.15.94:0/315217419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a8290 0x559bf81c1050 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.466+0000 7f9be2b91740 1 --2- 172.21.15.94:0/315217419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81a78e0 0x559bf81b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.466+0000 7f9be2b91740 1 -- 172.21.15.94:0/315217419 >> 172.21.15.94:0/315217419 conn(0x559bf81a1fb0 msgr2=0x559bf81a3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.466+0000 7f9be2b91740 1 -- 172.21.15.94:0/315217419 shutdown_connections 2024-04-21T08:40:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.467+0000 7f9be2b91740 1 -- 172.21.15.94:0/315217419 wait complete. 2024-04-21T08:40:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.467+0000 7f9be2b91740 1 -- start start 2024-04-21T08:40:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.467+0000 7f9be2b91740 1 Processor -- start 2024-04-21T08:40:38.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a78e0 0x559bf81b95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81a8290 0x559bf81b9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9be2b91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 0x559bf822aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9bddc5d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a78e0 0x559bf81b95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9bddc5d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a78e0 0x559bf81b95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39694/0 (socket says 172.21.15.94:39694) 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9bddc5d640 1 -- 172.21.15.94:0/4152741916 learned_addr learned my addr 172.21.15.94:0/4152741916 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd81ab860 con 0x559bf81a78e0 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81a8290 0x559bf81b9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd81accd0 con 0x559bf81b0e10 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd81abf30 con 0x559bf81a8290 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9bde45e640 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 0x559bf822aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.468+0000 7f9be2b91740 10 client.0 _ll_register_callbacks cb 0x7f9bd400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9bddc5d640 1 -- 172.21.15.94:0/4152741916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81a8290 msgr2=0x559bf81b9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81a8290 0x559bf81b9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9bddc5d640 1 -- 172.21.15.94:0/4152741916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 msgr2=0x559bf822aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 0x559bf822aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9bddc5d640 1 -- 172.21.15.94:0/4152741916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bf822afe0 con 0x559bf81a78e0 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9bddc5d640 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a78e0 0x559bf81b95f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4098b80 tx=0x7f9bd40d32c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d63a06b7a0d4c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9bc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9bc17fa640 1 -- 172.21.15.94:0/4152741916 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd409c080 con 0x559bf81a78e0 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9bc17fa640 1 -- 172.21.15.94:0/4152741916 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9bd409aa20 con 0x559bf81a78e0 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bc805b070 con 0x559bf81a78e0 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9bc17fa640 1 -- 172.21.15.94:0/4152741916 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9bd409c770 con 0x559bf81a78e0 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9be2b91740 10 client.16038 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.469+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559bf822c1b0 con 0x559bf81a78e0 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.470+0000 7f9bc17fa640 1 -- 172.21.15.94:0/4152741916 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bc805b070 con 0x559bf81a78e0 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.470+0000 7f9bc17fa640 1 -- 172.21.15.94:0/4152741916 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9bd40b5040 con 0x559bf81a78e0 2024-04-21T08:40:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.470+0000 7f9be2b91740 10 client.16038 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.470+0000 7f9be2b91740 10 client.16038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.470+0000 7f9be2b91740 20 client.16038 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.470+0000 7f9be2b91740 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559bf81c5560 0x559bf822e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.470+0000 7f9be2b91740 4 client.16038 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.470+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559bf81a8c10 con 0x559bf81c5560 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.470+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559bf81c5560 0x559bf822e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.471+0000 7f9bc17fa640 10 client.16038 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.471+0000 7f9bdd45c640 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559bf81c5560 0x559bf822e900 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d478699c08dcc586 server_cookie=481f0d4f3cca9669 in_seq=0 out_seq=0 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9bc17fa640 1 -- 172.21.15.94:0/4152741916 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559bf81a8c10 con 0x559bf81c5560 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9bc17fa640 10 client.16038 handle_command_reply: tid=0 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9be2b91740 1 client.16038 shutdown 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a78e0 msgr2=0x559bf81b95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9be2b91740 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a78e0 0x559bf81b95f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4098b80 tx=0x7f9bd40d32c0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 shutdown_connections 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9be2b91740 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559bf81c5560 0x559bf822e900 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9be2b91740 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf81a8290 0x559bf81b9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9bc17fa640 0 client.16038 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9be2b91740 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559bf81b0e10 0x559bf822aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9be2b91740 1 --2- 172.21.15.94:0/4152741916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559bf81a78e0 0x559bf81b95f0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.472+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 >> 172.21.15.94:0/4152741916 conn(0x559bf81a1fb0 msgr2=0x559bf81ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.473+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 shutdown_connections 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.473+0000 7f9be2b91740 1 -- 172.21.15.94:0/4152741916 wait complete. 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.473+0000 7f9be2b91740 20 client.16038 trim_cache size 0 max 16384 2024-04-21T08:40:38.476 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:40:38.477 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:40:38.477 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "a86abf41-02e7-40b9-9b51-b66433e82e1f", 2024-04-21T08:40:38.477 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:40:38.477 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.473+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bc40a3ec0 msgr2=0x7f9bc40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.473+0000 7f9bdfec1640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bc40a3ec0 0x7f9bc40a6380 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009230 tx=0x7f9bc8005fd0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.473+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bd8102180 msgr2=0x7f9bd81987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.473+0000 7f9bdfec1640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bd8102180 0x7f9bd81987a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400e9f0 tx=0x7f9bd400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.473+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 shutdown_connections 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.473+0000 7f9bdfec1640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bc40a3ec0 0x7f9bc40a6380 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.474+0000 7f9bdfec1640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bd819d010 0x7f9bd819d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.474+0000 7f9bdfec1640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bd8107250 0x7f9bd8198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.474+0000 7f9bdfec1640 1 --2- 172.21.15.94:0/2806333653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bd8102180 0x7f9bd81987a0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400e9f0 tx=0x7f9bd400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.474+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 >> 172.21.15.94:0/2806333653 conn(0x7f9bd80fd9c0 msgr2=0x7f9bd80fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.474+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 shutdown_connections 2024-04-21T08:40:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.474+0000 7f9bdfec1640 1 -- 172.21.15.94:0/2806333653 wait complete. 2024-04-21T08:40:38.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:40:38.488 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:40:38.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.610+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/160788311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b48102180 msgr2=0x7f7b48102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.610+0000 7f7b4ccc1640 1 --2- 172.21.15.94:0/160788311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b48102180 0x7f7b48102560 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009900 tx=0x7f7b3802bc30 comp rx=0 tx=0).stop 2024-04-21T08:40:38.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.610+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/160788311 shutdown_connections 2024-04-21T08:40:38.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.610+0000 7f7b4ccc1640 1 --2- 172.21.15.94:0/160788311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b48102b30 0x7f7b48106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.610+0000 7f7b4ccc1640 1 --2- 172.21.15.94:0/160788311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b48107250 0x7f7b4810ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.610+0000 7f7b4ccc1640 1 --2- 172.21.15.94:0/160788311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b48102180 0x7f7b48102560 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.610+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/160788311 >> 172.21.15.94:0/160788311 conn(0x7f7b480fda20 msgr2=0x7f7b480ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.610+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/160788311 shutdown_connections 2024-04-21T08:40:38.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.610+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/160788311 wait complete. 2024-04-21T08:40:38.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b4ccc1640 1 Processor -- start 2024-04-21T08:40:38.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b4ccc1640 1 -- start start 2024-04-21T08:40:38.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b4ccc1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b48102180 0x7f7b4819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b4ccc1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b48107250 0x7f7b4819e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b45d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b48107250 0x7f7b4819e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b4ccc1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b48198670 0x7f7b48198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b45d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b48107250 0x7f7b4819e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39714/0 (socket says 172.21.15.94:39714) 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b45d9b640 1 -- 172.21.15.94:0/534965996 learned_addr learned my addr 172.21.15.94:0/534965996 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481121b0 con 0x7f7b48107250 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48112030 con 0x7f7b48102180 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48112330 con 0x7f7b48198670 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b4659c640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b48102180 0x7f7b4819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.611+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b48198670 0x7f7b48198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b45d9b640 1 -- 172.21.15.94:0/534965996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b48198670 msgr2=0x7f7b48198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b45d9b640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b48198670 0x7f7b48198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b45d9b640 1 -- 172.21.15.94:0/534965996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b48102180 msgr2=0x7f7b4819e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b45d9b640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b48102180 0x7f7b4819e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b45d9b640 1 -- 172.21.15.94:0/534965996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b48199190 con 0x7f7b48107250 2024-04-21T08:40:38.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b4659c640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b48102180 0x7f7b4819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b45d9b640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b48107250 0x7f7b4819e9b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00d740 tx=0x7f7b3c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b377fe640 1 -- 172.21.15.94:0/534965996 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c017070 con 0x7f7b48107250 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b48199480 con 0x7f7b48107250 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b377fe640 1 -- 172.21.15.94:0/534965996 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b3c00de50 con 0x7f7b48107250 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b377fe640 1 -- 172.21.15.94:0/534965996 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c004ce0 con 0x7f7b48107250 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.612+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b48074180 con 0x7f7b48107250 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.615+0000 7f7b4f98e740 1 Processor -- start 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.615+0000 7f7b4f98e740 1 -- start start 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.615+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff295f8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.615+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29609d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.615+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff2960c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.615+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ff2955d1e0 con 0x55ff2960a290 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.615+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ff294fa390 con 0x55ff295f8390 2024-04-21T08:40:38.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.615+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ff295591a0 con 0x55ff295f8d40 2024-04-21T08:40:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b377fe640 1 -- 172.21.15.94:0/534965996 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f7b3c004e80 con 0x7f7b48107250 2024-04-21T08:40:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b4659c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29609d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b45d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff2960c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b4659c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29609d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:51400/0 (socket says 172.21.15.94:51400) 2024-04-21T08:40:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b4659c640 1 -- 172.21.15.94:0/3853521755 learned_addr learned my addr 172.21.15.94:0/3853521755 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:38.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/3853521755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff295f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b46d9d640 1 -- 172.21.15.94:0/3853521755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 msgr2=0x55ff29609d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/3853521755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29609d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b46d9d640 1 -- 172.21.15.94:0/3853521755 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 msgr2=0x55ff2960c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/3853521755 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff2960c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b46d9d640 1 -- 172.21.15.94:0/3853521755 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff293397d0 con 0x55ff295f8390 2024-04-21T08:40:38.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b4659c640 1 --2- 172.21.15.94:0/3853521755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29609d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.616+0000 7f7b45d9b640 1 --2- 172.21.15.94:0/3853521755 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff2960c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.617+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/3853521755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff295f8770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009e80 tx=0x7f7b400107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c95e0b02d4d4d543 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.617+0000 7f7b377fe640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b300a3ec0 0x7f7b300a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.617+0000 7f7b34ff9640 1 -- 172.21.15.94:0/3853521755 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40003b70 con 0x55ff295f8390 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.617+0000 7f7b4659c640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b300a3ec0 0x7f7b300a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.617+0000 7f7b377fe640 1 -- 172.21.15.94:0/534965996 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f7b3c0d07b0 con 0x7f7b48107250 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.617+0000 7f7b34ff9640 1 -- 172.21.15.94:0/3853521755 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b40015070 con 0x55ff295f8390 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.617+0000 7f7b34ff9640 1 -- 172.21.15.94:0/3853521755 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40011800 con 0x55ff295f8390 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 -- 172.21.15.94:0/3853521755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 msgr2=0x55ff295f8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/3853521755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff295f8770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009e80 tx=0x7f7b400107a0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4659c640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b300a3ec0 0x7f7b300a6380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x55ff295f3f40 tx=0x7f7b38005e40 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 -- 172.21.15.94:0/3853521755 shutdown_connections 2024-04-21T08:40:38.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/3853521755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29609d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/3853521755 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff2960c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/3853521755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff295f8770 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 -- 172.21.15.94:0/3853521755 >> 172.21.15.94:0/3853521755 conn(0x55ff295f2cb0 msgr2=0x55ff295f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 -- 172.21.15.94:0/3853521755 shutdown_connections 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 -- 172.21.15.94:0/3853521755 wait complete. 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 -- start start 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.618+0000 7f7b4f98e740 1 Processor -- start 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff29607310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29607850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29607d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ff2960d050 con 0x55ff2960a290 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ff295162f0 con 0x55ff295f8390 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ff2960d1d0 con 0x55ff295f8d40 2024-04-21T08:40:38.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b4f98e740 10 client.0 _ll_register_callbacks cb 0x55ff29523580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b46d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29607d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b46d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29607d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39736/0 (socket says 172.21.15.94:39736) 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b46d9d640 1 -- 172.21.15.94:0/1403401009 learned_addr learned my addr 172.21.15.94:0/1403401009 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b46d9d640 1 -- 172.21.15.94:0/1403401009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 msgr2=0x55ff29607850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b45d9b640 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29607850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29607850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b46d9d640 1 -- 172.21.15.94:0/1403401009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 msgr2=0x55ff29607310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff29607310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b46d9d640 1 -- 172.21.15.94:0/1403401009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff2926f3f0 con 0x55ff2960a290 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b45d9b640 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29607850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29607d90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7b40010ca0 tx=0x7f7b40011770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27215bdc61a6cd07 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.619+0000 7f7b2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1403401009 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4002a060 con 0x55ff2960a290 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1403401009 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f7b40011b70 con 0x55ff2960a290 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1403401009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b40015070 con 0x55ff2960a290 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1403401009 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7b40031050 con 0x55ff2960a290 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b4f98e740 10 client.16050 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1403401009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ff2922f0f0 con 0x55ff2960a290 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1403401009 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b40015070 con 0x55ff2960a290 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1403401009 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7b400243b0 con 0x55ff2960a290 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b4f98e740 10 client.16050 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b4f98e740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b4f98e740 20 client.16050 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ff2961a220 0x55ff2961c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b4f98e740 4 client.16050 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:38.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.620+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1403401009 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff2956ac00 con 0x55ff2961a220 2024-04-21T08:40:38.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.621+0000 7f7b4659c640 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ff2961a220 0x55ff2961c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.621+0000 7f7b2dffb640 10 client.16050 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:38.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.621+0000 7f7b4659c640 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ff2961a220 0x55ff2961c610 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=becacd27ed9d9675 server_cookie=daaa7d6873782a7e in_seq=0 out_seq=0 2024-04-21T08:40:38.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1403401009 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff2956ac00 con 0x55ff2961a220 2024-04-21T08:40:38.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b2dffb640 10 client.16050 handle_command_reply: tid=0 2024-04-21T08:40:38.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 client.16050 shutdown 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1403401009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 msgr2=0x55ff29607d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29607d90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7b40010ca0 tx=0x7f7b40011770 comp rx=0 tx=0).stop 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1403401009 shutdown_connections 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ff2961a220 0x55ff2961c610 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29607850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29607d90 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1403401009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff29607310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b2dffb640 0 client.16050 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1403401009 >> 172.21.15.94:0/1403401009 conn(0x55ff295f2cb0 msgr2=0x55ff295f4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1403401009 shutdown_connections 2024-04-21T08:40:38.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1403401009 wait complete. 2024-04-21T08:40:38.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.622+0000 7f7b4f98e740 20 client.16050 trim_cache size 0 max 16384 2024-04-21T08:40:38.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.639+0000 7f7b4f98e740 1 Processor -- start 2024-04-21T08:40:38.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.639+0000 7f7b4f98e740 1 -- start start 2024-04-21T08:40:38.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff29615f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29606bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29621be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481afbb0 con 0x55ff2960a290 2024-04-21T08:40:38.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481ad480 con 0x55ff295f8390 2024-04-21T08:40:38.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481ab2d0 con 0x55ff295f8d40 2024-04-21T08:40:38.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b46d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29621be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b46d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29621be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39740/0 (socket says 172.21.15.94:39740) 2024-04-21T08:40:38.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b46d9d640 1 -- 172.21.15.94:0/3857424198 learned_addr learned my addr 172.21.15.94:0/3857424198 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:38.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b4659c640 1 --2- 172.21.15.94:0/3857424198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff29615f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b45d9b640 1 --2- 172.21.15.94:0/3857424198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29606bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b46d9d640 1 -- 172.21.15.94:0/3857424198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 msgr2=0x55ff29606bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/3857424198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29606bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b46d9d640 1 -- 172.21.15.94:0/3857424198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 msgr2=0x55ff29615f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/3857424198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff29615f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b46d9d640 1 -- 172.21.15.94:0/3857424198 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff2960c680 con 0x55ff2960a290 2024-04-21T08:40:38.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.640+0000 7f7b46d9d640 1 --2- 172.21.15.94:0/3857424198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29621be0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7b40024920 tx=0x7f7b40009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56095b2fa85f4aba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.641+0000 7f7b2ffff640 1 -- 172.21.15.94:0/3857424198 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4002a070 con 0x55ff2960a290 2024-04-21T08:40:38.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.641+0000 7f7b2ffff640 1 -- 172.21.15.94:0/3857424198 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b40015070 con 0x55ff2960a290 2024-04-21T08:40:38.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.641+0000 7f7b2ffff640 1 -- 172.21.15.94:0/3857424198 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40011b70 con 0x55ff2960a290 2024-04-21T08:40:38.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.641+0000 7f7b4f98e740 1 -- 172.21.15.94:0/3857424198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 msgr2=0x55ff29621be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.641+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/3857424198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29621be0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7b40024920 tx=0x7f7b40009bf0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 -- 172.21.15.94:0/3857424198 shutdown_connections 2024-04-21T08:40:38.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/3857424198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8d40 0x55ff29606bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/3857424198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff2960a290 0x55ff29621be0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/3857424198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff295f8390 0x55ff29615f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 -- 172.21.15.94:0/3857424198 >> 172.21.15.94:0/3857424198 conn(0x55ff295f2cb0 msgr2=0x55ff295f48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 -- 172.21.15.94:0/3857424198 shutdown_connections 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 -- 172.21.15.94:0/3857424198 wait complete. 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 -- start start 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 Processor -- start 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8390 0x55ff29612830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff295f8d40 0x55ff29603f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff2960a290 0x55ff296044c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.642+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481ab940 con 0x55ff295f8d40 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481acdb0 con 0x55ff2960a290 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4f98e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481ac010 con 0x55ff295f8390 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b45d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff295f8d40 0x55ff29603f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4f98e740 10 client.0 _ll_register_callbacks cb 0x7f7b3c0d63e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:38.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b45d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff295f8d40 0x55ff29603f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39748/0 (socket says 172.21.15.94:39748) 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b45d9b640 1 -- 172.21.15.94:0/1992575063 learned_addr learned my addr 172.21.15.94:0/1992575063 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4659c640 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8390 0x55ff29612830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4659c640 1 -- 172.21.15.94:0/1992575063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff2960a290 msgr2=0x55ff296044c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4659c640 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff2960a290 0x55ff296044c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4659c640 1 -- 172.21.15.94:0/1992575063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff295f8d40 msgr2=0x55ff29603f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4659c640 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff295f8d40 0x55ff29603f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4659c640 1 -- 172.21.15.94:0/1992575063 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff29604cd0 con 0x55ff295f8390 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b45d9b640 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff295f8d40 0x55ff29603f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4659c640 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8390 0x55ff29612830 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b38006010 tx=0x7f7b38009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=892f42f5d6ef53b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1992575063 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38046070 con 0x55ff295f8390 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1992575063 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b40015070 con 0x55ff295f8390 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.643+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1992575063 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f7b38053460 con 0x55ff295f8390 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1992575063 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7b38053660 con 0x55ff295f8390 2024-04-21T08:40:38.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b4f98e740 10 client.25471 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1992575063 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ff29680480 con 0x55ff295f8390 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1992575063 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b40015070 con 0x55ff295f8390 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1992575063 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7b3805a070 con 0x55ff295f8390 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b4f98e740 10 client.25471 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b4f98e740 10 client.25471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b4f98e740 20 client.25471 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ff29680c30 0x55ff29683050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b4f98e740 4 client.25471 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1992575063 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ff29564a70 con 0x55ff29680c30 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.644+0000 7f7b45d9b640 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ff29680c30 0x55ff29683050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b2dffb640 10 client.25471 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b45d9b640 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ff29680c30 0x55ff29683050 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f06213a52c6ae567 server_cookie=54a08ebe9f75da6 in_seq=0 out_seq=0 2024-04-21T08:40:38.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b2dffb640 1 -- 172.21.15.94:0/1992575063 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ff29564a70 con 0x55ff29680c30 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b2dffb640 10 client.25471 handle_command_reply: tid=0 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 client.25471 shutdown 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1992575063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8390 msgr2=0x55ff29612830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8390 0x55ff29612830 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b38006010 tx=0x7f7b38009260 comp rx=0 tx=0).stop 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1992575063 shutdown_connections 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ff29680c30 0x55ff29683050 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b2dffb640 0 client.25471 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ff2960a290 0x55ff296044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ff295f8d40 0x55ff29603f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 --2- 172.21.15.94:0/1992575063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ff295f8390 0x55ff29612830 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1992575063 >> 172.21.15.94:0/1992575063 conn(0x55ff295f2cb0 msgr2=0x55ff295f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1992575063 shutdown_connections 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 1 -- 172.21.15.94:0/1992575063 wait complete. 2024-04-21T08:40:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.645+0000 7f7b4f98e740 20 client.25471 trim_cache size 0 max 16384 2024-04-21T08:40:38.652 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:40:38.652 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:40:38.652 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:40:38.652 INFO:teuthology.orchestra.run.smithi094.stdout: "a86abf41-02e7-40b9-9b51-b66433e82e1f": { 2024-04-21T08:40:38.652 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:40:38.652 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "a86abf41-02e7-40b9-9b51-b66433e82e1f", 2024-04-21T08:40:38.652 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:40:38.652 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:40:38.652 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:40:38.652 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:40:38.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b300a3ec0 msgr2=0x7f7b300a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b300a3ec0 0x7f7b300a6380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x55ff295f3f40 tx=0x7f7b38005e40 comp rx=0 tx=0).stop 2024-04-21T08:40:38.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b48107250 msgr2=0x7f7b4819e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:38.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b48107250 0x7f7b4819e9b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00d740 tx=0x7f7b3c00dc00 comp rx=0 tx=0).stop 2024-04-21T08:40:38.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 shutdown_connections 2024-04-21T08:40:38.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b300a3ec0 0x7f7b300a6380 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b48198670 0x7f7b48198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b48107250 0x7f7b4819e9b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 --2- 172.21.15.94:0/534965996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b48102180 0x7f7b4819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:38.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 >> 172.21.15.94:0/534965996 conn(0x7f7b480fda20 msgr2=0x7f7b480fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:38.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 shutdown_connections 2024-04-21T08:40:38.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:40:38.646+0000 7f7b4ccc1640 1 -- 172.21.15.94:0/534965996 wait complete. 2024-04-21T08:40:38.657 INFO:tasks.cephfs.filesystem:scrub status for tag:a86abf41-02e7-40b9-9b51-b66433e82e1f - {'path': '/', 'tag': 'a86abf41-02e7-40b9-9b51-b66433e82e1f', 'options': 'recursive,force'} 2024-04-21T08:40:39.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:39 smithi111 ceph-mon[31493]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T08:40:39.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:39 smithi094 ceph-mon[27988]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T08:40:39.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:39 smithi173 ceph-mon[31428]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T08:40:40.589 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:40 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:40:40.589 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:40 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:40.589 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:40 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:40:40.589 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:40 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:40.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:40 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:40:40.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:40 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:40.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:40 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:40:40.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:40 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:40.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:40 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:40:40.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:40 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:40.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:40 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:40:40.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:40 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:40:41.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:41 smithi111 ceph-mon[31493]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:40:41.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:41 smithi094 ceph-mon[27988]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:40:41.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:41 smithi173 ceph-mon[31428]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:40:42.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:42 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:40:42.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:42 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:40:42.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:42 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:40:43.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:43 smithi111 ceph-mon[31493]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:40:43.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:43 smithi094 ceph-mon[27988]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:40:43.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:43 smithi173 ceph-mon[31428]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:40:45.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:45 smithi173 ceph-mon[31428]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:40:45.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:45 smithi111 ceph-mon[31493]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:40:45.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:45 smithi094 ceph-mon[27988]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:40:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.907+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/1138246781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc28106170 msgr2=0x7fdc2810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.907+0000 7fdc2cdf4640 1 --2- 172.21.15.94:0/1138246781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc28106170 0x7fdc2810dd40 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009ca0 tx=0x7fdc2002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.907+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/1138246781 shutdown_connections 2024-04-21T08:40:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.907+0000 7fdc2cdf4640 1 --2- 172.21.15.94:0/1138246781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc28106170 0x7fdc2810dd40 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.907+0000 7fdc2cdf4640 1 --2- 172.21.15.94:0/1138246781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc28101aa0 0x7fdc28105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.907+0000 7fdc2cdf4640 1 --2- 172.21.15.94:0/1138246781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc281010f0 0x7fdc281014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.907+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/1138246781 >> 172.21.15.94:0/1138246781 conn(0x7fdc28077fd0 msgr2=0x7fdc280783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.907+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/1138246781 shutdown_connections 2024-04-21T08:40:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.907+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/1138246781 wait complete. 2024-04-21T08:40:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc2cdf4640 1 Processor -- start 2024-04-21T08:40:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc2cdf4640 1 -- start start 2024-04-21T08:40:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc2cdf4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc281010f0 0x7fdc281994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc2cdf4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc28106170 0x7fdc28199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc2cdf4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc28199fb0 0x7fdc281a4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc2cdf4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281110e0 con 0x7fdc28106170 2024-04-21T08:40:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc2cdf4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdc28110f60 con 0x7fdc281010f0 2024-04-21T08:40:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc2cdf4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdc28111260 con 0x7fdc28199fb0 2024-04-21T08:40:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc1ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc28106170 0x7fdc28199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc1ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc28106170 0x7fdc28199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40932/0 (socket says 172.21.15.94:40932) 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.908+0000 7fdc1ffff640 1 -- 172.21.15.94:0/346458897 learned_addr learned my addr 172.21.15.94:0/346458897 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.909+0000 7fdc2659c640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc281010f0 0x7fdc281994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.909+0000 7fdc26d9d640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc28199fb0 0x7fdc281a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.909+0000 7fdc1ffff640 1 -- 172.21.15.94:0/346458897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc28199fb0 msgr2=0x7fdc281a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.909+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc28199fb0 0x7fdc281a4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.909+0000 7fdc1ffff640 1 -- 172.21.15.94:0/346458897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc281010f0 msgr2=0x7fdc281994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.909+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc281010f0 0x7fdc281994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.909+0000 7fdc1ffff640 1 -- 172.21.15.94:0/346458897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc281a4d30 con 0x7fdc28106170 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.909+0000 7fdc26d9d640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc28199fb0 0x7fdc281a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.909+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc28106170 0x7fdc28199a20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000e9c0 tx=0x7fdc1000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.909+0000 7fdc1f7fe640 1 -- 172.21.15.94:0/346458897 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1000cd80 con 0x7fdc28106170 2024-04-21T08:40:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.910+0000 7fdc1f7fe640 1 -- 172.21.15.94:0/346458897 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc100054d0 con 0x7fdc28106170 2024-04-21T08:40:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.910+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/346458897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc281a4fc0 con 0x7fdc28106170 2024-04-21T08:40:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.910+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/346458897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc281a5530 con 0x7fdc28106170 2024-04-21T08:40:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.910+0000 7fdc1f7fe640 1 -- 172.21.15.94:0/346458897 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10010430 con 0x7fdc28106170 2024-04-21T08:40:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.912+0000 7fdc2f9e7740 1 Processor -- start 2024-04-21T08:40:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.912+0000 7fdc2f9e7740 1 -- start start 2024-04-21T08:40:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480998d0 0x55bf48098620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf48098b60 0x55bf48098fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf480a64f0 0x55bf480a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2f9e7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55bf47ffd8d0 con 0x55bf48098b60 2024-04-21T08:40:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2f9e7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bf47f9ae80 con 0x55bf480998d0 2024-04-21T08:40:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2f9e7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55bf47ff9890 con 0x55bf480a64f0 2024-04-21T08:40:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf48098b60 0x55bf48098fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf48098b60 0x55bf48098fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40936/0 (socket says 172.21.15.94:40936) 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2659c640 1 -- 172.21.15.94:0/2070313328 learned_addr learned my addr 172.21.15.94:0/2070313328 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc1f7fe640 1 -- 172.21.15.94:0/346458897 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7fdc10004fb0 con 0x7fdc28106170 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc26d9d640 1 --2- 172.21.15.94:0/2070313328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480998d0 0x55bf48098620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2659c640 1 -- 172.21.15.94:0/2070313328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf480a64f0 msgr2=0x55bf480a88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2659c640 1 --2- 172.21.15.94:0/2070313328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf480a64f0 0x55bf480a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2659c640 1 -- 172.21.15.94:0/2070313328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480998d0 msgr2=0x55bf48098620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2659c640 1 --2- 172.21.15.94:0/2070313328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480998d0 0x55bf48098620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc2659c640 1 -- 172.21.15.94:0/2070313328 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf47fbc320 con 0x55bf48098b60 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.913+0000 7fdc26d9d640 1 --2- 172.21.15.94:0/2070313328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480998d0 0x55bf48098620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.914+0000 7fdc2659c640 1 --2- 172.21.15.94:0/2070313328 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf48098b60 0x55bf48098fe0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009d20 tx=0x7fdc18011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb529b514b6501e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.914+0000 7fdc1f7fe640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdc140a3f60 0x7fdc140a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.914+0000 7fdc1cff9640 1 -- 172.21.15.94:0/2070313328 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1801e070 con 0x55bf48098b60 2024-04-21T08:40:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.914+0000 7fdc1f7fe640 1 -- 172.21.15.94:0/346458897 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fdc100d12b0 con 0x7fdc28106170 2024-04-21T08:40:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.914+0000 7fdc26d9d640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdc140a3f60 0x7fdc140a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.914+0000 7fdc1cff9640 1 -- 172.21.15.94:0/2070313328 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc18019ae0 con 0x55bf48098b60 2024-04-21T08:40:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.914+0000 7fdc1cff9640 1 -- 172.21.15.94:0/2070313328 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800ea80 con 0x55bf48098b60 2024-04-21T08:40:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.915+0000 7fdc26d9d640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdc140a3f60 0x7fdc140a6420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009a60 tx=0x7fdc20004a60 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.915+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/2070313328 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf48098b60 msgr2=0x55bf48098fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.915+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/2070313328 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf48098b60 0x55bf48098fe0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009d20 tx=0x7fdc18011810 comp rx=0 tx=0).stop 2024-04-21T08:40:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.915+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/2070313328 shutdown_connections 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.915+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/2070313328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf480a64f0 0x55bf480a88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.915+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/2070313328 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf48098b60 0x55bf48098fe0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.915+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/2070313328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480998d0 0x55bf48098620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.915+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/2070313328 >> 172.21.15.94:0/2070313328 conn(0x55bf4805fc90 msgr2=0x55bf48093850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.915+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/2070313328 shutdown_connections 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.915+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/2070313328 wait complete. 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc2f9e7740 1 -- start start 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc2f9e7740 1 Processor -- start 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf48098b60 0x55bf480a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480a31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf480a64f0 0x55bf480a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc2f9e7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55bf480a9270 con 0x55bf480998d0 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc2f9e7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bf47f85eb0 con 0x55bf48098b60 2024-04-21T08:40:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc2f9e7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55bf480a93f0 con 0x55bf480a64f0 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc2f9e7740 10 client.0 _ll_register_callbacks cb 0x55bf47fb2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc1ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480a31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc1ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480a31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40944/0 (socket says 172.21.15.94:40944) 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc1ffff640 1 -- 172.21.15.94:0/3729966208 learned_addr learned my addr 172.21.15.94:0/3729966208 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.916+0000 7fdc2659c640 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf48098b60 0x55bf480a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc26d9d640 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf480a64f0 0x55bf480a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc2659c640 1 -- 172.21.15.94:0/3729966208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf480a64f0 msgr2=0x55bf480a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc2659c640 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf480a64f0 0x55bf480a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc2659c640 1 -- 172.21.15.94:0/3729966208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 msgr2=0x55bf480a31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc2659c640 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480a31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc2659c640 1 -- 172.21.15.94:0/3729966208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf47d10210 con 0x55bf48098b60 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc26d9d640 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf480a64f0 0x55bf480a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480a31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc2659c640 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf48098b60 0x55bf480a5bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc18005890 tx=0x7fdc18004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=807029a685be8ceb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc0dffb640 1 -- 172.21.15.94:0/3729966208 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1801e070 con 0x55bf48098b60 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/3729966208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc18019ae0 con 0x55bf48098b60 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc0dffb640 1 -- 172.21.15.94:0/3729966208 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fdc18019430 con 0x55bf48098b60 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc0dffb640 1 -- 172.21.15.94:0/3729966208 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdc18030050 con 0x55bf48098b60 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc2f9e7740 10 client.25382 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/3729966208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bf47ddc620 con 0x55bf48098b60 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.917+0000 7fdc0dffb640 1 -- 172.21.15.94:0/3729966208 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc18019ae0 con 0x55bf48098b60 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.918+0000 7fdc0dffb640 1 -- 172.21.15.94:0/3729966208 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdc1800e800 con 0x55bf48098b60 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.918+0000 7fdc2f9e7740 10 client.25382 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.918+0000 7fdc2f9e7740 10 client.25382 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.918+0000 7fdc2f9e7740 20 client.25382 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.918+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf480ba7f0 0x55bf480bcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.918+0000 7fdc2f9e7740 4 client.25382 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:40:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.918+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/3729966208 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf48011880 con 0x55bf480ba7f0 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.918+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf480ba7f0 0x55bf480bcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.918+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf480ba7f0 0x55bf480bcbe0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c310c822f9471bb2 server_cookie=83f69e7dd8665bae in_seq=0 out_seq=0 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.918+0000 7fdc0dffb640 10 client.25382 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc0dffb640 1 -- 172.21.15.94:0/3729966208 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf48011880 con 0x55bf480ba7f0 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc0dffb640 10 client.25382 handle_command_reply: tid=0 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 client.25382 shutdown 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/3729966208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf48098b60 msgr2=0x55bf480a5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf48098b60 0x55bf480a5bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc18005890 tx=0x7fdc18004270 comp rx=0 tx=0).stop 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/3729966208 shutdown_connections 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf480ba7f0 0x55bf480bcbe0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf480a64f0 0x55bf480a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480a31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/3729966208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf48098b60 0x55bf480a5bb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/3729966208 >> 172.21.15.94:0/3729966208 conn(0x55bf4805fc90 msgr2=0x55bf480955a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc0dffb640 0 client.25382 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/3729966208 shutdown_connections 2024-04-21T08:40:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/3729966208 wait complete. 2024-04-21T08:40:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.919+0000 7fdc2f9e7740 20 client.25382 trim_cache size 0 max 16384 2024-04-21T08:40:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc2f9e7740 1 Processor -- start 2024-04-21T08:40:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc2f9e7740 1 -- start start 2024-04-21T08:40:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 0x55bf480a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480c22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480a64f0 0x55bf480c27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc2f9e7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281afaf0 con 0x55bf480998d0 2024-04-21T08:40:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc2f9e7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281ad3c0 con 0x55bf480a64f0 2024-04-21T08:40:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc2f9e7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281ab210 con 0x55bf48098b60 2024-04-21T08:40:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc1ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480c22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc1ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480c22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40960/0 (socket says 172.21.15.94:40960) 2024-04-21T08:40:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc1ffff640 1 -- 172.21.15.94:0/2802815042 learned_addr learned my addr 172.21.15.94:0/2802815042 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc2659c640 1 --2- 172.21.15.94:0/2802815042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 0x55bf480a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc1ffff640 1 -- 172.21.15.94:0/2802815042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 msgr2=0x55bf480a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/2802815042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 0x55bf480a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc1ffff640 1 -- 172.21.15.94:0/2802815042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480a64f0 msgr2=0x55bf480c27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/2802815042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480a64f0 0x55bf480c27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.946+0000 7fdc1ffff640 1 -- 172.21.15.94:0/2802815042 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf48097e30 con 0x55bf480998d0 2024-04-21T08:40:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.947+0000 7fdc2659c640 1 --2- 172.21.15.94:0/2802815042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 0x55bf480a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.947+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/2802815042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480c22a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdc10004960 tx=0x7fdc100049b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab348d366176720f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.947+0000 7fdc0ffff640 1 -- 172.21.15.94:0/2802815042 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc100046c0 con 0x55bf480998d0 2024-04-21T08:40:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.947+0000 7fdc0ffff640 1 -- 172.21.15.94:0/2802815042 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc1009d2c0 con 0x55bf480998d0 2024-04-21T08:40:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.947+0000 7fdc0ffff640 1 -- 172.21.15.94:0/2802815042 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc100bd0b0 con 0x55bf480998d0 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.948+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/2802815042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 msgr2=0x55bf480c22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.948+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/2802815042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480c22a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdc10004960 tx=0x7fdc100049b0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.948+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/2802815042 shutdown_connections 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.948+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/2802815042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480a64f0 0x55bf480c27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.948+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/2802815042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480c22a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.948+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/2802815042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 0x55bf480a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.948+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/2802815042 >> 172.21.15.94:0/2802815042 conn(0x55bf4805fc90 msgr2=0x55bf4809e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.948+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/2802815042 shutdown_connections 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.948+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/2802815042 wait complete. 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.948+0000 7fdc2f9e7740 1 -- start start 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc2f9e7740 1 Processor -- start 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 0x55bf480b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.951 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480b7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc2f9e7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480a64f0 0x55bf480b7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc2f9e7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281ab880 con 0x55bf480998d0 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc1ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480b7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc1ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480b7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40974/0 (socket says 172.21.15.94:40974) 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc1ffff640 1 -- 172.21.15.94:0/660227460 learned_addr learned my addr 172.21.15.94:0/660227460 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/660227460 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281accf0 con 0x55bf480a64f0 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/660227460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281abf50 con 0x55bf48098b60 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc2f9e7740 10 client.0 _ll_register_callbacks cb 0x7fdc1801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc2659c640 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 0x55bf480b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc26d9d640 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480a64f0 0x55bf480b7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc1ffff640 1 -- 172.21.15.94:0/660227460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 msgr2=0x55bf480b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 0x55bf480b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc1ffff640 1 -- 172.21.15.94:0/660227460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480a64f0 msgr2=0x55bf480b7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480a64f0 0x55bf480b7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.949+0000 7fdc1ffff640 1 -- 172.21.15.94:0/660227460 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf480a0720 con 0x55bf480998d0 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc1ffff640 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480b7450 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdc10099020 tx=0x7fdc100a58a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2dd0a99708c17b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc0dffb640 1 -- 172.21.15.94:0/660227460 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc100046c0 con 0x55bf480998d0 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc0dffb640 1 -- 172.21.15.94:0/660227460 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fdc100a53a0 con 0x55bf480998d0 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/660227460 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc1009d2c0 con 0x55bf480998d0 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc0dffb640 1 -- 172.21.15.94:0/660227460 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdc1009cbc0 con 0x55bf480998d0 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc2f9e7740 10 client.16074 fetch_fsmap learned FSMap version 24 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/660227460 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bf480a6e10 con 0x55bf480998d0 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc0dffb640 1 -- 172.21.15.94:0/660227460 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc1009d2c0 con 0x55bf480998d0 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc0dffb640 1 -- 172.21.15.94:0/660227460 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdc100989a0 con 0x55bf480998d0 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc2f9e7740 10 client.16074 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc2f9e7740 10 client.16074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc2f9e7740 20 client.16074 populate_metadata read hostname 'smithi094' 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf480a7410 0x55bf48122b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:46.953 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc2f9e7740 4 client.16074 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:40:46.954 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.950+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/660227460 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bf47f9d3d0 con 0x55bf480a7410 2024-04-21T08:40:46.954 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.951+0000 7fdc2659c640 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf480a7410 0x55bf48122b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:46.954 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.951+0000 7fdc0dffb640 10 client.16074 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:46.954 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.951+0000 7fdc2659c640 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf480a7410 0x55bf48122b20 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64a3f622fe5abe98 server_cookie=8bb952c038eed54 in_seq=0 out_seq=0 2024-04-21T08:40:46.955 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.954+0000 7fdc0dffb640 1 -- 172.21.15.94:0/660227460 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bf47f9d3d0 con 0x55bf480a7410 2024-04-21T08:40:46.955 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.954+0000 7fdc0dffb640 10 client.16074 handle_command_reply: tid=0 2024-04-21T08:40:46.955 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.954+0000 7fdc2f9e7740 1 client.16074 shutdown 2024-04-21T08:40:46.955 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/660227460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 msgr2=0x55bf480b7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:46.955 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480b7450 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdc10099020 tx=0x7fdc100a58a0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.955 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/660227460 shutdown_connections 2024-04-21T08:40:46.955 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf480a7410 0x55bf48122b20 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.955 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf480a64f0 0x55bf480b7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.955 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf480998d0 0x55bf480b7450 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.955 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc0dffb640 0 client.16074 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:40:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 1 --2- 172.21.15.94:0/660227460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf48098b60 0x55bf480b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/660227460 >> 172.21.15.94:0/660227460 conn(0x55bf4805fc90 msgr2=0x55bf48094ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/660227460 shutdown_connections 2024-04-21T08:40:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 1 -- 172.21.15.94:0/660227460 wait complete. 2024-04-21T08:40:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2f9e7740 20 client.16074 trim_cache size 0 max 16384 2024-04-21T08:40:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:{ 2024-04-21T08:40:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "message": "", 2024-04-21T08:40:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "return_code": 0 2024-04-21T08:40:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:} 2024-04-21T08:40:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/346458897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdc140a3f60 msgr2=0x7fdc140a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2cdf4640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdc140a3f60 0x7fdc140a6420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009a60 tx=0x7fdc20004a60 comp rx=0 tx=0).stop 2024-04-21T08:40:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/346458897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc28106170 msgr2=0x7fdc28199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2cdf4640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc28106170 0x7fdc28199a20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000e9c0 tx=0x7fdc1000ee80 comp rx=0 tx=0).stop 2024-04-21T08:40:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/346458897 shutdown_connections 2024-04-21T08:40:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2cdf4640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdc140a3f60 0x7fdc140a6420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2cdf4640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc28199fb0 0x7fdc281a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2cdf4640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc28106170 0x7fdc28199a20 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2cdf4640 1 --2- 172.21.15.94:0/346458897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc281010f0 0x7fdc281994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.955+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/346458897 >> 172.21.15.94:0/346458897 conn(0x7fdc28077fd0 msgr2=0x7fdc280fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.956+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/346458897 shutdown_connections 2024-04-21T08:40:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:40:46.956+0000 7fdc2cdf4640 1 -- 172.21.15.94:0/346458897 wait complete. 2024-04-21T08:40:47.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:47 smithi111 ceph-mon[31493]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-04-21T08:40:47.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:47 smithi094 ceph-mon[27988]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-04-21T08:40:47.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:47 smithi173 ceph-mon[31428]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-04-21T08:40:49.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:49 smithi111 ceph-mon[31493]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:40:49.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:49 smithi094 ceph-mon[27988]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:40:49.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:49 smithi173 ceph-mon[31428]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:40:51.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:51 smithi111 ceph-mon[31493]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-21T08:40:51.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:51 smithi094 ceph-mon[27988]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-21T08:40:51.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:51 smithi173 ceph-mon[31428]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-21T08:40:51.904 INFO:tasks.workunit.client.0.smithi094.stderr:Note: switching to '4f6bc5273643aaf5660909e85400647a8951c10b'. 2024-04-21T08:40:51.904 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-04-21T08:40:51.904 INFO:tasks.workunit.client.0.smithi094.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-21T08:40:51.904 INFO:tasks.workunit.client.0.smithi094.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-21T08:40:51.904 INFO:tasks.workunit.client.0.smithi094.stderr:state without impacting any branches by switching back to a branch. 2024-04-21T08:40:51.904 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-04-21T08:40:51.904 INFO:tasks.workunit.client.0.smithi094.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-21T08:40:51.904 INFO:tasks.workunit.client.0.smithi094.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-21T08:40:51.905 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-04-21T08:40:51.905 INFO:tasks.workunit.client.0.smithi094.stderr: git switch -c 2024-04-21T08:40:51.905 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-04-21T08:40:51.905 INFO:tasks.workunit.client.0.smithi094.stderr:Or undo this operation with: 2024-04-21T08:40:51.905 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-04-21T08:40:51.905 INFO:tasks.workunit.client.0.smithi094.stderr: git switch - 2024-04-21T08:40:51.905 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-04-21T08:40:51.905 INFO:tasks.workunit.client.0.smithi094.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-21T08:40:51.905 INFO:tasks.workunit.client.0.smithi094.stderr: 2024-04-21T08:40:51.905 INFO:tasks.workunit.client.0.smithi094.stderr:HEAD is now at 4f6bc527364 use ubuntu for fs/workload tests 2024-04-21T08:40:51.908 DEBUG:teuthology.orchestra.run.smithi094:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-21T08:40:51.936 INFO:tasks.workunit.client.0.smithi094.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-21T08:40:51.938 INFO:tasks.workunit.client.0.smithi094.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T08:40:51.938 INFO:tasks.workunit.client.0.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-21T08:40:52.567 INFO:tasks.workunit.client.0.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-21T08:40:52.624 INFO:tasks.workunit.client.0.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-21T08:40:52.664 INFO:tasks.workunit.client.0.smithi094.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T08:40:52.666 INFO:tasks.workunit.client.0.smithi094.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T08:40:52.666 INFO:tasks.workunit.client.0.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-21T08:40:52.703 INFO:tasks.workunit.client.0.smithi094.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T08:40:52.707 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:40:52.707 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-21T08:40:52.763 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-04-21T08:40:52.764 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-04-21T08:40:52.764 DEBUG:teuthology.orchestra.run.smithi094:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-04-21T08:40:52.824 INFO:tasks.workunit.client.0.smithi094.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-04-21T08:40:52.895 INFO:tasks.workunit.client.0.smithi094.stdout: Iozone: Performance Test of File I/O 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Version $Revision: 3.494 $ 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Compiled for 64 bit mode. 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Build: linux-AMD64 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-21T08:40:52.896 INFO:tasks.workunit.client.0.smithi094.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Run began: Sun Apr 21 08:40:52 2024 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Include close in write timing 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Include fsync in write timing 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: File size set to 1048576 kB 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Record Size 16 kB 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Output is in kBytes/sec 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Time Resolution = 0.000001 seconds. 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Processor cache size set to 1024 kBytes. 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Processor cache line size set to 32 bytes. 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: File stride size set to 17 * record size. 2024-04-21T08:40:52.897 INFO:tasks.workunit.client.0.smithi094.stdout: Throughput test with 1 process 2024-04-21T08:40:52.898 INFO:tasks.workunit.client.0.smithi094.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-04-21T08:40:53.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:53 smithi111 ceph-mon[31493]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T08:40:53.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:53 smithi094 ceph-mon[27988]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T08:40:53.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:53 smithi173 ceph-mon[31428]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T08:40:54.243 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:40:54.243 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 initial writers = 779218.69 kB/sec 2024-04-21T08:40:54.243 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 initial writers = 779108.81 kB/sec 2024-04-21T08:40:54.243 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 779218.69 kB/sec 2024-04-21T08:40:54.243 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 779218.69 kB/sec 2024-04-21T08:40:54.244 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 779218.69 kB/sec 2024-04-21T08:40:54.244 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 1048576.00 kB 2024-04-21T08:40:55.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:55 smithi111 ceph-mon[31493]: pgmap v31: 129 pgs: 129 active+clean; 517 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 43 MiB/s wr, 19 op/s 2024-04-21T08:40:55.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:55 smithi094 ceph-mon[27988]: pgmap v31: 129 pgs: 129 active+clean; 517 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 43 MiB/s wr, 19 op/s 2024-04-21T08:40:55.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:55 smithi173 ceph-mon[31428]: pgmap v31: 129 pgs: 129 active+clean; 517 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 43 MiB/s wr, 19 op/s 2024-04-21T08:40:57.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:57 smithi111 ceph-mon[31493]: pgmap v32: 129 pgs: 129 active+clean; 733 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 61 MiB/s wr, 23 op/s 2024-04-21T08:40:57.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:57 smithi094 ceph-mon[27988]: pgmap v32: 129 pgs: 129 active+clean; 733 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 61 MiB/s wr, 23 op/s 2024-04-21T08:40:57.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:57 smithi173 ceph-mon[31428]: pgmap v32: 129 pgs: 129 active+clean; 733 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 61 MiB/s wr, 23 op/s 2024-04-21T08:40:59.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:40:59 smithi111 ceph-mon[31493]: pgmap v33: 129 pgs: 129 active+clean; 733 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 61 MiB/s wr, 22 op/s 2024-04-21T08:40:59.705 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:40:59.705 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 rewriters = 842769.19 kB/sec 2024-04-21T08:40:59.705 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 rewriters = 842625.01 kB/sec 2024-04-21T08:40:59.705 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 842769.19 kB/sec 2024-04-21T08:40:59.705 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 842769.19 kB/sec 2024-04-21T08:40:59.705 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 842769.19 kB/sec 2024-04-21T08:40:59.705 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 1048576.00 kB 2024-04-21T08:40:59.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:40:59 smithi094 ceph-mon[27988]: pgmap v33: 129 pgs: 129 active+clean; 733 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 61 MiB/s wr, 22 op/s 2024-04-21T08:40:59.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:40:59 smithi173 ceph-mon[31428]: pgmap v33: 129 pgs: 129 active+clean; 733 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 61 MiB/s wr, 22 op/s 2024-04-21T08:41:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.045+0000 7f7b492ce640 1 -- 172.21.15.94:0/1108658162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b44106170 msgr2=0x7f7b4410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.045+0000 7f7b492ce640 1 --2- 172.21.15.94:0/1108658162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b44106170 0x7f7b4410dd40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009ca0 tx=0x7f7b3c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.046+0000 7f7b492ce640 1 -- 172.21.15.94:0/1108658162 shutdown_connections 2024-04-21T08:41:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.046+0000 7f7b492ce640 1 --2- 172.21.15.94:0/1108658162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b44101aa0 0x7f7b44105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.046+0000 7f7b492ce640 1 --2- 172.21.15.94:0/1108658162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b44106170 0x7f7b4410dd40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.046+0000 7f7b492ce640 1 --2- 172.21.15.94:0/1108658162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b441010f0 0x7f7b441014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.046+0000 7f7b492ce640 1 -- 172.21.15.94:0/1108658162 >> 172.21.15.94:0/1108658162 conn(0x7f7b44077fd0 msgr2=0x7f7b440783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.046+0000 7f7b492ce640 1 -- 172.21.15.94:0/1108658162 shutdown_connections 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.046+0000 7f7b492ce640 1 -- 172.21.15.94:0/1108658162 wait complete. 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.046+0000 7f7b492ce640 1 Processor -- start 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.046+0000 7f7b492ce640 1 -- start start 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b492ce640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b441010f0 0x7f7b441975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b492ce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b44106170 0x7f7b44197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b492ce640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4419be10 0x7f7b4419c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b492ce640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7b441110e0 con 0x7f7b441010f0 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b492ce640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b44110f60 con 0x7f7b4419be10 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b492ce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b44111260 con 0x7f7b44106170 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b42ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b441010f0 0x7f7b441975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b42ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b441010f0 0x7f7b441975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34134/0 (socket says 172.21.15.94:34134) 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b42ffd640 1 -- 172.21.15.94:0/4027650242 learned_addr learned my addr 172.21.15.94:0/4027650242 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b427fc640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b44106170 0x7f7b44197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b437fe640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4419be10 0x7f7b4419c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b42ffd640 1 -- 172.21.15.94:0/4027650242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b44106170 msgr2=0x7f7b44197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b44106170 0x7f7b44197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b42ffd640 1 -- 172.21.15.94:0/4027650242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4419be10 msgr2=0x7f7b4419c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4419be10 0x7f7b4419c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b42ffd640 1 -- 172.21.15.94:0/4027650242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4419c900 con 0x7f7b441010f0 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b427fc640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b44106170 0x7f7b44197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b437fe640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4419be10 0x7f7b4419c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b441010f0 0x7f7b441975a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009900 tx=0x7f7b34009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b2bfff640 1 -- 172.21.15.94:0/4027650242 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3400d030 con 0x7f7b441010f0 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b2bfff640 1 -- 172.21.15.94:0/4027650242 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b3400d1d0 con 0x7f7b441010f0 2024-04-21T08:41:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b492ce640 1 -- 172.21.15.94:0/4027650242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b4419cbf0 con 0x7f7b441010f0 2024-04-21T08:41:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b2bfff640 1 -- 172.21.15.94:0/4027650242 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34005280 con 0x7f7b441010f0 2024-04-21T08:41:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.047+0000 7f7b492ce640 1 -- 172.21.15.94:0/4027650242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b44069920 con 0x7f7b441010f0 2024-04-21T08:41:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.049+0000 7f7b2bfff640 1 -- 172.21.15.94:0/4027650242 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 23) ==== 138873+0+0 (secure 0 0 0) 0x7f7b34005420 con 0x7f7b441010f0 2024-04-21T08:41:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b4bf95740 1 Processor -- start 2024-04-21T08:41:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b4bf95740 1 -- start start 2024-04-21T08:41:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b2bfff640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b2c0a3ee0 0x7f7b2c0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b2bfff640 1 -- 172.21.15.94:0/4027650242 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f7b340d2280 con 0x7f7b441010f0 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b437fe640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b2c0a3ee0 0x7f7b2c0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c3bce0 0x559a72c3aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c3b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c444f0 0x559a72c468e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x559a72b9b8d0 con 0x559a72c3af70 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559a72b38e80 con 0x559a72c3bce0 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559a72b97890 con 0x559a72c444f0 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b427fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c3b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b427fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c3b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34150/0 (socket says 172.21.15.94:34150) 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.050+0000 7f7b427fc640 1 -- 172.21.15.94:0/2612071482 learned_addr learned my addr 172.21.15.94:0/2612071482 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b437fe640 1 --2- 172.21.15.94:0/2612071482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c444f0 0x559a72c468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/2612071482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c3bce0 0x559a72c3aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b427fc640 1 -- 172.21.15.94:0/2612071482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c444f0 msgr2=0x559a72c468e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b427fc640 1 --2- 172.21.15.94:0/2612071482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c444f0 0x559a72c468e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b427fc640 1 -- 172.21.15.94:0/2612071482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c3bce0 msgr2=0x559a72c3aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b427fc640 1 --2- 172.21.15.94:0/2612071482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c3bce0 0x559a72c3aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b427fc640 1 -- 172.21.15.94:0/2612071482 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a72b5a320 con 0x559a72c3af70 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b437fe640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b2c0a3ee0 0x7f7b2c0a63a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c006010 tx=0x7f7b3c032d90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b427fc640 1 --2- 172.21.15.94:0/2612071482 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c3b3f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009c20 tx=0x7f7b38010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e56912f71529d2f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b297fa640 1 -- 172.21.15.94:0/2612071482 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38004030 con 0x559a72c3af70 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b297fa640 1 -- 172.21.15.94:0/2612071482 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b380035f0 con 0x559a72c3af70 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.051+0000 7f7b297fa640 1 -- 172.21.15.94:0/2612071482 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38012650 con 0x559a72c3af70 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2612071482 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 msgr2=0x559a72c3b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/2612071482 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c3b3f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009c20 tx=0x7f7b38010740 comp rx=0 tx=0).stop 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2612071482 shutdown_connections 2024-04-21T08:41:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/2612071482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c444f0 0x559a72c468e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/2612071482 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c3b3f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/2612071482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c3bce0 0x559a72c3aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2612071482 >> 172.21.15.94:0/2612071482 conn(0x559a72bfdc90 msgr2=0x559a72c35c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2612071482 shutdown_connections 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2612071482 wait complete. 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 -- start start 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 Processor -- start 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c43eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3bce0 0x559a72c41240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c41850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x559a72c47220 con 0x559a72c3af70 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559a72b23eb0 con 0x559a72c444f0 2024-04-21T08:41:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559a72c473a0 con 0x559a72c3bce0 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.052+0000 7f7b4bf95740 10 client.0 _ll_register_callbacks cb 0x559a72b50b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b42ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c43eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b42ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c43eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34156/0 (socket says 172.21.15.94:34156) 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b42ffd640 1 -- 172.21.15.94:0/1472753032 learned_addr learned my addr 172.21.15.94:0/1472753032 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b427fc640 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3bce0 0x559a72c41240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b437fe640 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c41850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b42ffd640 1 -- 172.21.15.94:0/1472753032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3bce0 msgr2=0x559a72c41240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3bce0 0x559a72c41240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b42ffd640 1 -- 172.21.15.94:0/1472753032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 msgr2=0x559a72c41850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c41850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b42ffd640 1 -- 172.21.15.94:0/1472753032 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a728ae210 con 0x559a72c3af70 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c43eb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7b34007590 tx=0x7f7b340d2170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4d965bb85ef8aa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b26ff5640 1 -- 172.21.15.94:0/1472753032 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3401e040 con 0x559a72c3af70 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b26ff5640 1 -- 172.21.15.94:0/1472753032 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f7b340be0c0 con 0x559a72c3af70 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1472753032 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b380035f0 con 0x559a72c3af70 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b26ff5640 1 -- 172.21.15.94:0/1472753032 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7b340a1fa0 con 0x559a72c3af70 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b4bf95740 10 client.16086 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1472753032 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559a7297a620 con 0x559a72c3af70 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.053+0000 7f7b26ff5640 1 -- 172.21.15.94:0/1472753032 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b380035f0 con 0x559a72c3af70 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.054+0000 7f7b26ff5640 1 -- 172.21.15.94:0/1472753032 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7b3400c070 con 0x559a72c3af70 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.054+0000 7f7b4bf95740 10 client.16086 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.054+0000 7f7b4bf95740 10 client.16086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.054+0000 7f7b4bf95740 20 client.16086 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.054+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559a72c54580 0x559a72c56970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.054+0000 7f7b4bf95740 4 client.16086 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.054+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1472753032 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a72baf880 con 0x559a72c54580 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.054+0000 7f7b427fc640 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559a72c54580 0x559a72c56970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.054+0000 7f7b427fc640 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559a72c54580 0x559a72c56970 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d95b7630a1efb34 server_cookie=2b101babb0d1718b in_seq=0 out_seq=0 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.054+0000 7f7b26ff5640 10 client.16086 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b26ff5640 1 -- 172.21.15.94:0/1472753032 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a72baf880 con 0x559a72c54580 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b26ff5640 10 client.16086 handle_command_reply: tid=0 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 client.16086 shutdown 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1472753032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 msgr2=0x559a72c43eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c43eb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7b34007590 tx=0x7f7b340d2170 comp rx=0 tx=0).stop 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1472753032 shutdown_connections 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559a72c54580 0x559a72c56970 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3bce0 0x559a72c41240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c41850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/1472753032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c43eb0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b26ff5640 0 client.16086 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1472753032 >> 172.21.15.94:0/1472753032 conn(0x559a72bfdc90 msgr2=0x559a72c379b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1472753032 shutdown_connections 2024-04-21T08:41:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1472753032 wait complete. 2024-04-21T08:41:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.055+0000 7f7b4bf95740 20 client.16086 trim_cache size 0 max 16384 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b4bf95740 1 Processor -- start 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b4bf95740 1 -- start start 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3af70 0x559a72c50120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3bce0 0x559a72c40220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c43970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7b441af960 con 0x559a72c3bce0 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b441ad230 con 0x559a72c444f0 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b441ab080 con 0x559a72c3af70 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b427fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3bce0 0x559a72c40220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b427fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3bce0 0x559a72c40220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34166/0 (socket says 172.21.15.94:34166) 2024-04-21T08:41:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.070+0000 7f7b427fc640 1 -- 172.21.15.94:0/1507574033 learned_addr learned my addr 172.21.15.94:0/1507574033 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/1507574033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3af70 0x559a72c50120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b437fe640 1 --2- 172.21.15.94:0/1507574033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c43970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b427fc640 1 -- 172.21.15.94:0/1507574033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3af70 msgr2=0x559a72c50120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b427fc640 1 --2- 172.21.15.94:0/1507574033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3af70 0x559a72c50120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b427fc640 1 -- 172.21.15.94:0/1507574033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 msgr2=0x559a72c43970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b427fc640 1 --2- 172.21.15.94:0/1507574033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c43970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b427fc640 1 -- 172.21.15.94:0/1507574033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a72c3a240 con 0x559a72c3bce0 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b427fc640 1 --2- 172.21.15.94:0/1507574033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3bce0 0x559a72c40220 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7b38019000 tx=0x7f7b3801f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d64fb03abc1ba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b28ff9640 1 -- 172.21.15.94:0/1507574033 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38030070 con 0x559a72c3bce0 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b28ff9640 1 -- 172.21.15.94:0/1507574033 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b380035f0 con 0x559a72c3bce0 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.071+0000 7f7b28ff9640 1 -- 172.21.15.94:0/1507574033 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3802b3d0 con 0x559a72c3bce0 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1507574033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3bce0 msgr2=0x559a72c40220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/1507574033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3bce0 0x559a72c40220 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7b38019000 tx=0x7f7b3801f040 comp rx=0 tx=0).stop 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1507574033 shutdown_connections 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/1507574033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c43970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/1507574033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3bce0 0x559a72c40220 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/1507574033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3af70 0x559a72c50120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1507574033 >> 172.21.15.94:0/1507574033 conn(0x559a72bfdc90 msgr2=0x559a72c3fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1507574033 shutdown_connections 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 -- 172.21.15.94:0/1507574033 wait complete. 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 -- start start 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 Processor -- start 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c4c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3bce0 0x559a72c47c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c48150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7b441ab6f0 con 0x559a72c3af70 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b441acb60 con 0x559a72c444f0 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b441abdc0 con 0x559a72c3bce0 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b4bf95740 10 client.0 _ll_register_callbacks cb 0x7f7b3409d9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b42ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c4c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b42ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c4c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34174/0 (socket says 172.21.15.94:34174) 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b42ffd640 1 -- 172.21.15.94:0/2919195094 learned_addr learned my addr 172.21.15.94:0/2919195094 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.072+0000 7f7b427fc640 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3bce0 0x559a72c47c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b437fe640 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c48150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b42ffd640 1 -- 172.21.15.94:0/2919195094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3bce0 msgr2=0x559a72c47c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3bce0 0x559a72c47c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b42ffd640 1 -- 172.21.15.94:0/2919195094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 msgr2=0x559a72c48150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c48150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b42ffd640 1 -- 172.21.15.94:0/2919195094 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a72c48690 con 0x559a72c3af70 2024-04-21T08:41:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b42ffd640 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c4c700 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7b340cf9a0 tx=0x7f7b3409c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be213bfef751a4ed server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b26ff5640 1 -- 172.21.15.94:0/2919195094 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3401e070 con 0x559a72c3af70 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b26ff5640 1 -- 172.21.15.94:0/2919195094 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f7b340cf750 con 0x559a72c3af70 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2919195094 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b380035f0 con 0x559a72c3af70 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b26ff5640 1 -- 172.21.15.94:0/2919195094 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7b3409d550 con 0x559a72c3af70 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b4bf95740 10 client.16092 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2919195094 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559a72cba790 con 0x559a72c3af70 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b26ff5640 1 -- 172.21.15.94:0/2919195094 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b380035f0 con 0x559a72c3af70 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b26ff5640 1 -- 172.21.15.94:0/2919195094 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7b3400c070 con 0x559a72c3af70 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b4bf95740 10 client.16092 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b4bf95740 10 client.16092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b4bf95740 20 client.16092 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.073+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559a72cbad50 0x559a72cbd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.074+0000 7f7b4bf95740 4 client.16092 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:41:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.074+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2919195094 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559a72b3b3d0 con 0x559a72cbad50 2024-04-21T08:41:00.079 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.074+0000 7f7b427fc640 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559a72cbad50 0x559a72cbd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.079 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.074+0000 7f7b427fc640 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559a72cbad50 0x559a72cbd170 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b4be2b938f5da5e server_cookie=ac18bb22e6ebbed6 in_seq=0 out_seq=0 2024-04-21T08:41:00.079 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.074+0000 7f7b26ff5640 10 client.16092 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b26ff5640 1 -- 172.21.15.94:0/2919195094 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559a72b3b3d0 con 0x559a72cbad50 2024-04-21T08:41:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b26ff5640 10 client.16092 handle_command_reply: tid=0 2024-04-21T08:41:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 client.16092 shutdown 2024-04-21T08:41:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2919195094 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 msgr2=0x559a72c4c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c4c700 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7b340cf9a0 tx=0x7f7b3409c100 comp rx=0 tx=0).stop 2024-04-21T08:41:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2919195094 shutdown_connections 2024-04-21T08:41:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559a72cbad50 0x559a72cbd170 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559a72c3bce0 0x559a72c47c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a72c444f0 0x559a72c48150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 --2- 172.21.15.94:0/2919195094 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559a72c3af70 0x559a72c4c700 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b26ff5640 0 client.16092 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2919195094 >> 172.21.15.94:0/2919195094 conn(0x559a72bfdc90 msgr2=0x559a72c308d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2919195094 shutdown_connections 2024-04-21T08:41:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 1 -- 172.21.15.94:0/2919195094 wait complete. 2024-04-21T08:41:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.082+0000 7f7b4bf95740 20 client.16092 trim_cache size 0 max 16384 2024-04-21T08:41:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:{ 2024-04-21T08:41:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "message": "", 2024-04-21T08:41:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "return_code": 0 2024-04-21T08:41:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:} 2024-04-21T08:41:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 -- 172.21.15.94:0/4027650242 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b2c0a3ee0 msgr2=0x7f7b2c0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b2c0a3ee0 0x7f7b2c0a63a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c006010 tx=0x7f7b3c032d90 comp rx=0 tx=0).stop 2024-04-21T08:41:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 -- 172.21.15.94:0/4027650242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b441010f0 msgr2=0x7f7b441975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b441010f0 0x7f7b441975a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009900 tx=0x7f7b34009dc0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 -- 172.21.15.94:0/4027650242 shutdown_connections 2024-04-21T08:41:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7b2c0a3ee0 0x7f7b2c0a63a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b44106170 0x7f7b44197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4419be10 0x7f7b4419c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 --2- 172.21.15.94:0/4027650242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b441010f0 0x7f7b441975a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 -- 172.21.15.94:0/4027650242 >> 172.21.15.94:0/4027650242 conn(0x7f7b44077fd0 msgr2=0x7f7b440fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 -- 172.21.15.94:0/4027650242 shutdown_connections 2024-04-21T08:41:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:00.083+0000 7f7b492ce640 1 -- 172.21.15.94:0/4027650242 wait complete. 2024-04-21T08:41:01.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:41:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:41:00.687+0000 7f5c68cff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:41:01.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:41:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:41:00.687+0000 7f5c68cff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:41:01.010 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:41:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:41:00.687+0000 7f5c68cff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:41:01.011 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:41:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:41:00.687+0000 7f5c68cff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:41:01.011 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:41:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:41:00.687+0000 7f5c68cff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:41:01.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:01 smithi111 ceph-mon[31493]: pgmap v34: 129 pgs: 129 active+clean; 1.0 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 98 MiB/s wr, 34 op/s 2024-04-21T08:41:01.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:01 smithi094 ceph-mon[27988]: pgmap v34: 129 pgs: 129 active+clean; 1.0 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 98 MiB/s wr, 34 op/s 2024-04-21T08:41:01.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:01 smithi173 ceph-mon[31428]: pgmap v34: 129 pgs: 129 active+clean; 1.0 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 98 MiB/s wr, 34 op/s 2024-04-21T08:41:02.594 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:02.594 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 readers = 7063253.50 kB/sec 2024-04-21T08:41:02.595 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 readers = 7053477.68 kB/sec 2024-04-21T08:41:02.595 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 7063253.50 kB/sec 2024-04-21T08:41:02.595 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 7063253.50 kB/sec 2024-04-21T08:41:02.595 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 7063253.50 kB/sec 2024-04-21T08:41:02.595 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 1048576.00 kB 2024-04-21T08:41:03.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:03 smithi111 ceph-mon[31493]: pgmap v35: 129 pgs: 129 active+clean; 1.0 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 98 MiB/s wr, 31 op/s 2024-04-21T08:41:03.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:03 smithi111 ceph-mon[31493]: mgrmap e24: x(active, since 64s), standbys: y, z 2024-04-21T08:41:03.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:03 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:03 smithi094 ceph-mon[27988]: pgmap v35: 129 pgs: 129 active+clean; 1.0 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 98 MiB/s wr, 31 op/s 2024-04-21T08:41:03.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:03 smithi094 ceph-mon[27988]: mgrmap e24: x(active, since 64s), standbys: y, z 2024-04-21T08:41:03.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:03.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:03 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:03 smithi173 ceph-mon[31428]: pgmap v35: 129 pgs: 129 active+clean; 1.0 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 98 MiB/s wr, 31 op/s 2024-04-21T08:41:03.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:03 smithi173 ceph-mon[31428]: mgrmap e24: x(active, since 64s), standbys: y, z 2024-04-21T08:41:03.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:03.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:03.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:03 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:04.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:04 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:04.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:04 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:04.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:04 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:04.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:04 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:04.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:04 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:04.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:04 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:04.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:04 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:04.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:04 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:04.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:04 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:41:04.844 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:04.845 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 re-readers = 7057790.00 kB/sec 2024-04-21T08:41:04.845 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 re-readers = 7048436.04 kB/sec 2024-04-21T08:41:04.845 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 7057790.00 kB/sec 2024-04-21T08:41:04.845 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 7057790.00 kB/sec 2024-04-21T08:41:04.845 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 7057790.00 kB/sec 2024-04-21T08:41:04.845 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 1048576.00 kB 2024-04-21T08:41:05.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:05 smithi111 ceph-mon[31493]: pgmap v36: 129 pgs: 129 active+clean; 1.0 GiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 128 MiB/s wr, 42 op/s 2024-04-21T08:41:05.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:05 smithi094 ceph-mon[27988]: pgmap v36: 129 pgs: 129 active+clean; 1.0 GiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 128 MiB/s wr, 42 op/s 2024-04-21T08:41:05.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:05 smithi173 ceph-mon[31428]: pgmap v36: 129 pgs: 129 active+clean; 1.0 GiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 128 MiB/s wr, 42 op/s 2024-04-21T08:41:07.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:07 smithi111 ceph-mon[31493]: pgmap v37: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 128 MiB/s wr, 56 op/s 2024-04-21T08:41:07.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:07 smithi094 ceph-mon[27988]: pgmap v37: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 128 MiB/s wr, 56 op/s 2024-04-21T08:41:07.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:07 smithi173 ceph-mon[31428]: pgmap v37: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 128 MiB/s wr, 56 op/s 2024-04-21T08:41:08.658 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:41:08.658 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:41:08.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.777+0000 7f903e146640 1 -- 172.21.15.94:0/3578565893 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9038106180 msgr2=0x7f903810dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.777+0000 7f903e146640 1 --2- 172.21.15.94:0/3578565893 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9038106180 0x7f903810dd50 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9030009ca0 tx=0x7f903002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.777+0000 7f903e146640 1 -- 172.21.15.94:0/3578565893 shutdown_connections 2024-04-21T08:41:08.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.777+0000 7f903e146640 1 --2- 172.21.15.94:0/3578565893 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9038106180 0x7f903810dd50 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.777+0000 7f903e146640 1 --2- 172.21.15.94:0/3578565893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9038101a60 0x7f9038105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.777+0000 7f903e146640 1 --2- 172.21.15.94:0/3578565893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90381010b0 0x7f9038101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.777+0000 7f903e146640 1 -- 172.21.15.94:0/3578565893 >> 172.21.15.94:0/3578565893 conn(0x7f90380771d0 msgr2=0x7f90380775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:08.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.777+0000 7f903e146640 1 -- 172.21.15.94:0/3578565893 shutdown_connections 2024-04-21T08:41:08.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.777+0000 7f903e146640 1 -- 172.21.15.94:0/3578565893 wait complete. 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f903e146640 1 Processor -- start 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f903e146640 1 -- start start 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f903e146640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90381010b0 0x7f9038198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f903e146640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9038106180 0x7f9038198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f903e146640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f903819cf30 0x7f903819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f903e146640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f90381110f0 con 0x7f90381010b0 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f903e146640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9038110f70 con 0x7f9038106180 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f903e146640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9038111270 con 0x7f903819cf30 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f90377fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90381010b0 0x7f9038198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f90377fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90381010b0 0x7f9038198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54482/0 (socket says 172.21.15.94:54482) 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f90377fe640 1 -- 172.21.15.94:0/568574765 learned_addr learned my addr 172.21.15.94:0/568574765 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f9037fff640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f903819cf30 0x7f903819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f90377fe640 1 -- 172.21.15.94:0/568574765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f903819cf30 msgr2=0x7f903819d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f90377fe640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f903819cf30 0x7f903819d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f90377fe640 1 -- 172.21.15.94:0/568574765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9038106180 msgr2=0x7f9038198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f902ffff640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9038106180 0x7f9038198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f90377fe640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9038106180 0x7f9038198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f90377fe640 1 -- 172.21.15.94:0/568574765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903819da20 con 0x7f90381010b0 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.778+0000 7f90377fe640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90381010b0 0x7f9038198670 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f902800d770 tx=0x7f902800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.779+0000 7f90357fa640 1 -- 172.21.15.94:0/568574765 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028017070 con 0x7f90381010b0 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.779+0000 7f90357fa640 1 -- 172.21.15.94:0/568574765 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90280054d0 con 0x7f90381010b0 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.779+0000 7f903e146640 1 -- 172.21.15.94:0/568574765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f903819dd10 con 0x7f90381010b0 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.779+0000 7f90357fa640 1 -- 172.21.15.94:0/568574765 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028004c30 con 0x7f90381010b0 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.779+0000 7f903e146640 1 -- 172.21.15.94:0/568574765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90381a5500 con 0x7f90381010b0 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.781+0000 7f90357fa640 1 -- 172.21.15.94:0/568574765 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 24) ==== 138930+0+0 (secure 0 0 0) 0x7f9028004e10 con 0x7f90381010b0 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.781+0000 7f9040e0c740 1 Processor -- start 2024-04-21T08:41:08.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.781+0000 7f9040e0c740 1 -- start start 2024-04-21T08:41:08.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f90357fa640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f90240a3f50 0x7f90240a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x55656664f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566658540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x55656665ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f9040e0c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5565665b41e0 con 0x55656664fea0 2024-04-21T08:41:08.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f9040e0c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556566551390 con 0x556566658a80 2024-04-21T08:41:08.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f9040e0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5565665b01a0 con 0x55656664f580 2024-04-21T08:41:08.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f9037fff640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f90240a3f50 0x7f90240a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566658540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566658540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54494/0 (socket says 172.21.15.94:54494) 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902ffff640 1 -- 172.21.15.94:0/4293988670 learned_addr learned my addr 172.21.15.94:0/4293988670 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f90357fa640 1 -- 172.21.15.94:0/568574765 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f90280d0aa0 con 0x7f90381010b0 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f90377fe640 1 --2- 172.21.15.94:0/4293988670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x55656664f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f9037fff640 1 --2- 172.21.15.94:0/4293988670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x55656665ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902ffff640 1 -- 172.21.15.94:0/4293988670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 msgr2=0x55656664f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902ffff640 1 --2- 172.21.15.94:0/4293988670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x55656664f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902ffff640 1 -- 172.21.15.94:0/4293988670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 msgr2=0x55656665ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902ffff640 1 --2- 172.21.15.94:0/4293988670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x55656665ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902ffff640 1 -- 172.21.15.94:0/4293988670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565663907d0 con 0x55656664fea0 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902ffff640 1 --2- 172.21.15.94:0/4293988670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566658540 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f902000ba60 tx=0x7f902000bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ee9b76c4899f42 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f9037fff640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f90240a3f50 0x7f90240a6410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9030032460 tx=0x7f9030032de0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:08.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902dffb640 1 -- 172.21.15.94:0/4293988670 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020017420 con 0x55656664fea0 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.782+0000 7f902dffb640 1 -- 172.21.15.94:0/4293988670 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9020017a70 con 0x55656664fea0 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.783+0000 7f902dffb640 1 -- 172.21.15.94:0/4293988670 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902000f9d0 con 0x55656664fea0 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.783+0000 7f9040e0c740 1 -- 172.21.15.94:0/4293988670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 msgr2=0x556566658540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.783+0000 7f9040e0c740 1 --2- 172.21.15.94:0/4293988670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566658540 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f902000ba60 tx=0x7f902000bf20 comp rx=0 tx=0).stop 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.783+0000 7f9040e0c740 1 -- 172.21.15.94:0/4293988670 shutdown_connections 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.783+0000 7f9040e0c740 1 --2- 172.21.15.94:0/4293988670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x55656665ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.783+0000 7f9040e0c740 1 --2- 172.21.15.94:0/4293988670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566658540 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.783+0000 7f9040e0c740 1 --2- 172.21.15.94:0/4293988670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x55656664f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.783+0000 7f9040e0c740 1 -- 172.21.15.94:0/4293988670 >> 172.21.15.94:0/4293988670 conn(0x556566649cb0 msgr2=0x55656664a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.783+0000 7f9040e0c740 1 -- 172.21.15.94:0/4293988670 shutdown_connections 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.783+0000 7f9040e0c740 1 -- 172.21.15.94:0/4293988670 wait complete. 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f9040e0c740 1 -- start start 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f9040e0c740 1 Processor -- start 2024-04-21T08:41:08.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x556566655ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566656020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x556566656560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f9040e0c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55656665b870 con 0x55656664fea0 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f9040e0c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55656656d2f0 con 0x556566658a80 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f9040e0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55656665b9f0 con 0x55656664f580 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f9040e0c740 10 client.0 _ll_register_callbacks cb 0x55656657a580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f902ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566656020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f902ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566656020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54498/0 (socket says 172.21.15.94:54498) 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f902ffff640 1 -- 172.21.15.94:0/446536916 learned_addr learned my addr 172.21.15.94:0/446536916 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f9037fff640 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x556566656560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f90377fe640 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x556566655ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f902ffff640 1 -- 172.21.15.94:0/446536916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 msgr2=0x556566655ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f902ffff640 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x556566655ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.784+0000 7f902ffff640 1 -- 172.21.15.94:0/446536916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 msgr2=0x556566656560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f902ffff640 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x556566656560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f902ffff640 1 -- 172.21.15.94:0/446536916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565662c63f0 con 0x55656664fea0 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f90377fe640 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x556566655ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f902ffff640 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566656020 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9020019040 tx=0x7f9020017e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ea346f9c897fea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9016ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9016ffd640 1 -- 172.21.15.94:0/446536916 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020025070 con 0x55656664fea0 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9016ffd640 1 -- 172.21.15.94:0/446536916 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9020017830 con 0x55656664fea0 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9040e0c740 1 -- 172.21.15.94:0/446536916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9020017a70 con 0x55656664fea0 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9016ffd640 1 -- 172.21.15.94:0/446536916 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9020031050 con 0x55656664fea0 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9040e0c740 10 client.16110 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9040e0c740 1 -- 172.21.15.94:0/446536916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5565662860f0 con 0x55656664fea0 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9016ffd640 1 -- 172.21.15.94:0/446536916 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9020017a70 con 0x55656664fea0 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9016ffd640 1 -- 172.21.15.94:0/446536916 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f902000fcb0 con 0x55656664fea0 2024-04-21T08:41:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9040e0c740 10 client.16110 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9040e0c740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9040e0c740 20 client.16110 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9040e0c740 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5565666689e0 0x55656666add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9040e0c740 4 client.16110 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.785+0000 7f9040e0c740 1 -- 172.21.15.94:0/446536916 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565665c1c00 con 0x5565666689e0 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.786+0000 7f90377fe640 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5565666689e0 0x55656666add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.786+0000 7f90377fe640 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5565666689e0 0x55656666add0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9acda2a0f9abcfdf server_cookie=88427f10ea40c53a in_seq=0 out_seq=0 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.786+0000 7f9016ffd640 10 client.16110 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9016ffd640 1 -- 172.21.15.94:0/446536916 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565665c1c00 con 0x5565666689e0 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9016ffd640 10 client.16110 handle_command_reply: tid=0 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 client.16110 shutdown 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 -- 172.21.15.94:0/446536916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 msgr2=0x556566656020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566656020 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9020019040 tx=0x7f9020017e50 comp rx=0 tx=0).stop 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 -- 172.21.15.94:0/446536916 shutdown_connections 2024-04-21T08:41:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5565666689e0 0x55656666add0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x556566656560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664fea0 0x556566656020 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9016ffd640 0 client.16110 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 --2- 172.21.15.94:0/446536916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x556566655ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 -- 172.21.15.94:0/446536916 >> 172.21.15.94:0/446536916 conn(0x556566649cb0 msgr2=0x55656664ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 -- 172.21.15.94:0/446536916 shutdown_connections 2024-04-21T08:41:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 1 -- 172.21.15.94:0/446536916 wait complete. 2024-04-21T08:41:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.787+0000 7f9040e0c740 20 client.16110 trim_cache size 0 max 16384 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.805+0000 7f9040e0c740 1 Processor -- start 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.805+0000 7f9040e0c740 1 -- start start 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.805+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664f580 0x556566658210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.805+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664fea0 0x556566664730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.805+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x556566670460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.805+0000 7f9040e0c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f90381b1c10 con 0x55656664f580 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.805+0000 7f9040e0c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f90381af4e0 con 0x556566658a80 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.805+0000 7f9040e0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f90381ad330 con 0x55656664fea0 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f90377fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664f580 0x556566658210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f90377fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664f580 0x556566658210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54508/0 (socket says 172.21.15.94:54508) 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f90377fe640 1 -- 172.21.15.94:0/1616034188 learned_addr learned my addr 172.21.15.94:0/1616034188 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:08.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f90377fe640 1 -- 172.21.15.94:0/1616034188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664fea0 msgr2=0x556566664730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f90377fe640 1 --2- 172.21.15.94:0/1616034188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664fea0 0x556566664730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f90377fe640 1 -- 172.21.15.94:0/1616034188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 msgr2=0x556566670460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f90377fe640 1 --2- 172.21.15.94:0/1616034188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x556566670460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f90377fe640 1 -- 172.21.15.94:0/1616034188 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556566651690 con 0x55656664f580 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f90377fe640 1 --2- 172.21.15.94:0/1616034188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664f580 0x556566658210 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9028098d10 tx=0x7f9028009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce76b5286d0ff1e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f902d7fa640 1 -- 172.21.15.94:0/1616034188 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028020040 con 0x55656664f580 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f902d7fa640 1 -- 172.21.15.94:0/1616034188 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f902809c340 con 0x55656664f580 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.806+0000 7f902d7fa640 1 -- 172.21.15.94:0/1616034188 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902809a640 con 0x55656664f580 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 -- 172.21.15.94:0/1616034188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664f580 msgr2=0x556566658210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 --2- 172.21.15.94:0/1616034188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664f580 0x556566658210 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9028098d10 tx=0x7f9028009f90 comp rx=0 tx=0).stop 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 -- 172.21.15.94:0/1616034188 shutdown_connections 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 --2- 172.21.15.94:0/1616034188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664fea0 0x556566664730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 --2- 172.21.15.94:0/1616034188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556566658a80 0x556566670460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 --2- 172.21.15.94:0/1616034188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55656664f580 0x556566658210 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 -- 172.21.15.94:0/1616034188 >> 172.21.15.94:0/1616034188 conn(0x556566649cb0 msgr2=0x5565666543b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 -- 172.21.15.94:0/1616034188 shutdown_connections 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 -- 172.21.15.94:0/1616034188 wait complete. 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 -- start start 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.807+0000 7f9040e0c740 1 Processor -- start 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x55656665c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656664fea0 0x55656665cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9040e0c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556566658a80 0x55656665d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9040e0c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f90381ad9a0 con 0x556566658a80 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9040e0c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f90381aee10 con 0x55656664fea0 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9040e0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f90381ae070 con 0x55656664f580 2024-04-21T08:41:08.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9040e0c740 10 client.0 _ll_register_callbacks cb 0x7f902000b8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9037fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556566658a80 0x55656665d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9037fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556566658a80 0x55656665d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54522/0 (socket says 172.21.15.94:54522) 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9037fff640 1 -- 172.21.15.94:0/1028772900 learned_addr learned my addr 172.21.15.94:0/1028772900 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f902ffff640 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656664fea0 0x55656665cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f90377fe640 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x55656665c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9037fff640 1 -- 172.21.15.94:0/1028772900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 msgr2=0x55656665c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9037fff640 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x55656665c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9037fff640 1 -- 172.21.15.94:0/1028772900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656664fea0 msgr2=0x55656665cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9037fff640 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656664fea0 0x55656665cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9037fff640 1 -- 172.21.15.94:0/1028772900 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565666cdcd0 con 0x556566658a80 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f90377fe640 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x55656665c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f902ffff640 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656664fea0 0x55656665cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9037fff640 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556566658a80 0x55656665d020 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9030036000 tx=0x7f90300095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bc1f4e2e68589a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9016ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.808+0000 7f9016ffd640 1 -- 172.21.15.94:0/1028772900 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030054060 con 0x556566658a80 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9016ffd640 1 -- 172.21.15.94:0/1028772900 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9030004450 con 0x556566658a80 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9040e0c740 1 -- 172.21.15.94:0/1028772900 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f902809c340 con 0x556566658a80 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9016ffd640 1 -- 172.21.15.94:0/1028772900 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f903005b050 con 0x556566658a80 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9040e0c740 10 client.16116 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9040e0c740 1 -- 172.21.15.94:0/1028772900 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5565666cef20 con 0x556566658a80 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9016ffd640 1 -- 172.21.15.94:0/1028772900 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f902809c340 con 0x556566658a80 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9016ffd640 1 -- 172.21.15.94:0/1028772900 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9030034c20 con 0x556566658a80 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9040e0c740 10 client.16116 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9040e0c740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9040e0c740 20 client.16116 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9040e0c740 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5565666cf6a0 0x5565666d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9040e0c740 4 client.16116 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f9040e0c740 1 -- 172.21.15.94:0/1028772900 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5565665bba70 con 0x5565666cf6a0 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.809+0000 7f90377fe640 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5565666cf6a0 0x5565666d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9016ffd640 10 client.16116 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f90377fe640 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5565666cf6a0 0x5565666d1ac0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5149dd7acbb15089 server_cookie=dc5c64bbfa44c4de in_seq=0 out_seq=0 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9016ffd640 1 -- 172.21.15.94:0/1028772900 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5565665bba70 con 0x5565666cf6a0 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9016ffd640 10 client.16116 handle_command_reply: tid=0 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 client.16116 shutdown 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 -- 172.21.15.94:0/1028772900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556566658a80 msgr2=0x55656665d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556566658a80 0x55656665d020 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9030036000 tx=0x7f90300095c0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 -- 172.21.15.94:0/1028772900 shutdown_connections 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5565666cf6a0 0x5565666d1ac0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556566658a80 0x55656665d020 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656664fea0 0x55656665cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9016ffd640 0 client.16116 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:08.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 --2- 172.21.15.94:0/1028772900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55656664f580 0x55656665c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 -- 172.21.15.94:0/1028772900 >> 172.21.15.94:0/1028772900 conn(0x556566649cb0 msgr2=0x55656664b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:08.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 -- 172.21.15.94:0/1028772900 shutdown_connections 2024-04-21T08:41:08.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 1 -- 172.21.15.94:0/1028772900 wait complete. 2024-04-21T08:41:08.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.810+0000 7f9040e0c740 20 client.16116 trim_cache size 0 max 16384 2024-04-21T08:41:08.817 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:41:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:41:08.817 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:41:08.818 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:41:08.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 -- 172.21.15.94:0/568574765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f90240a3f50 msgr2=0x7f90240a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f90240a3f50 0x7f90240a6410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9030032460 tx=0x7f9030032de0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 -- 172.21.15.94:0/568574765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90381010b0 msgr2=0x7f9038198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90381010b0 0x7f9038198670 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f902800d770 tx=0x7f902800dc30 comp rx=0 tx=0).stop 2024-04-21T08:41:08.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 -- 172.21.15.94:0/568574765 shutdown_connections 2024-04-21T08:41:08.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f90240a3f50 0x7f90240a6410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f903819cf30 0x7f903819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9038106180 0x7f9038198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 --2- 172.21.15.94:0/568574765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90381010b0 0x7f9038198670 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 -- 172.21.15.94:0/568574765 >> 172.21.15.94:0/568574765 conn(0x7f90380771d0 msgr2=0x7f903806a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:08.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 -- 172.21.15.94:0/568574765 shutdown_connections 2024-04-21T08:41:08.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.811+0000 7f903e146640 1 -- 172.21.15.94:0/568574765 wait complete. 2024-04-21T08:41:08.822 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:41:08.823 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:08.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.922+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2956358567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7ac1011a0 msgr2=0x7fb7ac101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.922+0000 7fb7b43c3640 1 --2- 172.21.15.94:0/2956358567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7ac1011a0 0x7fb7ac101620 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009930 tx=0x7fb79c02bc20 comp rx=0 tx=0).stop 2024-04-21T08:41:08.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.922+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2956358567 shutdown_connections 2024-04-21T08:41:08.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.922+0000 7fb7b43c3640 1 --2- 172.21.15.94:0/2956358567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7ac101b60 0x7fb7ac10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.922+0000 7fb7b43c3640 1 --2- 172.21.15.94:0/2956358567 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7ac1011a0 0x7fb7ac101620 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.922+0000 7fb7b43c3640 1 --2- 172.21.15.94:0/2956358567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7ac105e30 0x7fb7ac106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.922+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2956358567 >> 172.21.15.94:0/2956358567 conn(0x7fb7ac077e30 msgr2=0x7fb7ac078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.922+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2956358567 shutdown_connections 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.922+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2956358567 wait complete. 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b43c3640 1 Processor -- start 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b43c3640 1 -- start start 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b43c3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7ac1011a0 0x7fb7ac1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b43c3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7ac101b60 0x7fb7ac198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b43c3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7ac105e30 0x7fb7ac19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b43c3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ac1110e0 con 0x7fb7ac1011a0 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b43c3640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ac110f60 con 0x7fb7ac101b60 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b43c3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ac111260 con 0x7fb7ac105e30 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b215f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7ac1011a0 0x7fb7ac1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b215f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7ac1011a0 0x7fb7ac1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54538/0 (socket says 172.21.15.94:54538) 2024-04-21T08:41:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b215f640 1 -- 172.21.15.94:0/2165629449 learned_addr learned my addr 172.21.15.94:0/2165629449 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b195e640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7ac101b60 0x7fb7ac198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b2960640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7ac105e30 0x7fb7ac19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b215f640 1 -- 172.21.15.94:0/2165629449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7ac105e30 msgr2=0x7fb7ac19cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b215f640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7ac105e30 0x7fb7ac19cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b215f640 1 -- 172.21.15.94:0/2165629449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7ac101b60 msgr2=0x7fb7ac198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b215f640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7ac101b60 0x7fb7ac198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.923+0000 7fb7b215f640 1 -- 172.21.15.94:0/2165629449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ac19d6e0 con 0x7fb7ac1011a0 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.924+0000 7fb7b215f640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7ac1011a0 0x7fb7ac1986f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800c920 tx=0x7fb7a800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.924+0000 7fb7a37fe640 1 -- 172.21.15.94:0/2165629449 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8007c70 con 0x7fb7ac1011a0 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.924+0000 7fb7a37fe640 1 -- 172.21.15.94:0/2165629449 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7a8007e10 con 0x7fb7ac1011a0 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.924+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2165629449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ac19d9d0 con 0x7fb7ac1011a0 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.924+0000 7fb7a37fe640 1 -- 172.21.15.94:0/2165629449 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8018820 con 0x7fb7ac1011a0 2024-04-21T08:41:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.924+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2165629449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ac1a5260 con 0x7fb7ac1011a0 2024-04-21T08:41:08.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.925+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2165629449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7ac068de0 con 0x7fb7ac1011a0 2024-04-21T08:41:08.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.926+0000 7fb7a37fe640 1 -- 172.21.15.94:0/2165629449 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 24) ==== 138930+0+0 (secure 0 0 0) 0x7fb7a800f450 con 0x7fb7ac1011a0 2024-04-21T08:41:08.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.926+0000 7fb7a37fe640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb7980a3ed0 0x7fb7980a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.927+0000 7fb7b195e640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb7980a3ed0 0x7fb7980a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.927+0000 7fb7a37fe640 1 -- 172.21.15.94:0/2165629449 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb7a80d0f70 con 0x7fb7ac1011a0 2024-04-21T08:41:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.927+0000 7fb7b195e640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb7980a3ed0 0x7fb7980a6390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb79c005ab0 tx=0x7fb79c005fd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:08.932+0000 7fb7a37fe640 1 -- 172.21.15.94:0/2165629449 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7a8014030 con 0x7fb7ac1011a0 2024-04-21T08:41:09.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.151+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2165629449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7ac10a3c0 con 0x7fb7ac1011a0 2024-04-21T08:41:09.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.152+0000 7fb7a37fe640 1 -- 172.21.15.94:0/2165629449 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7fb7a809cff0 con 0x7fb7ac1011a0 2024-04-21T08:41:09.152 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:41:09.152 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:41:09.152 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:41:09.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2165629449 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb7980a3ed0 msgr2=0x7fb7980a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb7980a3ed0 0x7fb7980a6390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb79c005ab0 tx=0x7fb79c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2165629449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7ac1011a0 msgr2=0x7fb7ac1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7ac1011a0 0x7fb7ac1986f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800c920 tx=0x7fb7a800cde0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2165629449 shutdown_connections 2024-04-21T08:41:09.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb7980a3ed0 0x7fb7980a6390 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7ac105e30 0x7fb7ac19cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7ac101b60 0x7fb7ac198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 --2- 172.21.15.94:0/2165629449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7ac1011a0 0x7fb7ac1986f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2165629449 >> 172.21.15.94:0/2165629449 conn(0x7fb7ac077e30 msgr2=0x7fb7ac0693c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2165629449 shutdown_connections 2024-04-21T08:41:09.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.154+0000 7fb7b43c3640 1 -- 172.21.15.94:0/2165629449 wait complete. 2024-04-21T08:41:09.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3932917621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc58102190 msgr2=0x7efc58102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 --2- 172.21.15.94:0/3932917621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc58102190 0x7efc58102570 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7efc50009ca0 tx=0x7efc5002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3932917621 shutdown_connections 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 --2- 172.21.15.94:0/3932917621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58102b40 0x7efc58106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 --2- 172.21.15.94:0/3932917621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc58107260 0x7efc5810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 --2- 172.21.15.94:0/3932917621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc58102190 0x7efc58102570 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3932917621 >> 172.21.15.94:0/3932917621 conn(0x7efc580fd990 msgr2=0x7efc580ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3932917621 shutdown_connections 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3932917621 wait complete. 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 Processor -- start 2024-04-21T08:41:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.266+0000 7efc5ffb6640 1 -- start start 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5ffb6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc58102190 0x7efc5819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5ffb6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58107260 0x7efc5819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5ffb6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc581985a0 0x7efc58198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5ffb6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efc58112220 con 0x7efc58102190 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5ffb6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc581120a0 con 0x7efc581985a0 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5ffb6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc581123a0 con 0x7efc58107260 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5dd52640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc58102190 0x7efc5819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5dd52640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc58102190 0x7efc5819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54550/0 (socket says 172.21.15.94:54550) 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5dd52640 1 -- 172.21.15.94:0/3968869210 learned_addr learned my addr 172.21.15.94:0/3968869210 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5dd52640 1 -- 172.21.15.94:0/3968869210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58107260 msgr2=0x7efc5819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5dd52640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58107260 0x7efc5819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5dd52640 1 -- 172.21.15.94:0/3968869210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc581985a0 msgr2=0x7efc58198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5dd52640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc581985a0 0x7efc58198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5dd52640 1 -- 172.21.15.94:0/3968869210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc581990c0 con 0x7efc58102190 2024-04-21T08:41:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5dd52640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc58102190 0x7efc5819e3f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7efc500098d0 tx=0x7efc50035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc4effd640 1 -- 172.21.15.94:0/3968869210 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc50039040 con 0x7efc58102190 2024-04-21T08:41:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc4effd640 1 -- 172.21.15.94:0/3968869210 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc50044070 con 0x7efc58102190 2024-04-21T08:41:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc4effd640 1 -- 172.21.15.94:0/3968869210 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc500042a0 con 0x7efc58102190 2024-04-21T08:41:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3968869210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc58199350 con 0x7efc58102190 2024-04-21T08:41:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.267+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3968869210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc581a55a0 con 0x7efc58102190 2024-04-21T08:41:09.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.269+0000 7efc4effd640 1 -- 172.21.15.94:0/3968869210 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 24) ==== 138930+0+0 (secure 0 0 0) 0x7efc50004440 con 0x7efc58102190 2024-04-21T08:41:09.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc62bac740 1 Processor -- start 2024-04-21T08:41:09.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc62bac740 1 -- start start 2024-04-21T08:41:09.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc62bac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 0x5640a192b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc62bac740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 0x5640a1934430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc62bac740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a1936e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc62bac740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5640a18901f0 con 0x5640a1934a00 2024-04-21T08:41:09.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc62bac740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5640a182d4b0 con 0x5640a192be80 2024-04-21T08:41:09.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc62bac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5640a188c1b0 con 0x5640a192b4d0 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc4effd640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc440a3f50 0x7efc440a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc5d551640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a1936e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc5d551640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a1936e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54564/0 (socket says 172.21.15.94:54564) 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc5d551640 1 -- 172.21.15.94:0/550445278 learned_addr learned my addr 172.21.15.94:0/550445278 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.270+0000 7efc5e553640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc440a3f50 0x7efc440a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc4effd640 1 -- 172.21.15.94:0/3968869210 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7efc500f3300 con 0x7efc58102190 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc5dd52640 1 --2- 172.21.15.94:0/550445278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 0x5640a1934430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc5e553640 1 --2- 172.21.15.94:0/550445278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 0x5640a192b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc5d551640 1 -- 172.21.15.94:0/550445278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 msgr2=0x5640a192b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc5d551640 1 --2- 172.21.15.94:0/550445278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 0x5640a192b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc5d551640 1 -- 172.21.15.94:0/550445278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 msgr2=0x5640a1934430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc5d551640 1 --2- 172.21.15.94:0/550445278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 0x5640a1934430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc5d551640 1 -- 172.21.15.94:0/550445278 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640a181f1b0 con 0x5640a1934a00 2024-04-21T08:41:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc5e553640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc440a3f50 0x7efc440a6410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efc5400cfd0 tx=0x7efc5400c8b0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc5d551640 1 --2- 172.21.15.94:0/550445278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a1936e50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7efc48009780 tx=0x7efc48009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd43c76864a6b8a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc43fff640 1 -- 172.21.15.94:0/550445278 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48015020 con 0x5640a1934a00 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc43fff640 1 -- 172.21.15.94:0/550445278 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc4800bd70 con 0x5640a1934a00 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.271+0000 7efc43fff640 1 -- 172.21.15.94:0/550445278 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48011e10 con 0x5640a1934a00 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 -- 172.21.15.94:0/550445278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 msgr2=0x5640a1936e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 --2- 172.21.15.94:0/550445278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a1936e50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7efc48009780 tx=0x7efc48009b80 comp rx=0 tx=0).stop 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 -- 172.21.15.94:0/550445278 shutdown_connections 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 --2- 172.21.15.94:0/550445278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a1936e50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 --2- 172.21.15.94:0/550445278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 0x5640a1934430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 --2- 172.21.15.94:0/550445278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 0x5640a192b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 -- 172.21.15.94:0/550445278 >> 172.21.15.94:0/550445278 conn(0x5640a1925ba0 msgr2=0x5640a1925f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 -- 172.21.15.94:0/550445278 shutdown_connections 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 -- 172.21.15.94:0/550445278 wait complete. 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 -- start start 2024-04-21T08:41:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.272+0000 7efc62bac740 1 Processor -- start 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc62bac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 0x5640a1934280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc62bac740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192be80 0x5640a1931610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc62bac740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a1934a00 0x5640a1931b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc62bac740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5640a1937970 con 0x5640a192be80 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc62bac740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5640a1849500 con 0x5640a1934a00 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc62bac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5640a1937af0 con 0x5640a192b4d0 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc62bac740 10 client.0 _ll_register_callbacks cb 0x5640a1817b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5d551640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192be80 0x5640a1931610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5d551640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192be80 0x5640a1931610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54566/0 (socket says 172.21.15.94:54566) 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5d551640 1 -- 172.21.15.94:0/119078237 learned_addr learned my addr 172.21.15.94:0/119078237 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5dd52640 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 0x5640a1934280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5d551640 1 -- 172.21.15.94:0/119078237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 msgr2=0x5640a1934280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5d551640 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 0x5640a1934280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5d551640 1 -- 172.21.15.94:0/119078237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a1934a00 msgr2=0x5640a1931b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5d551640 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a1934a00 0x5640a1931b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5d551640 1 -- 172.21.15.94:0/119078237 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640a15a23f0 con 0x5640a192be80 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5dd52640 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 0x5640a1934280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc5d551640 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192be80 0x5640a1931610 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7efc48006c30 tx=0x7efc480115d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb2ef04c408bbadf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc417fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc417fa640 1 -- 172.21.15.94:0/119078237 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48027060 con 0x5640a192be80 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc417fa640 1 -- 172.21.15.94:0/119078237 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7efc48011d60 con 0x5640a192be80 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc62bac740 1 -- 172.21.15.94:0/119078237 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc4800bd70 con 0x5640a192be80 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.273+0000 7efc417fa640 1 -- 172.21.15.94:0/119078237 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efc4802e050 con 0x5640a192be80 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc62bac740 10 client.16134 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc62bac740 1 -- 172.21.15.94:0/119078237 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5640a166c7d0 con 0x5640a192be80 2024-04-21T08:41:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc417fa640 1 -- 172.21.15.94:0/119078237 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc4800bd70 con 0x5640a192be80 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc417fa640 1 -- 172.21.15.94:0/119078237 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efc4801aaf0 con 0x5640a192be80 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc62bac740 10 client.16134 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc62bac740 10 client.16134 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc62bac740 20 client.16134 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc62bac740 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5640a1944a10 0x5640a1946e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc62bac740 4 client.16134 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc62bac740 1 -- 172.21.15.94:0/119078237 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640a182f930 con 0x5640a1944a10 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.274+0000 7efc5dd52640 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5640a1944a10 0x5640a1946e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.275+0000 7efc417fa640 10 client.16134 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.275+0000 7efc5dd52640 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5640a1944a10 0x5640a1946e00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2402799facf847a server_cookie=104bd8c385d90561 in_seq=0 out_seq=0 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.275+0000 7efc417fa640 1 -- 172.21.15.94:0/119078237 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640a182f930 con 0x5640a1944a10 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.275+0000 7efc417fa640 10 client.16134 handle_command_reply: tid=0 2024-04-21T08:41:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.275+0000 7efc62bac740 1 client.16134 shutdown 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 1 -- 172.21.15.94:0/119078237 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192be80 msgr2=0x5640a1931610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192be80 0x5640a1931610 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7efc48006c30 tx=0x7efc480115d0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 1 -- 172.21.15.94:0/119078237 shutdown_connections 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5640a1944a10 0x5640a1946e00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a1934a00 0x5640a1931b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192be80 0x5640a1931610 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 1 --2- 172.21.15.94:0/119078237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192b4d0 0x5640a1934280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc417fa640 0 client.16134 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 1 -- 172.21.15.94:0/119078237 >> 172.21.15.94:0/119078237 conn(0x5640a1925ba0 msgr2=0x5640a19278f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 1 -- 172.21.15.94:0/119078237 shutdown_connections 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 1 -- 172.21.15.94:0/119078237 wait complete. 2024-04-21T08:41:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.276+0000 7efc62bac740 20 client.16134 trim_cache size 0 max 16384 2024-04-21T08:41:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc62bac740 1 Processor -- start 2024-04-21T08:41:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc62bac740 1 -- start start 2024-04-21T08:41:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc62bac740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192b4d0 0x5640a1940830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc62bac740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 0x5640a1930770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc62bac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a1934a00 0x5640a19276e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc62bac740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efc581b1c50 con 0x5640a192b4d0 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc62bac740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc581af520 con 0x5640a192be80 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc62bac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc581ad370 con 0x5640a1934a00 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc5dd52640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192b4d0 0x5640a1940830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc5dd52640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192b4d0 0x5640a1940830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54580/0 (socket says 172.21.15.94:54580) 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc5dd52640 1 -- 172.21.15.94:0/2412893057 learned_addr learned my addr 172.21.15.94:0/2412893057 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc5d551640 1 --2- 172.21.15.94:0/2412893057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 0x5640a1930770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc5dd52640 1 -- 172.21.15.94:0/2412893057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a1934a00 msgr2=0x5640a19276e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc5dd52640 1 --2- 172.21.15.94:0/2412893057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a1934a00 0x5640a19276e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc5dd52640 1 -- 172.21.15.94:0/2412893057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 msgr2=0x5640a1930770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc5dd52640 1 --2- 172.21.15.94:0/2412893057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 0x5640a1930770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc5dd52640 1 -- 172.21.15.94:0/2412893057 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640a192a820 con 0x5640a192b4d0 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.289+0000 7efc5d551640 1 --2- 172.21.15.94:0/2412893057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 0x5640a1930770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc5dd52640 1 --2- 172.21.15.94:0/2412893057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192b4d0 0x5640a1940830 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7efc500bed20 tx=0x7efc500347b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f5a932e29acba1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc437fe640 1 -- 172.21.15.94:0/2412893057 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc500bed90 con 0x5640a192b4d0 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc437fe640 1 -- 172.21.15.94:0/2412893057 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc500c3140 con 0x5640a192b4d0 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc437fe640 1 -- 172.21.15.94:0/2412893057 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc50004730 con 0x5640a192b4d0 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc62bac740 1 -- 172.21.15.94:0/2412893057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192b4d0 msgr2=0x5640a1940830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc62bac740 1 --2- 172.21.15.94:0/2412893057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192b4d0 0x5640a1940830 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7efc500bed20 tx=0x7efc500347b0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc62bac740 1 -- 172.21.15.94:0/2412893057 shutdown_connections 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc62bac740 1 --2- 172.21.15.94:0/2412893057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a1934a00 0x5640a19276e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc62bac740 1 --2- 172.21.15.94:0/2412893057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192be80 0x5640a1930770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc62bac740 1 --2- 172.21.15.94:0/2412893057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a192b4d0 0x5640a1940830 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc62bac740 1 -- 172.21.15.94:0/2412893057 >> 172.21.15.94:0/2412893057 conn(0x5640a1925ba0 msgr2=0x5640a1926fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc62bac740 1 -- 172.21.15.94:0/2412893057 shutdown_connections 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.290+0000 7efc62bac740 1 -- 172.21.15.94:0/2412893057 wait complete. 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc62bac740 1 -- start start 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc62bac740 1 Processor -- start 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc62bac740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192b4d0 0x5640a193cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc62bac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192be80 0x5640a193d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc62bac740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a192d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc62bac740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efc581ad9e0 con 0x5640a1934a00 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc62bac740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc581aee50 con 0x5640a192b4d0 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc62bac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc581ae0b0 con 0x5640a192be80 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc62bac740 10 client.0 _ll_register_callbacks cb 0x7efc4800bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc5e553640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a192d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc5e553640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a192d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54586/0 (socket says 172.21.15.94:54586) 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc5e553640 1 -- 172.21.15.94:0/3103081947 learned_addr learned my addr 172.21.15.94:0/3103081947 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc5dd52640 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192b4d0 0x5640a193cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc5d551640 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192be80 0x5640a193d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc5e553640 1 -- 172.21.15.94:0/3103081947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192be80 msgr2=0x5640a193d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc5e553640 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192be80 0x5640a193d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc5e553640 1 -- 172.21.15.94:0/3103081947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192b4d0 msgr2=0x5640a193cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.291+0000 7efc5e553640 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192b4d0 0x5640a193cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc5e553640 1 -- 172.21.15.94:0/3103081947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640a192df00 con 0x5640a1934a00 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc5dd52640 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192b4d0 0x5640a193cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc5e553640 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a192d8c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7efc5400cb70 tx=0x7efc540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1d56764fbc6dda server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc417fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc417fa640 1 -- 172.21.15.94:0/3103081947 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc54022070 con 0x5640a1934a00 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc417fa640 1 -- 172.21.15.94:0/3103081947 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7efc54004020 con 0x5640a1934a00 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc62bac740 1 -- 172.21.15.94:0/3103081947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc500c3140 con 0x5640a1934a00 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc417fa640 1 -- 172.21.15.94:0/3103081947 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efc5402d050 con 0x5640a1934a00 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc62bac740 10 client.16140 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc62bac740 1 -- 172.21.15.94:0/3103081947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5640a19aa860 con 0x5640a1934a00 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc417fa640 1 -- 172.21.15.94:0/3103081947 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc500c3140 con 0x5640a1934a00 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc417fa640 1 -- 172.21.15.94:0/3103081947 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efc5401d040 con 0x5640a1934a00 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc62bac740 10 client.16140 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc62bac740 10 client.16140 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc62bac740 20 client.16140 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc62bac740 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5640a19aafe0 0x5640a19ad400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc62bac740 4 client.16140 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.292+0000 7efc62bac740 1 -- 172.21.15.94:0/3103081947 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5640a189dc10 con 0x5640a19aafe0 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc5dd52640 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5640a19aafe0 0x5640a19ad400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc5dd52640 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5640a19aafe0 0x5640a19ad400 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e3e9c284803447a server_cookie=d4aacf0f6158339d in_seq=0 out_seq=0 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc417fa640 10 client.16140 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc417fa640 1 -- 172.21.15.94:0/3103081947 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5640a189dc10 con 0x5640a19aafe0 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc417fa640 10 client.16140 handle_command_reply: tid=0 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 client.16140 shutdown 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 -- 172.21.15.94:0/3103081947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 msgr2=0x5640a192d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a192d8c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7efc5400cb70 tx=0x7efc540046c0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 -- 172.21.15.94:0/3103081947 shutdown_connections 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5640a19aafe0 0x5640a19ad400 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640a192be80 0x5640a193d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5640a1934a00 0x5640a192d8c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc417fa640 0 client.16140 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:41:09.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 --2- 172.21.15.94:0/3103081947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a192b4d0 0x5640a193cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 -- 172.21.15.94:0/3103081947 >> 172.21.15.94:0/3103081947 conn(0x5640a1925ba0 msgr2=0x5640a1927860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 -- 172.21.15.94:0/3103081947 shutdown_connections 2024-04-21T08:41:09.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.293+0000 7efc62bac740 1 -- 172.21.15.94:0/3103081947 wait complete. 2024-04-21T08:41:09.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc62bac740 20 client.16140 trim_cache size 0 max 16384 2024-04-21T08:41:09.306 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:41:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3968869210 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc440a3f50 msgr2=0x7efc440a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc440a3f50 0x7efc440a6410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efc5400cfd0 tx=0x7efc5400c8b0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3968869210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc58102190 msgr2=0x7efc5819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc58102190 0x7efc5819e3f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7efc500098d0 tx=0x7efc50035d60 comp rx=0 tx=0).stop 2024-04-21T08:41:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3968869210 shutdown_connections 2024-04-21T08:41:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc440a3f50 0x7efc440a6410 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58107260 0x7efc5819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc581985a0 0x7efc58198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 --2- 172.21.15.94:0/3968869210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc58102190 0x7efc5819e3f0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3968869210 >> 172.21.15.94:0/3968869210 conn(0x7efc580fd990 msgr2=0x7efc580fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3968869210 shutdown_connections 2024-04-21T08:41:09.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.294+0000 7efc5ffb6640 1 -- 172.21.15.94:0/3968869210 wait complete. 2024-04-21T08:41:09.309 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:41:09.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.409+0000 7f5ecbd52640 1 -- 172.21.15.94:0/4142247025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ec4106160 msgr2=0x7f5ec410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.409+0000 7f5ecbd52640 1 --2- 172.21.15.94:0/4142247025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ec4106160 0x7f5ec410dd30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc009ca0 tx=0x7f5ebc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 -- 172.21.15.94:0/4142247025 shutdown_connections 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 --2- 172.21.15.94:0/4142247025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ec4106160 0x7f5ec410dd30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 --2- 172.21.15.94:0/4142247025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ec4101a40 0x7f5ec4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 --2- 172.21.15.94:0/4142247025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec4101090 0x7f5ec4101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 -- 172.21.15.94:0/4142247025 >> 172.21.15.94:0/4142247025 conn(0x7f5ec4077f50 msgr2=0x7f5ec4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 -- 172.21.15.94:0/4142247025 shutdown_connections 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 -- 172.21.15.94:0/4142247025 wait complete. 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 Processor -- start 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 -- start start 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ec4101090 0x7f5ec4197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec4106160 0x7f5ec4197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ec419beb0 0x7f5ec419c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec4111120 con 0x7f5ec419beb0 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec4110fa0 con 0x7f5ec4101090 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5ecbd52640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec41112a0 con 0x7f5ec4106160 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5eca2ef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ec419beb0 0x7f5ec419c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5eca2ef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ec419beb0 0x7f5ec419c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54602/0 (socket says 172.21.15.94:54602) 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.410+0000 7f5eca2ef640 1 -- 172.21.15.94:0/1438227555 learned_addr learned my addr 172.21.15.94:0/1438227555 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5ec9aee640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ec4101090 0x7f5ec4197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec4106160 0x7f5ec4197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5eca2ef640 1 -- 172.21.15.94:0/1438227555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec4106160 msgr2=0x7f5ec4197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5eca2ef640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec4106160 0x7f5ec4197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5eca2ef640 1 -- 172.21.15.94:0/1438227555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ec4101090 msgr2=0x7f5ec4197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5eca2ef640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ec4101090 0x7f5ec4197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5eca2ef640 1 -- 172.21.15.94:0/1438227555 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec419c9a0 con 0x7f5ec419beb0 2024-04-21T08:41:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5eca2ef640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ec419beb0 0x7f5ec419c360 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0098d0 tx=0x7f5ebc035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5ebaffd640 1 -- 172.21.15.94:0/1438227555 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc039040 con 0x7f5ec419beb0 2024-04-21T08:41:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5ecbd52640 1 -- 172.21.15.94:0/1438227555 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ec419cc30 con 0x7f5ec419beb0 2024-04-21T08:41:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5ebaffd640 1 -- 172.21.15.94:0/1438227555 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ebc044070 con 0x7f5ec419beb0 2024-04-21T08:41:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5ebaffd640 1 -- 172.21.15.94:0/1438227555 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc0042a0 con 0x7f5ec419beb0 2024-04-21T08:41:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.411+0000 7f5ecbd52640 1 -- 172.21.15.94:0/1438227555 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ec41a43f0 con 0x7f5ec419beb0 2024-04-21T08:41:09.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.414+0000 7f5ebaffd640 1 -- 172.21.15.94:0/1438227555 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 24) ==== 138930+0+0 (secure 0 0 0) 0x7f5ebc004440 con 0x7f5ec419beb0 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.414+0000 7f5ecea22740 1 Processor -- start 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.414+0000 7f5ecea22740 1 -- start start 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eace80 0x55b6a4eb5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ecea22740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4e111f0 con 0x55b6a4eace80 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ecea22740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4dae4b0 con 0x55b6a4eb5a00 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ecea22740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4e0d1b0 con 0x55b6a4eac4d0 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec9aee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eace80 0x55b6a4eb5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec9aee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eace80 0x55b6a4eb5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54614/0 (socket says 172.21.15.94:54614) 2024-04-21T08:41:09.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec9aee640 1 -- 172.21.15.94:0/3225267467 learned_addr learned my addr 172.21.15.94:0/3225267467 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ebaffd640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5eb00a3ed0 0x7f5eb00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/3225267467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/3225267467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec9aee640 1 -- 172.21.15.94:0/3225267467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eac4d0 msgr2=0x55b6a4eac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec9aee640 1 --2- 172.21.15.94:0/3225267467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5eca2ef640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5eb00a3ed0 0x7f5eb00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec9aee640 1 -- 172.21.15.94:0/3225267467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 msgr2=0x55b6a4eb7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec9aee640 1 --2- 172.21.15.94:0/3225267467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec9aee640 1 -- 172.21.15.94:0/3225267467 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6a4da01b0 con 0x55b6a4eace80 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/3225267467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ebaffd640 1 -- 172.21.15.94:0/1438227555 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5ebc0f32d0 con 0x7f5ec419beb0 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5ec9aee640 1 --2- 172.21.15.94:0/3225267467 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eace80 0x55b6a4eb5430 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400ad00 tx=0x7f5eb400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979af1a71c74b653 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5eaffff640 1 -- 172.21.15.94:0/3225267467 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb401d070 con 0x55b6a4eace80 2024-04-21T08:41:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5eaffff640 1 -- 172.21.15.94:0/3225267467 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5eb4018af0 con 0x55b6a4eace80 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.415+0000 7f5eaffff640 1 -- 172.21.15.94:0/3225267467 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4010ae0 con 0x55b6a4eace80 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.416+0000 7f5ecea22740 1 -- 172.21.15.94:0/3225267467 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eace80 msgr2=0x55b6a4eb5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.416+0000 7f5eca2ef640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5eb00a3ed0 0x7f5eb00a6390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0f0570 tx=0x7f5ebc0907a0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.416+0000 7f5ecea22740 1 --2- 172.21.15.94:0/3225267467 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eace80 0x55b6a4eb5430 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400ad00 tx=0x7f5eb400f790 comp rx=0 tx=0).stop 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.416+0000 7f5ecea22740 1 -- 172.21.15.94:0/3225267467 shutdown_connections 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.416+0000 7f5ecea22740 1 --2- 172.21.15.94:0/3225267467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.416+0000 7f5ecea22740 1 --2- 172.21.15.94:0/3225267467 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eace80 0x55b6a4eb5430 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.416+0000 7f5ecea22740 1 --2- 172.21.15.94:0/3225267467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.416+0000 7f5ecea22740 1 -- 172.21.15.94:0/3225267467 >> 172.21.15.94:0/3225267467 conn(0x55b6a4ea6ba0 msgr2=0x55b6a4ea6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.416+0000 7f5ecea22740 1 -- 172.21.15.94:0/3225267467 shutdown_connections 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.416+0000 7f5ecea22740 1 -- 172.21.15.94:0/3225267467 wait complete. 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ecea22740 1 -- start start 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ecea22740 1 Processor -- start 2024-04-21T08:41:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eb5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eace80 0x55b6a4eb26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ecea22740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4eb8970 con 0x55b6a4eac4d0 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ecea22740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4dca500 con 0x55b6a4eb5a00 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ecea22740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4eb8af0 con 0x55b6a4eace80 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ecea22740 10 client.0 _ll_register_callbacks cb 0x55b6a4d98b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ec92ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eb5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ec92ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eb5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54620/0 (socket says 172.21.15.94:54620) 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ec92ed640 1 -- 172.21.15.94:0/3222148532 learned_addr learned my addr 172.21.15.94:0/3222148532 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ec9aee640 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eace80 0x55b6a4eb26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ec92ed640 1 -- 172.21.15.94:0/3222148532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eace80 msgr2=0x55b6a4eb26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eace80 0x55b6a4eb26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ec92ed640 1 -- 172.21.15.94:0/3222148532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 msgr2=0x55b6a4eb2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.417+0000 7f5ec92ed640 1 -- 172.21.15.94:0/3222148532 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6a4b233f0 con 0x55b6a4eac4d0 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eb5330 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000ced0 tx=0x7f5ec00047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c88319f30cdb85 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ead7fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ead7fa640 1 -- 172.21.15.94:0/3222148532 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0011b70 con 0x55b6a4eac4d0 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ead7fa640 1 -- 172.21.15.94:0/3222148532 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5ec0011d10 con 0x55b6a4eac4d0 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ecea22740 1 -- 172.21.15.94:0/3222148532 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5eb4018af0 con 0x55b6a4eac4d0 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ead7fa640 1 -- 172.21.15.94:0/3222148532 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5ec0026050 con 0x55b6a4eac4d0 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ecea22740 10 client.16152 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ecea22740 1 -- 172.21.15.94:0/3222148532 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b6a4bed7d0 con 0x55b6a4eac4d0 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ead7fa640 1 -- 172.21.15.94:0/3222148532 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5eb4018af0 con 0x55b6a4eac4d0 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ead7fa640 1 -- 172.21.15.94:0/3222148532 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5ec00213b0 con 0x55b6a4eac4d0 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ecea22740 10 client.16152 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ecea22740 10 client.16152 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ecea22740 20 client.16152 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ecea22740 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55b6a4ec5cb0 0x55b6a4ec8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ecea22740 4 client.16152 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ecea22740 1 -- 172.21.15.94:0/3222148532 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6a4db0930 con 0x55b6a4ec5cb0 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.418+0000 7f5ec9aee640 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55b6a4ec5cb0 0x55b6a4ec8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.419+0000 7f5ead7fa640 10 client.16152 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.419+0000 7f5ec9aee640 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55b6a4ec5cb0 0x55b6a4ec8110 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4024a903cd16ede7 server_cookie=fb1a7abf227cd78 in_seq=0 out_seq=0 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ead7fa640 1 -- 172.21.15.94:0/3222148532 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b6a4db0930 con 0x55b6a4ec5cb0 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ead7fa640 10 client.16152 handle_command_reply: tid=0 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 client.16152 shutdown 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 -- 172.21.15.94:0/3222148532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 msgr2=0x55b6a4eb5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eb5330 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000ced0 tx=0x7f5ec00047d0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 -- 172.21.15.94:0/3222148532 shutdown_connections 2024-04-21T08:41:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eace80 0x55b6a4eb26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55b6a4ec5cb0 0x55b6a4ec8110 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 --2- 172.21.15.94:0/3222148532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eb5330 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ead7fa640 0 client.16152 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:41:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 -- 172.21.15.94:0/3222148532 >> 172.21.15.94:0/3222148532 conn(0x55b6a4ea6ba0 msgr2=0x55b6a4ea88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 -- 172.21.15.94:0/3222148532 shutdown_connections 2024-04-21T08:41:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 1 -- 172.21.15.94:0/3222148532 wait complete. 2024-04-21T08:41:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.420+0000 7f5ecea22740 20 client.16152 trim_cache size 0 max 16384 2024-04-21T08:41:09.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ecea22740 1 Processor -- start 2024-04-21T08:41:09.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ecea22740 1 -- start start 2024-04-21T08:41:09.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4ec1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eace80 0x55b6a4ea5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ecea22740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec41b0b60 con 0x55b6a4eac4d0 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ecea22740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec41ae430 con 0x55b6a4eace80 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ecea22740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec41ac280 con 0x55b6a4eb5a00 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ec92ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4ec1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ec92ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4ec1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54628/0 (socket says 172.21.15.94:54628) 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.433+0000 7f5ec92ed640 1 -- 172.21.15.94:0/1498198287 learned_addr learned my addr 172.21.15.94:0/1498198287 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ec92ed640 1 -- 172.21.15.94:0/1498198287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eb5a00 msgr2=0x55b6a4eb4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/1498198287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ec92ed640 1 -- 172.21.15.94:0/1498198287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eace80 msgr2=0x55b6a4ea5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/1498198287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eace80 0x55b6a4ea5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ec92ed640 1 -- 172.21.15.94:0/1498198287 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6a4ec4850 con 0x55b6a4eac4d0 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/1498198287 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4ec1890 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0000c00 tx=0x7f5ec000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c332532f2e5012 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5eaf7fe640 1 -- 172.21.15.94:0/1498198287 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec000c4f0 con 0x55b6a4eac4d0 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5eaf7fe640 1 -- 172.21.15.94:0/1498198287 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ec001d070 con 0x55b6a4eac4d0 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5eaf7fe640 1 -- 172.21.15.94:0/1498198287 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0018480 con 0x55b6a4eac4d0 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ecea22740 1 -- 172.21.15.94:0/1498198287 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 msgr2=0x55b6a4ec1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ecea22740 1 --2- 172.21.15.94:0/1498198287 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4ec1890 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0000c00 tx=0x7f5ec000cd50 comp rx=0 tx=0).stop 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ecea22740 1 -- 172.21.15.94:0/1498198287 shutdown_connections 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ecea22740 1 --2- 172.21.15.94:0/1498198287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eb5a00 0x55b6a4eb4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ecea22740 1 --2- 172.21.15.94:0/1498198287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eace80 0x55b6a4ea5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ecea22740 1 --2- 172.21.15.94:0/1498198287 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eac4d0 0x55b6a4ec1890 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ecea22740 1 -- 172.21.15.94:0/1498198287 >> 172.21.15.94:0/1498198287 conn(0x55b6a4ea6ba0 msgr2=0x55b6a4eb16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.434+0000 7f5ecea22740 1 -- 172.21.15.94:0/1498198287 shutdown_connections 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ecea22740 1 -- 172.21.15.94:0/1498198287 wait complete. 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ecea22740 1 -- start start 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ecea22740 1 Processor -- start 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eb8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eace80 0x55b6a4eb9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ecea22740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eb5a00 0x55b6a4f2e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ecea22740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec41ac8f0 con 0x55b6a4eb5a00 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ecea22740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec41add60 con 0x55b6a4eac4d0 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ecea22740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec41acfc0 con 0x55b6a4eace80 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ecea22740 10 client.0 _ll_register_callbacks cb 0x7f5ec00165a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ec92ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eb5a00 0x55b6a4f2e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ec92ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eb5a00 0x55b6a4f2e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54640/0 (socket says 172.21.15.94:54640) 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ec92ed640 1 -- 172.21.15.94:0/2252760428 learned_addr learned my addr 172.21.15.94:0/2252760428 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ec9aee640 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eace80 0x55b6a4eb9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.435+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eb8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ec92ed640 1 -- 172.21.15.94:0/2252760428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eace80 msgr2=0x55b6a4eb9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eace80 0x55b6a4eb9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ec92ed640 1 -- 172.21.15.94:0/2252760428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eac4d0 msgr2=0x55b6a4eb8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eb8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ec92ed640 1 -- 172.21.15.94:0/2252760428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6a4f2ebe0 con 0x55b6a4eb5a00 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ec92ed640 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eb5a00 0x55b6a4f2e650 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0011730 tx=0x7f5ec0011bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb0dcdac226b605 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ead7fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ead7fa640 1 -- 172.21.15.94:0/2252760428 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0021bf0 con 0x55b6a4eb5a00 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ecea22740 1 -- 172.21.15.94:0/2252760428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ec001d070 con 0x55b6a4eb5a00 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ead7fa640 1 -- 172.21.15.94:0/2252760428 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5ec0021d90 con 0x55b6a4eb5a00 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ead7fa640 1 -- 172.21.15.94:0/2252760428 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5ec00059c0 con 0x55b6a4eb5a00 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ecea22740 10 client.16158 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ecea22740 1 -- 172.21.15.94:0/2252760428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b6a4f2fee0 con 0x55b6a4eb5a00 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ead7fa640 1 -- 172.21.15.94:0/2252760428 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ec001d070 con 0x55b6a4eb5a00 2024-04-21T08:41:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ead7fa640 1 -- 172.21.15.94:0/2252760428 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5ec00065d0 con 0x55b6a4eb5a00 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ecea22740 10 client.16158 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ecea22740 10 client.16158 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ecea22740 20 client.16158 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ecea22740 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55b6a4f30490 0x55b6a4f32880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ecea22740 4 client.16158 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.436+0000 7f5ecea22740 1 -- 172.21.15.94:0/2252760428 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b6a4e1ec10 con 0x55b6a4f30490 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ec9aee640 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55b6a4f30490 0x55b6a4f32880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ead7fa640 10 client.16158 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ec9aee640 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55b6a4f30490 0x55b6a4f32880 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a4c7cd185fb74726 server_cookie=69746eab904b747d in_seq=0 out_seq=0 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ead7fa640 1 -- 172.21.15.94:0/2252760428 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b6a4e1ec10 con 0x55b6a4f30490 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ead7fa640 10 client.16158 handle_command_reply: tid=0 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ecea22740 1 client.16158 shutdown 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ecea22740 1 -- 172.21.15.94:0/2252760428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eb5a00 msgr2=0x55b6a4f2e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ecea22740 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eb5a00 0x55b6a4f2e650 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0011730 tx=0x7f5ec0011bd0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ecea22740 1 -- 172.21.15.94:0/2252760428 shutdown_connections 2024-04-21T08:41:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ecea22740 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6a4eace80 0x55b6a4eb9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.437+0000 7f5ecea22740 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55b6a4f30490 0x55b6a4f32880 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecea22740 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b6a4eb5a00 0x55b6a4f2e650 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecea22740 1 --2- 172.21.15.94:0/2252760428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4eac4d0 0x55b6a4eb8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecea22740 1 -- 172.21.15.94:0/2252760428 >> 172.21.15.94:0/2252760428 conn(0x55b6a4ea6ba0 msgr2=0x55b6a4eb16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ead7fa640 0 client.16158 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:41:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecea22740 1 -- 172.21.15.94:0/2252760428 shutdown_connections 2024-04-21T08:41:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecea22740 1 -- 172.21.15.94:0/2252760428 wait complete. 2024-04-21T08:41:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecea22740 20 client.16158 trim_cache size 0 max 16384 2024-04-21T08:41:09.445 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:41:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 -- 172.21.15.94:0/1438227555 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5eb00a3ed0 msgr2=0x7f5eb00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5eb00a3ed0 0x7f5eb00a6390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0f0570 tx=0x7f5ebc0907a0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 -- 172.21.15.94:0/1438227555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ec419beb0 msgr2=0x7f5ec419c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ec419beb0 0x7f5ec419c360 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0098d0 tx=0x7f5ebc035d60 comp rx=0 tx=0).stop 2024-04-21T08:41:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 -- 172.21.15.94:0/1438227555 shutdown_connections 2024-04-21T08:41:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5eb00a3ed0 0x7f5eb00a6390 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec4106160 0x7f5ec4197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ec419beb0 0x7f5ec419c360 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 --2- 172.21.15.94:0/1438227555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ec4101090 0x7f5ec4197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 -- 172.21.15.94:0/1438227555 >> 172.21.15.94:0/1438227555 conn(0x7f5ec4077f50 msgr2=0x7f5ec40fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 -- 172.21.15.94:0/1438227555 shutdown_connections 2024-04-21T08:41:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.438+0000 7f5ecbd52640 1 -- 172.21.15.94:0/1438227555 wait complete. 2024-04-21T08:41:09.447 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:09 smithi094 ceph-mon[27988]: pgmap v38: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 110 MiB/s wr, 51 op/s 2024-04-21T08:41:09.447 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:09 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2165629449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:09.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:41:09.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.549+0000 7f9b86f75640 1 -- 172.21.15.94:0/783185187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b80101a40 msgr2=0x7f9b80105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.549+0000 7f9b86f75640 1 --2- 172.21.15.94:0/783185187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b80101a40 0x7f9b80105a80 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009900 tx=0x7f9b6c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.549+0000 7f9b86f75640 1 -- 172.21.15.94:0/783185187 shutdown_connections 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.549+0000 7f9b86f75640 1 --2- 172.21.15.94:0/783185187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b80106160 0x7f9b8010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.549+0000 7f9b86f75640 1 --2- 172.21.15.94:0/783185187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b80101a40 0x7f9b80105a80 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.549+0000 7f9b86f75640 1 --2- 172.21.15.94:0/783185187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b80101090 0x7f9b80101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.549+0000 7f9b86f75640 1 -- 172.21.15.94:0/783185187 >> 172.21.15.94:0/783185187 conn(0x7f9b800771d0 msgr2=0x7f9b800775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.549+0000 7f9b86f75640 1 -- 172.21.15.94:0/783185187 shutdown_connections 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.549+0000 7f9b86f75640 1 -- 172.21.15.94:0/783185187 wait complete. 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b86f75640 1 Processor -- start 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b86f75640 1 -- start start 2024-04-21T08:41:09.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b86f75640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b80101090 0x7f9b80197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b86f75640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b80106160 0x7f9b80197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b86f75640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8019bf40 0x7f9b8019c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b86f75640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801110a0 con 0x7f9b80101090 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b86f75640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9b80110f20 con 0x7f9b8019bf40 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b86f75640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9b80111220 con 0x7f9b80106160 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b84d11640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b80101090 0x7f9b80197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b84d11640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b80101090 0x7f9b80197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54664/0 (socket says 172.21.15.94:54664) 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b84d11640 1 -- 172.21.15.94:0/3733707760 learned_addr learned my addr 172.21.15.94:0/3733707760 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.550+0000 7f9b7ffff640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b80106160 0x7f9b80197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b85512640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8019bf40 0x7f9b8019c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b84d11640 1 -- 172.21.15.94:0/3733707760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b80106160 msgr2=0x7f9b80197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b84d11640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b80106160 0x7f9b80197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b84d11640 1 -- 172.21.15.94:0/3733707760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8019bf40 msgr2=0x7f9b8019c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b84d11640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8019bf40 0x7f9b8019c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b84d11640 1 -- 172.21.15.94:0/3733707760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8019ca30 con 0x7f9b80101090 2024-04-21T08:41:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b85512640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8019bf40 0x7f9b8019c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b7ffff640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b80106160 0x7f9b80197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b84d11640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b80101090 0x7f9b80197680 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9b7400eb00 tx=0x7f9b7400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b7dffb640 1 -- 172.21.15.94:0/3733707760 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7400bc30 con 0x7f9b80101090 2024-04-21T08:41:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b86f75640 1 -- 172.21.15.94:0/3733707760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b8019cd20 con 0x7f9b80101090 2024-04-21T08:41:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b7dffb640 1 -- 172.21.15.94:0/3733707760 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b7400bdd0 con 0x7f9b80101090 2024-04-21T08:41:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b7dffb640 1 -- 172.21.15.94:0/3733707760 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74017410 con 0x7f9b80101090 2024-04-21T08:41:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.551+0000 7f9b86f75640 1 -- 172.21.15.94:0/3733707760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b8006a850 con 0x7f9b80101090 2024-04-21T08:41:09.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.553+0000 7f9b7dffb640 1 -- 172.21.15.94:0/3733707760 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 24) ==== 138930+0+0 (secure 0 0 0) 0x7f9b74004f70 con 0x7f9b80101090 2024-04-21T08:41:09.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b89b65740 1 Processor -- start 2024-04-21T08:41:09.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b89b65740 1 -- start start 2024-04-21T08:41:09.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b7dffb640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9b700a3ed0 0x7f9b700a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef74d0 0x55a37cef78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef7e80 0x55a37cf00430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b85512640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9b700a3ed0 0x7f9b700a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 0x55a37cf02e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b89b65740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55a37ce5c1f0 con 0x55a37cef74d0 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b89b65740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a37cdf94b0 con 0x55a37cef7e80 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b89b65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a37ce581b0 con 0x55a37cf00a00 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b7dffb640 1 -- 172.21.15.94:0/3733707760 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9b740d28d0 con 0x7f9b80101090 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b84d11640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef74d0 0x55a37cef78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b84d11640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef74d0 0x55a37cef78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54670/0 (socket says 172.21.15.94:54670) 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b84d11640 1 -- 172.21.15.94:0/3338302168 learned_addr learned my addr 172.21.15.94:0/3338302168 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b85512640 1 --2- 172.21.15.94:0/3338302168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 0x55a37cf02e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b7ffff640 1 --2- 172.21.15.94:0/3338302168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef7e80 0x55a37cf00430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b84d11640 1 -- 172.21.15.94:0/3338302168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 msgr2=0x55a37cf02e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b84d11640 1 --2- 172.21.15.94:0/3338302168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 0x55a37cf02e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b84d11640 1 -- 172.21.15.94:0/3338302168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef7e80 msgr2=0x55a37cf00430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b84d11640 1 --2- 172.21.15.94:0/3338302168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef7e80 0x55a37cf00430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b84d11640 1 -- 172.21.15.94:0/3338302168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a37cdeb1b0 con 0x55a37cef74d0 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b7ffff640 1 --2- 172.21.15.94:0/3338302168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef7e80 0x55a37cf00430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b85512640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9b700a3ed0 0x7f9b700a6390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9b78005fd0 tx=0x7f9b78015a80 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.554+0000 7f9b84d11640 1 --2- 172.21.15.94:0/3338302168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef74d0 0x55a37cef78b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9b7409e1e0 tx=0x7f9b7409ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b726131e6ccceb5f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b6affd640 1 -- 172.21.15.94:0/3338302168 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7409c2f0 con 0x55a37cef74d0 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b6affd640 1 -- 172.21.15.94:0/3338302168 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b7409c550 con 0x55a37cef74d0 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b6affd640 1 -- 172.21.15.94:0/3338302168 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b740a2120 con 0x55a37cef74d0 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b89b65740 1 -- 172.21.15.94:0/3338302168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef74d0 msgr2=0x55a37cef78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b89b65740 1 --2- 172.21.15.94:0/3338302168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef74d0 0x55a37cef78b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9b7409e1e0 tx=0x7f9b7409ea60 comp rx=0 tx=0).stop 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b89b65740 1 -- 172.21.15.94:0/3338302168 shutdown_connections 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b89b65740 1 --2- 172.21.15.94:0/3338302168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 0x55a37cf02e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b89b65740 1 --2- 172.21.15.94:0/3338302168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef7e80 0x55a37cf00430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b89b65740 1 --2- 172.21.15.94:0/3338302168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef74d0 0x55a37cef78b0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b89b65740 1 -- 172.21.15.94:0/3338302168 >> 172.21.15.94:0/3338302168 conn(0x55a37cef1ba0 msgr2=0x55a37cef1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b89b65740 1 -- 172.21.15.94:0/3338302168 shutdown_connections 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.555+0000 7f9b89b65740 1 -- 172.21.15.94:0/3338302168 wait complete. 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b89b65740 1 -- start start 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b89b65740 1 Processor -- start 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef74d0 0x55a37cf00340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cefd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 0x55a37cefdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b89b65740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55a37cf03970 con 0x55a37cef7e80 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b89b65740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a37ce15500 con 0x55a37cef74d0 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b89b65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a37cf03af0 con 0x55a37cf00a00 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b89b65740 10 client.0 _ll_register_callbacks cb 0x55a37cde3b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b7ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cefd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b7ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cefd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54678/0 (socket says 172.21.15.94:54678) 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b7ffff640 1 -- 172.21.15.94:0/1508362468 learned_addr learned my addr 172.21.15.94:0/1508362468 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b84d11640 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef74d0 0x55a37cf00340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b85512640 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 0x55a37cefdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b7ffff640 1 -- 172.21.15.94:0/1508362468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 msgr2=0x55a37cefdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b7ffff640 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 0x55a37cefdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b7ffff640 1 -- 172.21.15.94:0/1508362468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef74d0 msgr2=0x55a37cf00340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b7ffff640 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef74d0 0x55a37cf00340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b7ffff640 1 -- 172.21.15.94:0/1508362468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a37cb6e3f0 con 0x55a37cef7e80 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b85512640 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 0x55a37cefdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.556+0000 7f9b84d11640 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef74d0 0x55a37cf00340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b7ffff640 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cefd6d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c005ab0 tx=0x7f9b6c02ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981b0bad4ecd9038 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b5ffff640 1 -- 172.21.15.94:0/1508362468 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c044070 con 0x55a37cef7e80 2024-04-21T08:41:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b5ffff640 1 -- 172.21.15.94:0/1508362468 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9b6c005450 con 0x55a37cef7e80 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b89b65740 1 -- 172.21.15.94:0/1508362468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b7409c550 con 0x55a37cef7e80 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b5ffff640 1 -- 172.21.15.94:0/1508362468 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9b6c050050 con 0x55a37cef7e80 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b89b65740 10 client.16170 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b89b65740 1 -- 172.21.15.94:0/1508362468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a37cc387d0 con 0x55a37cef7e80 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b5ffff640 1 -- 172.21.15.94:0/1508362468 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b7409c550 con 0x55a37cef7e80 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b5ffff640 1 -- 172.21.15.94:0/1508362468 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9b6c0035f0 con 0x55a37cef7e80 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b89b65740 10 client.16170 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b89b65740 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b89b65740 20 client.16170 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b89b65740 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a37cf10b40 0x55a37cf12fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b89b65740 4 client.16170 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.557+0000 7f9b89b65740 1 -- 172.21.15.94:0/1508362468 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a37cdfb930 con 0x55a37cf10b40 2024-04-21T08:41:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.558+0000 7f9b84d11640 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a37cf10b40 0x55a37cf12fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.558+0000 7f9b5ffff640 10 client.16170 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.558+0000 7f9b84d11640 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a37cf10b40 0x55a37cf12fb0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c2e4458925a0dc2 server_cookie=2a7b94d384e1ee83 in_seq=0 out_seq=0 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b5ffff640 1 -- 172.21.15.94:0/1508362468 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a37cdfb930 con 0x55a37cf10b40 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b5ffff640 10 client.16170 handle_command_reply: tid=0 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 client.16170 shutdown 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 -- 172.21.15.94:0/1508362468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 msgr2=0x55a37cefd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cefd6d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c005ab0 tx=0x7f9b6c02ca90 comp rx=0 tx=0).stop 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 -- 172.21.15.94:0/1508362468 shutdown_connections 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a37cf10b40 0x55a37cf12fb0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cf00a00 0x55a37cefdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cefd6d0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 --2- 172.21.15.94:0/1508362468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef74d0 0x55a37cf00340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b5ffff640 0 client.16170 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 -- 172.21.15.94:0/1508362468 >> 172.21.15.94:0/1508362468 conn(0x55a37cef1ba0 msgr2=0x55a37cef38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 -- 172.21.15.94:0/1508362468 shutdown_connections 2024-04-21T08:41:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 1 -- 172.21.15.94:0/1508362468 wait complete. 2024-04-21T08:41:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.559+0000 7f9b89b65740 20 client.16170 trim_cache size 0 max 16384 2024-04-21T08:41:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b89b65740 1 Processor -- start 2024-04-21T08:41:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b89b65740 1 -- start start 2024-04-21T08:41:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cef74d0 0x55a37cf0c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cef3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cf00a00 0x55a37ceffe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b89b65740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801afac0 con 0x55a37cef7e80 2024-04-21T08:41:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b89b65740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801ad390 con 0x55a37cf00a00 2024-04-21T08:41:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b89b65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801ab1e0 con 0x55a37cef74d0 2024-04-21T08:41:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b7ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cef3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b7ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cef3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54690/0 (socket says 172.21.15.94:54690) 2024-04-21T08:41:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.572+0000 7f9b7ffff640 1 -- 172.21.15.94:0/233699604 learned_addr learned my addr 172.21.15.94:0/233699604 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b7ffff640 1 -- 172.21.15.94:0/233699604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cef74d0 msgr2=0x55a37cf0c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b7ffff640 1 --2- 172.21.15.94:0/233699604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cef74d0 0x55a37cf0c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b7ffff640 1 -- 172.21.15.94:0/233699604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cf00a00 msgr2=0x55a37ceffe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b85512640 1 --2- 172.21.15.94:0/233699604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cf00a00 0x55a37ceffe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b7ffff640 1 --2- 172.21.15.94:0/233699604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cf00a00 0x55a37ceffe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b7ffff640 1 -- 172.21.15.94:0/233699604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a37cefe2e0 con 0x55a37cef7e80 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b85512640 1 --2- 172.21.15.94:0/233699604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cf00a00 0x55a37ceffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b7ffff640 1 --2- 172.21.15.94:0/233699604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cef3630 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009900 tx=0x7f9b6c009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2cef1b6f5853817 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b6a7fc640 1 -- 172.21.15.94:0/233699604 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c04d070 con 0x55a37cef7e80 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b6a7fc640 1 -- 172.21.15.94:0/233699604 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b6c02c5d0 con 0x55a37cef7e80 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.573+0000 7f9b6a7fc640 1 -- 172.21.15.94:0/233699604 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c044040 con 0x55a37cef7e80 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 -- 172.21.15.94:0/233699604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 msgr2=0x55a37cef3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 --2- 172.21.15.94:0/233699604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cef3630 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009900 tx=0x7f9b6c009660 comp rx=0 tx=0).stop 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 -- 172.21.15.94:0/233699604 shutdown_connections 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 --2- 172.21.15.94:0/233699604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cf00a00 0x55a37ceffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 --2- 172.21.15.94:0/233699604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cef7e80 0x55a37cef3630 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 --2- 172.21.15.94:0/233699604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cef74d0 0x55a37cf0c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 -- 172.21.15.94:0/233699604 >> 172.21.15.94:0/233699604 conn(0x55a37cef1ba0 msgr2=0x55a37cef2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 -- 172.21.15.94:0/233699604 shutdown_connections 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 -- 172.21.15.94:0/233699604 wait complete. 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 -- start start 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 Processor -- start 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cef74d0 0x55a37cf049c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef7e80 0x55a37cf04f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cf00a00 0x55a37cf05440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801ab850 con 0x55a37cf00a00 2024-04-21T08:41:09.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801accc0 con 0x55a37cef7e80 2024-04-21T08:41:09.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801abf20 con 0x55a37cef74d0 2024-04-21T08:41:09.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.574+0000 7f9b89b65740 10 client.0 _ll_register_callbacks cb 0x7f9b6c004490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:09.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b85512640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cf00a00 0x55a37cf05440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b85512640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cf00a00 0x55a37cf05440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54706/0 (socket says 172.21.15.94:54706) 2024-04-21T08:41:09.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b85512640 1 -- 172.21.15.94:0/2669734155 learned_addr learned my addr 172.21.15.94:0/2669734155 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:09.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b84d11640 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cef74d0 0x55a37cf049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b85512640 1 -- 172.21.15.94:0/2669734155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cef74d0 msgr2=0x55a37cf049c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b85512640 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cef74d0 0x55a37cf049c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b85512640 1 -- 172.21.15.94:0/2669734155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef7e80 msgr2=0x55a37cf04f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b85512640 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef7e80 0x55a37cf04f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b85512640 1 -- 172.21.15.94:0/2669734155 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a37cf76340 con 0x55a37cf00a00 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b84d11640 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cef74d0 0x55a37cf049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b85512640 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cf00a00 0x55a37cf05440 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9b78015490 tx=0x7f9b78004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6868421207f4028b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b5ffff640 1 -- 172.21.15.94:0/2669734155 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b78026070 con 0x55a37cf00a00 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b5ffff640 1 -- 172.21.15.94:0/2669734155 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9b78013950 con 0x55a37cf00a00 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b89b65740 1 -- 172.21.15.94:0/2669734155 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b6c02c5d0 con 0x55a37cf00a00 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b5ffff640 1 -- 172.21.15.94:0/2669734155 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9b78032050 con 0x55a37cf00a00 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b89b65740 10 client.16176 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:09.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b89b65740 1 -- 172.21.15.94:0/2669734155 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a37cf775b0 con 0x55a37cf00a00 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b5ffff640 1 -- 172.21.15.94:0/2669734155 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b6c02c5d0 con 0x55a37cf00a00 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.575+0000 7f9b5ffff640 1 -- 172.21.15.94:0/2669734155 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9b7801f040 con 0x55a37cf00a00 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b89b65740 10 client.16176 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b89b65740 10 client.16176 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b89b65740 20 client.16176 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b89b65740 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a37cf77d30 0x55a37cf7a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b89b65740 4 client.16176 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b89b65740 1 -- 172.21.15.94:0/2669734155 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a37ce69c10 con 0x55a37cf77d30 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b84d11640 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a37cf77d30 0x55a37cf7a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b5ffff640 10 client.16176 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b84d11640 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a37cf77d30 0x55a37cf7a150 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2f376f0b0412df3 server_cookie=c9f9957c19a70982 in_seq=0 out_seq=0 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b5ffff640 1 -- 172.21.15.94:0/2669734155 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a37ce69c10 con 0x55a37cf77d30 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b5ffff640 10 client.16176 handle_command_reply: tid=0 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.576+0000 7f9b89b65740 1 client.16176 shutdown 2024-04-21T08:41:09.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 1 -- 172.21.15.94:0/2669734155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cf00a00 msgr2=0x55a37cf05440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cf00a00 0x55a37cf05440 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9b78015490 tx=0x7f9b78004ab0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 1 -- 172.21.15.94:0/2669734155 shutdown_connections 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a37cf77d30 0x55a37cf7a150 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a37cf00a00 0x55a37cf05440 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a37cef7e80 0x55a37cf04f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b5ffff640 0 client.16176 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 1 --2- 172.21.15.94:0/2669734155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a37cef74d0 0x55a37cf049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 1 -- 172.21.15.94:0/2669734155 >> 172.21.15.94:0/2669734155 conn(0x55a37cef1ba0 msgr2=0x55a37cef37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 1 -- 172.21.15.94:0/2669734155 shutdown_connections 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 1 -- 172.21.15.94:0/2669734155 wait complete. 2024-04-21T08:41:09.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b89b65740 20 client.16176 trim_cache size 0 max 16384 2024-04-21T08:41:09.585 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:41:09.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 -- 172.21.15.94:0/3733707760 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9b700a3ed0 msgr2=0x7f9b700a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9b700a3ed0 0x7f9b700a6390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9b78005fd0 tx=0x7f9b78015a80 comp rx=0 tx=0).stop 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 -- 172.21.15.94:0/3733707760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b80101090 msgr2=0x7f9b80197680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b80101090 0x7f9b80197680 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9b7400eb00 tx=0x7f9b7400ef00 comp rx=0 tx=0).stop 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 -- 172.21.15.94:0/3733707760 shutdown_connections 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9b700a3ed0 0x7f9b700a6390 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b80106160 0x7f9b80197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8019bf40 0x7f9b8019c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 --2- 172.21.15.94:0/3733707760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b80101090 0x7f9b80197680 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 -- 172.21.15.94:0/3733707760 >> 172.21.15.94:0/3733707760 conn(0x7f9b800771d0 msgr2=0x7f9b800fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 -- 172.21.15.94:0/3733707760 shutdown_connections 2024-04-21T08:41:09.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:09.577+0000 7f9b86f75640 1 -- 172.21.15.94:0/3733707760 wait complete. 2024-04-21T08:41:09.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:09 smithi111 ceph-mon[31493]: pgmap v38: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 110 MiB/s wr, 51 op/s 2024-04-21T08:41:09.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:09 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2165629449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:09.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:09 smithi173 ceph-mon[31428]: pgmap v38: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 110 MiB/s wr, 51 op/s 2024-04-21T08:41:09.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:09 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2165629449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:10.589 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:41:10.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:41:10.589 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:41:10.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.707+0000 7fbc77d8f640 1 -- 172.21.15.94:0/2792561011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc70074380 msgr2=0x7fbc70074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.707+0000 7fbc77d8f640 1 --2- 172.21.15.94:0/2792561011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc70074380 0x7fbc70074760 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009ca0 tx=0x7fbc6c02c180 comp rx=0 tx=0).stop 2024-04-21T08:41:10.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 -- 172.21.15.94:0/2792561011 shutdown_connections 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 --2- 172.21.15.94:0/2792561011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc70074ca0 0x7fbc7010e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 --2- 172.21.15.94:0/2792561011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc7010e760 0x7fbc70110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 --2- 172.21.15.94:0/2792561011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc70074380 0x7fbc70074760 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 -- 172.21.15.94:0/2792561011 >> 172.21.15.94:0/2792561011 conn(0x7fbc700fd990 msgr2=0x7fbc700ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 -- 172.21.15.94:0/2792561011 shutdown_connections 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 -- 172.21.15.94:0/2792561011 wait complete. 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 Processor -- start 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 -- start start 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc70074380 0x7fbc701a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.708+0000 7fbc77d8f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc70074ca0 0x7fbc701a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc77d8f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc7010e760 0x7fbc7019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc77d8f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbc701143a0 con 0x7fbc70074380 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc77d8f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbc70114220 con 0x7fbc70074ca0 2024-04-21T08:41:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc77d8f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc70114520 con 0x7fbc7010e760 2024-04-21T08:41:10.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc75b2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc70074380 0x7fbc701a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc75b2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc70074380 0x7fbc701a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54720/0 (socket says 172.21.15.94:54720) 2024-04-21T08:41:10.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc75b2b640 1 -- 172.21.15.94:0/4101456489 learned_addr learned my addr 172.21.15.94:0/4101456489 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:10.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc7532a640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc70074ca0 0x7fbc701a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc7632c640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc7010e760 0x7fbc7019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc75b2b640 1 -- 172.21.15.94:0/4101456489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc7010e760 msgr2=0x7fbc7019a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc7010e760 0x7fbc7019a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc75b2b640 1 -- 172.21.15.94:0/4101456489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc70074ca0 msgr2=0x7fbc701a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc70074ca0 0x7fbc701a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc75b2b640 1 -- 172.21.15.94:0/4101456489 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7019b040 con 0x7fbc70074380 2024-04-21T08:41:10.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc70074380 0x7fbc701a0670 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c005e30 tx=0x7fbc6c0039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc66ffd640 1 -- 172.21.15.94:0/4101456489 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c039040 con 0x7fbc70074380 2024-04-21T08:41:10.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc77d8f640 1 -- 172.21.15.94:0/4101456489 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7019b2d0 con 0x7fbc70074380 2024-04-21T08:41:10.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc66ffd640 1 -- 172.21.15.94:0/4101456489 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c044070 con 0x7fbc70074380 2024-04-21T08:41:10.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc66ffd640 1 -- 172.21.15.94:0/4101456489 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c032960 con 0x7fbc70074380 2024-04-21T08:41:10.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.709+0000 7fbc77d8f640 1 -- 172.21.15.94:0/4101456489 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc701a7410 con 0x7fbc70074380 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.712+0000 7fbc66ffd640 1 -- 172.21.15.94:0/4101456489 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 24) ==== 138930+0+0 (secure 0 0 0) 0x7fbc6c004580 con 0x7fbc70074380 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.712+0000 7fbc7a97d740 1 Processor -- start 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.712+0000 7fbc7a97d740 1 -- start start 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc66ffd640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc5c0a3ed0 0x7fbc5c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x5563499ffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 0x556349a08840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a08e10 0x556349a0b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556349963f00 con 0x5563499ff8e0 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556349901590 con 0x556349a00290 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55634995fec0 con 0x556349a08e10 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7632c640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc5c0a3ed0 0x7fbc5c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc75b2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x5563499ffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc75b2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x5563499ffcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54728/0 (socket says 172.21.15.94:54728) 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc75b2b640 1 -- 172.21.15.94:0/3618647835 learned_addr learned my addr 172.21.15.94:0/3618647835 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc66ffd640 1 -- 172.21.15.94:0/4101456489 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fbc6c0f3310 con 0x7fbc70074380 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7532a640 1 --2- 172.21.15.94:0/3618647835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 0x556349a08840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7632c640 1 --2- 172.21.15.94:0/3618647835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a08e10 0x556349a0b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc75b2b640 1 -- 172.21.15.94:0/3618647835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a08e10 msgr2=0x556349a0b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/3618647835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a08e10 0x556349a0b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc75b2b640 1 -- 172.21.15.94:0/3618647835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 msgr2=0x556349a08840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/3618647835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 0x556349a08840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc75b2b640 1 -- 172.21.15.94:0/3618647835 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55634973e7e0 con 0x5563499ff8e0 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7532a640 1 --2- 172.21.15.94:0/3618647835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 0x556349a08840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/3618647835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x5563499ffcc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0f0690 tx=0x7fbc6c003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626408eda17dca52 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.713+0000 7fbc7632c640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc5c0a3ed0 0x7fbc5c0a6390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800a500 tx=0x7fbc68013830 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc5bfff640 1 -- 172.21.15.94:0/3618647835 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0be8e0 con 0x5563499ff8e0 2024-04-21T08:41:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc5bfff640 1 -- 172.21.15.94:0/3618647835 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c0bf4a0 con 0x5563499ff8e0 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc5bfff640 1 -- 172.21.15.94:0/3618647835 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c090a90 con 0x5563499ff8e0 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc7a97d740 1 -- 172.21.15.94:0/3618647835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 msgr2=0x5563499ffcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/3618647835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x5563499ffcc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0f0690 tx=0x7fbc6c003f50 comp rx=0 tx=0).stop 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc7a97d740 1 -- 172.21.15.94:0/3618647835 shutdown_connections 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/3618647835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a08e10 0x556349a0b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/3618647835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 0x556349a08840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/3618647835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x5563499ffcc0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc7a97d740 1 -- 172.21.15.94:0/3618647835 >> 172.21.15.94:0/3618647835 conn(0x5563499f9fb0 msgr2=0x5563499fa390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc7a97d740 1 -- 172.21.15.94:0/3618647835 shutdown_connections 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.714+0000 7fbc7a97d740 1 -- 172.21.15.94:0/3618647835 wait complete. 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7a97d740 1 -- start start 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7a97d740 1 Processor -- start 2024-04-21T08:41:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 0x556349a05d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a00290 0x556349a062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556349a0bb40 con 0x556349a08e10 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55634991d580 con 0x5563499ff8e0 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556349a0bcc0 con 0x556349a00290 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7a97d740 10 client.0 _ll_register_callbacks cb 0x556349923650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7632c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.715+0000 7fbc7632c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54742/0 (socket says 172.21.15.94:54742) 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7632c640 1 -- 172.21.15.94:0/2546591947 learned_addr learned my addr 172.21.15.94:0/2546591947 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 0x556349a05d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7532a640 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a00290 0x556349a062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7632c640 1 -- 172.21.15.94:0/2546591947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a00290 msgr2=0x556349a062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7632c640 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a00290 0x556349a062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7632c640 1 -- 172.21.15.94:0/2546591947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 msgr2=0x556349a05d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7632c640 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 0x556349a05d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7632c640 1 -- 172.21.15.94:0/2546591947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55634979ef70 con 0x556349a08e10 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7632c640 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a067f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800eb50 tx=0x7fbc68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad436917945f062 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc597fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc597fa640 1 -- 172.21.15.94:0/2546591947 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc68011db0 con 0x556349a08e10 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc597fa640 1 -- 172.21.15.94:0/2546591947 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fbc68005020 con 0x556349a08e10 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2546591947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc6c0bf4a0 con 0x556349a08e10 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc597fa640 1 -- 172.21.15.94:0/2546591947 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbc68030050 con 0x556349a08e10 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7a97d740 10 client.16188 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2546591947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556349636170 con 0x556349a08e10 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.716+0000 7fbc597fa640 1 -- 172.21.15.94:0/2546591947 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c0bf4a0 con 0x556349a08e10 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.717+0000 7fbc597fa640 1 -- 172.21.15.94:0/2546591947 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbc6801c040 con 0x556349a08e10 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.717+0000 7fbc7a97d740 10 client.16188 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.717+0000 7fbc7a97d740 10 client.16188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.717+0000 7fbc7a97d740 20 client.16188 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.717+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556349a19010 0x556349a1b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.717+0000 7fbc7a97d740 4 client.16188 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.717+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2546591947 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556349971920 con 0x556349a19010 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.717+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556349a19010 0x556349a1b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.718+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556349a19010 0x556349a1b4b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98ee824c41a13807 server_cookie=7cd66a08ce3f09e6 in_seq=0 out_seq=0 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.718+0000 7fbc597fa640 10 client.16188 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.718+0000 7fbc597fa640 1 -- 172.21.15.94:0/2546591947 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556349971920 con 0x556349a19010 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.718+0000 7fbc597fa640 10 client.16188 handle_command_reply: tid=0 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.718+0000 7fbc7a97d740 1 client.16188 shutdown 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.718+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2546591947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 msgr2=0x556349a067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.718+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a067f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800eb50 tx=0x7fbc68004730 comp rx=0 tx=0).stop 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.719+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2546591947 shutdown_connections 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.719+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556349a19010 0x556349a1b4b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.719+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a00290 0x556349a062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.719+0000 7fbc597fa640 0 client.16188 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.719+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a067f0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.719+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/2546591947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 0x556349a05d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.719+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2546591947 >> 172.21.15.94:0/2546591947 conn(0x5563499f9fb0 msgr2=0x5563499fbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.719+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2546591947 shutdown_connections 2024-04-21T08:41:10.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.719+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2546591947 wait complete. 2024-04-21T08:41:10.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.719+0000 7fbc7a97d740 20 client.16188 trim_cache size 0 max 16384 2024-04-21T08:41:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.736+0000 7fbc7a97d740 1 Processor -- start 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.736+0000 7fbc7a97d740 1 -- start start 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 0x556349a084a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a00290 0x556349a148b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a04cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbc701b3e90 con 0x556349a08e10 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbc701b1760 con 0x5563499ff8e0 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc701af5b0 con 0x556349a00290 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7632c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a04cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7632c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a04cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54758/0 (socket says 172.21.15.94:54758) 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7632c640 1 -- 172.21.15.94:0/2268083667 learned_addr learned my addr 172.21.15.94:0/2268083667 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/2268083667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 0x556349a084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7632c640 1 -- 172.21.15.94:0/2268083667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a00290 msgr2=0x556349a148b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7632c640 1 --2- 172.21.15.94:0/2268083667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a00290 0x556349a148b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7632c640 1 -- 172.21.15.94:0/2268083667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 msgr2=0x556349a084a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7632c640 1 --2- 172.21.15.94:0/2268083667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 0x556349a084a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7632c640 1 -- 172.21.15.94:0/2268083667 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563499f9de0 con 0x556349a08e10 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/2268083667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 0x556349a084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc7632c640 1 --2- 172.21.15.94:0/2268083667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a04cb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbc68004f40 tx=0x7fbc6800e750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb20d013e1fa3583 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc5b7fe640 1 -- 172.21.15.94:0/2268083667 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6801e440 con 0x556349a08e10 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc5b7fe640 1 -- 172.21.15.94:0/2268083667 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6802d040 con 0x556349a08e10 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.737+0000 7fbc5b7fe640 1 -- 172.21.15.94:0/2268083667 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6801ea20 con 0x556349a08e10 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2268083667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 msgr2=0x556349a04cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/2268083667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a04cb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbc68004f40 tx=0x7fbc6800e750 comp rx=0 tx=0).stop 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2268083667 shutdown_connections 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/2268083667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a00290 0x556349a148b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/2268083667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556349a08e10 0x556349a04cb0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/2268083667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563499ff8e0 0x556349a084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2268083667 >> 172.21.15.94:0/2268083667 conn(0x5563499f9fb0 msgr2=0x5563499f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2268083667 shutdown_connections 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 -- 172.21.15.94:0/2268083667 wait complete. 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 -- start start 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.738+0000 7fbc7a97d740 1 Processor -- start 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x556349a0cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 0x556349a0d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc7a97d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a08e10 0x556349a7e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbc701afc20 con 0x5563499ff8e0 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbc701b1090 con 0x556349a00290 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc7a97d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc701b02f0 con 0x556349a08e10 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc7a97d740 10 client.0 _ll_register_callbacks cb 0x7fbc6800e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc75b2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x556349a0cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc75b2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x556349a0cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54762/0 (socket says 172.21.15.94:54762) 2024-04-21T08:41:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc75b2b640 1 -- 172.21.15.94:0/934575269 learned_addr learned my addr 172.21.15.94:0/934575269 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc75b2b640 1 -- 172.21.15.94:0/934575269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a08e10 msgr2=0x556349a7e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc7532a640 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 0x556349a0d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a08e10 0x556349a7e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc75b2b640 1 -- 172.21.15.94:0/934575269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 msgr2=0x556349a0d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 0x556349a0d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc75b2b640 1 -- 172.21.15.94:0/934575269 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556349a7ebb0 con 0x5563499ff8e0 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc75b2b640 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x556349a0cf80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0904c0 tx=0x7fbc6c02c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4aa4b8e7de71ce9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.739+0000 7fbc597fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc597fa640 1 -- 172.21.15.94:0/934575269 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0d7070 con 0x5563499ff8e0 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc597fa640 1 -- 172.21.15.94:0/934575269 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fbc6c090c80 con 0x5563499ff8e0 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc7a97d740 1 -- 172.21.15.94:0/934575269 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc6802d040 con 0x5563499ff8e0 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc597fa640 1 -- 172.21.15.94:0/934575269 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbc6c0c2070 con 0x5563499ff8e0 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc7a97d740 10 client.16194 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc7a97d740 1 -- 172.21.15.94:0/934575269 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556349a7fe40 con 0x5563499ff8e0 2024-04-21T08:41:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc597fa640 1 -- 172.21.15.94:0/934575269 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6802d040 con 0x5563499ff8e0 2024-04-21T08:41:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc597fa640 1 -- 172.21.15.94:0/934575269 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbc6c03c030 con 0x5563499ff8e0 2024-04-21T08:41:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc7a97d740 10 client.16194 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc7a97d740 10 client.16194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc7a97d740 20 client.16194 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556349a804a0 0x556349a82890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc7a97d740 4 client.16194 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:41:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc7a97d740 1 -- 172.21.15.94:0/934575269 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55634996b790 con 0x556349a804a0 2024-04-21T08:41:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.740+0000 7fbc7532a640 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556349a804a0 0x556349a82890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.741+0000 7fbc597fa640 10 client.16194 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.741+0000 7fbc7532a640 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556349a804a0 0x556349a82890 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68b2c510be861be8 server_cookie=1d8c106736e21461 in_seq=0 out_seq=0 2024-04-21T08:41:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc597fa640 1 -- 172.21.15.94:0/934575269 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55634996b790 con 0x556349a804a0 2024-04-21T08:41:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc597fa640 10 client.16194 handle_command_reply: tid=0 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 client.16194 shutdown 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 -- 172.21.15.94:0/934575269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 msgr2=0x556349a0cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x556349a0cf80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0904c0 tx=0x7fbc6c02c180 comp rx=0 tx=0).stop 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 -- 172.21.15.94:0/934575269 shutdown_connections 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556349a804a0 0x556349a82890 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556349a08e10 0x556349a7e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556349a00290 0x556349a0d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc597fa640 0 client.16194 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 --2- 172.21.15.94:0/934575269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563499ff8e0 0x556349a0cf80 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 -- 172.21.15.94:0/934575269 >> 172.21.15.94:0/934575269 conn(0x5563499f9fb0 msgr2=0x5563499fb330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 -- 172.21.15.94:0/934575269 shutdown_connections 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 1 -- 172.21.15.94:0/934575269 wait complete. 2024-04-21T08:41:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.742+0000 7fbc7a97d740 20 client.16194 trim_cache size 0 max 16384 2024-04-21T08:41:10.748 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:41:10.748 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:41:10.748 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "13701888-61e7-46e7-b913-ba56362f5b30", 2024-04-21T08:41:10.748 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:41:10.748 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:41:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 -- 172.21.15.94:0/4101456489 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc5c0a3ed0 msgr2=0x7fbc5c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc5c0a3ed0 0x7fbc5c0a6390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800a500 tx=0x7fbc68013830 comp rx=0 tx=0).stop 2024-04-21T08:41:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 -- 172.21.15.94:0/4101456489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc70074380 msgr2=0x7fbc701a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc70074380 0x7fbc701a0670 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c005e30 tx=0x7fbc6c0039f0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 -- 172.21.15.94:0/4101456489 shutdown_connections 2024-04-21T08:41:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbc5c0a3ed0 0x7fbc5c0a6390 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc7010e760 0x7fbc7019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc70074ca0 0x7fbc701a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 --2- 172.21.15.94:0/4101456489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc70074380 0x7fbc701a0670 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 -- 172.21.15.94:0/4101456489 >> 172.21.15.94:0/4101456489 conn(0x7fbc700fd990 msgr2=0x7fbc700fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 -- 172.21.15.94:0/4101456489 shutdown_connections 2024-04-21T08:41:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.743+0000 7fbc77d8f640 1 -- 172.21.15.94:0/4101456489 wait complete. 2024-04-21T08:41:10.754 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:41:10.754 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:41:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.855+0000 7f4cea5a4640 1 -- 172.21.15.94:0/2084601033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ce4107240 msgr2=0x7f4ce410ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.855+0000 7f4cea5a4640 1 --2- 172.21.15.94:0/2084601033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ce4107240 0x7f4ce410ee10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009ca0 tx=0x7f4cdc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.855+0000 7f4cea5a4640 1 -- 172.21.15.94:0/2084601033 shutdown_connections 2024-04-21T08:41:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.855+0000 7f4cea5a4640 1 --2- 172.21.15.94:0/2084601033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ce4102b20 0x7f4ce4106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.855+0000 7f4cea5a4640 1 --2- 172.21.15.94:0/2084601033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ce4107240 0x7f4ce410ee10 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.855+0000 7f4cea5a4640 1 --2- 172.21.15.94:0/2084601033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ce4102170 0x7f4ce4102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.855+0000 7f4cea5a4640 1 -- 172.21.15.94:0/2084601033 >> 172.21.15.94:0/2084601033 conn(0x7f4ce40793a0 msgr2=0x7f4ce40797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.855+0000 7f4cea5a4640 1 -- 172.21.15.94:0/2084601033 shutdown_connections 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.855+0000 7f4cea5a4640 1 -- 172.21.15.94:0/2084601033 wait complete. 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4cea5a4640 1 Processor -- start 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4cea5a4640 1 -- start start 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4cea5a4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ce4102170 0x7f4ce4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4cea5a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ce4107240 0x7f4ce4198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4cea5a4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ce419d020 0x7f4ce419d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4cea5a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce4112220 con 0x7f4ce4107240 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4cea5a4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41120a0 con 0x7f4ce419d020 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4cea5a4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41123a0 con 0x7f4ce4102170 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4ce37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ce4107240 0x7f4ce4198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4ce37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ce4107240 0x7f4ce4198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54780/0 (socket says 172.21.15.94:54780) 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4ce37fe640 1 -- 172.21.15.94:0/1978365301 learned_addr learned my addr 172.21.15.94:0/1978365301 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4ce3fff640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ce4102170 0x7f4ce4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ce419d020 0x7f4ce419d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4ce37fe640 1 -- 172.21.15.94:0/1978365301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ce4102170 msgr2=0x7f4ce4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ce4102170 0x7f4ce4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4ce37fe640 1 -- 172.21.15.94:0/1978365301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ce419d020 msgr2=0x7f4ce419d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ce419d020 0x7f4ce419d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.856+0000 7f4ce37fe640 1 -- 172.21.15.94:0/1978365301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce419db10 con 0x7f4ce4107240 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.857+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ce4107240 0x7f4ce4198ca0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800c970 tx=0x7f4cd800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.857+0000 7f4ce17fa640 1 -- 172.21.15.94:0/1978365301 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8007bb0 con 0x7f4ce4107240 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.857+0000 7f4cea5a4640 1 -- 172.21.15.94:0/1978365301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce419de00 con 0x7f4ce4107240 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.857+0000 7f4ce17fa640 1 -- 172.21.15.94:0/1978365301 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cd8007d50 con 0x7f4ce4107240 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.857+0000 7f4ce17fa640 1 -- 172.21.15.94:0/1978365301 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8018800 con 0x7f4ce4107240 2024-04-21T08:41:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.857+0000 7f4cea5a4640 1 -- 172.21.15.94:0/1978365301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce41a55a0 con 0x7f4ce4107240 2024-04-21T08:41:10.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.859+0000 7f4ce17fa640 1 -- 172.21.15.94:0/1978365301 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 24) ==== 138930+0+0 (secure 0 0 0) 0x7f4cd800fa20 con 0x7f4ce4107240 2024-04-21T08:41:10.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.859+0000 7f4ced196740 1 Processor -- start 2024-04-21T08:41:10.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.859+0000 7f4ced196740 1 -- start start 2024-04-21T08:41:10.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.859+0000 7f4ced196740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024eb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.859+0000 7f4ce17fa640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4ccc0a3ed0 0x7f4ccc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.859+0000 7f4ced196740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024ebea0 0x5635024f4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.859+0000 7f4ced196740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.859+0000 7f4ced196740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5635024501e0 con 0x5635024f4a80 2024-04-21T08:41:10.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.859+0000 7f4ced196740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5635023ed390 con 0x5635024ebea0 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.859+0000 7f4ced196740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56350244c1a0 con 0x5635024eb580 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce8b41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce3fff640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4ccc0a3ed0 0x7f4ccc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce8b41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54794/0 (socket says 172.21.15.94:54794) 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce8b41640 1 -- 172.21.15.94:0/2222171773 learned_addr learned my addr 172.21.15.94:0/2222171773 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/2222171773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024ebea0 0x5635024f4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce17fa640 1 -- 172.21.15.94:0/1978365301 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4cd80d0f20 con 0x7f4ce4107240 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/2222171773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024eb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce8b41640 1 -- 172.21.15.94:0/2222171773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 msgr2=0x5635024eb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/2222171773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024eb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce8b41640 1 -- 172.21.15.94:0/2222171773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024ebea0 msgr2=0x5635024f4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/2222171773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024ebea0 0x5635024f4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce8b41640 1 -- 172.21.15.94:0/2222171773 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56350222c7d0 con 0x5635024f4a80 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/2222171773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024ebea0 0x5635024f4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce3fff640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4ccc0a3ed0 0x7f4ccc0a6390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009990 tx=0x7f4cd4009110 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/2222171773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f6e70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc037040 tx=0x7f4cdc02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c495b88b7420b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4cca7fc640 1 -- 172.21.15.94:0/2222171773 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc048070 con 0x5635024f4a80 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4cca7fc640 1 -- 172.21.15.94:0/2222171773 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cdc032970 con 0x5635024f4a80 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.860+0000 7f4cca7fc640 1 -- 172.21.15.94:0/2222171773 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc039040 con 0x5635024f4a80 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 -- 172.21.15.94:0/2222171773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 msgr2=0x5635024f6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 --2- 172.21.15.94:0/2222171773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f6e70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc037040 tx=0x7f4cdc02e040 comp rx=0 tx=0).stop 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 -- 172.21.15.94:0/2222171773 shutdown_connections 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 --2- 172.21.15.94:0/2222171773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f6e70 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 --2- 172.21.15.94:0/2222171773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024ebea0 0x5635024f4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 --2- 172.21.15.94:0/2222171773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024eb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 -- 172.21.15.94:0/2222171773 >> 172.21.15.94:0/2222171773 conn(0x5635024e5cb0 msgr2=0x5635024e6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 -- 172.21.15.94:0/2222171773 shutdown_connections 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 -- 172.21.15.94:0/2222171773 wait complete. 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 -- start start 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.861+0000 7f4ced196740 1 Processor -- start 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ced196740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024f1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ced196740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635024f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ced196740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 0x5635024f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ced196740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5635024f7810 con 0x5635024ebea0 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ced196740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5635024092f0 con 0x5635024f4a80 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ced196740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5635024f7990 con 0x5635024eb580 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ced196740 10 client.0 _ll_register_callbacks cb 0x563502416580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635024f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635024f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54800/0 (socket says 172.21.15.94:54800) 2024-04-21T08:41:10.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce37fe640 1 -- 172.21.15.94:0/474885908 learned_addr learned my addr 172.21.15.94:0/474885908 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 0x5635024f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce3fff640 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024f1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce37fe640 1 -- 172.21.15.94:0/474885908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 msgr2=0x5635024f1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024f1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce37fe640 1 -- 172.21.15.94:0/474885908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 msgr2=0x5635024f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 0x5635024f25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce37fe640 1 -- 172.21.15.94:0/474885908 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635021623f0 con 0x5635024ebea0 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 0x5635024f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635024f2090 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4cd809c830 tx=0x7f4cd8008c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7054291d8d148a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4cc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4cc37fe640 1 -- 172.21.15.94:0/474885908 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd809c200 con 0x5635024ebea0 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4cc37fe640 1 -- 172.21.15.94:0/474885908 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4cd80870f0 con 0x5635024ebea0 2024-04-21T08:41:10.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ced196740 1 -- 172.21.15.94:0/474885908 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cdc032970 con 0x5635024ebea0 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4cc37fe640 1 -- 172.21.15.94:0/474885908 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4cd8096050 con 0x5635024ebea0 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ced196740 10 client.16206 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.862+0000 7f4ced196740 1 -- 172.21.15.94:0/474885908 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5635021220f0 con 0x5635024ebea0 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.863+0000 7f4cc37fe640 1 -- 172.21.15.94:0/474885908 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cdc032970 con 0x5635024ebea0 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.863+0000 7f4cc37fe640 1 -- 172.21.15.94:0/474885908 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4cd8020030 con 0x5635024ebea0 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.863+0000 7f4ced196740 10 client.16206 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.863+0000 7f4ced196740 10 client.16206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.863+0000 7f4ced196740 20 client.16206 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.863+0000 7f4ced196740 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635025049c0 0x563502506e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.863+0000 7f4ced196740 4 client.16206 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.863+0000 7f4ced196740 1 -- 172.21.15.94:0/474885908 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56350245dc00 con 0x5635025049c0 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.863+0000 7f4ce3fff640 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635025049c0 0x563502506e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4cc37fe640 10 client.16206 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ce3fff640 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635025049c0 0x563502506e30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb71bf9f83f363c7 server_cookie=376b3a9a63fcbc4e in_seq=0 out_seq=0 2024-04-21T08:41:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4cc37fe640 1 -- 172.21.15.94:0/474885908 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56350245dc00 con 0x5635025049c0 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4cc37fe640 10 client.16206 handle_command_reply: tid=0 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 client.16206 shutdown 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 -- 172.21.15.94:0/474885908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 msgr2=0x5635024f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635024f2090 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4cd809c830 tx=0x7f4cd8008c80 comp rx=0 tx=0).stop 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 -- 172.21.15.94:0/474885908 shutdown_connections 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5635025049c0 0x563502506e30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 0x5635024f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4cc37fe640 0 client.16206 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635024f2090 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 --2- 172.21.15.94:0/474885908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024f1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 -- 172.21.15.94:0/474885908 >> 172.21.15.94:0/474885908 conn(0x5635024e5cb0 msgr2=0x5635024e7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 -- 172.21.15.94:0/474885908 shutdown_connections 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.864+0000 7f4ced196740 1 -- 172.21.15.94:0/474885908 wait complete. 2024-04-21T08:41:10.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.865+0000 7f4ced196740 20 client.16206 trim_cache size 0 max 16384 2024-04-21T08:41:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ced196740 1 Processor -- start 2024-04-21T08:41:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ced196740 1 -- start start 2024-04-21T08:41:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ced196740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024f4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ced196740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635025006d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ced196740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 0x5635024f2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ced196740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41b1c10 con 0x5635024ebea0 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ced196740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41af4e0 con 0x5635024f4a80 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ced196740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41ad330 con 0x5635024eb580 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635025006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635025006d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54808/0 (socket says 172.21.15.94:54808) 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce37fe640 1 -- 172.21.15.94:0/1641999476 learned_addr learned my addr 172.21.15.94:0/1641999476 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/1641999476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 0x5635024f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce37fe640 1 -- 172.21.15.94:0/1641999476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 msgr2=0x5635024f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/1641999476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce37fe640 1 -- 172.21.15.94:0/1641999476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 msgr2=0x5635024f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/1641999476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 0x5635024f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce37fe640 1 -- 172.21.15.94:0/1641999476 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635024f71e0 con 0x5635024ebea0 2024-04-21T08:41:10.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/1641999476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 0x5635024f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/1641999476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635025006d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8098580 tx=0x7f4cd800fdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a85a6d5af032fb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4cc9ffb640 1 -- 172.21.15.94:0/1641999476 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd80a4c80 con 0x5635024ebea0 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4cc9ffb640 1 -- 172.21.15.94:0/1641999476 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cd809cfa0 con 0x5635024ebea0 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.882+0000 7f4cc9ffb640 1 -- 172.21.15.94:0/1641999476 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd80a4e70 con 0x5635024ebea0 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 -- 172.21.15.94:0/1641999476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 msgr2=0x5635025006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 --2- 172.21.15.94:0/1641999476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635025006d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8098580 tx=0x7f4cd800fdc0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 -- 172.21.15.94:0/1641999476 shutdown_connections 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 --2- 172.21.15.94:0/1641999476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024f4a80 0x5635024f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 --2- 172.21.15.94:0/1641999476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024ebea0 0x5635025006d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 --2- 172.21.15.94:0/1641999476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024eb580 0x5635024f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 -- 172.21.15.94:0/1641999476 >> 172.21.15.94:0/1641999476 conn(0x5635024e5cb0 msgr2=0x5635024f1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 -- 172.21.15.94:0/1641999476 shutdown_connections 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 -- 172.21.15.94:0/1641999476 wait complete. 2024-04-21T08:41:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 -- start start 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.883+0000 7f4ced196740 1 Processor -- start 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ced196740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024eb580 0x5635024fcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ced196740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024ebea0 0x5635024fd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ced196740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ced196740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41ad9a0 con 0x5635024f4a80 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ced196740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41aee10 con 0x5635024eb580 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ced196740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41ae070 con 0x5635024ebea0 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ced196740 10 client.0 _ll_register_callbacks cb 0x7f4cd80058a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce8b41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce8b41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54824/0 (socket says 172.21.15.94:54824) 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce8b41640 1 -- 172.21.15.94:0/1708102448 learned_addr learned my addr 172.21.15.94:0/1708102448 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024ebea0 0x5635024fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce3fff640 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024eb580 0x5635024fcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce8b41640 1 -- 172.21.15.94:0/1708102448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024ebea0 msgr2=0x5635024fd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024ebea0 0x5635024fd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce8b41640 1 -- 172.21.15.94:0/1708102448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024eb580 msgr2=0x5635024fcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024eb580 0x5635024fcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce8b41640 1 -- 172.21.15.94:0/1708102448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635024fa430 con 0x5635024f4a80 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce37fe640 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024ebea0 0x5635024fd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce3fff640 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024eb580 0x5635024fcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ce8b41640 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f9bb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc02c130 tx=0x7f4cdc009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753805388cdf2c50 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4cc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4cc37fe640 1 -- 172.21.15.94:0/1708102448 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc039070 con 0x5635024f4a80 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4cc37fe640 1 -- 172.21.15.94:0/1708102448 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4cdc032780 con 0x5635024f4a80 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.884+0000 7f4ced196740 1 -- 172.21.15.94:0/1708102448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cd809cfa0 con 0x5635024f4a80 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4cc37fe640 1 -- 172.21.15.94:0/1708102448 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4cdc048040 con 0x5635024f4a80 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4ced196740 10 client.16212 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4ced196740 1 -- 172.21.15.94:0/1708102448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56350256ba20 con 0x5635024f4a80 2024-04-21T08:41:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4cc37fe640 1 -- 172.21.15.94:0/1708102448 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cd809cfa0 con 0x5635024f4a80 2024-04-21T08:41:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4cc37fe640 1 -- 172.21.15.94:0/1708102448 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4cdc03f040 con 0x5635024f4a80 2024-04-21T08:41:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4ced196740 10 client.16212 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4ced196740 10 client.16212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4ced196740 20 client.16212 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4ced196740 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56350256bdb0 0x56350256e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4ced196740 4 client.16212 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:41:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4ced196740 1 -- 172.21.15.94:0/1708102448 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563502457a70 con 0x56350256bdb0 2024-04-21T08:41:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.885+0000 7f4ce3fff640 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56350256bdb0 0x56350256e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4cc37fe640 10 client.16212 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ce3fff640 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56350256bdb0 0x56350256e1a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a87eb7818c8b09f5 server_cookie=aea2e914585f9dc0 in_seq=0 out_seq=0 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4cc37fe640 1 -- 172.21.15.94:0/1708102448 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563502457a70 con 0x56350256bdb0 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4cc37fe640 10 client.16212 handle_command_reply: tid=0 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 client.16212 shutdown 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 -- 172.21.15.94:0/1708102448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 msgr2=0x5635024f9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f9bb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc02c130 tx=0x7f4cdc009360 comp rx=0 tx=0).stop 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 -- 172.21.15.94:0/1708102448 shutdown_connections 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56350256bdb0 0x56350256e1a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5635024ebea0 0x5635024fd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5635024f4a80 0x5635024f9bb0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 --2- 172.21.15.94:0/1708102448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635024eb580 0x5635024fcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4cc37fe640 0 client.16212 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 -- 172.21.15.94:0/1708102448 >> 172.21.15.94:0/1708102448 conn(0x5635024e5cb0 msgr2=0x5635024e4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 -- 172.21.15.94:0/1708102448 shutdown_connections 2024-04-21T08:41:10.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 1 -- 172.21.15.94:0/1708102448 wait complete. 2024-04-21T08:41:10.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4ced196740 20 client.16212 trim_cache size 0 max 16384 2024-04-21T08:41:10.896 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:41:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:41:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:41:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "13701888-61e7-46e7-b913-ba56362f5b30": { 2024-04-21T08:41:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:41:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "13701888-61e7-46e7-b913-ba56362f5b30", 2024-04-21T08:41:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:41:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:41:10.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:41:10.897 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:41:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4cea5a4640 1 -- 172.21.15.94:0/1978365301 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4ccc0a3ed0 msgr2=0x7f4ccc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4cea5a4640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4ccc0a3ed0 0x7f4ccc0a6390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009990 tx=0x7f4cd4009110 comp rx=0 tx=0).stop 2024-04-21T08:41:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4cea5a4640 1 -- 172.21.15.94:0/1978365301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ce4107240 msgr2=0x7f4ce4198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.886+0000 7f4cea5a4640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ce4107240 0x7f4ce4198ca0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800c970 tx=0x7f4cd800cd70 comp rx=0 tx=0).stop 2024-04-21T08:41:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.887+0000 7f4cea5a4640 1 -- 172.21.15.94:0/1978365301 shutdown_connections 2024-04-21T08:41:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.887+0000 7f4cea5a4640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4ccc0a3ed0 0x7f4ccc0a6390 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.887+0000 7f4cea5a4640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ce419d020 0x7f4ce419d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.887+0000 7f4cea5a4640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ce4107240 0x7f4ce4198ca0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.887+0000 7f4cea5a4640 1 --2- 172.21.15.94:0/1978365301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ce4102170 0x7f4ce4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.887+0000 7f4cea5a4640 1 -- 172.21.15.94:0/1978365301 >> 172.21.15.94:0/1978365301 conn(0x7f4ce40793a0 msgr2=0x7f4ce40fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.887+0000 7f4cea5a4640 1 -- 172.21.15.94:0/1978365301 shutdown_connections 2024-04-21T08:41:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:10.887+0000 7f4cea5a4640 1 -- 172.21.15.94:0/1978365301 wait complete. 2024-04-21T08:41:10.899 INFO:tasks.cephfs.filesystem:scrub status for tag:13701888-61e7-46e7-b913-ba56362f5b30 - {'path': '/', 'tag': '13701888-61e7-46e7-b913-ba56362f5b30', 'options': 'recursive,force'} 2024-04-21T08:41:11.183 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:11.183 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:11.183 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:11.183 INFO:tasks.workunit.client.0.smithi094.stdout:iozone test complete. 2024-04-21T08:41:11.184 INFO:tasks.workunit.client.0.smithi094.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-04-21T08:41:11.247 INFO:tasks.workunit.client.0.smithi094.stdout: Iozone: Performance Test of File I/O 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Version $Revision: 3.494 $ 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Compiled for 64 bit mode. 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Build: linux-AMD64 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:11.248 INFO:tasks.workunit.client.0.smithi094.stdout: Run began: Sun Apr 21 08:41:11 2024 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: Include close in write timing 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: Include fsync in write timing 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: File size set to 1048576 kB 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: Record Size 1024 kB 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: Output is in kBytes/sec 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: Time Resolution = 0.000001 seconds. 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: Processor cache size set to 1024 kBytes. 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: Processor cache line size set to 32 bytes. 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: File stride size set to 17 * record size. 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: Throughput test with 1 process 2024-04-21T08:41:11.249 INFO:tasks.workunit.client.0.smithi094.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-04-21T08:41:11.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:11 smithi111 ceph-mon[31493]: pgmap v39: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 110 MiB/s wr, 53 op/s 2024-04-21T08:41:11.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:11 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:41:11.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:11 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:11.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:11 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:41:11.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:11 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:11.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:11 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:41:11.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:11 smithi173 ceph-mon[31428]: pgmap v39: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 110 MiB/s wr, 53 op/s 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:11 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:11 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:11 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:11 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:11 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:11 smithi094 ceph-mon[27988]: pgmap v39: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 110 MiB/s wr, 53 op/s 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:11 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:11 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:11 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:11 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:11.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:11 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:41:12.563 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:12.563 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 initial writers = 816691.25 kB/sec 2024-04-21T08:41:12.563 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 initial writers = 816555.72 kB/sec 2024-04-21T08:41:12.563 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 816691.25 kB/sec 2024-04-21T08:41:12.563 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 816691.25 kB/sec 2024-04-21T08:41:12.563 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 816691.25 kB/sec 2024-04-21T08:41:12.563 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 1048576.00 kB 2024-04-21T08:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.168+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/3919443127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff81004b0 msgr2=0x7f8ff8100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.168+0000 7f8ffdf9f640 1 --2- 172.21.15.94:0/3919443127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff81004b0 0x7f8ff8100890 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009900 tx=0x7f8fe802bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.168+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/3919443127 shutdown_connections 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.168+0000 7f8ffdf9f640 1 --2- 172.21.15.94:0/3919443127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff8100e60 0x7f8ff81069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.168+0000 7f8ffdf9f640 1 --2- 172.21.15.94:0/3919443127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ff8107120 0x7f8ff810eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.168+0000 7f8ffdf9f640 1 --2- 172.21.15.94:0/3919443127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff81004b0 0x7f8ff8100890 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.168+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/3919443127 >> 172.21.15.94:0/3919443127 conn(0x7f8ff80771c0 msgr2=0x7f8ff80fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.168+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/3919443127 shutdown_connections 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.168+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/3919443127 wait complete. 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffdf9f640 1 Processor -- start 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffdf9f640 1 -- start start 2024-04-21T08:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffdf9f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff81004b0 0x7f8ff81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffdf9f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff8107120 0x7f8ff8198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffdf9f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ff819ce90 0x7f8ff819d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffdf9f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8112030 con 0x7f8ff81004b0 2024-04-21T08:41:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffdf9f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8111eb0 con 0x7f8ff819ce90 2024-04-21T08:41:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffdf9f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81121b0 con 0x7f8ff8107120 2024-04-21T08:41:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffcf9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff81004b0 0x7f8ff81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffcf9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff81004b0 0x7f8ff81985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54838/0 (socket says 172.21.15.94:54838) 2024-04-21T08:41:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/528045442 learned_addr learned my addr 172.21.15.94:0/528045442 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.169+0000 7f8ff7fff640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff8107120 0x7f8ff8198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ff819ce90 0x7f8ff819d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/528045442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff8107120 msgr2=0x7f8ff8198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff8107120 0x7f8ff8198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/528045442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ff819ce90 msgr2=0x7f8ff819d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ff819ce90 0x7f8ff819d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/528045442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff819d980 con 0x7f8ff81004b0 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ff819ce90 0x7f8ff819d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff81004b0 0x7f8ff81985d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8005940 tx=0x7f8fe802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ff5ffb640 1 -- 172.21.15.94:0/528045442 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8043020 con 0x7f8ff81004b0 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/528045442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff819dc10 con 0x7f8ff81004b0 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ff5ffb640 1 -- 172.21.15.94:0/528045442 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe802cbe0 con 0x7f8ff81004b0 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ff5ffb640 1 -- 172.21.15.94:0/528045442 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8003bf0 con 0x7f8ff81004b0 2024-04-21T08:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.170+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/528045442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a5410 con 0x7f8ff81004b0 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.172+0000 7f8ff5ffb640 1 -- 172.21.15.94:0/528045442 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 24) ==== 138930+0+0 (secure 0 0 0) 0x7f8fe8046020 con 0x7f8ff81004b0 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f9001df0740 1 Processor -- start 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f9001df0740 1 -- start start 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f9001df0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c7ac0 0x5575cf7c6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f9001df0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f9001df0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7d02d0 0x5575cf7d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f9001df0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5575cf72b8d0 con 0x5575cf7c6d50 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f9001df0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5575cf6c8e80 con 0x5575cf7c7ac0 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ff5ffb640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8fe00a3ec0 0x7f8fe00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f9001df0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5575cf727890 con 0x5575cf7d02d0 2024-04-21T08:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ffcf9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ffcf9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7c71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54842/0 (socket says 172.21.15.94:54842) 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/121221213 learned_addr learned my addr 172.21.15.94:0/121221213 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8fe00a3ec0 0x7f8fe00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/121221213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c7ac0 0x5575cf7c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ff7fff640 1 --2- 172.21.15.94:0/121221213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7d02d0 0x5575cf7d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ff5ffb640 1 -- 172.21.15.94:0/528045442 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f8fe80f3030 con 0x7f8ff81004b0 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/121221213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7d02d0 msgr2=0x5575cf7d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/121221213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7d02d0 0x5575cf7d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/121221213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c7ac0 msgr2=0x5575cf7c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/121221213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c7ac0 0x5575cf7c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.173+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/121221213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575cf6ea320 con 0x5575cf7c6d50 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8fe00a3ec0 0x7f8fe00a6380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0005c40 tx=0x7f8ff0009d50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/121221213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7c71d0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80f0210 tx=0x7f8fe802cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2909b65a551f7168 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f8fdeffd640 1 -- 172.21.15.94:0/121221213 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe80c6430 con 0x5575cf7c6d50 2024-04-21T08:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f8fdeffd640 1 -- 172.21.15.94:0/121221213 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe80bf1c0 con 0x5575cf7c6d50 2024-04-21T08:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f8fdeffd640 1 -- 172.21.15.94:0/121221213 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe80beac0 con 0x5575cf7c6d50 2024-04-21T08:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f9001df0740 1 -- 172.21.15.94:0/121221213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 msgr2=0x5575cf7c71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f9001df0740 1 --2- 172.21.15.94:0/121221213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7c71d0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80f0210 tx=0x7f8fe802cd90 comp rx=0 tx=0).stop 2024-04-21T08:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f9001df0740 1 -- 172.21.15.94:0/121221213 shutdown_connections 2024-04-21T08:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f9001df0740 1 --2- 172.21.15.94:0/121221213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7d02d0 0x5575cf7d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f9001df0740 1 --2- 172.21.15.94:0/121221213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7c71d0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f9001df0740 1 --2- 172.21.15.94:0/121221213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c7ac0 0x5575cf7c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.174+0000 7f9001df0740 1 -- 172.21.15.94:0/121221213 >> 172.21.15.94:0/121221213 conn(0x5575cf78dc90 msgr2=0x5575cf7c1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 1 -- 172.21.15.94:0/121221213 shutdown_connections 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 1 -- 172.21.15.94:0/121221213 wait complete. 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 1 -- start start 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 1 Processor -- start 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c6d50 0x5575cf7cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c7ac0 0x5575cf7cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5575cf72b8d0 con 0x5575cf7d02d0 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5575cf6b3eb0 con 0x5575cf7c7ac0 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5575cf7d2dc0 con 0x5575cf7c6d50 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f9001df0740 10 client.0 _ll_register_callbacks cb 0x5575cf6e0b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f8ffd79e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f8ffd79e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7cd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54858/0 (socket says 172.21.15.94:54858) 2024-04-21T08:41:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f8ffd79e640 1 -- 172.21.15.94:0/4043429454 learned_addr learned my addr 172.21.15.94:0/4043429454 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c6d50 0x5575cf7cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.175+0000 7f8ff7fff640 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c7ac0 0x5575cf7cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8ffd79e640 1 -- 172.21.15.94:0/4043429454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c6d50 msgr2=0x5575cf7cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c6d50 0x5575cf7cfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8ffd79e640 1 -- 172.21.15.94:0/4043429454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c7ac0 msgr2=0x5575cf7cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c7ac0 0x5575cf7cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8ffd79e640 1 -- 172.21.15.94:0/4043429454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575cf43e210 con 0x5575cf7d02d0 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7cd5b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0010d20 tx=0x7f8ff0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cebe6ff936a3fc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8fd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8fd7fff640 1 -- 172.21.15.94:0/4043429454 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff001ee40 con 0x5575cf7d02d0 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8fd7fff640 1 -- 172.21.15.94:0/4043429454 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f8ff002a410 con 0x5575cf7d02d0 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f9001df0740 1 -- 172.21.15.94:0/4043429454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fe80bf1c0 con 0x5575cf7d02d0 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8fd7fff640 1 -- 172.21.15.94:0/4043429454 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8ff002a650 con 0x5575cf7d02d0 2024-04-21T08:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f9001df0740 10 client.16224 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f9001df0740 1 -- 172.21.15.94:0/4043429454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5575cf50a620 con 0x5575cf7d02d0 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8fd7fff640 1 -- 172.21.15.94:0/4043429454 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe80bf1c0 con 0x5575cf7d02d0 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f8fd7fff640 1 -- 172.21.15.94:0/4043429454 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f8ff001f070 con 0x5575cf7d02d0 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f9001df0740 10 client.16224 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f9001df0740 10 client.16224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.176+0000 7f9001df0740 20 client.16224 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.177+0000 7f9001df0740 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5575cf7dff70 0x5575cf7e2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.177+0000 7f9001df0740 4 client.16224 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.177+0000 7f9001df0740 1 -- 172.21.15.94:0/4043429454 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575cf73f880 con 0x5575cf7dff70 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.177+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5575cf7dff70 0x5575cf7e2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.177+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5575cf7dff70 0x5575cf7e2360 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46ce2b9ff7efdf5 server_cookie=112c5353a57423e7 in_seq=0 out_seq=0 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.177+0000 7f8fd7fff640 10 client.16224 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f8fd7fff640 1 -- 172.21.15.94:0/4043429454 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5575cf73f880 con 0x5575cf7dff70 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f8fd7fff640 10 client.16224 handle_command_reply: tid=0 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 client.16224 shutdown 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 -- 172.21.15.94:0/4043429454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 msgr2=0x5575cf7cd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7cd5b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0010d20 tx=0x7f8ff0005bd0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 -- 172.21.15.94:0/4043429454 shutdown_connections 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5575cf7dff70 0x5575cf7e2360 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7cd5b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f8fd7fff640 0 client.16224 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c7ac0 0x5575cf7cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 --2- 172.21.15.94:0/4043429454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c6d50 0x5575cf7cfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 -- 172.21.15.94:0/4043429454 >> 172.21.15.94:0/4043429454 conn(0x5575cf78dc90 msgr2=0x5575cf7c3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:13.187 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 -- 172.21.15.94:0/4043429454 shutdown_connections 2024-04-21T08:41:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 1 -- 172.21.15.94:0/4043429454 wait complete. 2024-04-21T08:41:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.178+0000 7f9001df0740 20 client.16224 trim_cache size 0 max 16384 2024-04-21T08:41:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.192+0000 7f9001df0740 1 Processor -- start 2024-04-21T08:41:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.192+0000 7f9001df0740 1 -- start start 2024-04-21T08:41:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f9001df0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7dbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f9001df0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c7ac0 0x5575cf7d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f9001df0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7d02d0 0x5575cf7cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f9001df0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81b1b80 con 0x5575cf7c6d50 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f9001df0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81af450 con 0x5575cf7d02d0 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f9001df0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81ad2a0 con 0x5575cf7c7ac0 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffcf9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7dbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffcf9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7dbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54868/0 (socket says 172.21.15.94:54868) 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/1323790246 learned_addr learned my addr 172.21.15.94:0/1323790246 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/1323790246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c7ac0 msgr2=0x5575cf7d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/1323790246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c7ac0 0x5575cf7d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/1323790246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7d02d0 msgr2=0x5575cf7cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/1323790246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7d02d0 0x5575cf7cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/1323790246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7d02d0 0x5575cf7cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffcf9d640 1 -- 172.21.15.94:0/1323790246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575cf7dc6b0 con 0x5575cf7c6d50 2024-04-21T08:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/1323790246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7d02d0 0x5575cf7cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/1323790246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7dbb40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80baf20 tx=0x7f8fe80be6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257b00f6c70d147f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8fde7fc640 1 -- 172.21.15.94:0/1323790246 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe80bf1c0 con 0x5575cf7c6d50 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8fde7fc640 1 -- 172.21.15.94:0/1323790246 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe80c61f0 con 0x5575cf7c6d50 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.193+0000 7f8fde7fc640 1 -- 172.21.15.94:0/1323790246 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe80c63e0 con 0x5575cf7c6d50 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 -- 172.21.15.94:0/1323790246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 msgr2=0x5575cf7dbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 --2- 172.21.15.94:0/1323790246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7dbb40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80baf20 tx=0x7f8fe80be6c0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 -- 172.21.15.94:0/1323790246 shutdown_connections 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 --2- 172.21.15.94:0/1323790246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c7ac0 0x5575cf7d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 --2- 172.21.15.94:0/1323790246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7d02d0 0x5575cf7cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 --2- 172.21.15.94:0/1323790246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7c6d50 0x5575cf7dbb40 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 -- 172.21.15.94:0/1323790246 >> 172.21.15.94:0/1323790246 conn(0x5575cf78dc90 msgr2=0x5575cf7c8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 -- 172.21.15.94:0/1323790246 shutdown_connections 2024-04-21T08:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 -- 172.21.15.94:0/1323790246 wait complete. 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 -- start start 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 Processor -- start 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c6d50 0x5575cf7d8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c7ac0 0x5575cf7d8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81ad910 con 0x5575cf7d02d0 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81aed80 con 0x5575cf7c6d50 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81adfe0 con 0x5575cf7c7ac0 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.194+0000 7f9001df0740 10 client.0 _ll_register_callbacks cb 0x7f8ff0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ffd79e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ffd79e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7ca2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54876/0 (socket says 172.21.15.94:54876) 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ffd79e640 1 -- 172.21.15.94:0/3036877510 learned_addr learned my addr 172.21.15.94:0/3036877510 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c6d50 0x5575cf7d8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ff7fff640 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c7ac0 0x5575cf7d8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ffd79e640 1 -- 172.21.15.94:0/3036877510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c7ac0 msgr2=0x5575cf7d8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c7ac0 0x5575cf7d8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ffd79e640 1 -- 172.21.15.94:0/3036877510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c6d50 msgr2=0x5575cf7d8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c6d50 0x5575cf7d8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ffd79e640 1 -- 172.21.15.94:0/3036877510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575cf7caad0 con 0x5575cf7d02d0 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ff7fff640 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c7ac0 0x5575cf7d8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8ffd79e640 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7ca2d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0010870 tx=0x7f8ff0009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d991e27e8b2962b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8fd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8fd7fff640 1 -- 172.21.15.94:0/3036877510 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0009910 con 0x5575cf7d02d0 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8fd7fff640 1 -- 172.21.15.94:0/3036877510 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f8ff0009ab0 con 0x5575cf7d02d0 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f9001df0740 1 -- 172.21.15.94:0/3036877510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5575cf6cb3d0 con 0x5575cf7d02d0 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8fd7fff640 1 -- 172.21.15.94:0/3036877510 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8ff0040050 con 0x5575cf7d02d0 2024-04-21T08:41:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f9001df0740 10 client.16230 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f9001df0740 1 -- 172.21.15.94:0/3036877510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5575cf7d10c0 con 0x5575cf7d02d0 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.195+0000 7f8fd7fff640 1 -- 172.21.15.94:0/3036877510 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5575cf6cb3d0 con 0x5575cf7d02d0 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.196+0000 7f8fd7fff640 1 -- 172.21.15.94:0/3036877510 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f8ff0035020 con 0x5575cf7d02d0 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.196+0000 7f9001df0740 10 client.16230 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.196+0000 7f9001df0740 10 client.16230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.196+0000 7f9001df0740 20 client.16230 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.196+0000 7f9001df0740 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5575cf846670 0x5575cf848a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.196+0000 7f9001df0740 4 client.16230 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.196+0000 7f9001df0740 1 -- 172.21.15.94:0/3036877510 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575cf7392f0 con 0x5575cf846670 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.196+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5575cf846670 0x5575cf848a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.196+0000 7f8ffcf9d640 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5575cf846670 0x5575cf848a90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deeadead1fda9bcd server_cookie=b494f53d6c9022b3 in_seq=0 out_seq=0 2024-04-21T08:41:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.196+0000 7f8fd7fff640 10 client.16230 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f8fd7fff640 1 -- 172.21.15.94:0/3036877510 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575cf7392f0 con 0x5575cf846670 2024-04-21T08:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f8fd7fff640 10 client.16230 handle_command_reply: tid=0 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 client.16230 shutdown 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 -- 172.21.15.94:0/3036877510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 msgr2=0x5575cf7ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7ca2d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0010870 tx=0x7f8ff0009fd0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 -- 172.21.15.94:0/3036877510 shutdown_connections 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5575cf846670 0x5575cf848a90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575cf7c7ac0 0x5575cf7d8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5575cf7d02d0 0x5575cf7ca2d0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 --2- 172.21.15.94:0/3036877510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575cf7c6d50 0x5575cf7d8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f8fd7fff640 0 client.16230 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 -- 172.21.15.94:0/3036877510 >> 172.21.15.94:0/3036877510 conn(0x5575cf78dc90 msgr2=0x5575cf7cb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 -- 172.21.15.94:0/3036877510 shutdown_connections 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 1 -- 172.21.15.94:0/3036877510 wait complete. 2024-04-21T08:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.206+0000 7f9001df0740 20 client.16230 trim_cache size 0 max 16384 2024-04-21T08:41:13.208 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:{ 2024-04-21T08:41:13.208 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "message": "", 2024-04-21T08:41:13.208 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "return_code": 0 2024-04-21T08:41:13.208 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:} 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/528045442 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8fe00a3ec0 msgr2=0x7f8fe00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8fe00a3ec0 0x7f8fe00a6380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0005c40 tx=0x7f8ff0009d50 comp rx=0 tx=0).stop 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/528045442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff81004b0 msgr2=0x7f8ff81985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff81004b0 0x7f8ff81985d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8005940 tx=0x7f8fe802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/528045442 shutdown_connections 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8fe00a3ec0 0x7f8fe00a6380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff8107120 0x7f8ff8198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ff819ce90 0x7f8ff819d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 --2- 172.21.15.94:0/528045442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff81004b0 0x7f8ff81985d0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/528045442 >> 172.21.15.94:0/528045442 conn(0x7f8ff80771c0 msgr2=0x7f8ff80fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/528045442 shutdown_connections 2024-04-21T08:41:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:13.207+0000 7f8ffdf9f640 1 -- 172.21.15.94:0/528045442 wait complete. 2024-04-21T08:41:13.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:13 smithi111 ceph-mon[31493]: pgmap v40: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 72 MiB/s wr, 41 op/s 2024-04-21T08:41:13.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:13 smithi094 ceph-mon[27988]: pgmap v40: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 72 MiB/s wr, 41 op/s 2024-04-21T08:41:13.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:13 smithi173 ceph-mon[31428]: pgmap v40: 129 pgs: 129 active+clean; 1.0 GiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 72 MiB/s wr, 41 op/s 2024-04-21T08:41:15.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:15 smithi111 ceph-mon[31493]: pgmap v41: 129 pgs: 129 active+clean; 1.6 GiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 127 MiB/s wr, 74 op/s 2024-04-21T08:41:15.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:15 smithi094 ceph-mon[27988]: pgmap v41: 129 pgs: 129 active+clean; 1.6 GiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 127 MiB/s wr, 74 op/s 2024-04-21T08:41:15.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:15 smithi173 ceph-mon[31428]: pgmap v41: 129 pgs: 129 active+clean; 1.6 GiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 127 MiB/s wr, 74 op/s 2024-04-21T08:41:16.255 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:16.256 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 rewriters = 864136.44 kB/sec 2024-04-21T08:41:16.256 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 rewriters = 863961.27 kB/sec 2024-04-21T08:41:16.256 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 864136.44 kB/sec 2024-04-21T08:41:16.256 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 864136.44 kB/sec 2024-04-21T08:41:16.256 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 864136.44 kB/sec 2024-04-21T08:41:16.256 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 1048576.00 kB 2024-04-21T08:41:17.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:17 smithi111 ceph-mon[31493]: pgmap v42: 129 pgs: 129 active+clean; 2.0 GiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 128 MiB/s wr, 83 op/s 2024-04-21T08:41:17.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:17 smithi094 ceph-mon[27988]: pgmap v42: 129 pgs: 129 active+clean; 2.0 GiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 128 MiB/s wr, 83 op/s 2024-04-21T08:41:17.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:17 smithi173 ceph-mon[31428]: pgmap v42: 129 pgs: 129 active+clean; 2.0 GiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 128 MiB/s wr, 83 op/s 2024-04-21T08:41:19.580 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:19.580 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 readers = 7493156.00 kB/sec 2024-04-21T08:41:19.580 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 readers = 7482192.92 kB/sec 2024-04-21T08:41:19.580 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 7493156.00 kB/sec 2024-04-21T08:41:19.580 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 7493156.00 kB/sec 2024-04-21T08:41:19.580 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 7493156.00 kB/sec 2024-04-21T08:41:19.581 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 1048576.00 kB 2024-04-21T08:41:19.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:19 smithi111 ceph-mon[31493]: pgmap v43: 129 pgs: 129 active+clean; 2.0 GiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 MiB/s wr, 55 op/s 2024-04-21T08:41:19.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:19 smithi094 ceph-mon[27988]: pgmap v43: 129 pgs: 129 active+clean; 2.0 GiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 MiB/s wr, 55 op/s 2024-04-21T08:41:19.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:19 smithi173 ceph-mon[31428]: pgmap v43: 129 pgs: 129 active+clean; 2.0 GiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 MiB/s wr, 55 op/s 2024-04-21T08:41:21.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:21 smithi111 ceph-mon[31493]: pgmap v44: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 79 op/s 2024-04-21T08:41:21.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:21 smithi094 ceph-mon[27988]: pgmap v44: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 79 op/s 2024-04-21T08:41:21.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:21 smithi173 ceph-mon[31428]: pgmap v44: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 79 op/s 2024-04-21T08:41:21.804 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:21.804 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 re-readers = 7137440.00 kB/sec 2024-04-21T08:41:21.804 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 re-readers = 7126233.91 kB/sec 2024-04-21T08:41:21.804 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 7137440.00 kB/sec 2024-04-21T08:41:21.804 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 7137440.00 kB/sec 2024-04-21T08:41:21.804 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 7137440.00 kB/sec 2024-04-21T08:41:21.804 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 1048576.00 kB 2024-04-21T08:41:23.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:23 smithi111 ceph-mon[31493]: pgmap v45: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 76 op/s 2024-04-21T08:41:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:23 smithi094 ceph-mon[27988]: pgmap v45: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 76 op/s 2024-04-21T08:41:23.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:23 smithi173 ceph-mon[31428]: pgmap v45: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 76 op/s 2024-04-21T08:41:25.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:25 smithi111 ceph-mon[31493]: pgmap v46: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 79 op/s 2024-04-21T08:41:25.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:25 smithi094 ceph-mon[27988]: pgmap v46: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 79 op/s 2024-04-21T08:41:25.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:25 smithi173 ceph-mon[31428]: pgmap v46: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 79 op/s 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.298+0000 7feb314eb640 1 -- 172.21.15.94:0/3578613576 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb2c1011a0 msgr2=0x7feb2c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.298+0000 7feb314eb640 1 --2- 172.21.15.94:0/3578613576 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb2c1011a0 0x7feb2c101620 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7feb20009900 tx=0x7feb2002bc80 comp rx=0 tx=0).stop 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.298+0000 7feb314eb640 1 -- 172.21.15.94:0/3578613576 shutdown_connections 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.298+0000 7feb314eb640 1 --2- 172.21.15.94:0/3578613576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb2c101b60 0x7feb2c10dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.298+0000 7feb314eb640 1 --2- 172.21.15.94:0/3578613576 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb2c1011a0 0x7feb2c101620 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.298+0000 7feb314eb640 1 --2- 172.21.15.94:0/3578613576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb2c106fb0 0x7feb2c107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.298+0000 7feb314eb640 1 -- 172.21.15.94:0/3578613576 >> 172.21.15.94:0/3578613576 conn(0x7feb2c076c50 msgr2=0x7feb2c077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.298+0000 7feb314eb640 1 -- 172.21.15.94:0/3578613576 shutdown_connections 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.298+0000 7feb314eb640 1 -- 172.21.15.94:0/3578613576 wait complete. 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.299+0000 7feb314eb640 1 Processor -- start 2024-04-21T08:41:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.299+0000 7feb314eb640 1 -- start start 2024-04-21T08:41:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.299+0000 7feb314eb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb2c1011a0 0x7feb2c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.299+0000 7feb314eb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb2c101b60 0x7feb2c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.299+0000 7feb314eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb2c106fb0 0x7feb2c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.299+0000 7feb314eb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c111080 con 0x7feb2c106fb0 2024-04-21T08:41:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.299+0000 7feb314eb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c110f00 con 0x7feb2c101b60 2024-04-21T08:41:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.299+0000 7feb314eb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c111200 con 0x7feb2c1011a0 2024-04-21T08:41:26.303 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb2c106fb0 0x7feb2c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.303 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb2c106fb0 0x7feb2c19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55634/0 (socket says 172.21.15.94:55634) 2024-04-21T08:41:26.303 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2b7fe640 1 -- 172.21.15.94:0/1655538515 learned_addr learned my addr 172.21.15.94:0/1655538515 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:26.303 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2affd640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb2c1011a0 0x7feb2c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.303 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb2c101b60 0x7feb2c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.303 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2b7fe640 1 -- 172.21.15.94:0/1655538515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb2c1011a0 msgr2=0x7feb2c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.303 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2b7fe640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb2c1011a0 0x7feb2c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2b7fe640 1 -- 172.21.15.94:0/1655538515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb2c101b60 msgr2=0x7feb2c198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2b7fe640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb2c101b60 0x7feb2c198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2b7fe640 1 -- 172.21.15.94:0/1655538515 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2c19d680 con 0x7feb2c106fb0 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2affd640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb2c1011a0 0x7feb2c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb2c101b60 0x7feb2c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb2b7fe640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb2c106fb0 0x7feb2c19cf50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7feb2400c970 tx=0x7feb2400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb13fff640 1 -- 172.21.15.94:0/1655538515 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb24007bb0 con 0x7feb2c106fb0 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb13fff640 1 -- 172.21.15.94:0/1655538515 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb24007d50 con 0x7feb2c106fb0 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb314eb640 1 -- 172.21.15.94:0/1655538515 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb2c19d970 con 0x7feb2c106fb0 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb13fff640 1 -- 172.21.15.94:0/1655538515 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb24004e60 con 0x7feb2c106fb0 2024-04-21T08:41:26.304 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.300+0000 7feb314eb640 1 -- 172.21.15.94:0/1655538515 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb2c1a5200 con 0x7feb2c106fb0 2024-04-21T08:41:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.302+0000 7feb13fff640 1 -- 172.21.15.94:0/1655538515 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 24) ==== 138930+0+0 (secure 0 0 0) 0x7feb24005020 con 0x7feb2c106fb0 2024-04-21T08:41:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb341b8740 1 Processor -- start 2024-04-21T08:41:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb341b8740 1 -- start start 2024-04-21T08:41:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb13fff640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb140a3f70 0x7feb140a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb341b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ed1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb341b8740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 0x562d24ed1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb341b8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24edd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb341b8740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x562d24e368d0 con 0x562d24ed1b60 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb341b8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562d24dd3e80 con 0x562d24edb0e0 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb341b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562d24e32890 con 0x562d24ed28d0 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb140a3f70 0x7feb140a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb2affd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ed1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb2affd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ed1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:59434/0 (socket says 172.21.15.94:59434) 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb2affd640 1 -- 172.21.15.94:0/1496819709 learned_addr learned my addr 172.21.15.94:0/1496819709 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb2b7fe640 1 --2- 172.21.15.94:0/1496819709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24edd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/1496819709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 0x562d24ed1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.303+0000 7feb13fff640 1 -- 172.21.15.94:0/1655538515 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7feb240d0c40 con 0x7feb2c106fb0 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.304+0000 7feb2affd640 1 -- 172.21.15.94:0/1496819709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 msgr2=0x562d24edd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.304+0000 7feb2affd640 1 --2- 172.21.15.94:0/1496819709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24edd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.304+0000 7feb2affd640 1 -- 172.21.15.94:0/1496819709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 msgr2=0x562d24ed1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.304+0000 7feb2affd640 1 --2- 172.21.15.94:0/1496819709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 0x562d24ed1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.304+0000 7feb2affd640 1 -- 172.21.15.94:0/1496819709 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d24df5320 con 0x562d24ed28d0 2024-04-21T08:41:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.304+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb140a3f70 0x7feb140a6430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7feb2c100c70 tx=0x7feb20004150 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.304+0000 7feb2affd640 1 --2- 172.21.15.94:0/1496819709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ed1620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00af40 tx=0x7feb1c00f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab85dd82150e4ab5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.304+0000 7feb117fa640 1 -- 172.21.15.94:0/1496819709 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c004d20 con 0x562d24ed28d0 2024-04-21T08:41:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.304+0000 7feb117fa640 1 -- 172.21.15.94:0/1496819709 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb1c011b00 con 0x562d24ed28d0 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.304+0000 7feb117fa640 1 -- 172.21.15.94:0/1496819709 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0055f0 con 0x562d24ed28d0 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 -- 172.21.15.94:0/1496819709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 msgr2=0x562d24ed1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 --2- 172.21.15.94:0/1496819709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ed1620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00af40 tx=0x7feb1c00f7f0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 -- 172.21.15.94:0/1496819709 shutdown_connections 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 --2- 172.21.15.94:0/1496819709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24edd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 --2- 172.21.15.94:0/1496819709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 0x562d24ed1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 --2- 172.21.15.94:0/1496819709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ed1620 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 -- 172.21.15.94:0/1496819709 >> 172.21.15.94:0/1496819709 conn(0x562d24e98c90 msgr2=0x562d24ecc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 -- 172.21.15.94:0/1496819709 shutdown_connections 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 -- 172.21.15.94:0/1496819709 wait complete. 2024-04-21T08:41:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 -- start start 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 Processor -- start 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed1b60 0x562d24edaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.305+0000 7feb341b8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24ed83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb341b8740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x562d24edde10 con 0x562d24ed28d0 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb341b8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562d24dbeeb0 con 0x562d24edb0e0 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb341b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562d24eddf90 con 0x562d24ed1b60 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb341b8740 10 client.0 _ll_register_callbacks cb 0x562d24debb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb2a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb2a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55656/0 (socket says 172.21.15.94:55656) 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb2a7fc640 1 -- 172.21.15.94:0/2619940596 learned_addr learned my addr 172.21.15.94:0/2619940596 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb2b7fe640 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24ed83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb2a7fc640 1 -- 172.21.15.94:0/2619940596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed1b60 msgr2=0x562d24edaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed1b60 0x562d24edaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb2a7fc640 1 -- 172.21.15.94:0/2619940596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 msgr2=0x562d24ed83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24ed83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb2a7fc640 1 -- 172.21.15.94:0/2619940596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d24b49210 con 0x562d24ed28d0 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed7e20 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7feb20009900 tx=0x7feb20000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c250aeb8f4be42a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb0eff5640 1 -- 172.21.15.94:0/2619940596 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb200048d0 con 0x562d24ed28d0 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb0eff5640 1 -- 172.21.15.94:0/2619940596 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7feb20004a70 con 0x562d24ed28d0 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb341b8740 1 -- 172.21.15.94:0/2619940596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb1c011b00 con 0x562d24ed28d0 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb0eff5640 1 -- 172.21.15.94:0/2619940596 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7feb2005c050 con 0x562d24ed28d0 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb341b8740 10 client.16242 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.306+0000 7feb341b8740 1 -- 172.21.15.94:0/2619940596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562d24c15620 con 0x562d24ed28d0 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.307+0000 7feb0eff5640 1 -- 172.21.15.94:0/2619940596 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb1c011b00 con 0x562d24ed28d0 2024-04-21T08:41:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.307+0000 7feb0eff5640 1 -- 172.21.15.94:0/2619940596 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7feb20040040 con 0x562d24ed28d0 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.307+0000 7feb341b8740 10 client.16242 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.307+0000 7feb341b8740 10 client.16242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.307+0000 7feb341b8740 20 client.16242 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.307+0000 7feb341b8740 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562d24eeb140 0x562d24eed570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.307+0000 7feb341b8740 4 client.16242 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.307+0000 7feb341b8740 1 -- 172.21.15.94:0/2619940596 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d24e4a880 con 0x562d24eeb140 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.307+0000 7feb2affd640 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562d24eeb140 0x562d24eed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.308+0000 7feb0eff5640 10 client.16242 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.308+0000 7feb2affd640 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562d24eeb140 0x562d24eed570 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1b355b5a2b67106 server_cookie=e6b523f1e1c4b717 in_seq=0 out_seq=0 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.308+0000 7feb0eff5640 1 -- 172.21.15.94:0/2619940596 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d24e4a880 con 0x562d24eeb140 2024-04-21T08:41:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.308+0000 7feb0eff5640 10 client.16242 handle_command_reply: tid=0 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.308+0000 7feb341b8740 1 client.16242 shutdown 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.308+0000 7feb341b8740 1 -- 172.21.15.94:0/2619940596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 msgr2=0x562d24ed7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.308+0000 7feb341b8740 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed7e20 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7feb20009900 tx=0x7feb20000bc0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.309+0000 7feb341b8740 1 -- 172.21.15.94:0/2619940596 shutdown_connections 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.309+0000 7feb341b8740 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562d24eeb140 0x562d24eed570 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.309+0000 7feb341b8740 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24ed83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.309+0000 7feb341b8740 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed7e20 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.309+0000 7feb341b8740 1 --2- 172.21.15.94:0/2619940596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed1b60 0x562d24edaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.309+0000 7feb0eff5640 0 client.16242 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.309+0000 7feb341b8740 1 -- 172.21.15.94:0/2619940596 >> 172.21.15.94:0/2619940596 conn(0x562d24e98c90 msgr2=0x562d24ece5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.309+0000 7feb341b8740 1 -- 172.21.15.94:0/2619940596 shutdown_connections 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.309+0000 7feb341b8740 1 -- 172.21.15.94:0/2619940596 wait complete. 2024-04-21T08:41:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.309+0000 7feb341b8740 20 client.16242 trim_cache size 0 max 16384 2024-04-21T08:41:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.323+0000 7feb341b8740 1 Processor -- start 2024-04-21T08:41:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.323+0000 7feb341b8740 1 -- start start 2024-04-21T08:41:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.323+0000 7feb341b8740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 0x562d24ee6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.323+0000 7feb341b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ef2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.323+0000 7feb341b8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24ef33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.323+0000 7feb341b8740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1b1c60 con 0x562d24ed1b60 2024-04-21T08:41:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.323+0000 7feb341b8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1af530 con 0x562d24edb0e0 2024-04-21T08:41:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.323+0000 7feb341b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1ad380 con 0x562d24ed28d0 2024-04-21T08:41:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 0x562d24ee6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 0x562d24ee6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55672/0 (socket says 172.21.15.94:55672) 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2affd640 1 -- 172.21.15.94:0/1968405884 learned_addr learned my addr 172.21.15.94:0/1968405884 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2b7fe640 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24ef33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2affd640 1 -- 172.21.15.94:0/1968405884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 msgr2=0x562d24ef2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ef2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2affd640 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ef2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2affd640 1 -- 172.21.15.94:0/1968405884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 msgr2=0x562d24ef33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2affd640 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24ef33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2affd640 1 -- 172.21.15.94:0/1968405884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d24eedbf0 con 0x562d24ed1b60 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2b7fe640 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24ef33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ef2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb2affd640 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 0x562d24ee6d10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00acd0 tx=0x7feb1c00afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366a645eac1e5b94 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb10ff9640 1 -- 172.21.15.94:0/1968405884 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c011850 con 0x562d24ed1b60 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb10ff9640 1 -- 172.21.15.94:0/1968405884 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb1c004e90 con 0x562d24ed1b60 2024-04-21T08:41:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.324+0000 7feb10ff9640 1 -- 172.21.15.94:0/1968405884 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c005770 con 0x562d24ed1b60 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 -- 172.21.15.94:0/1968405884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 msgr2=0x562d24ee6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 0x562d24ee6d10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00acd0 tx=0x7feb1c00afd0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 -- 172.21.15.94:0/1968405884 shutdown_connections 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24ed28d0 0x562d24ef2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24edb0e0 0x562d24ef33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 --2- 172.21.15.94:0/1968405884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed1b60 0x562d24ee6d10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 -- 172.21.15.94:0/1968405884 >> 172.21.15.94:0/1968405884 conn(0x562d24e98c90 msgr2=0x562d24ecc380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 -- 172.21.15.94:0/1968405884 shutdown_connections 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 -- 172.21.15.94:0/1968405884 wait complete. 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 -- start start 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 Processor -- start 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24ed1b60 0x562d24ee36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24edb0e0 0x562d24ed4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1ad9f0 con 0x562d24ed28d0 2024-04-21T08:41:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1aee60 con 0x562d24ed1b60 2024-04-21T08:41:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1ae0c0 con 0x562d24edb0e0 2024-04-21T08:41:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb341b8740 10 client.0 _ll_register_callbacks cb 0x7feb2002c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb2a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb2a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55680/0 (socket says 172.21.15.94:55680) 2024-04-21T08:41:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb2a7fc640 1 -- 172.21.15.94:0/4034607797 learned_addr learned my addr 172.21.15.94:0/4034607797 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.325+0000 7feb2b7fe640 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24edb0e0 0x562d24ed4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb2affd640 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24ed1b60 0x562d24ee36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb2a7fc640 1 -- 172.21.15.94:0/4034607797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24edb0e0 msgr2=0x562d24ed4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24edb0e0 0x562d24ed4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb2a7fc640 1 -- 172.21.15.94:0/4034607797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24ed1b60 msgr2=0x562d24ee36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24ed1b60 0x562d24ee36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb2a7fc640 1 -- 172.21.15.94:0/4034607797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d24ed0e30 con 0x562d24ed28d0 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb2affd640 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24ed1b60 0x562d24ee36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb2a7fc640 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed4970 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7feb2002bcc0 tx=0x7feb2002c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9176a2c141967130 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb0eff5640 1 -- 172.21.15.94:0/4034607797 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb200048d0 con 0x562d24ed28d0 2024-04-21T08:41:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb0eff5640 1 -- 172.21.15.94:0/4034607797 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7feb20004a70 con 0x562d24ed28d0 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb341b8740 1 -- 172.21.15.94:0/4034607797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb1c004e90 con 0x562d24ed28d0 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb0eff5640 1 -- 172.21.15.94:0/4034607797 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7feb20004c70 con 0x562d24ed28d0 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb341b8740 10 client.16248 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb341b8740 1 -- 172.21.15.94:0/4034607797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562d24f515a0 con 0x562d24ed28d0 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb0eff5640 1 -- 172.21.15.94:0/4034607797 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb1c004e90 con 0x562d24ed28d0 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb0eff5640 1 -- 172.21.15.94:0/4034607797 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7feb20043870 con 0x562d24ed28d0 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb341b8740 10 client.16248 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb341b8740 10 client.16248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb341b8740 20 client.16248 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb341b8740 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562d24f51cc0 0x562d24f540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb341b8740 4 client.16248 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.326+0000 7feb341b8740 1 -- 172.21.15.94:0/4034607797 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562d24dd63d0 con 0x562d24f51cc0 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.327+0000 7feb2affd640 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562d24f51cc0 0x562d24f540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.327+0000 7feb0eff5640 10 client.16248 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.327+0000 7feb2affd640 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562d24f51cc0 0x562d24f540e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55d763ee08670b8 server_cookie=479959db2b505a58 in_seq=0 out_seq=0 2024-04-21T08:41:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb0eff5640 1 -- 172.21.15.94:0/4034607797 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562d24dd63d0 con 0x562d24f51cc0 2024-04-21T08:41:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb0eff5640 10 client.16248 handle_command_reply: tid=0 2024-04-21T08:41:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 client.16248 shutdown 2024-04-21T08:41:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 -- 172.21.15.94:0/4034607797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 msgr2=0x562d24ed4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed4970 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7feb2002bcc0 tx=0x7feb2002c9e0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 -- 172.21.15.94:0/4034607797 shutdown_connections 2024-04-21T08:41:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562d24f51cc0 0x562d24f540e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562d24edb0e0 0x562d24ed4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562d24ed28d0 0x562d24ed4970 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb0eff5640 0 client.16248 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 --2- 172.21.15.94:0/4034607797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562d24ed1b60 0x562d24ee36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 -- 172.21.15.94:0/4034607797 >> 172.21.15.94:0/4034607797 conn(0x562d24e98c90 msgr2=0x562d24ed6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 -- 172.21.15.94:0/4034607797 shutdown_connections 2024-04-21T08:41:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 1 -- 172.21.15.94:0/4034607797 wait complete. 2024-04-21T08:41:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.337+0000 7feb341b8740 20 client.16248 trim_cache size 0 max 16384 2024-04-21T08:41:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:{ 2024-04-21T08:41:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "message": "", 2024-04-21T08:41:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "return_code": 0 2024-04-21T08:41:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:} 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 -- 172.21.15.94:0/1655538515 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb140a3f70 msgr2=0x7feb140a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb140a3f70 0x7feb140a6430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7feb2c100c70 tx=0x7feb20004150 comp rx=0 tx=0).stop 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 -- 172.21.15.94:0/1655538515 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb2c106fb0 msgr2=0x7feb2c19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb2c106fb0 0x7feb2c19cf50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7feb2400c970 tx=0x7feb2400ce30 comp rx=0 tx=0).stop 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 -- 172.21.15.94:0/1655538515 shutdown_connections 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb140a3f70 0x7feb140a6430 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb2c106fb0 0x7feb2c19cf50 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb2c101b60 0x7feb2c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 --2- 172.21.15.94:0/1655538515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb2c1011a0 0x7feb2c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 -- 172.21.15.94:0/1655538515 >> 172.21.15.94:0/1655538515 conn(0x7feb2c076c50 msgr2=0x7feb2c06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 -- 172.21.15.94:0/1655538515 shutdown_connections 2024-04-21T08:41:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:26.338+0000 7feb314eb640 1 -- 172.21.15.94:0/1655538515 wait complete. 2024-04-21T08:41:27.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:27 smithi111 ceph-mon[31493]: pgmap v47: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 116 MiB/s wr, 47 op/s 2024-04-21T08:41:27.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:27 smithi094 ceph-mon[27988]: pgmap v47: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 116 MiB/s wr, 47 op/s 2024-04-21T08:41:27.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:27 smithi173 ceph-mon[31428]: pgmap v47: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 116 MiB/s wr, 47 op/s 2024-04-21T08:41:28.137 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:28.137 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:28.137 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:28.138 INFO:tasks.workunit.client.0.smithi094.stdout:iozone test complete. 2024-04-21T08:41:28.138 INFO:tasks.workunit.client.0.smithi094.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-04-21T08:41:28.204 INFO:tasks.workunit.client.0.smithi094.stdout: Iozone: Performance Test of File I/O 2024-04-21T08:41:28.204 INFO:tasks.workunit.client.0.smithi094.stdout: Version $Revision: 3.494 $ 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Compiled for 64 bit mode. 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Build: linux-AMD64 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:28.205 INFO:tasks.workunit.client.0.smithi094.stdout: Run began: Sun Apr 21 08:41:28 2024 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: Include close in write timing 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: Include fsync in write timing 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: File size set to 10485760 kB 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: Record Size 1024 kB 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: Output is in kBytes/sec 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: Time Resolution = 0.000001 seconds. 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: Processor cache size set to 1024 kBytes. 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: Processor cache line size set to 32 bytes. 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: File stride size set to 17 * record size. 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: Throughput test with 1 process 2024-04-21T08:41:28.206 INFO:tasks.workunit.client.0.smithi094.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-04-21T08:41:29.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:29 smithi111 ceph-mon[31493]: pgmap v48: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 85 MiB/s wr, 26 op/s 2024-04-21T08:41:29.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:29 smithi094 ceph-mon[27988]: pgmap v48: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 85 MiB/s wr, 26 op/s 2024-04-21T08:41:29.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:29 smithi173 ceph-mon[31428]: pgmap v48: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 85 MiB/s wr, 26 op/s 2024-04-21T08:41:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:31 smithi094 ceph-mon[27988]: pgmap v49: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-04-21T08:41:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:31 smithi094 ceph-mon[27988]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-04-21T08:41:31.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:31 smithi173 ceph-mon[31428]: pgmap v49: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-04-21T08:41:31.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:31 smithi173 ceph-mon[31428]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-04-21T08:41:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:31 smithi111 ceph-mon[31493]: pgmap v49: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-04-21T08:41:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:31 smithi111 ceph-mon[31493]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-04-21T08:41:33.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:33 smithi094 ceph-mon[27988]: pgmap v50: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9.4 KiB/s wr, 11 op/s 2024-04-21T08:41:33.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:33 smithi173 ceph-mon[31428]: pgmap v50: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9.4 KiB/s wr, 11 op/s 2024-04-21T08:41:33.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:33 smithi111 ceph-mon[31493]: pgmap v50: 129 pgs: 129 active+clean; 2.0 GiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9.4 KiB/s wr, 11 op/s 2024-04-21T08:41:35.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:35 smithi173 ceph-mon[31428]: pgmap v51: 129 pgs: 129 active+clean; 4.3 GiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 250 MiB/s wr, 274 op/s 2024-04-21T08:41:35.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:35 smithi094 ceph-mon[27988]: pgmap v51: 129 pgs: 129 active+clean; 4.3 GiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 250 MiB/s wr, 274 op/s 2024-04-21T08:41:35.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:35 smithi111 ceph-mon[31493]: pgmap v51: 129 pgs: 129 active+clean; 4.3 GiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 250 MiB/s wr, 274 op/s 2024-04-21T08:41:38.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:37 smithi173 ceph-mon[31428]: pgmap v52: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 401 MiB/s wr, 317 op/s 2024-04-21T08:41:38.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:37 smithi094 ceph-mon[27988]: pgmap v52: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 401 MiB/s wr, 317 op/s 2024-04-21T08:41:38.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:37 smithi111 ceph-mon[31493]: pgmap v52: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 401 MiB/s wr, 317 op/s 2024-04-21T08:41:39.041 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:39.041 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 initial writers = 970489.19 kB/sec 2024-04-21T08:41:39.041 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 initial writers = 970467.89 kB/sec 2024-04-21T08:41:39.041 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 970489.19 kB/sec 2024-04-21T08:41:39.041 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 970489.19 kB/sec 2024-04-21T08:41:39.041 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 970489.19 kB/sec 2024-04-21T08:41:39.041 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 10485760.00 kB 2024-04-21T08:41:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 -- 172.21.15.94:0/817924617 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63fc10e7f0 msgr2=0x7f63fc110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 --2- 172.21.15.94:0/817924617 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63fc10e7f0 0x7f63fc110be0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009ca0 tx=0x7f63f402c180 comp rx=0 tx=0).stop 2024-04-21T08:41:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 -- 172.21.15.94:0/817924617 shutdown_connections 2024-04-21T08:41:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 --2- 172.21.15.94:0/817924617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63fc074c20 0x7f63fc10e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 --2- 172.21.15.94:0/817924617 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63fc10e7f0 0x7f63fc110be0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 --2- 172.21.15.94:0/817924617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63fc074300 0x7f63fc0746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 -- 172.21.15.94:0/817924617 >> 172.21.15.94:0/817924617 conn(0x7f63fc0fda00 msgr2=0x7f63fc0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 -- 172.21.15.94:0/817924617 shutdown_connections 2024-04-21T08:41:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 -- 172.21.15.94:0/817924617 wait complete. 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 Processor -- start 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.434+0000 7f6401470640 1 -- start start 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f6401470640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63fc074300 0x7f63fc1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f6401470640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63fc074c20 0x7f63fc1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f6401470640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63fc10e7f0 0x7f63fc19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f6401470640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1143a0 con 0x7f63fc10e7f0 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f6401470640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc114220 con 0x7f63fc074c20 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f6401470640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc114520 con 0x7f63fc074300 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63fc10e7f0 0x7f63fc19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63fc10e7f0 0x7f63fc19a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51880/0 (socket says 172.21.15.94:51880) 2024-04-21T08:41:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63fb7fe640 1 -- 172.21.15.94:0/1892203507 learned_addr learned my addr 172.21.15.94:0/1892203507 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63faffd640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63fc074300 0x7f63fc1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63f3fff640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63fc074c20 0x7f63fc1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63fb7fe640 1 -- 172.21.15.94:0/1892203507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63fc074300 msgr2=0x7f63fc1a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63fc074300 0x7f63fc1a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63fb7fe640 1 -- 172.21.15.94:0/1892203507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63fc074c20 msgr2=0x7f63fc1a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63fc074c20 0x7f63fc1a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63fb7fe640 1 -- 172.21.15.94:0/1892203507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63fc19af40 con 0x7f63fc10e7f0 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63faffd640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63fc074300 0x7f63fc1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63fc10e7f0 0x7f63fc19a680 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f63f402c790 tx=0x7f63f4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63f8ff9640 1 -- 172.21.15.94:0/1892203507 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4039040 con 0x7f63fc10e7f0 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63f8ff9640 1 -- 172.21.15.94:0/1892203507 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63f4044070 con 0x7f63fc10e7f0 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f6401470640 1 -- 172.21.15.94:0/1892203507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63fc19b1d0 con 0x7f63fc10e7f0 2024-04-21T08:41:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.435+0000 7f63f8ff9640 1 -- 172.21.15.94:0/1892203507 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4032850 con 0x7f63fc10e7f0 2024-04-21T08:41:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.436+0000 7f6401470640 1 -- 172.21.15.94:0/1892203507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63fc1a7360 con 0x7f63fc10e7f0 2024-04-21T08:41:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.438+0000 7f63f8ff9640 1 -- 172.21.15.94:0/1892203507 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f63f4004240 con 0x7f63fc10e7f0 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.438+0000 7f640405a740 1 Processor -- start 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.438+0000 7f63f8ff9640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e80a3dc0 0x7f63e80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.438+0000 7f640405a740 1 -- start start 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.438+0000 7f63f8ff9640 1 -- 172.21.15.94:0/1892203507 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f63f40f30a0 con 0x7f63fc10e7f0 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.438+0000 7f63f3fff640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e80a3dc0 0x7f63e80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f640405a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 0x56455ece7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f640405a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ece7b60 0x56455ece7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f640405a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecf34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f640405a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56455ec4c8d0 con 0x56455ecf10e0 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f640405a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56455ebe9e80 con 0x56455ece88d0 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f640405a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56455ec48890 con 0x56455ece7b60 2024-04-21T08:41:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecf34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecf34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51896/0 (socket says 172.21.15.94:51896) 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63fb7fe640 1 -- 172.21.15.94:0/1790781124 learned_addr learned my addr 172.21.15.94:0/1790781124 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63f3fff640 1 --2- 172.21.15.94:0/1790781124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ece7b60 0x56455ece7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63faffd640 1 --2- 172.21.15.94:0/1790781124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 0x56455ece7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63f3fff640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e80a3dc0 0x7f63e80a6280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f63fc19bed0 tx=0x7f63e4009ee0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63fb7fe640 1 -- 172.21.15.94:0/1790781124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ece7b60 msgr2=0x56455ece7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/1790781124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ece7b60 0x56455ece7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63fb7fe640 1 -- 172.21.15.94:0/1790781124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 msgr2=0x56455ece7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/1790781124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 0x56455ece7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63fb7fe640 1 -- 172.21.15.94:0/1790781124 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56455ec0b320 con 0x56455ecf10e0 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/1790781124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecf34d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f63f40f0a10 tx=0x7f63f40bb0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839e7db42625c495 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63f17fa640 1 -- 172.21.15.94:0/1790781124 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f40d8070 con 0x56455ecf10e0 2024-04-21T08:41:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63f17fa640 1 -- 172.21.15.94:0/1790781124 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63f4003e10 con 0x56455ecf10e0 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.439+0000 7f63f17fa640 1 -- 172.21.15.94:0/1790781124 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4004530 con 0x56455ecf10e0 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.440+0000 7f640405a740 1 -- 172.21.15.94:0/1790781124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 msgr2=0x56455ecf34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.440+0000 7f640405a740 1 --2- 172.21.15.94:0/1790781124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecf34d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f63f40f0a10 tx=0x7f63f40bb0e0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.440+0000 7f640405a740 1 -- 172.21.15.94:0/1790781124 shutdown_connections 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.440+0000 7f640405a740 1 --2- 172.21.15.94:0/1790781124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ece7b60 0x56455ece7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.440+0000 7f640405a740 1 --2- 172.21.15.94:0/1790781124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecf34d0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.440+0000 7f640405a740 1 --2- 172.21.15.94:0/1790781124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 0x56455ece7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.440+0000 7f640405a740 1 -- 172.21.15.94:0/1790781124 >> 172.21.15.94:0/1790781124 conn(0x56455ecaec90 msgr2=0x56455ece2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.440+0000 7f640405a740 1 -- 172.21.15.94:0/1790781124 shutdown_connections 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.440+0000 7f640405a740 1 -- 172.21.15.94:0/1790781124 wait complete. 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f640405a740 1 -- start start 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f640405a740 1 Processor -- start 2024-04-21T08:41:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f640405a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 0x56455ecf0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f640405a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ece88d0 0x56455ecede50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f640405a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecee460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f640405a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56455ecf3bd0 con 0x56455ecf10e0 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f640405a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56455ebd4eb0 con 0x56455ece7b60 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f640405a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56455ecf3d50 con 0x56455ece88d0 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f640405a740 10 client.0 _ll_register_callbacks cb 0x56455ec01b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f63fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecee460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f63fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecee460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51910/0 (socket says 172.21.15.94:51910) 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f63fb7fe640 1 -- 172.21.15.94:0/2916142267 learned_addr learned my addr 172.21.15.94:0/2916142267 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f63f3fff640 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ece88d0 0x56455ecede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f63faffd640 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 0x56455ecf0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f63fb7fe640 1 -- 172.21.15.94:0/2916142267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ece88d0 msgr2=0x56455ecede50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:39.449 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ece88d0 0x56455ecede50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f63fb7fe640 1 -- 172.21.15.94:0/2916142267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 msgr2=0x56455ecf0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 0x56455ecf0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.441+0000 7f63fb7fe640 1 -- 172.21.15.94:0/2916142267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56455e95f210 con 0x56455ecf10e0 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecee460 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f63f40bb1d0 tx=0x7f63f4035f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b6ae8b3a22483f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f63e27fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f63e27fc640 1 -- 172.21.15.94:0/2916142267 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f40bf470 con 0x56455ecf10e0 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f63e27fc640 1 -- 172.21.15.94:0/2916142267 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f63f40f0600 con 0x56455ecf10e0 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f640405a740 1 -- 172.21.15.94:0/2916142267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56455ec60880 con 0x56455ecf10e0 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f63e27fc640 1 -- 172.21.15.94:0/2916142267 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f63f40f0820 con 0x56455ecf10e0 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f640405a740 10 client.16260 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f640405a740 1 -- 172.21.15.94:0/2916142267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56455ea2b620 con 0x56455ecf10e0 2024-04-21T08:41:39.450 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f63e27fc640 1 -- 172.21.15.94:0/2916142267 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56455ec60880 con 0x56455ecf10e0 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f63e27fc640 1 -- 172.21.15.94:0/2916142267 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f63f403c030 con 0x56455ecf10e0 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f640405a740 10 client.16260 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f640405a740 10 client.16260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f640405a740 20 client.16260 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f640405a740 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56455ed00cc0 0x56455ed030b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f640405a740 4 client.16260 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.442+0000 7f640405a740 1 -- 172.21.15.94:0/2916142267 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56455ebec3d0 con 0x56455ed00cc0 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.443+0000 7f63faffd640 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56455ed00cc0 0x56455ed030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.443+0000 7f63e27fc640 10 client.16260 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.443+0000 7f63faffd640 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56455ed00cc0 0x56455ed030b0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20da7d7ccd5fa07c server_cookie=8c7269a4a52412a8 in_seq=0 out_seq=0 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f63e27fc640 1 -- 172.21.15.94:0/2916142267 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56455ebec3d0 con 0x56455ed00cc0 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f63e27fc640 10 client.16260 handle_command_reply: tid=0 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 client.16260 shutdown 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 -- 172.21.15.94:0/2916142267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 msgr2=0x56455ecee460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:39.451 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecee460 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f63f40bb1d0 tx=0x7f63f4035f90 comp rx=0 tx=0).stop 2024-04-21T08:41:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 -- 172.21.15.94:0/2916142267 shutdown_connections 2024-04-21T08:41:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56455ed00cc0 0x56455ed030b0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ece88d0 0x56455ecede50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ecf10e0 0x56455ecee460 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f63e27fc640 0 client.16260 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 --2- 172.21.15.94:0/2916142267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 0x56455ecf0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 -- 172.21.15.94:0/2916142267 >> 172.21.15.94:0/2916142267 conn(0x56455ecaec90 msgr2=0x56455ece45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 -- 172.21.15.94:0/2916142267 shutdown_connections 2024-04-21T08:41:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 1 -- 172.21.15.94:0/2916142267 wait complete. 2024-04-21T08:41:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.444+0000 7f640405a740 20 client.16260 trim_cache size 0 max 16384 2024-04-21T08:41:39.460 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f640405a740 1 Processor -- start 2024-04-21T08:41:39.460 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f640405a740 1 -- start start 2024-04-21T08:41:39.465 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f640405a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 0x56455ecece30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.465 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f640405a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece88d0 0x56455ecf0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f640405a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 0x56455ecf0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f640405a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1b3e70 con 0x56455ece88d0 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f640405a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1b1740 con 0x56455ece7b60 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f640405a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1af590 con 0x56455ecf10e0 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f63f3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece88d0 0x56455ecf0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f63faffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 0x56455ecece30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f63f3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece88d0 0x56455ecf0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51926/0 (socket says 172.21.15.94:51926) 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f63f3fff640 1 -- 172.21.15.94:0/2053582024 learned_addr learned my addr 172.21.15.94:0/2053582024 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/2053582024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 0x56455ecf0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f63f3fff640 1 -- 172.21.15.94:0/2053582024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 msgr2=0x56455ecf0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f63f3fff640 1 --2- 172.21.15.94:0/2053582024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 0x56455ecf0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f63f3fff640 1 -- 172.21.15.94:0/2053582024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 msgr2=0x56455ecece30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.461+0000 7f63f3fff640 1 --2- 172.21.15.94:0/2053582024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 0x56455ecece30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.462+0000 7f63f3fff640 1 -- 172.21.15.94:0/2053582024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56455ed03640 con 0x56455ece88d0 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.462+0000 7f63fb7fe640 1 --2- 172.21.15.94:0/2053582024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 0x56455ecf0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.462+0000 7f63faffd640 1 --2- 172.21.15.94:0/2053582024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 0x56455ecece30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.462+0000 7f63f3fff640 1 --2- 172.21.15.94:0/2053582024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece88d0 0x56455ecf0580 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009fd0 tx=0x7f63e4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fc1072014ab54 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.462+0000 7f63f0ff9640 1 -- 172.21.15.94:0/2053582024 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e4004050 con 0x56455ece88d0 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.462+0000 7f63f0ff9640 1 -- 172.21.15.94:0/2053582024 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e40041f0 con 0x56455ece88d0 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.462+0000 7f63f0ff9640 1 -- 172.21.15.94:0/2053582024 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e401f680 con 0x56455ece88d0 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 -- 172.21.15.94:0/2053582024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece88d0 msgr2=0x56455ecf0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 --2- 172.21.15.94:0/2053582024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece88d0 0x56455ecf0580 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009fd0 tx=0x7f63e4016000 comp rx=0 tx=0).stop 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 -- 172.21.15.94:0/2053582024 shutdown_connections 2024-04-21T08:41:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 --2- 172.21.15.94:0/2053582024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 0x56455ecf0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 --2- 172.21.15.94:0/2053582024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece88d0 0x56455ecf0580 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 --2- 172.21.15.94:0/2053582024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece7b60 0x56455ecece30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 -- 172.21.15.94:0/2053582024 >> 172.21.15.94:0/2053582024 conn(0x56455ecaec90 msgr2=0x56455ecec720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 -- 172.21.15.94:0/2053582024 shutdown_connections 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 -- 172.21.15.94:0/2053582024 wait complete. 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 -- start start 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 Processor -- start 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece7b60 0x56455ecf1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 0x56455ecf21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 0x56455ecf2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1afc00 con 0x56455ece7b60 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1b1070 con 0x56455ece88d0 2024-04-21T08:41:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1b02d0 con 0x56455ecf10e0 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.463+0000 7f640405a740 10 client.0 _ll_register_callbacks cb 0x7f63f40d6070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63faffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece7b60 0x56455ecf1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63f3fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 0x56455ecf21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63fb7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 0x56455ecf2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63faffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece7b60 0x56455ecf1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51938/0 (socket says 172.21.15.94:51938) 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63faffd640 1 -- 172.21.15.94:0/854910278 learned_addr learned my addr 172.21.15.94:0/854910278 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63f3fff640 1 -- 172.21.15.94:0/854910278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 msgr2=0x56455ecf2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63f3fff640 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 0x56455ecf2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63f3fff640 1 -- 172.21.15.94:0/854910278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece7b60 msgr2=0x56455ecf1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63f3fff640 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece7b60 0x56455ecf1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63f3fff640 1 -- 172.21.15.94:0/854910278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56455ece6e30 con 0x56455ece88d0 2024-04-21T08:41:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63faffd640 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece7b60 0x56455ecf1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63f3fff640 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 0x56455ecf21d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f63e40099b0 tx=0x7f63e40056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bb516b92cfbe962 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63e27fc640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63e27fc640 1 -- 172.21.15.94:0/854910278 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e4023070 con 0x56455ece88d0 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f640405a740 1 -- 172.21.15.94:0/854910278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63e40041f0 con 0x56455ece88d0 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63e27fc640 1 -- 172.21.15.94:0/854910278 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f63e4003fb0 con 0x56455ece88d0 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.464+0000 7f63e27fc640 1 -- 172.21.15.94:0/854910278 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f63e403b050 con 0x56455ece88d0 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f640405a740 10 client.25490 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f640405a740 1 -- 172.21.15.94:0/854910278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56455ed67200 con 0x56455ece88d0 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f63e27fc640 1 -- 172.21.15.94:0/854910278 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e40041f0 con 0x56455ece88d0 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f63e27fc640 1 -- 172.21.15.94:0/854910278 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f63e40354d0 con 0x56455ece88d0 2024-04-21T08:41:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f640405a740 10 client.25490 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f640405a740 10 client.25490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f640405a740 20 client.25490 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f640405a740 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56455ed67900 0x56455ed69d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f640405a740 4 client.25490 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:41:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f640405a740 1 -- 172.21.15.94:0/854910278 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56455ec5a2f0 con 0x56455ed67900 2024-04-21T08:41:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f63faffd640 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56455ed67900 0x56455ed69d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f63faffd640 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56455ed67900 0x56455ed69d20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13aa9993fbe7769b server_cookie=1d2fc7e0e31849a1 in_seq=0 out_seq=0 2024-04-21T08:41:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.465+0000 7f63e27fc640 10 client.25490 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.473+0000 7f63e27fc640 1 -- 172.21.15.94:0/854910278 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56455ec5a2f0 con 0x56455ed67900 2024-04-21T08:41:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.473+0000 7f63e27fc640 10 client.25490 handle_command_reply: tid=0 2024-04-21T08:41:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.473+0000 7f640405a740 1 client.25490 shutdown 2024-04-21T08:41:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 1 -- 172.21.15.94:0/854910278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 msgr2=0x56455ecf21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 0x56455ecf21d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f63e40099b0 tx=0x7f63e40056d0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 1 -- 172.21.15.94:0/854910278 shutdown_connections 2024-04-21T08:41:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56455ed67900 0x56455ed69d20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56455ecf10e0 0x56455ecf2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f63e27fc640 0 client.25490 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56455ece88d0 0x56455ecf21d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 1 --2- 172.21.15.94:0/854910278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56455ece7b60 0x56455ecf1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 1 -- 172.21.15.94:0/854910278 >> 172.21.15.94:0/854910278 conn(0x56455ecaec90 msgr2=0x56455ececfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 1 -- 172.21.15.94:0/854910278 shutdown_connections 2024-04-21T08:41:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 1 -- 172.21.15.94:0/854910278 wait complete. 2024-04-21T08:41:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f640405a740 20 client.25490 trim_cache size 0 max 16384 2024-04-21T08:41:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:{ 2024-04-21T08:41:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "message": "", 2024-04-21T08:41:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "return_code": 0 2024-04-21T08:41:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:} 2024-04-21T08:41:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f6401470640 1 -- 172.21.15.94:0/1892203507 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e80a3dc0 msgr2=0x7f63e80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f6401470640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e80a3dc0 0x7f63e80a6280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f63fc19bed0 tx=0x7f63e4009ee0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f6401470640 1 -- 172.21.15.94:0/1892203507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63fc10e7f0 msgr2=0x7f63fc19a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f6401470640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63fc10e7f0 0x7f63fc19a680 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f63f402c790 tx=0x7f63f4003c20 comp rx=0 tx=0).stop 2024-04-21T08:41:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f6401470640 1 -- 172.21.15.94:0/1892203507 shutdown_connections 2024-04-21T08:41:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f6401470640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e80a3dc0 0x7f63e80a6280 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f6401470640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63fc10e7f0 0x7f63fc19a680 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f6401470640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63fc074c20 0x7f63fc1a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f6401470640 1 --2- 172.21.15.94:0/1892203507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63fc074300 0x7f63fc1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.474+0000 7f6401470640 1 -- 172.21.15.94:0/1892203507 >> 172.21.15.94:0/1892203507 conn(0x7f63fc0fda00 msgr2=0x7f63fc0fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.475+0000 7f6401470640 1 -- 172.21.15.94:0/1892203507 shutdown_connections 2024-04-21T08:41:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:39.475+0000 7f6401470640 1 -- 172.21.15.94:0/1892203507 wait complete. 2024-04-21T08:41:40.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:39 smithi094 ceph-mon[27988]: pgmap v53: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 401 MiB/s wr, 317 op/s 2024-04-21T08:41:40.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:39 smithi173 ceph-mon[31428]: pgmap v53: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 401 MiB/s wr, 317 op/s 2024-04-21T08:41:40.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:39 smithi111 ceph-mon[31493]: pgmap v53: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 401 MiB/s wr, 317 op/s 2024-04-21T08:41:40.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:41:40.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:41:41.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.017+0000 7f3f531b2640 1 -- 172.21.15.94:0/2378764464 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06a860 msgr2=0x7f3f4c06ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.017+0000 7f3f531b2640 1 --2- 172.21.15.94:0/2378764464 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06a860 0x7f3f4c06ac40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009900 tx=0x7f3f3c02bc50 comp rx=0 tx=0).stop 2024-04-21T08:41:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 -- 172.21.15.94:0/2378764464 shutdown_connections 2024-04-21T08:41:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 --2- 172.21.15.94:0/2378764464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f4c06b210 0x7f3f4c106be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 --2- 172.21.15.94:0/2378764464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f4c107310 0x7f3f4c10ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 --2- 172.21.15.94:0/2378764464 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06a860 0x7f3f4c06ac40 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 -- 172.21.15.94:0/2378764464 >> 172.21.15.94:0/2378764464 conn(0x7f3f4c077c40 msgr2=0x7f3f4c078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 -- 172.21.15.94:0/2378764464 shutdown_connections 2024-04-21T08:41:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 -- 172.21.15.94:0/2378764464 wait complete. 2024-04-21T08:41:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 Processor -- start 2024-04-21T08:41:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 -- start start 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06a860 0x7f3f4c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f4c107310 0x7f3f4c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f4c19cfb0 0x7f3f4c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c112220 con 0x7f3f4c06a860 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1120a0 con 0x7f3f4c107310 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f531b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1123a0 con 0x7f3f4c19cfb0 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.018+0000 7f3f50f4e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06a860 0x7f3f4c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f50f4e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06a860 0x7f3f4c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51968/0 (socket says 172.21.15.94:51968) 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f50f4e640 1 -- 172.21.15.94:0/1435251399 learned_addr learned my addr 172.21.15.94:0/1435251399 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f5174f640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f4c19cfb0 0x7f3f4c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f4c107310 0x7f3f4c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f50f4e640 1 -- 172.21.15.94:0/1435251399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f4c19cfb0 msgr2=0x7f3f4c19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f4c19cfb0 0x7f3f4c19d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f50f4e640 1 -- 172.21.15.94:0/1435251399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f4c107310 msgr2=0x7f3f4c198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f4c107310 0x7f3f4c198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f50f4e640 1 -- 172.21.15.94:0/1435251399 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4c19daa0 con 0x7f3f4c06a860 2024-04-21T08:41:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f4c107310 0x7f3f4c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:41.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06a860 0x7f3f4c198740 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c005940 tx=0x7f3f3c02c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f49ffb640 1 -- 172.21.15.94:0/1435251399 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c043020 con 0x7f3f4c06a860 2024-04-21T08:41:41.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f49ffb640 1 -- 172.21.15.94:0/1435251399 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f3c02cc80 con 0x7f3f4c06a860 2024-04-21T08:41:41.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f49ffb640 1 -- 172.21.15.94:0/1435251399 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c003c30 con 0x7f3f4c06a860 2024-04-21T08:41:41.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f531b2640 1 -- 172.21.15.94:0/1435251399 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4c19dd30 con 0x7f3f4c06a860 2024-04-21T08:41:41.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.019+0000 7f3f531b2640 1 -- 172.21.15.94:0/1435251399 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f4c1a55d0 con 0x7f3f4c06a860 2024-04-21T08:41:41.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.021+0000 7f3f49ffb640 1 -- 172.21.15.94:0/1435251399 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f3f3c046020 con 0x7f3f4c06a860 2024-04-21T08:41:41.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f55da7740 1 Processor -- start 2024-04-21T08:41:41.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f55da7740 1 -- start start 2024-04-21T08:41:41.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f49ffb640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3f340a3d20 0x7f3f340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e70580 0x55d240e70960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70ea0 0x55d240e79540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e79a80 0x55d240e7be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f55da7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d240dd51e0 con 0x55d240e70ea0 2024-04-21T08:41:41.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f55da7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d240d72390 con 0x55d240e79a80 2024-04-21T08:41:41.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f55da7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d240dd11a0 con 0x55d240e70580 2024-04-21T08:41:41.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f5174f640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3f340a3d20 0x7f3f340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f49ffb640 1 -- 172.21.15.94:0/1435251399 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3f3c0f2f80 con 0x7f3f4c06a860 2024-04-21T08:41:41.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f4bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70ea0 0x55d240e79540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f4bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70ea0 0x55d240e79540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51984/0 (socket says 172.21.15.94:51984) 2024-04-21T08:41:41.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f4bfff640 1 -- 172.21.15.94:0/4088732162 learned_addr learned my addr 172.21.15.94:0/4088732162 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.022+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/4088732162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e70580 0x55d240e70960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f5174f640 1 --2- 172.21.15.94:0/4088732162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e79a80 0x55d240e7be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f4bfff640 1 -- 172.21.15.94:0/4088732162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e70580 msgr2=0x55d240e70960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/4088732162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e70580 0x55d240e70960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f4bfff640 1 -- 172.21.15.94:0/4088732162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e79a80 msgr2=0x55d240e7be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/4088732162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e79a80 0x55d240e7be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f4bfff640 1 -- 172.21.15.94:0/4088732162 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d240bb17d0 con 0x55d240e70ea0 2024-04-21T08:41:41.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f5174f640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3f340a3d20 0x7f3f340a61e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3f44007f10 tx=0x7f3f440127c0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/4088732162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70ea0 0x55d240e79540 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000a940 tx=0x7f3f4000ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe4ea6e5c22b1cf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f32ffd640 1 -- 172.21.15.94:0/4088732162 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40015070 con 0x55d240e70ea0 2024-04-21T08:41:41.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f32ffd640 1 -- 172.21.15.94:0/4088732162 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f4000cc00 con 0x55d240e70ea0 2024-04-21T08:41:41.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.023+0000 7f3f32ffd640 1 -- 172.21.15.94:0/4088732162 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40018560 con 0x55d240e70ea0 2024-04-21T08:41:41.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 -- 172.21.15.94:0/4088732162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70ea0 msgr2=0x55d240e79540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 --2- 172.21.15.94:0/4088732162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70ea0 0x55d240e79540 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000a940 tx=0x7f3f4000ae00 comp rx=0 tx=0).stop 2024-04-21T08:41:41.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 -- 172.21.15.94:0/4088732162 shutdown_connections 2024-04-21T08:41:41.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 --2- 172.21.15.94:0/4088732162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e79a80 0x55d240e7be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 --2- 172.21.15.94:0/4088732162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70ea0 0x55d240e79540 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 --2- 172.21.15.94:0/4088732162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e70580 0x55d240e70960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 -- 172.21.15.94:0/4088732162 >> 172.21.15.94:0/4088732162 conn(0x55d240e6acb0 msgr2=0x55d240e6b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 -- 172.21.15.94:0/4088732162 shutdown_connections 2024-04-21T08:41:41.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 -- 172.21.15.94:0/4088732162 wait complete. 2024-04-21T08:41:41.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 -- start start 2024-04-21T08:41:41.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.024+0000 7f3f55da7740 1 Processor -- start 2024-04-21T08:41:41.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240e76a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240e76fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 0x55d240e77500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f55da7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d240e7c870 con 0x55d240e70580 2024-04-21T08:41:41.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f55da7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d240d8e2f0 con 0x55d240e70ea0 2024-04-21T08:41:41.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f55da7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d240e7c9f0 con 0x55d240e79a80 2024-04-21T08:41:41.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f55da7740 10 client.0 _ll_register_callbacks cb 0x55d240d9b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:41.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f50f4e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240e76a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f50f4e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240e76a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51994/0 (socket says 172.21.15.94:51994) 2024-04-21T08:41:41.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f50f4e640 1 -- 172.21.15.94:0/2846588907 learned_addr learned my addr 172.21.15.94:0/2846588907 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240e76fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f5174f640 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 0x55d240e77500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f50f4e640 1 -- 172.21.15.94:0/2846588907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 msgr2=0x55d240e77500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 0x55d240e77500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f50f4e640 1 -- 172.21.15.94:0/2846588907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 msgr2=0x55d240e76fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240e76fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f50f4e640 1 -- 172.21.15.94:0/2846588907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d240ae73f0 con 0x55d240e70580 2024-04-21T08:41:41.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240e76fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:41.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f5174f640 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 0x55d240e77500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:41.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240e76a80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c000c00 tx=0x7f3f3c0be670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94185f2647be9815 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f2bfff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:41.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f2bfff640 1 -- 172.21.15.94:0/2846588907 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0bf350 con 0x55d240e70580 2024-04-21T08:41:41.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f2bfff640 1 -- 172.21.15.94:0/2846588907 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3f3c0d8790 con 0x55d240e70580 2024-04-21T08:41:41.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f55da7740 1 -- 172.21.15.94:0/2846588907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f4000cc00 con 0x55d240e70580 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f2bfff640 1 -- 172.21.15.94:0/2846588907 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3f3c0bec50 con 0x55d240e70580 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.025+0000 7f3f55da7740 10 client.16278 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.026+0000 7f3f55da7740 1 -- 172.21.15.94:0/2846588907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d240aa70f0 con 0x55d240e70580 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.026+0000 7f3f2bfff640 1 -- 172.21.15.94:0/2846588907 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f4000cc00 con 0x55d240e70580 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.026+0000 7f3f2bfff640 1 -- 172.21.15.94:0/2846588907 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3f3c0df0b0 con 0x55d240e70580 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.026+0000 7f3f55da7740 10 client.16278 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.026+0000 7f3f55da7740 10 client.16278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.026+0000 7f3f55da7740 20 client.16278 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.026+0000 7f3f55da7740 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d240e89aa0 0x55d240e8bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.026+0000 7f3f55da7740 4 client.16278 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.026+0000 7f3f55da7740 1 -- 172.21.15.94:0/2846588907 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d240de2c00 con 0x55d240e89aa0 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.026+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d240e89aa0 0x55d240e8bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d240e89aa0 0x55d240e8bef0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63ffc04047500151 server_cookie=92f6bd0dbe133257 in_seq=0 out_seq=0 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f2bfff640 10 client.16278 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f2bfff640 1 -- 172.21.15.94:0/2846588907 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d240de2c00 con 0x55d240e89aa0 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f2bfff640 10 client.16278 handle_command_reply: tid=0 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f55da7740 1 client.16278 shutdown 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f55da7740 1 -- 172.21.15.94:0/2846588907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 msgr2=0x55d240e76a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f55da7740 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240e76a80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c000c00 tx=0x7f3f3c0be670 comp rx=0 tx=0).stop 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f55da7740 1 -- 172.21.15.94:0/2846588907 shutdown_connections 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f55da7740 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d240e89aa0 0x55d240e8bef0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f55da7740 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 0x55d240e77500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f55da7740 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240e76fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f55da7740 1 --2- 172.21.15.94:0/2846588907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240e76a80 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f2bfff640 0 client.16278 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.027+0000 7f3f55da7740 1 -- 172.21.15.94:0/2846588907 >> 172.21.15.94:0/2846588907 conn(0x55d240e6acb0 msgr2=0x55d240e6ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.028+0000 7f3f55da7740 1 -- 172.21.15.94:0/2846588907 shutdown_connections 2024-04-21T08:41:41.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.028+0000 7f3f55da7740 1 -- 172.21.15.94:0/2846588907 wait complete. 2024-04-21T08:41:41.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.028+0000 7f3f55da7740 20 client.16278 trim_cache size 0 max 16384 2024-04-21T08:41:41.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.045+0000 7f3f55da7740 1 Processor -- start 2024-04-21T08:41:41.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.045+0000 7f3f55da7740 1 -- start start 2024-04-21T08:41:41.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.045+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e70580 0x55d240e791b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.045+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240e85760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.045+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e79a80 0x55d240e82050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.045+0000 7f3f55da7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1b1b90 con 0x55d240e79a80 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.045+0000 7f3f55da7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1af460 con 0x55d240e70ea0 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.045+0000 7f3f55da7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1ad2b0 con 0x55d240e70580 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.045+0000 7f3f5174f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e79a80 0x55d240e82050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f5174f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e79a80 0x55d240e82050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52006/0 (socket says 172.21.15.94:52006) 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f5174f640 1 -- 172.21.15.94:0/4072622522 learned_addr learned my addr 172.21.15.94:0/4072622522 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/4072622522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e70580 0x55d240e791b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/4072622522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240e85760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f5174f640 1 -- 172.21.15.94:0/4072622522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e70580 msgr2=0x55d240e791b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f5174f640 1 --2- 172.21.15.94:0/4072622522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e70580 0x55d240e791b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f5174f640 1 -- 172.21.15.94:0/4072622522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 msgr2=0x55d240e85760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f5174f640 1 --2- 172.21.15.94:0/4072622522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240e85760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f5174f640 1 -- 172.21.15.94:0/4072622522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d240e7be70 con 0x55d240e79a80 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f5174f640 1 --2- 172.21.15.94:0/4072622522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e79a80 0x55d240e82050 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3f44007cf0 tx=0x7f3f44015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72f911a8b03ee60 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f327fc640 1 -- 172.21.15.94:0/4072622522 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4401e850 con 0x55d240e79a80 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f327fc640 1 -- 172.21.15.94:0/4072622522 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f44028410 con 0x55d240e79a80 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.046+0000 7f3f327fc640 1 -- 172.21.15.94:0/4072622522 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4401d630 con 0x55d240e79a80 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 -- 172.21.15.94:0/4072622522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e79a80 msgr2=0x55d240e82050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 --2- 172.21.15.94:0/4072622522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e79a80 0x55d240e82050 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3f44007cf0 tx=0x7f3f44015000 comp rx=0 tx=0).stop 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 -- 172.21.15.94:0/4072622522 shutdown_connections 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 --2- 172.21.15.94:0/4072622522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e79a80 0x55d240e82050 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 --2- 172.21.15.94:0/4072622522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240e85760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 --2- 172.21.15.94:0/4072622522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e70580 0x55d240e791b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 -- 172.21.15.94:0/4072622522 >> 172.21.15.94:0/4072622522 conn(0x55d240e6acb0 msgr2=0x55d240e694e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 -- 172.21.15.94:0/4072622522 shutdown_connections 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 -- 172.21.15.94:0/4072622522 wait complete. 2024-04-21T08:41:41.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 -- start start 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.047+0000 7f3f55da7740 1 Processor -- start 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240ee8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240ee8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f55da7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 0x55d240ee8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f55da7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1ad920 con 0x55d240e70580 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f55da7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1aed90 con 0x55d240e70ea0 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f55da7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1adff0 con 0x55d240e79a80 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f55da7740 10 client.0 _ll_register_callbacks cb 0x7f3f3c0057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f50f4e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240ee8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f50f4e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240ee8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52022/0 (socket says 172.21.15.94:52022) 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f50f4e640 1 -- 172.21.15.94:0/1294718441 learned_addr learned my addr 172.21.15.94:0/1294718441 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f5174f640 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 0x55d240ee8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240ee8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f50f4e640 1 -- 172.21.15.94:0/1294718441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 msgr2=0x55d240ee8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 0x55d240ee8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f50f4e640 1 -- 172.21.15.94:0/1294718441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 msgr2=0x55d240ee8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240ee8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f50f4e640 1 -- 172.21.15.94:0/1294718441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d240ef3350 con 0x55d240e70580 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f50f4e640 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240ee8030 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0bcf00 tx=0x7f3f3c003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1515adba305fd25 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f2bfff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f2bfff640 1 -- 172.21.15.94:0/1294718441 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0bea20 con 0x55d240e70580 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f2bfff640 1 -- 172.21.15.94:0/1294718441 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3f3c0c7630 con 0x55d240e70580 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f55da7740 1 -- 172.21.15.94:0/1294718441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f44028410 con 0x55d240e70580 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f2bfff640 1 -- 172.21.15.94:0/1294718441 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3f3c0be790 con 0x55d240e70580 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f55da7740 10 client.16284 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.048+0000 7f3f55da7740 1 -- 172.21.15.94:0/1294718441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d240ef4640 con 0x55d240e70580 2024-04-21T08:41:41.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f2bfff640 1 -- 172.21.15.94:0/1294718441 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f44028410 con 0x55d240e70580 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f2bfff640 1 -- 172.21.15.94:0/1294718441 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3f3c083bd0 con 0x55d240e70580 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f55da7740 10 client.16284 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f55da7740 10 client.16284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f55da7740 20 client.16284 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f55da7740 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d240ef4c00 0x55d240ef7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f55da7740 4 client.16284 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f55da7740 1 -- 172.21.15.94:0/1294718441 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d240ddca70 con 0x55d240ef4c00 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d240ef4c00 0x55d240ef7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f2bfff640 10 client.16284 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.049+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d240ef4c00 0x55d240ef7020 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fb21eb06f357be server_cookie=67192ef830522daf in_seq=0 out_seq=0 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f2bfff640 1 -- 172.21.15.94:0/1294718441 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d240ddca70 con 0x55d240ef4c00 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f2bfff640 10 client.16284 handle_command_reply: tid=0 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 client.16284 shutdown 2024-04-21T08:41:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 -- 172.21.15.94:0/1294718441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 msgr2=0x55d240ee8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240ee8030 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0bcf00 tx=0x7f3f3c003760 comp rx=0 tx=0).stop 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 -- 172.21.15.94:0/1294718441 shutdown_connections 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d240ef4c00 0x55d240ef7020 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d240e79a80 0x55d240ee8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d240e70ea0 0x55d240ee8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f2bfff640 0 client.16284 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 --2- 172.21.15.94:0/1294718441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d240e70580 0x55d240ee8030 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 -- 172.21.15.94:0/1294718441 >> 172.21.15.94:0/1294718441 conn(0x55d240e6acb0 msgr2=0x55d240e76a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 -- 172.21.15.94:0/1294718441 shutdown_connections 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 1 -- 172.21.15.94:0/1294718441 wait complete. 2024-04-21T08:41:41.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f55da7740 20 client.16284 trim_cache size 0 max 16384 2024-04-21T08:41:41.058 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:41:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:41:41.058 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:41:41.058 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f531b2640 1 -- 172.21.15.94:0/1435251399 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3f340a3d20 msgr2=0x7f3f340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f531b2640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3f340a3d20 0x7f3f340a61e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3f44007f10 tx=0x7f3f440127c0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f531b2640 1 -- 172.21.15.94:0/1435251399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06a860 msgr2=0x7f3f4c198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.050+0000 7f3f531b2640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06a860 0x7f3f4c198740 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c005940 tx=0x7f3f3c02c880 comp rx=0 tx=0).stop 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.051+0000 7f3f531b2640 1 -- 172.21.15.94:0/1435251399 shutdown_connections 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.051+0000 7f3f531b2640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3f340a3d20 0x7f3f340a61e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.051+0000 7f3f531b2640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f4c19cfb0 0x7f3f4c19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.051+0000 7f3f531b2640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f4c107310 0x7f3f4c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.051+0000 7f3f531b2640 1 --2- 172.21.15.94:0/1435251399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06a860 0x7f3f4c198740 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.051+0000 7f3f531b2640 1 -- 172.21.15.94:0/1435251399 >> 172.21.15.94:0/1435251399 conn(0x7f3f4c077c40 msgr2=0x7f3f4c0ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.051+0000 7f3f531b2640 1 -- 172.21.15.94:0/1435251399 shutdown_connections 2024-04-21T08:41:41.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.051+0000 7f3f531b2640 1 -- 172.21.15.94:0/1435251399 wait complete. 2024-04-21T08:41:41.061 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:41:41.062 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:41.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 -- 172.21.15.94:0/1481432073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83c4101b60 msgr2=0x7f83c410dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 --2- 172.21.15.94:0/1481432073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83c4101b60 0x7f83c410dc90 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009cd0 tx=0x7f83bc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 -- 172.21.15.94:0/1481432073 shutdown_connections 2024-04-21T08:41:41.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 --2- 172.21.15.94:0/1481432073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83c4101b60 0x7f83c410dc90 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 --2- 172.21.15.94:0/1481432073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83c41011a0 0x7f83c4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 --2- 172.21.15.94:0/1481432073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83c4105e30 0x7f83c4106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 -- 172.21.15.94:0/1481432073 >> 172.21.15.94:0/1481432073 conn(0x7f83c4077e30 msgr2=0x7f83c4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 -- 172.21.15.94:0/1481432073 shutdown_connections 2024-04-21T08:41:41.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 -- 172.21.15.94:0/1481432073 wait complete. 2024-04-21T08:41:41.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 Processor -- start 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.161+0000 7f83ca167640 1 -- start start 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83ca167640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83c41011a0 0x7f83c41986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83ca167640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83c4101b60 0x7f83c4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83ca167640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83c4105e30 0x7f83c419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83ca167640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f83c4112200 con 0x7f83c4101b60 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83ca167640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f83c4112080 con 0x7f83c4105e30 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83ca167640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f83c4112380 con 0x7f83c41011a0 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83baffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83c4101b60 0x7f83c4198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83baffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83c4101b60 0x7f83c4198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52038/0 (socket says 172.21.15.94:52038) 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83baffd640 1 -- 172.21.15.94:0/2830405665 learned_addr learned my addr 172.21.15.94:0/2830405665 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83c3fff640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83c4105e30 0x7f83c419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83c37fe640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83c41011a0 0x7f83c41986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83baffd640 1 -- 172.21.15.94:0/2830405665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83c41011a0 msgr2=0x7f83c41986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83baffd640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83c41011a0 0x7f83c41986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83baffd640 1 -- 172.21.15.94:0/2830405665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83c4105e30 msgr2=0x7f83c419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83baffd640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83c4105e30 0x7f83c419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83baffd640 1 -- 172.21.15.94:0/2830405665 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c419d690 con 0x7f83c4101b60 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83c3fff640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83c4105e30 0x7f83c419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83baffd640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83c4101b60 0x7f83c4198be0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00c890 tx=0x7f83ac00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83c17fa640 1 -- 172.21.15.94:0/2830405665 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac004220 con 0x7f83c4101b60 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83ca167640 1 -- 172.21.15.94:0/2830405665 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c419d980 con 0x7f83c4101b60 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83c17fa640 1 -- 172.21.15.94:0/2830405665 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83ac0043c0 con 0x7f83c4101b60 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83c17fa640 1 -- 172.21.15.94:0/2830405665 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac004a00 con 0x7f83c4101b60 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.162+0000 7f83ca167640 1 -- 172.21.15.94:0/2830405665 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c41a5210 con 0x7f83c4101b60 2024-04-21T08:41:41.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.163+0000 7f83ca167640 1 -- 172.21.15.94:0/2830405665 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83c4068de0 con 0x7f83c4101b60 2024-04-21T08:41:41.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.164+0000 7f83c17fa640 1 -- 172.21.15.94:0/2830405665 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f83ac0079b0 con 0x7f83c4101b60 2024-04-21T08:41:41.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.165+0000 7f83c17fa640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f83b00a3d20 0x7f83b00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.165+0000 7f83c37fe640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f83b00a3d20 0x7f83b00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.165+0000 7f83c17fa640 1 -- 172.21.15.94:0/2830405665 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f83ac0d0de0 con 0x7f83c4101b60 2024-04-21T08:41:41.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.166+0000 7f83c37fe640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f83b00a3d20 0x7f83b00a61e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f83b4005fd0 tx=0x7f83b4009470 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.169+0000 7f83c17fa640 1 -- 172.21.15.94:0/2830405665 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f83ac014030 con 0x7f83c4101b60 2024-04-21T08:41:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.387+0000 7f83ca167640 1 -- 172.21.15.94:0/2830405665 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83c4109380 con 0x7f83c4101b60 2024-04-21T08:41:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.387+0000 7f83c17fa640 1 -- 172.21.15.94:0/2830405665 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f83ac09d0a0 con 0x7f83c4101b60 2024-04-21T08:41:41.387 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:41:41.387 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:41:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:41:41.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 -- 172.21.15.94:0/2830405665 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f83b00a3d20 msgr2=0x7f83b00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f83b00a3d20 0x7f83b00a61e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f83b4005fd0 tx=0x7f83b4009470 comp rx=0 tx=0).stop 2024-04-21T08:41:41.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 -- 172.21.15.94:0/2830405665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83c4101b60 msgr2=0x7f83c4198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83c4101b60 0x7f83c4198be0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00c890 tx=0x7f83ac00cd50 comp rx=0 tx=0).stop 2024-04-21T08:41:41.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 -- 172.21.15.94:0/2830405665 shutdown_connections 2024-04-21T08:41:41.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f83b00a3d20 0x7f83b00a61e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83c4105e30 0x7f83c419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83c4101b60 0x7f83c4198be0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 --2- 172.21.15.94:0/2830405665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83c41011a0 0x7f83c41986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 -- 172.21.15.94:0/2830405665 >> 172.21.15.94:0/2830405665 conn(0x7f83c4077e30 msgr2=0x7f83c40fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 -- 172.21.15.94:0/2830405665 shutdown_connections 2024-04-21T08:41:41.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.389+0000 7f83ca167640 1 -- 172.21.15.94:0/2830405665 wait complete. 2024-04-21T08:41:41.401 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:41:41.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.501+0000 7f8540f2b640 1 -- 172.21.15.94:0/3961927504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f853c103ad0 msgr2=0x7f853c10bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.501+0000 7f8540f2b640 1 --2- 172.21.15.94:0/3961927504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f853c103ad0 0x7f853c10bfb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8530009900 tx=0x7f853002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.501+0000 7f8540f2b640 1 -- 172.21.15.94:0/3961927504 shutdown_connections 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.501+0000 7f8540f2b640 1 --2- 172.21.15.94:0/3961927504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f853c10c580 0x7f853c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.501+0000 7f8540f2b640 1 --2- 172.21.15.94:0/3961927504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f853c103ad0 0x7f853c10bfb0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.501+0000 7f8540f2b640 1 --2- 172.21.15.94:0/3961927504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f853c1031b0 0x7f853c103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.501+0000 7f8540f2b640 1 -- 172.21.15.94:0/3961927504 >> 172.21.15.94:0/3961927504 conn(0x7f853c0771d0 msgr2=0x7f853c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f8540f2b640 1 -- 172.21.15.94:0/3961927504 shutdown_connections 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f8540f2b640 1 -- 172.21.15.94:0/3961927504 wait complete. 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f8540f2b640 1 Processor -- start 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f8540f2b640 1 -- start start 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f8540f2b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f853c1031b0 0x7f853c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f8540f2b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f853c103ad0 0x7f853c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f8540f2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f853c10c580 0x7f853c19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f8540f2b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f853c112210 con 0x7f853c10c580 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f8540f2b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f853c112090 con 0x7f853c103ad0 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f8540f2b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f853c112390 con 0x7f853c1031b0 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.502+0000 7f853ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f853c10c580 0x7f853c19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f853ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f853c10c580 0x7f853c19cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52058/0 (socket says 172.21.15.94:52058) 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f853ad9d640 1 -- 172.21.15.94:0/1308022036 learned_addr learned my addr 172.21.15.94:0/1308022036 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f853a59c640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f853c1031b0 0x7f853c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f8539d9b640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f853c103ad0 0x7f853c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f853ad9d640 1 -- 172.21.15.94:0/1308022036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f853c1031b0 msgr2=0x7f853c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f853ad9d640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f853c1031b0 0x7f853c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f853ad9d640 1 -- 172.21.15.94:0/1308022036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f853c103ad0 msgr2=0x7f853c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f853ad9d640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f853c103ad0 0x7f853c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f853ad9d640 1 -- 172.21.15.94:0/1308022036 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c19d6e0 con 0x7f853c10c580 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f853ad9d640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f853c10c580 0x7f853c19cfb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f853400e9d0 tx=0x7f853400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f852b7fe640 1 -- 172.21.15.94:0/1308022036 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534004d20 con 0x7f853c10c580 2024-04-21T08:41:41.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f852b7fe640 1 -- 172.21.15.94:0/1308022036 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8534005020 con 0x7f853c10c580 2024-04-21T08:41:41.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f8540f2b640 1 -- 172.21.15.94:0/1308022036 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853c19d9d0 con 0x7f853c10c580 2024-04-21T08:41:41.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f852b7fe640 1 -- 172.21.15.94:0/1308022036 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85340116b0 con 0x7f853c10c580 2024-04-21T08:41:41.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.503+0000 7f8540f2b640 1 -- 172.21.15.94:0/1308022036 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853c1a5260 con 0x7f853c10c580 2024-04-21T08:41:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.505+0000 7f852b7fe640 1 -- 172.21.15.94:0/1308022036 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f85340051d0 con 0x7f853c10c580 2024-04-21T08:41:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f8543bfa740 1 Processor -- start 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f8543bfa740 1 -- start start 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c59494d0 0x5560c59498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c5952430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c5954e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f8543bfa740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5560c58ae1f0 con 0x5560c5949e80 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f8543bfa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5560c584b4b0 con 0x5560c5952a00 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f8543bfa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5560c58aa1b0 con 0x5560c59494d0 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f853a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c5952430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f852b7fe640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f85240a3da0 0x7f85240a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f853a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c5952430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52070/0 (socket says 172.21.15.94:52070) 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f853a59c640 1 -- 172.21.15.94:0/950558947 learned_addr learned my addr 172.21.15.94:0/950558947 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f8539d9b640 1 --2- 172.21.15.94:0/950558947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c59494d0 0x5560c59498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f8539d9b640 1 --2- 172.21.15.94:0/950558947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c5954e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.506+0000 7f853ad9d640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f85240a3da0 0x7f85240a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f852b7fe640 1 -- 172.21.15.94:0/1308022036 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f85340d1040 con 0x7f853c10c580 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f853a59c640 1 -- 172.21.15.94:0/950558947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c59494d0 msgr2=0x5560c59498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f853a59c640 1 --2- 172.21.15.94:0/950558947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c59494d0 0x5560c59498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f853a59c640 1 -- 172.21.15.94:0/950558947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 msgr2=0x5560c5954e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f853a59c640 1 --2- 172.21.15.94:0/950558947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c5954e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f853a59c640 1 -- 172.21.15.94:0/950558947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560c583d1b0 con 0x5560c5949e80 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f8539d9b640 1 --2- 172.21.15.94:0/950558947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c59494d0 0x5560c59498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f853a59c640 1 --2- 172.21.15.94:0/950558947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c5952430 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f852c00ad00 tx=0x7f852c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78c6585c68c68bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f853ad9d640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f85240a3da0 0x7f85240a6260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f853409c900 tx=0x7f85340bd040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f8528ff9640 1 -- 172.21.15.94:0/950558947 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c01d070 con 0x5560c5949e80 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f8528ff9640 1 -- 172.21.15.94:0/950558947 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f852c018af0 con 0x5560c5949e80 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.507+0000 7f8528ff9640 1 -- 172.21.15.94:0/950558947 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c010b40 con 0x5560c5949e80 2024-04-21T08:41:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 -- 172.21.15.94:0/950558947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 msgr2=0x5560c5952430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 --2- 172.21.15.94:0/950558947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c5952430 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f852c00ad00 tx=0x7f852c00f790 comp rx=0 tx=0).stop 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 -- 172.21.15.94:0/950558947 shutdown_connections 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 --2- 172.21.15.94:0/950558947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c5954e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 --2- 172.21.15.94:0/950558947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c5952430 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 --2- 172.21.15.94:0/950558947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c59494d0 0x5560c59498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 -- 172.21.15.94:0/950558947 >> 172.21.15.94:0/950558947 conn(0x5560c5943ba0 msgr2=0x5560c5943f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 -- 172.21.15.94:0/950558947 shutdown_connections 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 -- 172.21.15.94:0/950558947 wait complete. 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 -- start start 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 Processor -- start 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c59494d0 0x5560c5952350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c5952a00 0x5560c594fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5560c5955970 con 0x5560c5949e80 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5560c5867500 con 0x5560c59494d0 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5560c5955af0 con 0x5560c5952a00 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.508+0000 7f8543bfa740 10 client.0 _ll_register_callbacks cb 0x5560c5835b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f853a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f853a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52078/0 (socket says 172.21.15.94:52078) 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f853a59c640 1 -- 172.21.15.94:0/4282097772 learned_addr learned my addr 172.21.15.94:0/4282097772 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f8539d9b640 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c5952a00 0x5560c594fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f8539d9b640 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c59494d0 0x5560c5952350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f853a59c640 1 -- 172.21.15.94:0/4282097772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c5952a00 msgr2=0x5560c594fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f853a59c640 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c5952a00 0x5560c594fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f853a59c640 1 -- 172.21.15.94:0/4282097772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c59494d0 msgr2=0x5560c5952350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f853a59c640 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c59494d0 0x5560c5952350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f853a59c640 1 -- 172.21.15.94:0/4282097772 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560c55c03f0 con 0x5560c5949e80 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f853a59c640 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594f6e0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f852c000c00 tx=0x7f852c018670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11664a64dfa383b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f8521ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f8521ffb640 1 -- 172.21.15.94:0/4282097772 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c026020 con 0x5560c5949e80 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f8521ffb640 1 -- 172.21.15.94:0/4282097772 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f852c010880 con 0x5560c5949e80 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f8543bfa740 1 -- 172.21.15.94:0/4282097772 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f852c018af0 con 0x5560c5949e80 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f8521ffb640 1 -- 172.21.15.94:0/4282097772 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f852c031050 con 0x5560c5949e80 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f8543bfa740 10 client.16302 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f8543bfa740 1 -- 172.21.15.94:0/4282097772 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5560c568a7d0 con 0x5560c5949e80 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.509+0000 7f8521ffb640 1 -- 172.21.15.94:0/4282097772 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f852c018af0 con 0x5560c5949e80 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.510+0000 7f8521ffb640 1 -- 172.21.15.94:0/4282097772 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f852c005470 con 0x5560c5949e80 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.510+0000 7f8543bfa740 10 client.16302 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.510+0000 7f8543bfa740 10 client.16302 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.510+0000 7f8543bfa740 20 client.16302 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.510+0000 7f8543bfa740 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5560c5962bc0 0x5560c5964fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.510+0000 7f8543bfa740 4 client.16302 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:41.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.510+0000 7f8543bfa740 1 -- 172.21.15.94:0/4282097772 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560c584d930 con 0x5560c5962bc0 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.510+0000 7f8539d9b640 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5560c5962bc0 0x5560c5964fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.510+0000 7f8521ffb640 10 client.16302 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.510+0000 7f8539d9b640 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5560c5962bc0 0x5560c5964fb0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=94201abcbef8d51e server_cookie=e044150c2a76f1a5 in_seq=0 out_seq=0 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8521ffb640 1 -- 172.21.15.94:0/4282097772 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560c584d930 con 0x5560c5962bc0 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8521ffb640 10 client.16302 handle_command_reply: tid=0 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 client.16302 shutdown 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 -- 172.21.15.94:0/4282097772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 msgr2=0x5560c594f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594f6e0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f852c000c00 tx=0x7f852c018670 comp rx=0 tx=0).stop 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 -- 172.21.15.94:0/4282097772 shutdown_connections 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5560c5962bc0 0x5560c5964fb0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c5952a00 0x5560c594fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594f6e0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8521ffb640 0 client.16302 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:41:41.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 --2- 172.21.15.94:0/4282097772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c59494d0 0x5560c5952350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 -- 172.21.15.94:0/4282097772 >> 172.21.15.94:0/4282097772 conn(0x5560c5943ba0 msgr2=0x5560c59458f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 -- 172.21.15.94:0/4282097772 shutdown_connections 2024-04-21T08:41:41.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 1 -- 172.21.15.94:0/4282097772 wait complete. 2024-04-21T08:41:41.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.511+0000 7f8543bfa740 20 client.16302 trim_cache size 0 max 16384 2024-04-21T08:41:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8543bfa740 1 Processor -- start 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8543bfa740 1 -- start start 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c59494d0 0x5560c595e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c5949e80 0x5560c5951e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c5952350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8543bfa740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b1dd0 con 0x5560c59494d0 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8543bfa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1af6a0 con 0x5560c5952a00 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8543bfa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1ad4f0 con 0x5560c5949e80 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8539d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c59494d0 0x5560c595e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8539d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c59494d0 0x5560c595e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52090/0 (socket says 172.21.15.94:52090) 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8539d9b640 1 -- 172.21.15.94:0/2896126734 learned_addr learned my addr 172.21.15.94:0/2896126734 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8539d9b640 1 -- 172.21.15.94:0/2896126734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c5949e80 msgr2=0x5560c5951e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8539d9b640 1 --2- 172.21.15.94:0/2896126734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c5949e80 0x5560c5951e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8539d9b640 1 -- 172.21.15.94:0/2896126734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 msgr2=0x5560c5952350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8539d9b640 1 --2- 172.21.15.94:0/2896126734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c5952350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8539d9b640 1 -- 172.21.15.94:0/2896126734 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560c593e040 con 0x5560c59494d0 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8539d9b640 1 --2- 172.21.15.94:0/2896126734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c59494d0 0x5560c595e860 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8530006010 tx=0x7f853002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968c6e74611ff125 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8523fff640 1 -- 172.21.15.94:0/2896126734 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853004ab50 con 0x5560c59494d0 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8523fff640 1 -- 172.21.15.94:0/2896126734 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f853002cd30 con 0x5560c59494d0 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.525+0000 7f8523fff640 1 -- 172.21.15.94:0/2896126734 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853004a720 con 0x5560c59494d0 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 -- 172.21.15.94:0/2896126734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c59494d0 msgr2=0x5560c595e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 --2- 172.21.15.94:0/2896126734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c59494d0 0x5560c595e860 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8530006010 tx=0x7f853002e040 comp rx=0 tx=0).stop 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 -- 172.21.15.94:0/2896126734 shutdown_connections 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 --2- 172.21.15.94:0/2896126734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c5949e80 0x5560c5951e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 --2- 172.21.15.94:0/2896126734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c5952350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 --2- 172.21.15.94:0/2896126734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c59494d0 0x5560c595e860 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 -- 172.21.15.94:0/2896126734 >> 172.21.15.94:0/2896126734 conn(0x5560c5943ba0 msgr2=0x5560c5942c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 -- 172.21.15.94:0/2896126734 shutdown_connections 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 -- 172.21.15.94:0/2896126734 wait complete. 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 -- start start 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.526+0000 7f8543bfa740 1 Processor -- start 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c59494d0 0x5560c595b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8543bfa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c594c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8543bfa740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1adb60 con 0x5560c5949e80 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8543bfa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1aefd0 con 0x5560c5952a00 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8543bfa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1ae230 con 0x5560c59494d0 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8543bfa740 10 client.0 _ll_register_callbacks cb 0x7f852c00ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f853a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f853a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52104/0 (socket says 172.21.15.94:52104) 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f853a59c640 1 -- 172.21.15.94:0/3543478479 learned_addr learned my addr 172.21.15.94:0/3543478479 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8539d9b640 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c594c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f853a59c640 1 -- 172.21.15.94:0/3543478479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c59494d0 msgr2=0x5560c595b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f853a59c640 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c59494d0 0x5560c595b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f853a59c640 1 -- 172.21.15.94:0/3543478479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 msgr2=0x5560c594c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f853a59c640 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c594c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f853a59c640 1 -- 172.21.15.94:0/3543478479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560c594cfd0 con 0x5560c5949e80 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f853a59c640 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594c1c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f852c004880 tx=0x7f852c00a670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf74dd1444eccab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8521ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8521ffb640 1 -- 172.21.15.94:0/3543478479 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c03a060 con 0x5560c5949e80 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8521ffb640 1 -- 172.21.15.94:0/3543478479 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f852c0188e0 con 0x5560c5949e80 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8543bfa740 1 -- 172.21.15.94:0/3543478479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f853002cd30 con 0x5560c5949e80 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8521ffb640 1 -- 172.21.15.94:0/3543478479 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f852c041050 con 0x5560c5949e80 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8543bfa740 10 client.16308 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.527+0000 7f8543bfa740 1 -- 172.21.15.94:0/3543478479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5560c59c8ef0 con 0x5560c5949e80 2024-04-21T08:41:41.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.528+0000 7f8521ffb640 1 -- 172.21.15.94:0/3543478479 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f853002cd30 con 0x5560c5949e80 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.528+0000 7f8521ffb640 1 -- 172.21.15.94:0/3543478479 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f852c010430 con 0x5560c5949e80 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.528+0000 7f8543bfa740 10 client.16308 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.528+0000 7f8543bfa740 10 client.16308 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.528+0000 7f8543bfa740 20 client.16308 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.528+0000 7f8543bfa740 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5560c59c9590 0x5560c59cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.528+0000 7f8543bfa740 4 client.16308 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.528+0000 7f8543bfa740 1 -- 172.21.15.94:0/3543478479 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5560c58bbc10 con 0x5560c59c9590 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.528+0000 7f8539d9b640 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5560c59c9590 0x5560c59cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8521ffb640 10 client.16308 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8539d9b640 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5560c59c9590 0x5560c59cb9b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b74f534e94ab6ca server_cookie=a0d4fdccfb73b4c6 in_seq=0 out_seq=0 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8521ffb640 1 -- 172.21.15.94:0/3543478479 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5560c58bbc10 con 0x5560c59c9590 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8521ffb640 10 client.16308 handle_command_reply: tid=0 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 client.16308 shutdown 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 -- 172.21.15.94:0/3543478479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 msgr2=0x5560c594c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594c1c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f852c004880 tx=0x7f852c00a670 comp rx=0 tx=0).stop 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 -- 172.21.15.94:0/3543478479 shutdown_connections 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5560c59c9590 0x5560c59cb9b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560c5952a00 0x5560c594c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5560c5949e80 0x5560c594c1c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 --2- 172.21.15.94:0/3543478479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560c59494d0 0x5560c595b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8521ffb640 0 client.16308 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 -- 172.21.15.94:0/3543478479 >> 172.21.15.94:0/3543478479 conn(0x5560c5943ba0 msgr2=0x5560c594a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 -- 172.21.15.94:0/3543478479 shutdown_connections 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 1 -- 172.21.15.94:0/3543478479 wait complete. 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8543bfa740 20 client.16308 trim_cache size 0 max 16384 2024-04-21T08:41:41.535 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:41:41.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8540f2b640 1 -- 172.21.15.94:0/1308022036 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f85240a3da0 msgr2=0x7f85240a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8540f2b640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f85240a3da0 0x7f85240a6260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f853409c900 tx=0x7f85340bd040 comp rx=0 tx=0).stop 2024-04-21T08:41:41.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8540f2b640 1 -- 172.21.15.94:0/1308022036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f853c10c580 msgr2=0x7f853c19cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.529+0000 7f8540f2b640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f853c10c580 0x7f853c19cfb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f853400e9d0 tx=0x7f853400edd0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.530+0000 7f8540f2b640 1 -- 172.21.15.94:0/1308022036 shutdown_connections 2024-04-21T08:41:41.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.530+0000 7f8540f2b640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f85240a3da0 0x7f85240a6260 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.530+0000 7f8540f2b640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f853c10c580 0x7f853c19cfb0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.530+0000 7f8540f2b640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f853c103ad0 0x7f853c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.530+0000 7f8540f2b640 1 --2- 172.21.15.94:0/1308022036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f853c1031b0 0x7f853c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.530+0000 7f8540f2b640 1 -- 172.21.15.94:0/1308022036 >> 172.21.15.94:0/1308022036 conn(0x7f853c0771d0 msgr2=0x7f853c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.530+0000 7f8540f2b640 1 -- 172.21.15.94:0/1308022036 shutdown_connections 2024-04-21T08:41:41.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.530+0000 7f8540f2b640 1 -- 172.21.15.94:0/1308022036 wait complete. 2024-04-21T08:41:41.541 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:41:41.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.641+0000 7feb8044a640 1 -- 172.21.15.94:0/2659143729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb78101090 msgr2=0x7feb78101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.641+0000 7feb8044a640 1 --2- 172.21.15.94:0/2659143729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb78101090 0x7feb78101470 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7feb74009ca0 tx=0x7feb7402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 -- 172.21.15.94:0/2659143729 shutdown_connections 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 --2- 172.21.15.94:0/2659143729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb78101a40 0x7feb78105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 --2- 172.21.15.94:0/2659143729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb78106160 0x7feb7810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 --2- 172.21.15.94:0/2659143729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb78101090 0x7feb78101470 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 -- 172.21.15.94:0/2659143729 >> 172.21.15.94:0/2659143729 conn(0x7feb78077f50 msgr2=0x7feb78078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 -- 172.21.15.94:0/2659143729 shutdown_connections 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 -- 172.21.15.94:0/2659143729 wait complete. 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 Processor -- start 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 -- start start 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb78101090 0x7feb78198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb78106160 0x7feb78198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb7819d020 0x7feb7819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7feb781121e0 con 0x7feb7819d020 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feb78112060 con 0x7feb78101090 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.642+0000 7feb8044a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb78112360 con 0x7feb78106160 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7e9e7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb7819d020 0x7feb7819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7e9e7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb7819d020 0x7feb7819d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52114/0 (socket says 172.21.15.94:52114) 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7e9e7640 1 -- 172.21.15.94:0/1771050963 learned_addr learned my addr 172.21.15.94:0/1771050963 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb78101090 0x7feb78198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb78106160 0x7feb78198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7e9e7640 1 -- 172.21.15.94:0/1771050963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb78106160 msgr2=0x7feb78198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7e9e7640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb78106160 0x7feb78198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7e9e7640 1 -- 172.21.15.94:0/1771050963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb78101090 msgr2=0x7feb78198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7e9e7640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb78101090 0x7feb78198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7e9e7640 1 -- 172.21.15.94:0/1771050963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7819db10 con 0x7feb7819d020 2024-04-21T08:41:41.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb7e9e7640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb7819d020 0x7feb7819d4d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7feb7000d950 tx=0x7feb7000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb6f7fe640 1 -- 172.21.15.94:0/1771050963 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb70017030 con 0x7feb7819d020 2024-04-21T08:41:41.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb8044a640 1 -- 172.21.15.94:0/1771050963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7819de00 con 0x7feb7819d020 2024-04-21T08:41:41.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb6f7fe640 1 -- 172.21.15.94:0/1771050963 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb70004020 con 0x7feb7819d020 2024-04-21T08:41:41.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb6f7fe640 1 -- 172.21.15.94:0/1771050963 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb70003b50 con 0x7feb7819d020 2024-04-21T08:41:41.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.643+0000 7feb8044a640 1 -- 172.21.15.94:0/1771050963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb781a55a0 con 0x7feb7819d020 2024-04-21T08:41:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.645+0000 7feb6f7fe640 1 -- 172.21.15.94:0/1771050963 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7feb7001a070 con 0x7feb7819d020 2024-04-21T08:41:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb8303f740 1 Processor -- start 2024-04-21T08:41:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb8303f740 1 -- start start 2024-04-21T08:41:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb8303f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eac4d0 0x5579d0eac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb8303f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb6f7fe640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb640a3d20 0x7feb640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb8303f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eb5a00 0x5579d0eb7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb8303f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5579d0e111f0 con 0x5579d0eace80 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb8303f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5579d0dae4b0 con 0x5579d0eac4d0 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb8303f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5579d0e0d1b0 con 0x5579d0eb5a00 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb640a3d20 0x7feb640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb7e1e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb7e1e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52120/0 (socket says 172.21.15.94:52120) 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb7e1e6640 1 -- 172.21.15.94:0/745715674 learned_addr learned my addr 172.21.15.94:0/745715674 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/745715674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eac4d0 0x5579d0eac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb6f7fe640 1 -- 172.21.15.94:0/1771050963 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7feb700d0290 con 0x7feb7819d020 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.646+0000 7feb7e9e7640 1 --2- 172.21.15.94:0/745715674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eb5a00 0x5579d0eb7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.647+0000 7feb7e1e6640 1 -- 172.21.15.94:0/745715674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eb5a00 msgr2=0x5579d0eb7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.647+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/745715674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eb5a00 0x5579d0eb7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.647+0000 7feb7e1e6640 1 -- 172.21.15.94:0/745715674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eac4d0 msgr2=0x5579d0eac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.647+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/745715674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eac4d0 0x5579d0eac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.647+0000 7feb7e1e6640 1 -- 172.21.15.94:0/745715674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579d0da01b0 con 0x5579d0eace80 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.647+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb640a3d20 0x7feb640a61e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feb78100a40 tx=0x7feb6800bb30 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.647+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/745715674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb5430 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7feb7402c700 tx=0x7feb740323f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0a639917fb22e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.647+0000 7feb6cbfe640 1 -- 172.21.15.94:0/745715674 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74043070 con 0x5579d0eace80 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.647+0000 7feb6cbfe640 1 -- 172.21.15.94:0/745715674 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb74032970 con 0x5579d0eace80 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.647+0000 7feb6cbfe640 1 -- 172.21.15.94:0/745715674 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74039040 con 0x5579d0eace80 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 -- 172.21.15.94:0/745715674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 msgr2=0x5579d0eb5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 --2- 172.21.15.94:0/745715674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb5430 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7feb7402c700 tx=0x7feb740323f0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 -- 172.21.15.94:0/745715674 shutdown_connections 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 --2- 172.21.15.94:0/745715674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eb5a00 0x5579d0eb7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 --2- 172.21.15.94:0/745715674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb5430 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 --2- 172.21.15.94:0/745715674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eac4d0 0x5579d0eac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 -- 172.21.15.94:0/745715674 >> 172.21.15.94:0/745715674 conn(0x5579d0ea6ba0 msgr2=0x5579d0ea6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 -- 172.21.15.94:0/745715674 shutdown_connections 2024-04-21T08:41:41.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 -- 172.21.15.94:0/745715674 wait complete. 2024-04-21T08:41:41.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 -- start start 2024-04-21T08:41:41.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 Processor -- start 2024-04-21T08:41:41.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eac4d0 0x5579d0eb52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eb5a00 0x5579d0eb2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5579d0eb8910 con 0x5579d0eace80 2024-04-21T08:41:41.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5579d0dca500 con 0x5579d0eac4d0 2024-04-21T08:41:41.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5579d0eb8a90 con 0x5579d0eb5a00 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb8303f740 10 client.0 _ll_register_callbacks cb 0x5579d0d98b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.648+0000 7feb7d9e5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb7d9e5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52128/0 (socket says 172.21.15.94:52128) 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb7d9e5640 1 -- 172.21.15.94:0/3307515347 learned_addr learned my addr 172.21.15.94:0/3307515347 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb7e9e7640 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eb5a00 0x5579d0eb2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eac4d0 0x5579d0eb52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb7d9e5640 1 -- 172.21.15.94:0/3307515347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eb5a00 msgr2=0x5579d0eb2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eb5a00 0x5579d0eb2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb7d9e5640 1 -- 172.21.15.94:0/3307515347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eac4d0 msgr2=0x5579d0eb52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eac4d0 0x5579d0eb52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb7d9e5640 1 -- 172.21.15.94:0/3307515347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579d0b233f0 con 0x5579d0eace80 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb2670 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7feb68010c40 tx=0x7feb6800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0f4487ea7d217a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb61ffb640 1 -- 172.21.15.94:0/3307515347 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68010c70 con 0x5579d0eace80 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb61ffb640 1 -- 172.21.15.94:0/3307515347 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7feb6802a410 con 0x5579d0eace80 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb8303f740 1 -- 172.21.15.94:0/3307515347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb74032970 con 0x5579d0eace80 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb61ffb640 1 -- 172.21.15.94:0/3307515347 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7feb68010e10 con 0x5579d0eace80 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb8303f740 10 client.16320 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb8303f740 1 -- 172.21.15.94:0/3307515347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5579d0bed7d0 con 0x5579d0eace80 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.649+0000 7feb61ffb640 1 -- 172.21.15.94:0/3307515347 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb74032970 con 0x5579d0eace80 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.650+0000 7feb61ffb640 1 -- 172.21.15.94:0/3307515347 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7feb6801f070 con 0x5579d0eace80 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.650+0000 7feb8303f740 10 client.16320 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.650+0000 7feb8303f740 10 client.16320 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.650+0000 7feb8303f740 20 client.16320 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.650+0000 7feb8303f740 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5579d0ec5b70 0x5579d0ec8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.650+0000 7feb8303f740 4 client.16320 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.650+0000 7feb8303f740 1 -- 172.21.15.94:0/3307515347 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579d0db0930 con 0x5579d0ec5b70 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.650+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5579d0ec5b70 0x5579d0ec8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.650+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5579d0ec5b70 0x5579d0ec8010 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e7d97a8dce97955b server_cookie=d479ce503fcc3f35 in_seq=0 out_seq=0 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.650+0000 7feb61ffb640 10 client.16320 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb61ffb640 1 -- 172.21.15.94:0/3307515347 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5579d0db0930 con 0x5579d0ec5b70 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb61ffb640 10 client.16320 handle_command_reply: tid=0 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 client.16320 shutdown 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 -- 172.21.15.94:0/3307515347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 msgr2=0x5579d0eb2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb2670 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7feb68010c40 tx=0x7feb6800b410 comp rx=0 tx=0).stop 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 -- 172.21.15.94:0/3307515347 shutdown_connections 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eb5a00 0x5579d0eb2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5579d0ec5b70 0x5579d0ec8010 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eace80 0x5579d0eb2670 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 --2- 172.21.15.94:0/3307515347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eac4d0 0x5579d0eb52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb61ffb640 0 client.16320 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:41:41.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 -- 172.21.15.94:0/3307515347 >> 172.21.15.94:0/3307515347 conn(0x5579d0ea6ba0 msgr2=0x5579d0ea88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 -- 172.21.15.94:0/3307515347 shutdown_connections 2024-04-21T08:41:41.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 1 -- 172.21.15.94:0/3307515347 wait complete. 2024-04-21T08:41:41.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.651+0000 7feb8303f740 20 client.16320 trim_cache size 0 max 16384 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.664+0000 7feb8303f740 1 Processor -- start 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.664+0000 7feb8303f740 1 -- start start 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb8303f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eac4d0 0x5579d0ec17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb8303f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eace80 0x5579d0eb17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb8303f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eb5a00 0x5579d0ea86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb8303f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7feb781b1d00 con 0x5579d0eb5a00 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb8303f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feb781af5d0 con 0x5579d0eace80 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb8303f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb781ad420 con 0x5579d0eac4d0 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7e9e7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eb5a00 0x5579d0ea86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7e9e7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eb5a00 0x5579d0ea86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52130/0 (socket says 172.21.15.94:52130) 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7e9e7640 1 -- 172.21.15.94:0/1704727483 learned_addr learned my addr 172.21.15.94:0/1704727483 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/1704727483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eac4d0 0x5579d0ec17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/1704727483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eace80 0x5579d0eb17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7e9e7640 1 -- 172.21.15.94:0/1704727483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eac4d0 msgr2=0x5579d0ec17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7e9e7640 1 --2- 172.21.15.94:0/1704727483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eac4d0 0x5579d0ec17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7e9e7640 1 -- 172.21.15.94:0/1704727483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eace80 msgr2=0x5579d0eb17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7e9e7640 1 --2- 172.21.15.94:0/1704727483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eace80 0x5579d0eb17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7e9e7640 1 -- 172.21.15.94:0/1704727483 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579d0eb2f40 con 0x5579d0eb5a00 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/1704727483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eace80 0x5579d0eb17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb7e9e7640 1 --2- 172.21.15.94:0/1704727483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eb5a00 0x5579d0ea86e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7feb700cd6b0 tx=0x7feb700a1990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ffb1b11dd70bdb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb63fff640 1 -- 172.21.15.94:0/1704727483 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb700a0200 con 0x5579d0eb5a00 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb63fff640 1 -- 172.21.15.94:0/1704727483 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb7009e070 con 0x5579d0eb5a00 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.665+0000 7feb63fff640 1 -- 172.21.15.94:0/1704727483 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb70003cf0 con 0x5579d0eb5a00 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 -- 172.21.15.94:0/1704727483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eb5a00 msgr2=0x5579d0ea86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 --2- 172.21.15.94:0/1704727483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eb5a00 0x5579d0ea86e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7feb700cd6b0 tx=0x7feb700a1990 comp rx=0 tx=0).stop 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 -- 172.21.15.94:0/1704727483 shutdown_connections 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 --2- 172.21.15.94:0/1704727483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eb5a00 0x5579d0ea86e0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 --2- 172.21.15.94:0/1704727483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eace80 0x5579d0eb17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 --2- 172.21.15.94:0/1704727483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eac4d0 0x5579d0ec17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 -- 172.21.15.94:0/1704727483 >> 172.21.15.94:0/1704727483 conn(0x5579d0ea6ba0 msgr2=0x5579d0ea7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 -- 172.21.15.94:0/1704727483 shutdown_connections 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 -- 172.21.15.94:0/1704727483 wait complete. 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 -- start start 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 Processor -- start 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eac4d0 0x5579d0ebe000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eace80 0x5579d0eb93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eb5a00 0x5579d0eb9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7feb781ada90 con 0x5579d0eac4d0 2024-04-21T08:41:41.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feb781aef00 con 0x5579d0eb5a00 2024-04-21T08:41:41.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb781ae160 con 0x5579d0eace80 2024-04-21T08:41:41.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb8303f740 10 client.0 _ll_register_callbacks cb 0x7feb68010940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:41.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb7e1e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eac4d0 0x5579d0ebe000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb7e1e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eac4d0 0x5579d0ebe000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52138/0 (socket says 172.21.15.94:52138) 2024-04-21T08:41:41.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.666+0000 7feb7e1e6640 1 -- 172.21.15.94:0/3139421916 learned_addr learned my addr 172.21.15.94:0/3139421916 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb7e1e6640 1 -- 172.21.15.94:0/3139421916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eace80 msgr2=0x5579d0eb93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eace80 0x5579d0eb93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eace80 0x5579d0eb93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb7e1e6640 1 -- 172.21.15.94:0/3139421916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eb5a00 msgr2=0x5579d0eb9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eb5a00 0x5579d0eb9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb7e1e6640 1 -- 172.21.15.94:0/3139421916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579d0eba100 con 0x5579d0eac4d0 2024-04-21T08:41:41.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eace80 0x5579d0eb93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:41.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb7e1e6640 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eac4d0 0x5579d0ebe000 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7feb74009ca0 tx=0x7feb74003b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7eee1b5c8cb988 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:41.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb61ffb640 1 -- 172.21.15.94:0/3139421916 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74039040 con 0x5579d0eac4d0 2024-04-21T08:41:41.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb61ffb640 1 -- 172.21.15.94:0/3139421916 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7feb7404f070 con 0x5579d0eac4d0 2024-04-21T08:41:41.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb8303f740 1 -- 172.21.15.94:0/3139421916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb7009e070 con 0x5579d0eac4d0 2024-04-21T08:41:41.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb61ffb640 1 -- 172.21.15.94:0/3139421916 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7feb7405c050 con 0x5579d0eac4d0 2024-04-21T08:41:41.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb8303f740 10 client.16326 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:41.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb8303f740 1 -- 172.21.15.94:0/3139421916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5579d0f2c030 con 0x5579d0eac4d0 2024-04-21T08:41:41.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb61ffb640 1 -- 172.21.15.94:0/3139421916 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb7009e070 con 0x5579d0eac4d0 2024-04-21T08:41:41.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb61ffb640 1 -- 172.21.15.94:0/3139421916 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7feb740484e0 con 0x5579d0eac4d0 2024-04-21T08:41:41.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb8303f740 10 client.16326 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:41.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb8303f740 10 client.16326 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:41:41.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb8303f740 20 client.16326 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:41.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb8303f740 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5579d0f2c310 0x5579d0f2e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb8303f740 4 client.16326 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.667+0000 7feb8303f740 1 -- 172.21.15.94:0/3139421916 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5579d0e1ec10 con 0x5579d0f2c310 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5579d0f2c310 0x5579d0f2e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb7d9e5640 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5579d0f2c310 0x5579d0f2e700 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=28ea789acf7e996d server_cookie=618c1848e2edcac2 in_seq=0 out_seq=0 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb61ffb640 10 client.16326 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb61ffb640 1 -- 172.21.15.94:0/3139421916 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5579d0e1ec10 con 0x5579d0f2c310 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb61ffb640 10 client.16326 handle_command_reply: tid=0 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb8303f740 1 client.16326 shutdown 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb8303f740 1 -- 172.21.15.94:0/3139421916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eac4d0 msgr2=0x5579d0ebe000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb8303f740 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eac4d0 0x5579d0ebe000 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7feb74009ca0 tx=0x7feb74003b10 comp rx=0 tx=0).stop 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb8303f740 1 -- 172.21.15.94:0/3139421916 shutdown_connections 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb8303f740 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5579d0eace80 0x5579d0eb93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb8303f740 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5579d0f2c310 0x5579d0f2e700 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb8303f740 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5579d0eb5a00 0x5579d0eb9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb61ffb640 0 client.16326 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:41:41.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb8303f740 1 --2- 172.21.15.94:0/3139421916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5579d0eac4d0 0x5579d0ebe000 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.668+0000 7feb8303f740 1 -- 172.21.15.94:0/3139421916 >> 172.21.15.94:0/3139421916 conn(0x5579d0ea6ba0 msgr2=0x5579d0ea87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8303f740 1 -- 172.21.15.94:0/3139421916 shutdown_connections 2024-04-21T08:41:41.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8303f740 1 -- 172.21.15.94:0/3139421916 wait complete. 2024-04-21T08:41:41.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8303f740 20 client.16326 trim_cache size 0 max 16384 2024-04-21T08:41:41.677 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:41:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 -- 172.21.15.94:0/1771050963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb640a3d20 msgr2=0x7feb640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb640a3d20 0x7feb640a61e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feb78100a40 tx=0x7feb6800bb30 comp rx=0 tx=0).stop 2024-04-21T08:41:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 -- 172.21.15.94:0/1771050963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb7819d020 msgr2=0x7feb7819d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb7819d020 0x7feb7819d4d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7feb7000d950 tx=0x7feb7000de10 comp rx=0 tx=0).stop 2024-04-21T08:41:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 -- 172.21.15.94:0/1771050963 shutdown_connections 2024-04-21T08:41:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feb640a3d20 0x7feb640a61e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb78106160 0x7feb78198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb7819d020 0x7feb7819d4d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 --2- 172.21.15.94:0/1771050963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb78101090 0x7feb78198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 -- 172.21.15.94:0/1771050963 >> 172.21.15.94:0/1771050963 conn(0x7feb78077f50 msgr2=0x7feb780ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 -- 172.21.15.94:0/1771050963 shutdown_connections 2024-04-21T08:41:41.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.669+0000 7feb8044a640 1 -- 172.21.15.94:0/1771050963 wait complete. 2024-04-21T08:41:41.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:41:41.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.781+0000 7f139794a640 1 -- 172.21.15.94:0/2435712962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1390107260 msgr2=0x7f139010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.781+0000 7f139794a640 1 --2- 172.21.15.94:0/2435712962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1390107260 0x7f139010ee30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1388009ca0 tx=0x7f138802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.781+0000 7f139794a640 1 -- 172.21.15.94:0/2435712962 shutdown_connections 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.781+0000 7f139794a640 1 --2- 172.21.15.94:0/2435712962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1390107260 0x7f139010ee30 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.781+0000 7f139794a640 1 --2- 172.21.15.94:0/2435712962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1390102b40 0x7f1390106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.781+0000 7f139794a640 1 --2- 172.21.15.94:0/2435712962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1390102190 0x7f1390102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.781+0000 7f139794a640 1 -- 172.21.15.94:0/2435712962 >> 172.21.15.94:0/2435712962 conn(0x7f13900793a0 msgr2=0x7f13900797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.781+0000 7f139794a640 1 -- 172.21.15.94:0/2435712962 shutdown_connections 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.781+0000 7f139794a640 1 -- 172.21.15.94:0/2435712962 wait complete. 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.781+0000 7f139794a640 1 Processor -- start 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f139794a640 1 -- start start 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f139794a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1390102190 0x7f13901986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f139794a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1390107260 0x7f1390198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f139794a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f139019cfb0 0x7f139019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f139794a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f13901121b0 con 0x7f1390102190 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f139794a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1390112030 con 0x7f139019cfb0 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f139794a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1390112330 con 0x7f1390107260 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13956e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1390102190 0x7f13901986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13956e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1390102190 0x7f13901986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52160/0 (socket says 172.21.15.94:52160) 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13956e6640 1 -- 172.21.15.94:0/2490678944 learned_addr learned my addr 172.21.15.94:0/2490678944 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13956e6640 1 -- 172.21.15.94:0/2490678944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1390107260 msgr2=0x7f1390198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13956e6640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1390107260 0x7f1390198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13956e6640 1 -- 172.21.15.94:0/2490678944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f139019cfb0 msgr2=0x7f139019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13956e6640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f139019cfb0 0x7f139019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13956e6640 1 -- 172.21.15.94:0/2490678944 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139019daa0 con 0x7f1390102190 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13956e6640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1390102190 0x7f13901986f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f138c00c890 tx=0x7f138c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13867fc640 1 -- 172.21.15.94:0/2490678944 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c004220 con 0x7f1390102190 2024-04-21T08:41:41.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13867fc640 1 -- 172.21.15.94:0/2490678944 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138c0043c0 con 0x7f1390102190 2024-04-21T08:41:41.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f139794a640 1 -- 172.21.15.94:0/2490678944 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f139019dd90 con 0x7f1390102190 2024-04-21T08:41:41.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f13867fc640 1 -- 172.21.15.94:0/2490678944 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c004a00 con 0x7f1390102190 2024-04-21T08:41:41.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.782+0000 7f139794a640 1 -- 172.21.15.94:0/2490678944 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13901a5580 con 0x7f1390102190 2024-04-21T08:41:41.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.784+0000 7f13867fc640 1 -- 172.21.15.94:0/2490678944 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f138c0079b0 con 0x7f1390102190 2024-04-21T08:41:41.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f139a60f740 1 Processor -- start 2024-04-21T08:41:41.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f139a60f740 1 -- start start 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f139a60f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f139a60f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b67e80 0x557cd8b70430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f139a60f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 0x557cd8b72e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f13867fc640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f137c0a3d20 0x7f137c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f139a60f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x557cd8acc1f0 con 0x557cd8b674d0 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f139a60f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557cd8a694b0 con 0x557cd8b67e80 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f139a60f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557cd8ac81b0 con 0x557cd8b70a00 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f1395ee7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f1395ee7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b678b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52176/0 (socket says 172.21.15.94:52176) 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f1395ee7640 1 -- 172.21.15.94:0/546399799 learned_addr learned my addr 172.21.15.94:0/546399799 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f1395ee7640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f137c0a3d20 0x7f137c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f13867fc640 1 -- 172.21.15.94:0/2490678944 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f138c0d09e0 con 0x7f1390102190 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f13956e6640 1 --2- 172.21.15.94:0/546399799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b67e80 0x557cd8b70430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.785+0000 7f1394ee5640 1 --2- 172.21.15.94:0/546399799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 0x557cd8b72e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.786+0000 7f1395ee7640 1 -- 172.21.15.94:0/546399799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 msgr2=0x557cd8b72e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.786+0000 7f1395ee7640 1 --2- 172.21.15.94:0/546399799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 0x557cd8b72e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.786+0000 7f1395ee7640 1 -- 172.21.15.94:0/546399799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b67e80 msgr2=0x557cd8b70430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.786+0000 7f1395ee7640 1 --2- 172.21.15.94:0/546399799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b67e80 0x557cd8b70430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.786+0000 7f1395ee7640 1 -- 172.21.15.94:0/546399799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cd8a5b1b0 con 0x557cd8b674d0 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.786+0000 7f1395ee7640 1 --2- 172.21.15.94:0/546399799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b678b0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1388009a60 tx=0x7f1388032540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ccecb5805d624d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.786+0000 7f1395ee7640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f137c0a3d20 0x7f137c0a61e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1388005cc0 tx=0x7f1388045000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.786+0000 7f137b7fe640 1 -- 172.21.15.94:0/546399799 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388039040 con 0x557cd8b674d0 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.786+0000 7f137b7fe640 1 -- 172.21.15.94:0/546399799 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138804e070 con 0x557cd8b674d0 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.786+0000 7f137b7fe640 1 -- 172.21.15.94:0/546399799 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388034810 con 0x557cd8b674d0 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 -- 172.21.15.94:0/546399799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 msgr2=0x557cd8b678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 --2- 172.21.15.94:0/546399799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b678b0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1388009a60 tx=0x7f1388032540 comp rx=0 tx=0).stop 2024-04-21T08:41:41.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 -- 172.21.15.94:0/546399799 shutdown_connections 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 --2- 172.21.15.94:0/546399799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 0x557cd8b72e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 --2- 172.21.15.94:0/546399799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b67e80 0x557cd8b70430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 --2- 172.21.15.94:0/546399799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b678b0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 -- 172.21.15.94:0/546399799 >> 172.21.15.94:0/546399799 conn(0x557cd8b61ba0 msgr2=0x557cd8b61f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 -- 172.21.15.94:0/546399799 shutdown_connections 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 -- 172.21.15.94:0/546399799 wait complete. 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 -- start start 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.787+0000 7f139a60f740 1 Processor -- start 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f139a60f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b674d0 0x557cd8b703c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f139a60f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b6d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f139a60f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b70a00 0x557cd8b6de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f139a60f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x557cd8b73910 con 0x557cd8b67e80 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f139a60f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557cd8a85500 con 0x557cd8b70a00 2024-04-21T08:41:41.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f139a60f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557cd8b73a90 con 0x557cd8b674d0 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f139a60f740 10 client.0 _ll_register_callbacks cb 0x557cd8a53b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1394ee5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b6d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1394ee5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b6d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52186/0 (socket says 172.21.15.94:52186) 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1394ee5640 1 -- 172.21.15.94:0/795102597 learned_addr learned my addr 172.21.15.94:0/795102597 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f13956e6640 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b674d0 0x557cd8b703c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1394ee5640 1 -- 172.21.15.94:0/795102597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b674d0 msgr2=0x557cd8b703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1394ee5640 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b674d0 0x557cd8b703c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1394ee5640 1 -- 172.21.15.94:0/795102597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b70a00 msgr2=0x557cd8b6de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1394ee5640 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b70a00 0x557cd8b6de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1394ee5640 1 -- 172.21.15.94:0/795102597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cd87de3f0 con 0x557cd8b67e80 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f13956e6640 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b674d0 0x557cd8b703c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1394ee5640 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b6d750 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f138000e680 tx=0x7f138000eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58a7d4cd82a6d42 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1378ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:41.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1378ff9640 1 -- 172.21.15.94:0/795102597 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380004220 con 0x557cd8b67e80 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1378ff9640 1 -- 172.21.15.94:0/795102597 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f13800043c0 con 0x557cd8b67e80 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f139a60f740 1 -- 172.21.15.94:0/795102597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f138804e070 con 0x557cd8b67e80 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f1378ff9640 1 -- 172.21.15.94:0/795102597 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1380023050 con 0x557cd8b67e80 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f139a60f740 10 client.16338 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.788+0000 7f139a60f740 1 -- 172.21.15.94:0/795102597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557cd88a87d0 con 0x557cd8b67e80 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f1378ff9640 1 -- 172.21.15.94:0/795102597 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138804e070 con 0x557cd8b67e80 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f1378ff9640 1 -- 172.21.15.94:0/795102597 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1380003a30 con 0x557cd8b67e80 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f139a60f740 10 client.16338 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f139a60f740 10 client.16338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f139a60f740 20 client.16338 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f139a60f740 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557cd8b80d00 0x557cd8b83170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f139a60f740 4 client.16338 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f139a60f740 1 -- 172.21.15.94:0/795102597 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cd8a6b930 con 0x557cd8b80d00 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f13956e6640 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557cd8b80d00 0x557cd8b83170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f1378ff9640 10 client.16338 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.789+0000 7f13956e6640 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557cd8b80d00 0x557cd8b83170 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=331502cb04816760 server_cookie=440cbe973137a90c in_seq=0 out_seq=0 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f1378ff9640 1 -- 172.21.15.94:0/795102597 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557cd8a6b930 con 0x557cd8b80d00 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f1378ff9640 10 client.16338 handle_command_reply: tid=0 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 client.16338 shutdown 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 -- 172.21.15.94:0/795102597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 msgr2=0x557cd8b6d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b6d750 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f138000e680 tx=0x7f138000eb40 comp rx=0 tx=0).stop 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 -- 172.21.15.94:0/795102597 shutdown_connections 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557cd8b80d00 0x557cd8b83170 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b70a00 0x557cd8b6de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b6d750 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 --2- 172.21.15.94:0/795102597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b674d0 0x557cd8b703c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 -- 172.21.15.94:0/795102597 >> 172.21.15.94:0/795102597 conn(0x557cd8b61ba0 msgr2=0x557cd8b638f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f1378ff9640 0 client.16338 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:41.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 -- 172.21.15.94:0/795102597 shutdown_connections 2024-04-21T08:41:41.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.790+0000 7f139a60f740 1 -- 172.21.15.94:0/795102597 wait complete. 2024-04-21T08:41:41.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.791+0000 7f139a60f740 20 client.16338 trim_cache size 0 max 16384 2024-04-21T08:41:41.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f139a60f740 1 Processor -- start 2024-04-21T08:41:41.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f139a60f740 1 -- start start 2024-04-21T08:41:41.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f139a60f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b7c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f139a60f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b67e80 0x557cd8b82d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f139a60f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 0x557cd8b88a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f139a60f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f13901b1cb0 con 0x557cd8b674d0 2024-04-21T08:41:41.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f139a60f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f13901af580 con 0x557cd8b67e80 2024-04-21T08:41:41.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f139a60f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f13901ad3d0 con 0x557cd8b70a00 2024-04-21T08:41:41.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f13956e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b7c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f13956e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b7c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52192/0 (socket says 172.21.15.94:52192) 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f13956e6640 1 -- 172.21.15.94:0/3067120840 learned_addr learned my addr 172.21.15.94:0/3067120840 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f13956e6640 1 -- 172.21.15.94:0/3067120840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 msgr2=0x557cd8b88a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f13956e6640 1 --2- 172.21.15.94:0/3067120840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 0x557cd8b88a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f13956e6640 1 -- 172.21.15.94:0/3067120840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b67e80 msgr2=0x557cd8b82d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f13956e6640 1 --2- 172.21.15.94:0/3067120840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b67e80 0x557cd8b82d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f13956e6640 1 -- 172.21.15.94:0/3067120840 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cd8b732b0 con 0x557cd8b674d0 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f13956e6640 1 --2- 172.21.15.94:0/3067120840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b7c7d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f138c0cde00 tx=0x7f138c00a520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971b77d83c46d4a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f137affd640 1 -- 172.21.15.94:0/3067120840 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c01d040 con 0x557cd8b674d0 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f137affd640 1 -- 172.21.15.94:0/3067120840 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138c09c5a0 con 0x557cd8b674d0 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.804+0000 7f137affd640 1 -- 172.21.15.94:0/3067120840 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c098c10 con 0x557cd8b674d0 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 -- 172.21.15.94:0/3067120840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 msgr2=0x557cd8b7c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 --2- 172.21.15.94:0/3067120840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b7c7d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f138c0cde00 tx=0x7f138c00a520 comp rx=0 tx=0).stop 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 -- 172.21.15.94:0/3067120840 shutdown_connections 2024-04-21T08:41:41.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 --2- 172.21.15.94:0/3067120840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 0x557cd8b88a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 --2- 172.21.15.94:0/3067120840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b67e80 0x557cd8b82d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 --2- 172.21.15.94:0/3067120840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b674d0 0x557cd8b7c7d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 -- 172.21.15.94:0/3067120840 >> 172.21.15.94:0/3067120840 conn(0x557cd8b61ba0 msgr2=0x557cd8b61f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 -- 172.21.15.94:0/3067120840 shutdown_connections 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 -- 172.21.15.94:0/3067120840 wait complete. 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 -- start start 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.805+0000 7f139a60f740 1 Processor -- start 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f139a60f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b674d0 0x557cd8b78ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f139a60f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b69fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f139a60f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 0x557cd8b6a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f139a60f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f13901ada40 con 0x557cd8b67e80 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f139a60f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f13901aeeb0 con 0x557cd8b674d0 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f139a60f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f13901ae110 con 0x557cd8b70a00 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f139a60f740 10 client.0 _ll_register_callbacks cb 0x7f138000cd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:41.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1394ee5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b69fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1394ee5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b69fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52206/0 (socket says 172.21.15.94:52206) 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1394ee5640 1 -- 172.21.15.94:0/3214524613 learned_addr learned my addr 172.21.15.94:0/3214524613 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f13956e6640 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b674d0 0x557cd8b78ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1394ee5640 1 -- 172.21.15.94:0/3214524613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 msgr2=0x557cd8b6a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1394ee5640 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 0x557cd8b6a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1394ee5640 1 -- 172.21.15.94:0/3214524613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b674d0 msgr2=0x557cd8b78ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1394ee5640 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b674d0 0x557cd8b78ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1394ee5640 1 -- 172.21.15.94:0/3214524613 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cd8b6aac0 con 0x557cd8b67e80 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1394ee5640 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b69fc0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1380001eb0 tx=0x7f1380005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9c9b089c9be3a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1378ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1378ff9640 1 -- 172.21.15.94:0/3214524613 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380020040 con 0x557cd8b67e80 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f1378ff9640 1 -- 172.21.15.94:0/3214524613 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f13800041f0 con 0x557cd8b67e80 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.806+0000 7f139a60f740 1 -- 172.21.15.94:0/3214524613 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f138c09c5a0 con 0x557cd8b67e80 2024-04-21T08:41:41.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f1378ff9640 1 -- 172.21.15.94:0/3214524613 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f138001a6a0 con 0x557cd8b67e80 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f139a60f740 10 client.16344 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f139a60f740 1 -- 172.21.15.94:0/3214524613 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557cd8be70e0 con 0x557cd8b67e80 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f1378ff9640 1 -- 172.21.15.94:0/3214524613 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138c09c5a0 con 0x557cd8b67e80 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f1378ff9640 1 -- 172.21.15.94:0/3214524613 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f138000aac0 con 0x557cd8b67e80 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f139a60f740 10 client.16344 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f139a60f740 10 client.16344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f139a60f740 20 client.16344 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f139a60f740 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557cd8be7890 0x557cd8be9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f139a60f740 4 client.16344 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f139a60f740 1 -- 172.21.15.94:0/3214524613 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557cd8ad9c10 con 0x557cd8be7890 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.807+0000 7f13956e6640 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557cd8be7890 0x557cd8be9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f1378ff9640 10 client.16344 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:41.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f13956e6640 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557cd8be7890 0x557cd8be9cb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d03e6a85a290812f server_cookie=91a690bc5638acef in_seq=0 out_seq=0 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f1378ff9640 1 -- 172.21.15.94:0/3214524613 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557cd8ad9c10 con 0x557cd8be7890 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f1378ff9640 10 client.16344 handle_command_reply: tid=0 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 client.16344 shutdown 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 -- 172.21.15.94:0/3214524613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 msgr2=0x557cd8b69fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b69fc0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1380001eb0 tx=0x7f1380005cc0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 -- 172.21.15.94:0/3214524613 shutdown_connections 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557cd8be7890 0x557cd8be9cb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cd8b70a00 0x557cd8b6a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f1378ff9640 0 client.16344 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557cd8b67e80 0x557cd8b69fc0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 --2- 172.21.15.94:0/3214524613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cd8b674d0 0x557cd8b78ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 -- 172.21.15.94:0/3214524613 >> 172.21.15.94:0/3214524613 conn(0x557cd8b61ba0 msgr2=0x557cd8b82d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 -- 172.21.15.94:0/3214524613 shutdown_connections 2024-04-21T08:41:41.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 1 -- 172.21.15.94:0/3214524613 wait complete. 2024-04-21T08:41:41.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139a60f740 20 client.16344 trim_cache size 0 max 16384 2024-04-21T08:41:41.816 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139794a640 1 -- 172.21.15.94:0/2490678944 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f137c0a3d20 msgr2=0x7f137c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139794a640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f137c0a3d20 0x7f137c0a61e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1388005cc0 tx=0x7f1388045000 comp rx=0 tx=0).stop 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139794a640 1 -- 172.21.15.94:0/2490678944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1390102190 msgr2=0x7f13901986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.808+0000 7f139794a640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1390102190 0x7f13901986f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f138c00c890 tx=0x7f138c00cd50 comp rx=0 tx=0).stop 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.809+0000 7f139794a640 1 -- 172.21.15.94:0/2490678944 shutdown_connections 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.809+0000 7f139794a640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f137c0a3d20 0x7f137c0a61e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.809+0000 7f139794a640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1390107260 0x7f1390198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.809+0000 7f139794a640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f139019cfb0 0x7f139019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.809+0000 7f139794a640 1 --2- 172.21.15.94:0/2490678944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1390102190 0x7f13901986f0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.809+0000 7f139794a640 1 -- 172.21.15.94:0/2490678944 >> 172.21.15.94:0/2490678944 conn(0x7f13900793a0 msgr2=0x7f13900fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.809+0000 7f139794a640 1 -- 172.21.15.94:0/2490678944 shutdown_connections 2024-04-21T08:41:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:41.809+0000 7f139794a640 1 -- 172.21.15.94:0/2490678944 wait complete. 2024-04-21T08:41:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:41 smithi094 ceph-mon[27988]: pgmap v54: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 846 MiB/s wr, 464 op/s 2024-04-21T08:41:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:41 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2830405665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:42.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:41 smithi173 ceph-mon[31428]: pgmap v54: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 846 MiB/s wr, 464 op/s 2024-04-21T08:41:42.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:41 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2830405665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:42.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:41 smithi111 ceph-mon[31493]: pgmap v54: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 846 MiB/s wr, 464 op/s 2024-04-21T08:41:42.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:41 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2830405665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:42.821 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:41:42.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:41:42.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:41:43.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.114+0000 7f3e3a282640 1 -- 172.21.15.94:0/789018481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e34074c10 msgr2=0x7f3e341127f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.114+0000 7f3e3a282640 1 --2- 172.21.15.94:0/789018481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e34074c10 0x7f3e341127f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009ca0 tx=0x7f3e2c02c1c0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.118+0000 7f3e3a282640 1 -- 172.21.15.94:0/789018481 shutdown_connections 2024-04-21T08:41:43.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.118+0000 7f3e3a282640 1 --2- 172.21.15.94:0/789018481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e34112d30 0x7f3e34115120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.118+0000 7f3e3a282640 1 --2- 172.21.15.94:0/789018481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e34074c10 0x7f3e341127f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.118+0000 7f3e3a282640 1 --2- 172.21.15.94:0/789018481 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3e340742f0 0x7f3e340746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.118+0000 7f3e3a282640 1 -- 172.21.15.94:0/789018481 >> 172.21.15.94:0/789018481 conn(0x7f3e340fd850 msgr2=0x7f3e340ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.123+0000 7f3e3a282640 1 -- 172.21.15.94:0/789018481 shutdown_connections 2024-04-21T08:41:43.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.125+0000 7f3e3a282640 1 -- 172.21.15.94:0/789018481 wait complete. 2024-04-21T08:41:43.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.126+0000 7f3e3a282640 1 Processor -- start 2024-04-21T08:41:43.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.128+0000 7f3e3a282640 1 -- start start 2024-04-21T08:41:43.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.129+0000 7f3e3a282640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e340742f0 0x7f3e34073780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.129+0000 7f3e3a282640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e34074c10 0x7f3e3406c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.129+0000 7f3e3a282640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3e34112d30 0x7f3e3406cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.129+0000 7f3e3a282640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3e34118620 con 0x7f3e34112d30 2024-04-21T08:41:43.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.129+0000 7f3e3a282640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341184a0 con 0x7f3e34074c10 2024-04-21T08:41:43.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.129+0000 7f3e3a282640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341187a0 con 0x7f3e340742f0 2024-04-21T08:41:43.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.137+0000 7f3e39280640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e340742f0 0x7f3e34073780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.137+0000 7f3e39280640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e340742f0 0x7f3e34073780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:49936/0 (socket says 172.21.15.94:49936) 2024-04-21T08:41:43.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.137+0000 7f3e39280640 1 -- 172.21.15.94:0/2619416523 learned_addr learned my addr 172.21.15.94:0/2619416523 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:43.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.137+0000 7f3e38a7f640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e34074c10 0x7f3e3406c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.139+0000 7f3e39280640 1 -- 172.21.15.94:0/2619416523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e34074c10 msgr2=0x7f3e3406c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.139+0000 7f3e39280640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e34074c10 0x7f3e3406c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.139+0000 7f3e39280640 1 -- 172.21.15.94:0/2619416523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3e34112d30 msgr2=0x7f3e3406cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.139+0000 7f3e39280640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3e34112d30 0x7f3e3406cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.139+0000 7f3e39280640 1 -- 172.21.15.94:0/2619416523 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3406d3d0 con 0x7f3e340742f0 2024-04-21T08:41:43.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.142+0000 7f3e39280640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e340742f0 0x7f3e34073780 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c920 tx=0x7f3e2400cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.144+0000 7f3e2a7fc640 1 -- 172.21.15.94:0/2619416523 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24007c10 con 0x7f3e340742f0 2024-04-21T08:41:43.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.144+0000 7f3e3a282640 1 -- 172.21.15.94:0/2619416523 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3406d6c0 con 0x7f3e340742f0 2024-04-21T08:41:43.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.144+0000 7f3e3a282640 1 -- 172.21.15.94:0/2619416523 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e340715f0 con 0x7f3e340742f0 2024-04-21T08:41:43.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.144+0000 7f3e2a7fc640 1 -- 172.21.15.94:0/2619416523 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e24007db0 con 0x7f3e340742f0 2024-04-21T08:41:43.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.144+0000 7f3e2a7fc640 1 -- 172.21.15.94:0/2619416523 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e240187c0 con 0x7f3e340742f0 2024-04-21T08:41:43.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.148+0000 7f3e3e1b3740 1 Processor -- start 2024-04-21T08:41:43.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.152+0000 7f3e3e1b3740 1 -- start start 2024-04-21T08:41:43.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.153+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df59aa0 0x555b4df59e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.153+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df5a450 0x555b4df62a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.153+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df65420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.153+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x555b4debdf00 con 0x555b4df62fd0 2024-04-21T08:41:43.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.153+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555b4de5b590 con 0x555b4df59aa0 2024-04-21T08:41:43.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.153+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555b4deb9ec0 con 0x555b4df5a450 2024-04-21T08:41:43.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.154+0000 7f3e38a7f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df65420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.154+0000 7f3e38a7f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df65420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52228/0 (socket says 172.21.15.94:52228) 2024-04-21T08:41:43.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.154+0000 7f3e38a7f640 1 -- 172.21.15.94:0/4142535931 learned_addr learned my addr 172.21.15.94:0/4142535931 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:43.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.154+0000 7f3e39280640 1 --2- 172.21.15.94:0/4142535931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df5a450 0x555b4df62a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.154+0000 7f3e38a7f640 1 -- 172.21.15.94:0/4142535931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df5a450 msgr2=0x555b4df62a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.154+0000 7f3e38a7f640 1 --2- 172.21.15.94:0/4142535931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df5a450 0x555b4df62a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.155+0000 7f3e38a7f640 1 -- 172.21.15.94:0/4142535931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df59aa0 msgr2=0x555b4df59e80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:41:43.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.155+0000 7f3e38a7f640 1 --2- 172.21.15.94:0/4142535931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df59aa0 0x555b4df59e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.155+0000 7f3e38a7f640 1 -- 172.21.15.94:0/4142535931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b4dc987e0 con 0x555b4df62fd0 2024-04-21T08:41:43.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.156+0000 7f3e2a7fc640 1 -- 172.21.15.94:0/2619416523 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f3e24018a40 con 0x7f3e340742f0 2024-04-21T08:41:43.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.157+0000 7f3e38a7f640 1 --2- 172.21.15.94:0/4142535931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df65420 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009bf0 tx=0x7f3e2c0099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f23bc63e9bc548f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.157+0000 7f3e2a7fc640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3e200a3ef0 0x7f3e200a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.158+0000 7f3e2a7fc640 1 -- 172.21.15.94:0/2619416523 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3e240d0b50 con 0x7f3e340742f0 2024-04-21T08:41:43.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.158+0000 7f3e1f7fe640 1 -- 172.21.15.94:0/4142535931 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c039040 con 0x555b4df62fd0 2024-04-21T08:41:43.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.158+0000 7f3e1f7fe640 1 -- 172.21.15.94:0/4142535931 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e2c003b70 con 0x555b4df62fd0 2024-04-21T08:41:43.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.158+0000 7f3e39a81640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3e200a3ef0 0x7f3e200a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.159+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4142535931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 msgr2=0x555b4df65420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.159+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/4142535931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df65420 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009bf0 tx=0x7f3e2c0099b0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.162+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4142535931 shutdown_connections 2024-04-21T08:41:43.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.162+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/4142535931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df5a450 0x555b4df62a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.162+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/4142535931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df65420 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.162+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/4142535931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df59aa0 0x555b4df59e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.162+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4142535931 >> 172.21.15.94:0/4142535931 conn(0x555b4df54170 msgr2=0x555b4df54550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.163+0000 7f3e39a81640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3e200a3ef0 0x7f3e200a63b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3e30006fd0 tx=0x7f3e30006c40 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.169+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4142535931 shutdown_connections 2024-04-21T08:41:43.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.173+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4142535931 wait complete. 2024-04-21T08:41:43.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.173+0000 7f3e3e1b3740 1 -- start start 2024-04-21T08:41:43.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.174+0000 7f3e3e1b3740 1 Processor -- start 2024-04-21T08:41:43.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.178+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df59aa0 0x555b4df5ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.178+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 0x555b4df60520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.178+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df62fd0 0x555b4df60a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.178+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x555b4df65f30 con 0x555b4df59aa0 2024-04-21T08:41:43.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.178+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555b4de77580 con 0x555b4df5a450 2024-04-21T08:41:43.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.178+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555b4df660b0 con 0x555b4df62fd0 2024-04-21T08:41:43.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.179+0000 7f3e3e1b3740 10 client.0 _ll_register_callbacks cb 0x555b4de7d650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:43.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.179+0000 7f3e39280640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df59aa0 0x555b4df5ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.180+0000 7f3e39280640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df59aa0 0x555b4df5ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52238/0 (socket says 172.21.15.94:52238) 2024-04-21T08:41:43.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.180+0000 7f3e39280640 1 -- 172.21.15.94:0/3395044883 learned_addr learned my addr 172.21.15.94:0/3395044883 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:43.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.182+0000 7f3e38a7f640 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 0x555b4df60520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.182+0000 7f3e39280640 1 -- 172.21.15.94:0/3395044883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df62fd0 msgr2=0x555b4df60a60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:41:43.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.183+0000 7f3e39280640 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df62fd0 0x555b4df60a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.183+0000 7f3e39280640 1 -- 172.21.15.94:0/3395044883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 msgr2=0x555b4df60520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.183+0000 7f3e39280640 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 0x555b4df60520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.183+0000 7f3e39280640 1 -- 172.21.15.94:0/3395044883 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b4dcf8f70 con 0x555b4df59aa0 2024-04-21T08:41:43.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.183+0000 7f3e38a7f640 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 0x555b4df60520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:43.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.187+0000 7f3e39280640 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df59aa0 0x555b4df5ffe0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c8f0 tx=0x7f3e2409abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2c0947217d677e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.187+0000 7f3e1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:43.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.191+0000 7f3e1cff9640 1 -- 172.21.15.94:0/3395044883 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e240d5040 con 0x555b4df59aa0 2024-04-21T08:41:43.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.191+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/3395044883 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555b4decb920 con 0x555b4df59aa0 2024-04-21T08:41:43.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.191+0000 7f3e1cff9640 1 -- 172.21.15.94:0/3395044883 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3e240ce0e0 con 0x555b4df59aa0 2024-04-21T08:41:43.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.192+0000 7f3e1cff9640 1 -- 172.21.15.94:0/3395044883 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3e240ce2e0 con 0x555b4df59aa0 2024-04-21T08:41:43.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.192+0000 7f3e3e1b3740 10 client.16350 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:43.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.192+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/3395044883 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555b4db90170 con 0x555b4df59aa0 2024-04-21T08:41:43.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.193+0000 7f3e1cff9640 1 -- 172.21.15.94:0/3395044883 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555b4decb920 con 0x555b4df59aa0 2024-04-21T08:41:43.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.194+0000 7f3e1cff9640 1 -- 172.21.15.94:0/3395044883 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3e240a4e90 con 0x555b4df59aa0 2024-04-21T08:41:43.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.194+0000 7f3e3e1b3740 10 client.16350 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:43.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.194+0000 7f3e3e1b3740 10 client.16350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:43.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.194+0000 7f3e3e1b3740 20 client.16350 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:43.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.194+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b4df738b0 0x555b4df75d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.194+0000 7f3e3e1b3740 4 client.16350 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:43.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.194+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/3395044883 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b4dec5790 con 0x555b4df738b0 2024-04-21T08:41:43.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.201+0000 7f3e38a7f640 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b4df738b0 0x555b4df75d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.212+0000 7f3e1cff9640 10 client.16350 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:43.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.212+0000 7f3e38a7f640 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b4df738b0 0x555b4df75d70 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27afffa708b30748 server_cookie=c6bfa01c0255f873 in_seq=0 out_seq=0 2024-04-21T08:41:43.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e1cff9640 1 -- 172.21.15.94:0/3395044883 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b4dec5790 con 0x555b4df738b0 2024-04-21T08:41:43.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e1cff9640 10 client.16350 handle_command_reply: tid=0 2024-04-21T08:41:43.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e3e1b3740 1 client.16350 shutdown 2024-04-21T08:41:43.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/3395044883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df59aa0 msgr2=0x555b4df5ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df59aa0 0x555b4df5ffe0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c8f0 tx=0x7f3e2409abc0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/3395044883 shutdown_connections 2024-04-21T08:41:43.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b4df738b0 0x555b4df75d70 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df62fd0 0x555b4df60a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 0x555b4df60520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/3395044883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df59aa0 0x555b4df5ffe0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/3395044883 >> 172.21.15.94:0/3395044883 conn(0x555b4df54170 msgr2=0x555b4df55ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.218+0000 7f3e1cff9640 0 client.16350 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:43.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.219+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/3395044883 shutdown_connections 2024-04-21T08:41:43.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.219+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/3395044883 wait complete. 2024-04-21T08:41:43.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.223+0000 7f3e3e1b3740 20 client.16350 trim_cache size 0 max 16384 2024-04-21T08:41:43.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.265+0000 7f3e3e1b3740 1 Processor -- start 2024-04-21T08:41:43.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.270+0000 7f3e3e1b3740 1 -- start start 2024-04-21T08:41:43.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.270+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df59aa0 0x555b4df62710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.270+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 0x555b4df6ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.270+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df53210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.270+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18012880 con 0x555b4df62fd0 2024-04-21T08:41:43.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.270+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18010150 con 0x555b4df5a450 2024-04-21T08:41:43.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.270+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1800e050 con 0x555b4df59aa0 2024-04-21T08:41:43.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.272+0000 7f3e38a7f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 0x555b4df6ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.272+0000 7f3e38a7f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 0x555b4df6ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:52018/0 (socket says 172.21.15.94:52018) 2024-04-21T08:41:43.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.272+0000 7f3e38a7f640 1 -- 172.21.15.94:0/278658913 learned_addr learned my addr 172.21.15.94:0/278658913 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:43.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.272+0000 7f3e39a81640 1 --2- 172.21.15.94:0/278658913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df53210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.272+0000 7f3e39a81640 1 -- 172.21.15.94:0/278658913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df59aa0 msgr2=0x555b4df62710 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:41:43.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.272+0000 7f3e39a81640 1 --2- 172.21.15.94:0/278658913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df59aa0 0x555b4df62710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.272+0000 7f3e39a81640 1 -- 172.21.15.94:0/278658913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 msgr2=0x555b4df6ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.272+0000 7f3e39a81640 1 --2- 172.21.15.94:0/278658913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 0x555b4df6ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.273+0000 7f3e39a81640 1 -- 172.21.15.94:0/278658913 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b4df4fdb0 con 0x555b4df62fd0 2024-04-21T08:41:43.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.273+0000 7f3e39a81640 1 --2- 172.21.15.94:0/278658913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df53210 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3e30013850 tx=0x7f3e300047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a2f6bae3606159 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.275+0000 7f3e1effd640 1 -- 172.21.15.94:0/278658913 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e30012a90 con 0x555b4df62fd0 2024-04-21T08:41:43.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.275+0000 7f3e1effd640 1 -- 172.21.15.94:0/278658913 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e30022070 con 0x555b4df62fd0 2024-04-21T08:41:43.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.275+0000 7f3e1effd640 1 -- 172.21.15.94:0/278658913 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e30010360 con 0x555b4df62fd0 2024-04-21T08:41:43.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.276+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/278658913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 msgr2=0x555b4df53210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.276+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/278658913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df53210 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3e30013850 tx=0x7f3e300047e0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.277+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/278658913 shutdown_connections 2024-04-21T08:41:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.278+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/278658913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df53210 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.278+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/278658913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df5a450 0x555b4df6ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.278+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/278658913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df59aa0 0x555b4df62710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.278+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/278658913 >> 172.21.15.94:0/278658913 conn(0x555b4df54170 msgr2=0x555b4df52b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.283+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/278658913 shutdown_connections 2024-04-21T08:41:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.284+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/278658913 wait complete. 2024-04-21T08:41:43.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.284+0000 7f3e3e1b3740 1 -- start start 2024-04-21T08:41:43.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.285+0000 7f3e3e1b3740 1 Processor -- start 2024-04-21T08:41:43.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df59aa0 0x555b4df5cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df5a450 0x555b4df5d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e3e1b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df5d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1800e610 con 0x555b4df62fd0 2024-04-21T08:41:43.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1800fa80 con 0x555b4df59aa0 2024-04-21T08:41:43.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e3e1b3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1800ece0 con 0x555b4df5a450 2024-04-21T08:41:43.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e3e1b3740 10 client.0 _ll_register_callbacks cb 0x7f3e2400cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:43.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e39a81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df5d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e39a81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df5d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52268/0 (socket says 172.21.15.94:52268) 2024-04-21T08:41:43.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e39a81640 1 -- 172.21.15.94:0/4164904601 learned_addr learned my addr 172.21.15.94:0/4164904601 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:43.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.286+0000 7f3e39280640 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df59aa0 0x555b4df5cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.287+0000 7f3e39a81640 1 -- 172.21.15.94:0/4164904601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df5a450 msgr2=0x555b4df5d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.287+0000 7f3e39a81640 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df5a450 0x555b4df5d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.287+0000 7f3e39a81640 1 -- 172.21.15.94:0/4164904601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df59aa0 msgr2=0x555b4df5cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.287+0000 7f3e39a81640 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df59aa0 0x555b4df5cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.287+0000 7f3e39a81640 1 -- 172.21.15.94:0/4164904601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b4dfd9140 con 0x555b4df62fd0 2024-04-21T08:41:43.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.290+0000 7f3e39a81640 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df5d9f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3e30012ac0 tx=0x7f3e30012af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39deaa8b2ba36d7c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.291+0000 7f3e1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:43.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.292+0000 7f3e1cff9640 1 -- 172.21.15.94:0/4164904601 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e30030610 con 0x555b4df62fd0 2024-04-21T08:41:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.292+0000 7f3e1cff9640 1 -- 172.21.15.94:0/4164904601 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3e300307b0 con 0x555b4df62fd0 2024-04-21T08:41:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.292+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4164904601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e30022070 con 0x555b4df62fd0 2024-04-21T08:41:43.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.295+0000 7f3e1cff9640 1 -- 172.21.15.94:0/4164904601 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3e30039050 con 0x555b4df62fd0 2024-04-21T08:41:43.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.295+0000 7f3e3e1b3740 10 client.16356 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:43.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.296+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4164904601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555b4dfda460 con 0x555b4df62fd0 2024-04-21T08:41:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.299+0000 7f3e1cff9640 1 -- 172.21.15.94:0/4164904601 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e30022070 con 0x555b4df62fd0 2024-04-21T08:41:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.299+0000 7f3e1cff9640 1 -- 172.21.15.94:0/4164904601 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3e30012070 con 0x555b4df62fd0 2024-04-21T08:41:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.299+0000 7f3e3e1b3740 10 client.16356 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.299+0000 7f3e3e1b3740 10 client.16356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.299+0000 7f3e3e1b3740 20 client.16356 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.299+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b4dfdaa40 0x555b4dfdce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.299+0000 7f3e3e1b3740 4 client.16356 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:41:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.299+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4164904601 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555b4de41c50 con 0x555b4dfdaa40 2024-04-21T08:41:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.300+0000 7f3e39280640 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b4dfdaa40 0x555b4dfdce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.303+0000 7f3e39280640 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b4dfdaa40 0x555b4dfdce60 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daafd1bd2f2a0a16 server_cookie=25c8d26b2fba3dfc in_seq=0 out_seq=0 2024-04-21T08:41:43.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.303+0000 7f3e1cff9640 10 client.16356 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:43.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.306+0000 7f3e1cff9640 1 -- 172.21.15.94:0/4164904601 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555b4de41c50 con 0x555b4dfdaa40 2024-04-21T08:41:43.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.306+0000 7f3e1cff9640 10 client.16356 handle_command_reply: tid=0 2024-04-21T08:41:43.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.306+0000 7f3e3e1b3740 1 client.16356 shutdown 2024-04-21T08:41:43.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.311+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4164904601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 msgr2=0x555b4df5d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.311+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df5d9f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3e30012ac0 tx=0x7f3e30012af0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.314+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4164904601 shutdown_connections 2024-04-21T08:41:43.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.314+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b4dfdaa40 0x555b4dfdce60 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.314+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b4df5a450 0x555b4df5d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.314+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b4df62fd0 0x555b4df5d9f0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.314+0000 7f3e3e1b3740 1 --2- 172.21.15.94:0/4164904601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b4df59aa0 0x555b4df5cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.314+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4164904601 >> 172.21.15.94:0/4164904601 conn(0x555b4df54170 msgr2=0x555b4df53390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.314+0000 7f3e1cff9640 0 client.16356 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:43.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.314+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4164904601 shutdown_connections 2024-04-21T08:41:43.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.315+0000 7f3e3e1b3740 1 -- 172.21.15.94:0/4164904601 wait complete. 2024-04-21T08:41:43.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.316+0000 7f3e3e1b3740 20 client.16356 trim_cache size 0 max 16384 2024-04-21T08:41:43.318 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:41:43.318 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:41:43.318 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "6b3dade7-6e21-4c85-83b4-1546e52e1d20", 2024-04-21T08:41:43.319 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:41:43.319 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:41:43.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.322+0000 7f3e3a282640 1 -- 172.21.15.94:0/2619416523 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3e200a3ef0 msgr2=0x7f3e200a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.322+0000 7f3e3a282640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3e200a3ef0 0x7f3e200a63b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3e30006fd0 tx=0x7f3e30006c40 comp rx=0 tx=0).stop 2024-04-21T08:41:43.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.322+0000 7f3e3a282640 1 -- 172.21.15.94:0/2619416523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e340742f0 msgr2=0x7f3e34073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.322+0000 7f3e3a282640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e340742f0 0x7f3e34073780 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c920 tx=0x7f3e2400cde0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.322+0000 7f3e3a282640 1 -- 172.21.15.94:0/2619416523 shutdown_connections 2024-04-21T08:41:43.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.322+0000 7f3e3a282640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3e200a3ef0 0x7f3e200a63b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.322+0000 7f3e3a282640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3e34112d30 0x7f3e3406cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.322+0000 7f3e3a282640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e34074c10 0x7f3e3406c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.322+0000 7f3e3a282640 1 --2- 172.21.15.94:0/2619416523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e340742f0 0x7f3e34073780 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c920 tx=0x7f3e2400cde0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.322+0000 7f3e3a282640 1 -- 172.21.15.94:0/2619416523 >> 172.21.15.94:0/2619416523 conn(0x7f3e340fd850 msgr2=0x7f3e340ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.327+0000 7f3e3a282640 1 -- 172.21.15.94:0/2619416523 shutdown_connections 2024-04-21T08:41:43.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.329+0000 7f3e3a282640 1 -- 172.21.15.94:0/2619416523 wait complete. 2024-04-21T08:41:43.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:41:43.378 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:41:43.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.616+0000 7efed8b0d640 1 -- 172.21.15.94:0/3751223226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efed41088e0 msgr2=0x7efed4108cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.616+0000 7efed8b0d640 1 --2- 172.21.15.94:0/3751223226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efed41088e0 0x7efed4108cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efec4009930 tx=0x7efec402bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:43.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.617+0000 7efed8b0d640 1 -- 172.21.15.94:0/3751223226 shutdown_connections 2024-04-21T08:41:43.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.617+0000 7efed8b0d640 1 --2- 172.21.15.94:0/3751223226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efed410d900 0x7efed4115480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.617+0000 7efed8b0d640 1 --2- 172.21.15.94:0/3751223226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efed4109290 0x7efed410d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.617+0000 7efed8b0d640 1 --2- 172.21.15.94:0/3751223226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efed41088e0 0x7efed4108cc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.617+0000 7efed8b0d640 1 -- 172.21.15.94:0/3751223226 >> 172.21.15.94:0/3751223226 conn(0x7efed4077dc0 msgr2=0x7efed4106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.617+0000 7efed8b0d640 1 -- 172.21.15.94:0/3751223226 shutdown_connections 2024-04-21T08:41:43.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.617+0000 7efed8b0d640 1 -- 172.21.15.94:0/3751223226 wait complete. 2024-04-21T08:41:43.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.618+0000 7efed8b0d640 1 Processor -- start 2024-04-21T08:41:43.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.618+0000 7efed8b0d640 1 -- start start 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efed8b0d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efed41088e0 0x7efed419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efed8b0d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efed410d900 0x7efed419f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efed8b0d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efed41a3600 0x7efed41a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efed259c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efed41088e0 0x7efed419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efed259c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efed41088e0 0x7efed419ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:45736/0 (socket says 172.21.15.94:45736) 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efed259c640 1 -- 172.21.15.94:0/86771180 learned_addr learned my addr 172.21.15.94:0/86771180 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efecbfff640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efed410d900 0x7efed419f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efed4118780 con 0x7efed410d900 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efed4118600 con 0x7efed41088e0 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efed2d9d640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efed41a3600 0x7efed41a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.619+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efed4118900 con 0x7efed41a3600 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.620+0000 7efed259c640 1 -- 172.21.15.94:0/86771180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efed41a3600 msgr2=0x7efed41a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.620+0000 7efed259c640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efed41a3600 0x7efed41a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.620+0000 7efed259c640 1 -- 172.21.15.94:0/86771180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efed410d900 msgr2=0x7efed419f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.620+0000 7efed259c640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efed410d900 0x7efed419f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.620+0000 7efed259c640 1 -- 172.21.15.94:0/86771180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed41a40f0 con 0x7efed41088e0 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.620+0000 7efed259c640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efed41088e0 0x7efed419ed40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efec4009230 tx=0x7efec4004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.620+0000 7efecb7fe640 1 -- 172.21.15.94:0/86771180 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec4043020 con 0x7efed41088e0 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.620+0000 7efecb7fe640 1 -- 172.21.15.94:0/86771180 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efec4004440 con 0x7efed41088e0 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.620+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed41a4380 con 0x7efed41088e0 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.620+0000 7efecb7fe640 1 -- 172.21.15.94:0/86771180 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec4005360 con 0x7efed41088e0 2024-04-21T08:41:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.621+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed41abbd0 con 0x7efed41088e0 2024-04-21T08:41:43.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.624+0000 7efedb7d0740 1 Processor -- start 2024-04-21T08:41:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.624+0000 7efedb7d0740 1 -- start start 2024-04-21T08:41:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efecb7fe640 1 -- 172.21.15.94:0/86771180 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7efec4046020 con 0x7efed41088e0 2024-04-21T08:41:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0baa960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bb5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efedb7d0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55dbe0b0f1e0 con 0x55dbe0bb3a80 2024-04-21T08:41:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efedb7d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dbe0aac390 con 0x55dbe0baa580 2024-04-21T08:41:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efedb7d0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dbe0b0b1a0 con 0x55dbe0baaea0 2024-04-21T08:41:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efecbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bb5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efecbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bb5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49290/0 (socket says 172.21.15.94:49290) 2024-04-21T08:41:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efecbfff640 1 -- 172.21.15.94:0/2473941973 learned_addr learned my addr 172.21.15.94:0/2473941973 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.625+0000 7efed2d9d640 1 --2- 172.21.15.94:0/2473941973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0baa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.626+0000 7efecb7fe640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efec00a3d20 0x7efec00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.626+0000 7efecbfff640 1 -- 172.21.15.94:0/2473941973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 msgr2=0x55dbe0bb3540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:41:43.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.626+0000 7efecbfff640 1 --2- 172.21.15.94:0/2473941973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb3540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.626+0000 7efecbfff640 1 -- 172.21.15.94:0/2473941973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 msgr2=0x55dbe0baa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.626+0000 7efecbfff640 1 --2- 172.21.15.94:0/2473941973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0baa960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.626+0000 7efecbfff640 1 -- 172.21.15.94:0/2473941973 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbe08eb7d0 con 0x55dbe0bb3a80 2024-04-21T08:41:43.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.626+0000 7efed2d9d640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efec00a3d20 0x7efec00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.626+0000 7efecb7fe640 1 -- 172.21.15.94:0/86771180 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7efec40f2530 con 0x7efed41088e0 2024-04-21T08:41:43.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.626+0000 7efecbfff640 1 --2- 172.21.15.94:0/2473941973 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bb5e70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7efebc009a00 tx=0x7efebc009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fb8694e8937a2b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.626+0000 7efed2d9d640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efec00a3d20 0x7efec00a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efecc005fd0 tx=0x7efecc004a40 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.627+0000 7efec8ff9640 1 -- 172.21.15.94:0/2473941973 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc017070 con 0x55dbe0bb3a80 2024-04-21T08:41:43.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.627+0000 7efec8ff9640 1 -- 172.21.15.94:0/2473941973 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efebc013470 con 0x55dbe0bb3a80 2024-04-21T08:41:43.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.627+0000 7efec8ff9640 1 -- 172.21.15.94:0/2473941973 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc012630 con 0x55dbe0bb3a80 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.628+0000 7efedb7d0740 1 -- 172.21.15.94:0/2473941973 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 msgr2=0x55dbe0bb5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.628+0000 7efedb7d0740 1 --2- 172.21.15.94:0/2473941973 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bb5e70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7efebc009a00 tx=0x7efebc009ec0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.628+0000 7efedb7d0740 1 -- 172.21.15.94:0/2473941973 shutdown_connections 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.628+0000 7efedb7d0740 1 --2- 172.21.15.94:0/2473941973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.628+0000 7efedb7d0740 1 --2- 172.21.15.94:0/2473941973 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bb5e70 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.628+0000 7efedb7d0740 1 --2- 172.21.15.94:0/2473941973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0baa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.628+0000 7efedb7d0740 1 -- 172.21.15.94:0/2473941973 >> 172.21.15.94:0/2473941973 conn(0x55dbe0ba4cb0 msgr2=0x55dbe0ba5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.628+0000 7efedb7d0740 1 -- 172.21.15.94:0/2473941973 shutdown_connections 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.628+0000 7efedb7d0740 1 -- 172.21.15.94:0/2473941973 wait complete. 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.628+0000 7efedb7d0740 1 -- start start 2024-04-21T08:41:43.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efedb7d0740 1 Processor -- start 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0baa580 0x55dbe0bb0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bb15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efedb7d0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55dbe0bb6870 con 0x55dbe0baa580 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efedb7d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dbe0ac82f0 con 0x55dbe0bb3a80 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efedb7d0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dbe0bb69f0 con 0x55dbe0baaea0 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efedb7d0740 10 client.0 _ll_register_callbacks cb 0x55dbe0ad5580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efed259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0baa580 0x55dbe0bb0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efed259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0baa580 0x55dbe0bb0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49304/0 (socket says 172.21.15.94:49304) 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.629+0000 7efed259c640 1 -- 172.21.15.94:0/572092578 learned_addr learned my addr 172.21.15.94:0/572092578 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.630+0000 7efecbfff640 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.630+0000 7efed259c640 1 -- 172.21.15.94:0/572092578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 msgr2=0x55dbe0bb1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.630+0000 7efed259c640 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.630+0000 7efed259c640 1 -- 172.21.15.94:0/572092578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0bb3a80 msgr2=0x55dbe0bb15c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:41:43.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.630+0000 7efed259c640 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bb15c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.630+0000 7efed259c640 1 -- 172.21.15.94:0/572092578 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbe08213f0 con 0x55dbe0baa580 2024-04-21T08:41:43.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.630+0000 7efed259c640 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0baa580 0x55dbe0bb0b40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7efec40efea0 tx=0x7efec40bde00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b4ee474a3e8bae server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.630+0000 7efeb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:43.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.630+0000 7efeb9ffb640 1 -- 172.21.15.94:0/572092578 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec40d6020 con 0x55dbe0baa580 2024-04-21T08:41:43.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.631+0000 7efedb7d0740 1 -- 172.21.15.94:0/572092578 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efebc013470 con 0x55dbe0baa580 2024-04-21T08:41:43.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.631+0000 7efeb9ffb640 1 -- 172.21.15.94:0/572092578 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7efec40ba870 con 0x55dbe0baa580 2024-04-21T08:41:43.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.631+0000 7efeb9ffb640 1 -- 172.21.15.94:0/572092578 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efec40dd170 con 0x55dbe0baa580 2024-04-21T08:41:43.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.632+0000 7efedb7d0740 10 client.16365 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:43.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.632+0000 7efedb7d0740 1 -- 172.21.15.94:0/572092578 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dbe07e10f0 con 0x55dbe0baa580 2024-04-21T08:41:43.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.633+0000 7efeb9ffb640 1 -- 172.21.15.94:0/572092578 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efebc013470 con 0x55dbe0baa580 2024-04-21T08:41:43.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.633+0000 7efeb9ffb640 1 -- 172.21.15.94:0/572092578 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efec4005500 con 0x55dbe0baa580 2024-04-21T08:41:43.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.633+0000 7efedb7d0740 10 client.16365 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:43.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.633+0000 7efedb7d0740 10 client.16365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:43.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.633+0000 7efedb7d0740 20 client.16365 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:43.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.633+0000 7efedb7d0740 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dbe0bd4960 0x55dbe0bd6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.633+0000 7efedb7d0740 4 client.16365 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:43.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.633+0000 7efedb7d0740 1 -- 172.21.15.94:0/572092578 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbe0b1cc00 con 0x55dbe0bd4960 2024-04-21T08:41:43.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.635+0000 7efecbfff640 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dbe0bd4960 0x55dbe0bd6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.638+0000 7efecbfff640 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dbe0bd4960 0x55dbe0bd6da0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f16798bdcb21799 server_cookie=c9336ba6f8f6f225 in_seq=0 out_seq=0 2024-04-21T08:41:43.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.639+0000 7efeb9ffb640 10 client.16365 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:43.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efeb9ffb640 1 -- 172.21.15.94:0/572092578 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbe0b1cc00 con 0x55dbe0bd4960 2024-04-21T08:41:43.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efeb9ffb640 10 client.16365 handle_command_reply: tid=0 2024-04-21T08:41:43.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efedb7d0740 1 client.16365 shutdown 2024-04-21T08:41:43.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efedb7d0740 1 -- 172.21.15.94:0/572092578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0baa580 msgr2=0x55dbe0bb0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efedb7d0740 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0baa580 0x55dbe0bb0b40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7efec40efea0 tx=0x7efec40bde00 comp rx=0 tx=0).stop 2024-04-21T08:41:43.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efedb7d0740 1 -- 172.21.15.94:0/572092578 shutdown_connections 2024-04-21T08:41:43.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efedb7d0740 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dbe0bd4960 0x55dbe0bd6da0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efedb7d0740 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efedb7d0740 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bb15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efedb7d0740 1 --2- 172.21.15.94:0/572092578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0baa580 0x55dbe0bb0b40 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efedb7d0740 1 -- 172.21.15.94:0/572092578 >> 172.21.15.94:0/572092578 conn(0x55dbe0ba4cb0 msgr2=0x55dbe0ba6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.643+0000 7efeb9ffb640 0 client.16365 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:43.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.647+0000 7efedb7d0740 1 -- 172.21.15.94:0/572092578 shutdown_connections 2024-04-21T08:41:43.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.648+0000 7efedb7d0740 1 -- 172.21.15.94:0/572092578 wait complete. 2024-04-21T08:41:43.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.648+0000 7efedb7d0740 20 client.16365 trim_cache size 0 max 16384 2024-04-21T08:41:43.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.705+0000 7efedb7d0740 1 Processor -- start 2024-04-21T08:41:43.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.705+0000 7efedb7d0740 1 -- start start 2024-04-21T08:41:43.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.705+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0bb3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.706+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.706+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0ba3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.706+0000 7efedb7d0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efed41afb40 con 0x55dbe0bb3a80 2024-04-21T08:41:43.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.706+0000 7efedb7d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efed41ad410 con 0x55dbe0baa580 2024-04-21T08:41:43.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.706+0000 7efedb7d0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efed40735e0 con 0x55dbe0baaea0 2024-04-21T08:41:43.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.708+0000 7efed259c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0bb3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.708+0000 7efed259c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0bb3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:45768/0 (socket says 172.21.15.94:45768) 2024-04-21T08:41:43.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.708+0000 7efed259c640 1 -- 172.21.15.94:0/2462310488 learned_addr learned my addr 172.21.15.94:0/2462310488 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:43.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.710+0000 7efed2d9d640 1 --2- 172.21.15.94:0/2462310488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0ba3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.711+0000 7efecbfff640 1 --2- 172.21.15.94:0/2462310488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.713+0000 7efed259c640 1 -- 172.21.15.94:0/2462310488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 msgr2=0x55dbe0bb13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.713+0000 7efed259c640 1 --2- 172.21.15.94:0/2462310488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.713+0000 7efed259c640 1 -- 172.21.15.94:0/2462310488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 msgr2=0x55dbe0ba3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.713+0000 7efed259c640 1 --2- 172.21.15.94:0/2462310488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0ba3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.713+0000 7efed259c640 1 -- 172.21.15.94:0/2462310488 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbe0baf090 con 0x55dbe0baa580 2024-04-21T08:41:43.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.716+0000 7efed259c640 1 --2- 172.21.15.94:0/2462310488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0bb3270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efec40dd310 tx=0x7efec40bc710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4b37e3b357e084f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.718+0000 7efebbfff640 1 -- 172.21.15.94:0/2462310488 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec40d6040 con 0x55dbe0baa580 2024-04-21T08:41:43.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.718+0000 7efebbfff640 1 -- 172.21.15.94:0/2462310488 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efec40c6e70 con 0x55dbe0baa580 2024-04-21T08:41:43.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.718+0000 7efebbfff640 1 -- 172.21.15.94:0/2462310488 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec40c25b0 con 0x55dbe0baa580 2024-04-21T08:41:43.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.720+0000 7efedb7d0740 1 -- 172.21.15.94:0/2462310488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 msgr2=0x55dbe0bb3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.720+0000 7efedb7d0740 1 --2- 172.21.15.94:0/2462310488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0bb3270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efec40dd310 tx=0x7efec40bc710 comp rx=0 tx=0).stop 2024-04-21T08:41:43.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.720+0000 7efedb7d0740 1 -- 172.21.15.94:0/2462310488 shutdown_connections 2024-04-21T08:41:43.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.720+0000 7efedb7d0740 1 --2- 172.21.15.94:0/2462310488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0bb13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.720+0000 7efedb7d0740 1 --2- 172.21.15.94:0/2462310488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0ba3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.720+0000 7efedb7d0740 1 --2- 172.21.15.94:0/2462310488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0bb3270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.720+0000 7efedb7d0740 1 -- 172.21.15.94:0/2462310488 >> 172.21.15.94:0/2462310488 conn(0x55dbe0ba4cb0 msgr2=0x55dbe0ba34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.723+0000 7efedb7d0740 1 -- 172.21.15.94:0/2462310488 shutdown_connections 2024-04-21T08:41:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.724+0000 7efedb7d0740 1 -- 172.21.15.94:0/2462310488 wait complete. 2024-04-21T08:41:43.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.725+0000 7efedb7d0740 1 -- start start 2024-04-21T08:41:43.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.725+0000 7efedb7d0740 1 Processor -- start 2024-04-21T08:41:43.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.725+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0bad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.725+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0badbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.725+0000 7efedb7d0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bae130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.725+0000 7efedb7d0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efed4073c50 con 0x55dbe0bb3a80 2024-04-21T08:41:43.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.725+0000 7efedb7d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efed41acd40 con 0x55dbe0baa580 2024-04-21T08:41:43.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.725+0000 7efedb7d0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efed41abfa0 con 0x55dbe0baaea0 2024-04-21T08:41:43.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.725+0000 7efedb7d0740 10 client.0 _ll_register_callbacks cb 0x7efec4009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.728+0000 7efed2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.728+0000 7efed2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bae130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49320/0 (socket says 172.21.15.94:49320) 2024-04-21T08:41:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.728+0000 7efed2d9d640 1 -- 172.21.15.94:0/682450879 learned_addr learned my addr 172.21.15.94:0/682450879 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.729+0000 7efecbfff640 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0badbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.731+0000 7efecbfff640 1 -- 172.21.15.94:0/682450879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 msgr2=0x55dbe0bad6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.731+0000 7efed259c640 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0bad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.731+0000 7efecbfff640 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0bad6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.731+0000 7efecbfff640 1 -- 172.21.15.94:0/682450879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 msgr2=0x55dbe0bae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.731+0000 7efecbfff640 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.731+0000 7efecbfff640 1 -- 172.21.15.94:0/682450879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbe0c4af30 con 0x55dbe0baaea0 2024-04-21T08:41:43.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.737+0000 7efecbfff640 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0badbf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55dbe0bb03a0 tx=0x7efebc0047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9dfdbc8282f9b40 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.737+0000 7efeb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-21T08:41:43.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.740+0000 7efeb9ffb640 1 -- 172.21.15.94:0/682450879 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc02b060 con 0x55dbe0baaea0 2024-04-21T08:41:43.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.740+0000 7efedb7d0740 1 -- 172.21.15.94:0/682450879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efec40c6e70 con 0x55dbe0baaea0 2024-04-21T08:41:43.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.740+0000 7efeb9ffb640 1 -- 172.21.15.94:0/682450879 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7efebc025440 con 0x55dbe0baaea0 2024-04-21T08:41:43.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.744+0000 7efeb9ffb640 1 -- 172.21.15.94:0/682450879 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efebc017040 con 0x55dbe0baaea0 2024-04-21T08:41:43.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.744+0000 7efedb7d0740 10 client.25660 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:43.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.744+0000 7efedb7d0740 1 -- 172.21.15.94:0/682450879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dbe0c4c150 con 0x55dbe0baaea0 2024-04-21T08:41:43.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.745+0000 7efeb9ffb640 1 -- 172.21.15.94:0/682450879 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efec40c6e70 con 0x55dbe0baaea0 2024-04-21T08:41:43.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.745+0000 7efeb9ffb640 1 -- 172.21.15.94:0/682450879 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efebc006020 con 0x55dbe0baaea0 2024-04-21T08:41:43.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.746+0000 7efedb7d0740 10 client.25660 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:43.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.746+0000 7efedb7d0740 10 client.25660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:43.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.746+0000 7efedb7d0740 20 client.25660 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:43.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.746+0000 7efedb7d0740 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dbe0c4c850 0x55dbe0c4ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.746+0000 7efedb7d0740 4 client.25660 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:41:43.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.746+0000 7efedb7d0740 1 -- 172.21.15.94:0/682450879 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dbe0b16a70 con 0x55dbe0c4c850 2024-04-21T08:41:43.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.751+0000 7efed259c640 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dbe0c4c850 0x55dbe0c4ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.758+0000 7efed259c640 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dbe0c4c850 0x55dbe0c4ec70 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45719e4736d7635a server_cookie=e8e91cbd1d1ac6ad in_seq=0 out_seq=0 2024-04-21T08:41:43.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.758+0000 7efeb9ffb640 10 client.25660 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:43.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.760+0000 7efeb9ffb640 1 -- 172.21.15.94:0/682450879 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dbe0b16a70 con 0x55dbe0c4c850 2024-04-21T08:41:43.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.760+0000 7efeb9ffb640 10 client.25660 handle_command_reply: tid=0 2024-04-21T08:41:43.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.761+0000 7efedb7d0740 1 client.25660 shutdown 2024-04-21T08:41:43.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.763+0000 7efedb7d0740 1 -- 172.21.15.94:0/682450879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 msgr2=0x55dbe0badbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.763+0000 7efedb7d0740 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0badbf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55dbe0bb03a0 tx=0x7efebc0047d0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.767+0000 7efedb7d0740 1 -- 172.21.15.94:0/682450879 shutdown_connections 2024-04-21T08:41:43.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.767+0000 7efedb7d0740 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dbe0c4c850 0x55dbe0c4ec70 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.767+0000 7efedb7d0740 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dbe0baaea0 0x55dbe0badbf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.767+0000 7efedb7d0740 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dbe0bb3a80 0x55dbe0bae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.767+0000 7efedb7d0740 1 --2- 172.21.15.94:0/682450879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dbe0baa580 0x55dbe0bad6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.767+0000 7efedb7d0740 1 -- 172.21.15.94:0/682450879 >> 172.21.15.94:0/682450879 conn(0x55dbe0ba4cb0 msgr2=0x55dbe0ba3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.767+0000 7efeb9ffb640 0 client.25660 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:43.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.773+0000 7efedb7d0740 1 -- 172.21.15.94:0/682450879 shutdown_connections 2024-04-21T08:41:43.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.775+0000 7efedb7d0740 1 -- 172.21.15.94:0/682450879 wait complete. 2024-04-21T08:41:43.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.776+0000 7efedb7d0740 20 client.25660 trim_cache size 0 max 16384 2024-04-21T08:41:43.779 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:41:43.779 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:41:43.779 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:41:43.779 INFO:teuthology.orchestra.run.smithi094.stdout: "6b3dade7-6e21-4c85-83b4-1546e52e1d20": { 2024-04-21T08:41:43.779 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:41:43.779 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "6b3dade7-6e21-4c85-83b4-1546e52e1d20", 2024-04-21T08:41:43.779 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:41:43.779 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:41:43.779 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:41:43.779 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:41:43.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.783+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efec00a3d20 msgr2=0x7efec00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.783+0000 7efed8b0d640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efec00a3d20 0x7efec00a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efecc005fd0 tx=0x7efecc004a40 comp rx=0 tx=0).stop 2024-04-21T08:41:43.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.784+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efed41088e0 msgr2=0x7efed419ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.784+0000 7efed8b0d640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efed41088e0 0x7efed419ed40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efec4009230 tx=0x7efec4004040 comp rx=0 tx=0).stop 2024-04-21T08:41:43.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.786+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 shutdown_connections 2024-04-21T08:41:43.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.786+0000 7efed8b0d640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efec00a3d20 0x7efec00a61e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.786+0000 7efed8b0d640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efed41a3600 0x7efed41a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.786+0000 7efed8b0d640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efed410d900 0x7efed419f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.786+0000 7efed8b0d640 1 --2- 172.21.15.94:0/86771180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efed41088e0 0x7efed419ed40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.786+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 >> 172.21.15.94:0/86771180 conn(0x7efed4077dc0 msgr2=0x7efed410ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.788+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 shutdown_connections 2024-04-21T08:41:43.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:41:43.789+0000 7efed8b0d640 1 -- 172.21.15.94:0/86771180 wait complete. 2024-04-21T08:41:43.820 INFO:tasks.cephfs.filesystem:scrub status for tag:6b3dade7-6e21-4c85-83b4-1546e52e1d20 - {'path': '/', 'tag': '6b3dade7-6e21-4c85-83b4-1546e52e1d20', 'options': 'recursive,force'} 2024-04-21T08:41:44.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:43 smithi111 ceph-mon[31493]: pgmap v55: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 846 MiB/s wr, 455 op/s 2024-04-21T08:41:44.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:43 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:41:44.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:43 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:44.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:43 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:41:44.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:43 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:43 smithi094 ceph-mon[27988]: pgmap v55: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 846 MiB/s wr, 455 op/s 2024-04-21T08:41:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:43 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:41:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:43 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:43 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:41:44.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:43 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:44.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:43 smithi173 ceph-mon[31428]: pgmap v55: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 846 MiB/s wr, 455 op/s 2024-04-21T08:41:44.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:43 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:41:44.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:43 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:44.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:43 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:41:44.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:43 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:41:46.062 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:45 smithi173 ceph-mon[31428]: pgmap v56: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 921 MiB/s wr, 516 op/s 2024-04-21T08:41:46.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:45 smithi111 ceph-mon[31493]: pgmap v56: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 921 MiB/s wr, 516 op/s 2024-04-21T08:41:46.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:45 smithi094 ceph-mon[27988]: pgmap v56: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 921 MiB/s wr, 516 op/s 2024-04-21T08:41:47.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:46 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:41:47.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:46 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:41:47.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:46 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:41:48.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:47 smithi111 ceph-mon[31493]: pgmap v57: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 714 MiB/s wr, 280 op/s 2024-04-21T08:41:48.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:47 smithi094 ceph-mon[27988]: pgmap v57: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 714 MiB/s wr, 280 op/s 2024-04-21T08:41:48.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:47 smithi173 ceph-mon[31428]: pgmap v57: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 714 MiB/s wr, 280 op/s 2024-04-21T08:41:50.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:49 smithi111 ceph-mon[31493]: pgmap v58: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 563 MiB/s wr, 234 op/s 2024-04-21T08:41:50.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:49 smithi173 ceph-mon[31428]: pgmap v58: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 563 MiB/s wr, 234 op/s 2024-04-21T08:41:50.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:49 smithi094 ceph-mon[27988]: pgmap v58: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 563 MiB/s wr, 234 op/s 2024-04-21T08:41:52.061 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:51 smithi094 ceph-mon[27988]: pgmap v59: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 966 MiB/s wr, 431 op/s 2024-04-21T08:41:52.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:51 smithi111 ceph-mon[31493]: pgmap v59: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 966 MiB/s wr, 431 op/s 2024-04-21T08:41:52.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:51 smithi173 ceph-mon[31428]: pgmap v59: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 966 MiB/s wr, 431 op/s 2024-04-21T08:41:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.615+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/4036706833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f18101ce0 msgr2=0x7f6f18105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.615+0000 7f6f1d0fd640 1 --2- 172.21.15.94:0/4036706833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f18101ce0 0x7f6f18105af0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009900 tx=0x7f6f0c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.615+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/4036706833 shutdown_connections 2024-04-21T08:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.615+0000 7f6f1d0fd640 1 --2- 172.21.15.94:0/4036706833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f181061d0 0x7f6f1810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.615+0000 7f6f1d0fd640 1 --2- 172.21.15.94:0/4036706833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f18101ce0 0x7f6f18105af0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.615+0000 7f6f1d0fd640 1 --2- 172.21.15.94:0/4036706833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f18101330 0x7f6f18101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.615+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/4036706833 >> 172.21.15.94:0/4036706833 conn(0x7f6f18077d30 msgr2=0x7f6f18078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.615+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/4036706833 shutdown_connections 2024-04-21T08:41:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.615+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/4036706833 wait complete. 2024-04-21T08:41:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.618+0000 7f6f1d0fd640 1 Processor -- start 2024-04-21T08:41:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.618+0000 7f6f1d0fd640 1 -- start start 2024-04-21T08:41:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.618+0000 7f6f1d0fd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f18101330 0x7f6f1819d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.618+0000 7f6f1d0fd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f181061d0 0x7f6f18196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.618+0000 7f6f1d0fd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f18196b50 0x7f6f18197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.618+0000 7f6f1d0fd640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6f18110050 con 0x7f6f181061d0 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.618+0000 7f6f1d0fd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1810fed0 con 0x7f6f18196b50 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.618+0000 7f6f1d0fd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f181101d0 con 0x7f6f18101330 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.619+0000 7f6f1659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f181061d0 0x7f6f18196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.619+0000 7f6f1659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f181061d0 0x7f6f18196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49340/0 (socket says 172.21.15.94:49340) 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.619+0000 7f6f1659c640 1 -- 172.21.15.94:0/2434035070 learned_addr learned my addr 172.21.15.94:0/2434035070 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.619+0000 7f6f1659c640 1 -- 172.21.15.94:0/2434035070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f18101330 msgr2=0x7f6f1819d540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.619+0000 7f6f1659c640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f18101330 0x7f6f1819d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.619+0000 7f6f1659c640 1 -- 172.21.15.94:0/2434035070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f18196b50 msgr2=0x7f6f18197000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.619+0000 7f6f1659c640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f18196b50 0x7f6f18197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.619+0000 7f6f1659c640 1 -- 172.21.15.94:0/2434035070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1819b190 con 0x7f6f181061d0 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.620+0000 7f6f1659c640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f181061d0 0x7f6f18196610 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c000c00 tx=0x7f6f0c02dce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.620+0000 7f6efffff640 1 -- 172.21.15.94:0/2434035070 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c054020 con 0x7f6f181061d0 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.620+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/2434035070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1819b420 con 0x7f6f181061d0 2024-04-21T08:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.620+0000 7f6efffff640 1 -- 172.21.15.94:0/2434035070 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f0c004250 con 0x7f6f181061d0 2024-04-21T08:41:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.620+0000 7f6efffff640 1 -- 172.21.15.94:0/2434035070 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c0038e0 con 0x7f6f181061d0 2024-04-21T08:41:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.620+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/2434035070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1819b970 con 0x7f6f181061d0 2024-04-21T08:41:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.624+0000 7f6f1fdc8740 1 Processor -- start 2024-04-21T08:41:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.624+0000 7f6f1fdc8740 1 -- start start 2024-04-21T08:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e8259ff620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e8259fffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 0x55e825a0b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1fdc8740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55e8259648d0 con 0x55e825a008d0 2024-04-21T08:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1fdc8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e825901e80 con 0x55e825a090e0 2024-04-21T08:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1fdc8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e825960890 con 0x55e8259ffb60 2024-04-21T08:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e8259ff620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e8259ff620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49348/0 (socket says 172.21.15.94:49348) 2024-04-21T08:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1759e640 1 -- 172.21.15.94:0/3821768327 learned_addr learned my addr 172.21.15.94:0/3821768327 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1759e640 1 -- 172.21.15.94:0/3821768327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 msgr2=0x55e8259fffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.625+0000 7f6f1759e640 1 --2- 172.21.15.94:0/3821768327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e8259fffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.626+0000 7f6f1759e640 1 -- 172.21.15.94:0/3821768327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 msgr2=0x55e825a0b4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:41:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.626+0000 7f6f1759e640 1 --2- 172.21.15.94:0/3821768327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 0x55e825a0b4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.626+0000 7f6f1759e640 1 -- 172.21.15.94:0/3821768327 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e825923320 con 0x55e825a008d0 2024-04-21T08:41:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.626+0000 7f6efffff640 1 -- 172.21.15.94:0/2434035070 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f6f0c057020 con 0x7f6f181061d0 2024-04-21T08:41:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.626+0000 7f6f1759e640 1 --2- 172.21.15.94:0/3821768327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e8259ff620 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009790 tx=0x7f6f10009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db42a0bdf9872251 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.626+0000 7f6efd7fa640 1 -- 172.21.15.94:0/3821768327 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f10015020 con 0x55e825a008d0 2024-04-21T08:41:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.626+0000 7f6efd7fa640 1 -- 172.21.15.94:0/3821768327 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f10011420 con 0x55e825a008d0 2024-04-21T08:41:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.626+0000 7f6efd7fa640 1 -- 172.21.15.94:0/3821768327 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f100105e0 con 0x55e825a008d0 2024-04-21T08:41:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.627+0000 7f6efffff640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f000a3e70 0x7f6f000a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.627+0000 7f6f16d9d640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f000a3e70 0x7f6f000a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.627+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/3821768327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 msgr2=0x55e8259ff620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.627+0000 7f6efffff640 1 -- 172.21.15.94:0/2434035070 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f6f0c103860 con 0x7f6f181061d0 2024-04-21T08:41:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.627+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/3821768327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e8259ff620 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009790 tx=0x7f6f10009c50 comp rx=0 tx=0).stop 2024-04-21T08:41:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.628+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/3821768327 shutdown_connections 2024-04-21T08:41:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.628+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/3821768327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e8259fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.628+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/3821768327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 0x55e825a0b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.628+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/3821768327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e8259ff620 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.628+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/3821768327 >> 172.21.15.94:0/3821768327 conn(0x55e8259c6c90 msgr2=0x55e8259fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.628+0000 7f6f16d9d640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f000a3e70 0x7f6f000a6330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55e825a01030 tx=0x7f6f080093c0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.628+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/3821768327 shutdown_connections 2024-04-21T08:41:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.629+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/3821768327 wait complete. 2024-04-21T08:41:52.629 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.629+0000 7f6f1fdc8740 1 -- start start 2024-04-21T08:41:52.629 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.629+0000 7f6f1fdc8740 1 Processor -- start 2024-04-21T08:41:52.631 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.632+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e8259ffb60 0x55e825a08ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.631 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.632+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e825a008d0 0x55e825a05ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.631 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.632+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 0x55e825a06540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.631 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.632+0000 7f6f16d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e8259ffb60 0x55e825a08ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.632+0000 7f6f16d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e8259ffb60 0x55e825a08ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49354/0 (socket says 172.21.15.94:49354) 2024-04-21T08:41:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.632+0000 7f6f16d9d640 1 -- 172.21.15.94:0/118512785 learned_addr learned my addr 172.21.15.94:0/118512785 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.632+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55e825a0be10 con 0x55e8259ffb60 2024-04-21T08:41:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.632+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e8258eceb0 con 0x55e825a090e0 2024-04-21T08:41:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.632+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e825a0bf90 con 0x55e825a008d0 2024-04-21T08:41:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.633+0000 7f6f1fdc8740 10 client.0 _ll_register_callbacks cb 0x55e825919b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.633+0000 7f6f16d9d640 1 -- 172.21.15.94:0/118512785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e825a008d0 msgr2=0x55e825a05ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:41:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.633+0000 7f6f16d9d640 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e825a008d0 0x55e825a05ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.633+0000 7f6f16d9d640 1 -- 172.21.15.94:0/118512785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 msgr2=0x55e825a06540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.633+0000 7f6f16d9d640 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 0x55e825a06540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.633+0000 7f6f16d9d640 1 -- 172.21.15.94:0/118512785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e825677210 con 0x55e8259ffb60 2024-04-21T08:41:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.633+0000 7f6f16d9d640 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e8259ffb60 0x55e825a08ac0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6f080117f0 tx=0x7f6f08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e215a92c71624b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.634+0000 7f6efaff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.634+0000 7f6efaff5640 1 -- 172.21.15.94:0/118512785 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f08011e30 con 0x55e8259ffb60 2024-04-21T08:41:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.634+0000 7f6efaff5640 1 -- 172.21.15.94:0/118512785 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f6f080108d0 con 0x55e8259ffb60 2024-04-21T08:41:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.634+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f10011420 con 0x55e8259ffb60 2024-04-21T08:41:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.634+0000 7f6efaff5640 1 -- 172.21.15.94:0/118512785 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6f08010ad0 con 0x55e8259ffb60 2024-04-21T08:41:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.634+0000 7f6f1fdc8740 10 client.16383 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.634+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e825743620 con 0x55e8259ffb60 2024-04-21T08:41:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.635+0000 7f6efaff5640 1 -- 172.21.15.94:0/118512785 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f10011420 con 0x55e8259ffb60 2024-04-21T08:41:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.636+0000 7f6efaff5640 1 -- 172.21.15.94:0/118512785 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6f0800f040 con 0x55e8259ffb60 2024-04-21T08:41:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.636+0000 7f6f1fdc8740 10 client.16383 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.636+0000 7f6f1fdc8740 10 client.16383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.636+0000 7f6f1fdc8740 20 client.16383 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.636+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e825a19180 0x55e825a1b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.636+0000 7f6f1fdc8740 4 client.16383 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:41:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.636+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e825978880 con 0x55e825a19180 2024-04-21T08:41:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.638+0000 7f6f1659c640 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e825a19180 0x55e825a1b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.639+0000 7f6efaff5640 10 client.16383 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.639+0000 7f6f1659c640 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e825a19180 0x55e825a1b5b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42b8534b9be9015c server_cookie=bbf29c377a145c85 in_seq=0 out_seq=0 2024-04-21T08:41:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6efaff5640 1 -- 172.21.15.94:0/118512785 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e825978880 con 0x55e825a19180 2024-04-21T08:41:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6efaff5640 10 client.16383 handle_command_reply: tid=0 2024-04-21T08:41:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 client.16383 shutdown 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e8259ffb60 msgr2=0x55e825a08ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e8259ffb60 0x55e825a08ac0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6f080117f0 tx=0x7f6f08004a40 comp rx=0 tx=0).stop 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 shutdown_connections 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e825a19180 0x55e825a1b5b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e825a008d0 0x55e825a05ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 0x55e825a06540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/118512785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e8259ffb60 0x55e825a08ac0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6efaff5640 0 client.16383 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 >> 172.21.15.94:0/118512785 conn(0x55e8259c6c90 msgr2=0x55e8259fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 shutdown_connections 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.641+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/118512785 wait complete. 2024-04-21T08:41:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.642+0000 7f6f1fdc8740 20 client.16383 trim_cache size 0 max 16384 2024-04-21T08:41:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.660+0000 7f6f1fdc8740 1 Processor -- start 2024-04-21T08:41:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.660+0000 7f6f1fdc8740 1 -- start start 2024-04-21T08:41:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.660+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e825a057a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.660+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a008d0 0x55e8259fc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.660+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a090e0 0x55e825a085f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.660+0000 7f6f1fdc8740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6f181b8990 con 0x55e825a090e0 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.660+0000 7f6f1fdc8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6f181b6260 con 0x55e825a008d0 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.660+0000 7f6f1fdc8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f180734c0 con 0x55e8259ffb60 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.660+0000 7f6f1759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a090e0 0x55e825a085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6f1759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a090e0 0x55e825a085f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49358/0 (socket says 172.21.15.94:49358) 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6f1759e640 1 -- 172.21.15.94:0/727031412 learned_addr learned my addr 172.21.15.94:0/727031412 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6f16d9d640 1 --2- 172.21.15.94:0/727031412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e825a057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6f1759e640 1 -- 172.21.15.94:0/727031412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 msgr2=0x55e825a057a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6f1759e640 1 --2- 172.21.15.94:0/727031412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e825a057a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6f1759e640 1 -- 172.21.15.94:0/727031412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a008d0 msgr2=0x55e8259fc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6f1759e640 1 --2- 172.21.15.94:0/727031412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a008d0 0x55e8259fc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6f1759e640 1 -- 172.21.15.94:0/727031412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8259faa60 con 0x55e825a090e0 2024-04-21T08:41:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6f16d9d640 1 --2- 172.21.15.94:0/727031412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e825a057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:41:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6f1759e640 1 --2- 172.21.15.94:0/727031412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a090e0 0x55e825a085f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009440 tx=0x7f6f10009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6a4cd6bc0db0f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6efcff9640 1 -- 172.21.15.94:0/727031412 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f10015070 con 0x55e825a090e0 2024-04-21T08:41:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6efcff9640 1 -- 172.21.15.94:0/727031412 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f10011420 con 0x55e825a090e0 2024-04-21T08:41:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.661+0000 7f6efcff9640 1 -- 172.21.15.94:0/727031412 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1001a680 con 0x55e825a090e0 2024-04-21T08:41:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.662+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/727031412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a090e0 msgr2=0x55e825a085f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.662+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/727031412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a090e0 0x55e825a085f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009440 tx=0x7f6f10009470 comp rx=0 tx=0).stop 2024-04-21T08:41:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.662+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/727031412 shutdown_connections 2024-04-21T08:41:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.662+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/727031412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a090e0 0x55e825a085f0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.662+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/727031412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a008d0 0x55e8259fc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.662+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/727031412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e825a057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.662+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/727031412 >> 172.21.15.94:0/727031412 conn(0x55e8259c6c90 msgr2=0x55e8259fbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.662+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/727031412 shutdown_connections 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.662+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/727031412 wait complete. 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1fdc8740 1 -- start start 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1fdc8740 1 Processor -- start 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e825a0c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e825a0c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1fdc8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 0x55e825a7d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1fdc8740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6f18073b30 con 0x55e825a008d0 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1fdc8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6f181b5b90 con 0x55e825a090e0 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1fdc8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f181b4df0 con 0x55e8259ffb60 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1fdc8740 10 client.0 _ll_register_callbacks cb 0x7f6f080094f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:41:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e825a0c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e825a0c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49364/0 (socket says 172.21.15.94:49364) 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f16d9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e825a0c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1659c640 1 -- 172.21.15.94:0/1619453348 learned_addr learned my addr 172.21.15.94:0/1619453348 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1659c640 1 -- 172.21.15.94:0/1619453348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 msgr2=0x55e825a0c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1659c640 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e825a0c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1659c640 1 -- 172.21.15.94:0/1619453348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 msgr2=0x55e825a7d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1659c640 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 0x55e825a7d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.663+0000 7f6f1659c640 1 -- 172.21.15.94:0/1619453348 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e825a7dc30 con 0x55e825a008d0 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6f1659c640 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e825a0c910 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0cbea0 tx=0x7f6f0c0ceea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535df870af4cbe21 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6efaff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6efaff5640 1 -- 172.21.15.94:0/1619453348 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c0cf4d0 con 0x55e825a008d0 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6efaff5640 1 -- 172.21.15.94:0/1619453348 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f6f0c04ee70 con 0x55e825a008d0 2024-04-21T08:41:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/1619453348 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f10011420 con 0x55e825a008d0 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6efaff5640 1 -- 172.21.15.94:0/1619453348 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6f0c0d3880 con 0x55e825a008d0 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6f1fdc8740 10 client.16389 fetch_fsmap learned FSMap version 24 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/1619453348 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e825a7eee0 con 0x55e825a008d0 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6efaff5640 1 -- 172.21.15.94:0/1619453348 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f10011420 con 0x55e825a008d0 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6efaff5640 1 -- 172.21.15.94:0/1619453348 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6f0c0b4ae0 con 0x55e825a008d0 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6f1fdc8740 10 client.16389 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6f1fdc8740 10 client.16389 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6f1fdc8740 20 client.16389 populate_metadata read hostname 'smithi094' 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e825a7f5f0 0x55e825a81a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6f1fdc8740 4 client.16389 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.664+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/1619453348 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e8259043d0 con 0x55e825a7f5f0 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.665+0000 7f6f16d9d640 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e825a7f5f0 0x55e825a81a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.665+0000 7f6efaff5640 10 client.16389 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.665+0000 7f6f16d9d640 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e825a7f5f0 0x55e825a81a10 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9e734798cb32a18 server_cookie=5e2a9042802121e9 in_seq=0 out_seq=0 2024-04-21T08:41:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.680+0000 7f6efaff5640 1 -- 172.21.15.94:0/1619453348 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e8259043d0 con 0x55e825a7f5f0 2024-04-21T08:41:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.680+0000 7f6efaff5640 10 client.16389 handle_command_reply: tid=0 2024-04-21T08:41:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.680+0000 7f6f1fdc8740 1 client.16389 shutdown 2024-04-21T08:41:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.681+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/1619453348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 msgr2=0x55e825a0c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.681+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e825a0c910 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0cbea0 tx=0x7f6f0c0ceea0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.681+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/1619453348 shutdown_connections 2024-04-21T08:41:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.681+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55e825a7f5f0 0x55e825a81a10 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.681+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e825a090e0 0x55e825a7d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.681+0000 7f6efaff5640 0 client.16389 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:41:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.681+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e825a008d0 0x55e825a0c910 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.681+0000 7f6f1fdc8740 1 --2- 172.21.15.94:0/1619453348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e8259ffb60 0x55e825a0c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.681+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/1619453348 >> 172.21.15.94:0/1619453348 conn(0x55e8259c6c90 msgr2=0x55e8259fc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.682+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/1619453348 shutdown_connections 2024-04-21T08:41:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.682+0000 7f6f1fdc8740 1 -- 172.21.15.94:0/1619453348 wait complete. 2024-04-21T08:41:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.682+0000 7f6f1fdc8740 20 client.16389 trim_cache size 0 max 16384 2024-04-21T08:41:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:{ 2024-04-21T08:41:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "message": "", 2024-04-21T08:41:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "return_code": 0 2024-04-21T08:41:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:} 2024-04-21T08:41:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.682+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/2434035070 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f000a3e70 msgr2=0x7f6f000a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.682+0000 7f6f1d0fd640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f000a3e70 0x7f6f000a6330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55e825a01030 tx=0x7f6f080093c0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.682+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/2434035070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f181061d0 msgr2=0x7f6f18196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.682+0000 7f6f1d0fd640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f181061d0 0x7f6f18196610 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c000c00 tx=0x7f6f0c02dce0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.683+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/2434035070 shutdown_connections 2024-04-21T08:41:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.683+0000 7f6f1d0fd640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f000a3e70 0x7f6f000a6330 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.683+0000 7f6f1d0fd640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f18196b50 0x7f6f18197000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.683+0000 7f6f1d0fd640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f181061d0 0x7f6f18196610 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.683+0000 7f6f1d0fd640 1 --2- 172.21.15.94:0/2434035070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f18101330 0x7f6f1819d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.683+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/2434035070 >> 172.21.15.94:0/2434035070 conn(0x7f6f18077d30 msgr2=0x7f6f180ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.683+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/2434035070 shutdown_connections 2024-04-21T08:41:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:41:52.683+0000 7f6f1d0fd640 1 -- 172.21.15.94:0/2434035070 wait complete. 2024-04-21T08:41:52.702 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:52.702 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 rewriters = 995879.69 kB/sec 2024-04-21T08:41:52.702 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 rewriters = 995856.56 kB/sec 2024-04-21T08:41:52.702 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 995879.69 kB/sec 2024-04-21T08:41:52.702 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 995879.69 kB/sec 2024-04-21T08:41:52.702 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 995879.69 kB/sec 2024-04-21T08:41:52.703 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 10485760.00 kB 2024-04-21T08:41:54.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:53 smithi111 ceph-mon[31493]: pgmap v60: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 521 MiB/s wr, 284 op/s 2024-04-21T08:41:54.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:53 smithi094 ceph-mon[27988]: pgmap v60: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 521 MiB/s wr, 284 op/s 2024-04-21T08:41:54.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:53 smithi173 ceph-mon[31428]: pgmap v60: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 521 MiB/s wr, 284 op/s 2024-04-21T08:41:56.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:55 smithi111 ceph-mon[31493]: pgmap v61: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 748 MiB/s wr, 345 op/s 2024-04-21T08:41:56.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:55 smithi094 ceph-mon[27988]: pgmap v61: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 748 MiB/s wr, 345 op/s 2024-04-21T08:41:56.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:55 smithi173 ceph-mon[31428]: pgmap v61: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 748 MiB/s wr, 345 op/s 2024-04-21T08:41:58.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:57 smithi111 ceph-mon[31493]: pgmap v62: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 786 MiB/s wr, 314 op/s 2024-04-21T08:41:58.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:57 smithi094 ceph-mon[27988]: pgmap v62: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 786 MiB/s wr, 314 op/s 2024-04-21T08:41:58.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:57 smithi173 ceph-mon[31428]: pgmap v62: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 786 MiB/s wr, 314 op/s 2024-04-21T08:41:59.109 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:41:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 readers = 7900852.00 kB/sec 2024-04-21T08:41:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 readers = 7899585.20 kB/sec 2024-04-21T08:41:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 7900852.00 kB/sec 2024-04-21T08:41:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 7900852.00 kB/sec 2024-04-21T08:41:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 7900852.00 kB/sec 2024-04-21T08:41:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 10485760.00 kB 2024-04-21T08:42:00.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:41:59 smithi111 ceph-mon[31493]: pgmap v63: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 742 MiB/s wr, 286 op/s 2024-04-21T08:42:00.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:41:59 smithi094 ceph-mon[27988]: pgmap v63: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 742 MiB/s wr, 286 op/s 2024-04-21T08:42:00.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:41:59 smithi173 ceph-mon[31428]: pgmap v63: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 742 MiB/s wr, 286 op/s 2024-04-21T08:42:02.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:01 smithi111 ceph-mon[31493]: pgmap v64: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 743 MiB/s wr, 289 op/s 2024-04-21T08:42:02.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:01 smithi094 ceph-mon[27988]: pgmap v64: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 743 MiB/s wr, 289 op/s 2024-04-21T08:42:02.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:01 smithi173 ceph-mon[31428]: pgmap v64: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 743 MiB/s wr, 289 op/s 2024-04-21T08:42:02.530 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:42:02.531 INFO:tasks.workunit.client.0.smithi094.stdout: Children see throughput for 1 re-readers = 7881380.00 kB/sec 2024-04-21T08:42:02.531 INFO:tasks.workunit.client.0.smithi094.stdout: Parent sees throughput for 1 re-readers = 7880054.35 kB/sec 2024-04-21T08:42:02.531 INFO:tasks.workunit.client.0.smithi094.stdout: Min throughput per process = 7881380.00 kB/sec 2024-04-21T08:42:02.531 INFO:tasks.workunit.client.0.smithi094.stdout: Max throughput per process = 7881380.00 kB/sec 2024-04-21T08:42:02.531 INFO:tasks.workunit.client.0.smithi094.stdout: Avg throughput per process = 7881380.00 kB/sec 2024-04-21T08:42:02.531 INFO:tasks.workunit.client.0.smithi094.stdout: Min xfer = 10485760.00 kB 2024-04-21T08:42:04.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:03 smithi111 ceph-mon[31493]: pgmap v65: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 340 MiB/s wr, 92 op/s 2024-04-21T08:42:04.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:03 smithi094 ceph-mon[27988]: pgmap v65: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 340 MiB/s wr, 92 op/s 2024-04-21T08:42:04.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:03 smithi173 ceph-mon[31428]: pgmap v65: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 340 MiB/s wr, 92 op/s 2024-04-21T08:42:05.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:04 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:05.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:04 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:05.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:04 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:05.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:04 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:42:05.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:04 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:05.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:04 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:05.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:04 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:05.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:04 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:42:05.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:04 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:05.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:04 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:05.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:04 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:05.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:04 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:42:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 -- 172.21.15.94:0/1741785312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f0100fb0 msgr2=0x7ff8f0110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 --2- 172.21.15.94:0/1741785312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f0100fb0 0x7ff8f0110be0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009900 tx=0x7ff8e802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 -- 172.21.15.94:0/1741785312 shutdown_connections 2024-04-21T08:42:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 --2- 172.21.15.94:0/1741785312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f0103250 0x7ff8f0100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 --2- 172.21.15.94:0/1741785312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f0100fb0 0x7ff8f0110be0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 --2- 172.21.15.94:0/1741785312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f0102e70 0x7ff8f0100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 -- 172.21.15.94:0/1741785312 >> 172.21.15.94:0/1741785312 conn(0x7ff8f0077390 msgr2=0x7ff8f00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 -- 172.21.15.94:0/1741785312 shutdown_connections 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 -- 172.21.15.94:0/1741785312 wait complete. 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 Processor -- start 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.780+0000 7ff8f8414640 1 -- start start 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f8414640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f0100fb0 0x7ff8f019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f8414640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f0102e70 0x7ff8f019acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f8414640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f0103250 0x7ff8f019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f8414640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f0114390 con 0x7ff8f0102e70 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f8414640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f0114210 con 0x7ff8f0103250 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f8414640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f0114510 con 0x7ff8f0100fb0 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f59af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f0102e70 0x7ff8f019acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f59af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f0102e70 0x7ff8f019acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39490/0 (socket says 172.21.15.94:39490) 2024-04-21T08:42:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f59af640 1 -- 172.21.15.94:0/766921894 learned_addr learned my addr 172.21.15.94:0/766921894 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f69b1640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f0103250 0x7ff8f019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f0100fb0 0x7ff8f019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f59af640 1 -- 172.21.15.94:0/766921894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f0100fb0 msgr2=0x7ff8f019a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f59af640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f0100fb0 0x7ff8f019a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f59af640 1 -- 172.21.15.94:0/766921894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f0103250 msgr2=0x7ff8f019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f59af640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f0103250 0x7ff8f019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f59af640 1 -- 172.21.15.94:0/766921894 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f019f730 con 0x7ff8f0102e70 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8f59af640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f0102e70 0x7ff8f019acd0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000c920 tx=0x7ff8e000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.781+0000 7ff8e77fe640 1 -- 172.21.15.94:0/766921894 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0007c70 con 0x7ff8f0102e70 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.782+0000 7ff8e77fe640 1 -- 172.21.15.94:0/766921894 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e0007e10 con 0x7ff8f0102e70 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.782+0000 7ff8f8414640 1 -- 172.21.15.94:0/766921894 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f019fa20 con 0x7ff8f0102e70 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.782+0000 7ff8e77fe640 1 -- 172.21.15.94:0/766921894 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0018820 con 0x7ff8f0102e70 2024-04-21T08:42:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.782+0000 7ff8f8414640 1 -- 172.21.15.94:0/766921894 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f01a72b0 con 0x7ff8f0102e70 2024-04-21T08:42:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.784+0000 7ff8e77fe640 1 -- 172.21.15.94:0/766921894 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff8e000f450 con 0x7ff8f0102e70 2024-04-21T08:42:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.784+0000 7ff8fb008740 1 Processor -- start 2024-04-21T08:42:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.784+0000 7ff8fb008740 1 -- start start 2024-04-21T08:42:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c08d0 0x55f9694bf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694bfb60 0x55f9694bffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 0x55f9694cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8fb008740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55f9694248d0 con 0x55f9694bfb60 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8e77fe640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8dc0a3e40 0x7ff8dc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8fb008740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f9693c1e80 con 0x55f9694c08d0 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8fb008740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f969420890 con 0x55f9694c90e0 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f61b0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694bfb60 0x55f9694bffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f69b1640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8dc0a3e40 0x7ff8dc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f61b0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694bfb60 0x55f9694bffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39496/0 (socket says 172.21.15.94:39496) 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f61b0640 1 -- 172.21.15.94:0/619890040 learned_addr learned my addr 172.21.15.94:0/619890040 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8e77fe640 1 -- 172.21.15.94:0/766921894 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff8e00d0c30 con 0x7ff8f0102e70 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f69b1640 1 --2- 172.21.15.94:0/619890040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c08d0 0x55f9694bf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f59af640 1 --2- 172.21.15.94:0/619890040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 0x55f9694cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f61b0640 1 -- 172.21.15.94:0/619890040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 msgr2=0x55f9694cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/619890040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 0x55f9694cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f61b0640 1 -- 172.21.15.94:0/619890040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c08d0 msgr2=0x55f9694bf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/619890040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c08d0 0x55f9694bf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f61b0640 1 -- 172.21.15.94:0/619890040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9693e3320 con 0x55f9694bfb60 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f69b1640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8dc0a3e40 0x7ff8dc0a6300 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8005ab0 tx=0x7ff8e8005fd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.785+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/619890040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694bfb60 0x55f9694bffe0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00ae90 tx=0x7ff8ec00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d30ebd658d54be0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.786+0000 7ff8e4bfe640 1 -- 172.21.15.94:0/619890040 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec011850 con 0x55f9694bfb60 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.786+0000 7ff8e4bfe640 1 -- 172.21.15.94:0/619890040 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8ec01a480 con 0x55f9694bfb60 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.786+0000 7ff8e4bfe640 1 -- 172.21.15.94:0/619890040 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec010610 con 0x55f9694bfb60 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.786+0000 7ff8fb008740 1 -- 172.21.15.94:0/619890040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694bfb60 msgr2=0x55f9694bffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.786+0000 7ff8fb008740 1 --2- 172.21.15.94:0/619890040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694bfb60 0x55f9694bffe0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00ae90 tx=0x7ff8ec00f730 comp rx=0 tx=0).stop 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.786+0000 7ff8fb008740 1 -- 172.21.15.94:0/619890040 shutdown_connections 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.786+0000 7ff8fb008740 1 --2- 172.21.15.94:0/619890040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 0x55f9694cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 --2- 172.21.15.94:0/619890040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694bfb60 0x55f9694bffe0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 --2- 172.21.15.94:0/619890040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c08d0 0x55f9694bf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 -- 172.21.15.94:0/619890040 >> 172.21.15.94:0/619890040 conn(0x55f969486c90 msgr2=0x55f9694ba850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 -- 172.21.15.94:0/619890040 shutdown_connections 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 -- 172.21.15.94:0/619890040 wait complete. 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 -- start start 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 Processor -- start 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694bfb60 0x55f9694c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 0x55f9694c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55f9694cbe10 con 0x55f9694c08d0 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f9693aceb0 con 0x55f9694bfb60 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f9694cbf90 con 0x55f9694c90e0 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8f59af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8fb008740 10 client.0 _ll_register_callbacks cb 0x55f9693d9b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8f59af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694c5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39512/0 (socket says 172.21.15.94:39512) 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8f59af640 1 -- 172.21.15.94:0/1383165693 learned_addr learned my addr 172.21.15.94:0/1383165693 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.787+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694bfb60 0x55f9694c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8f69b1640 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 0x55f9694c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8f59af640 1 -- 172.21.15.94:0/1383165693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 msgr2=0x55f9694c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8f59af640 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 0x55f9694c63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8f59af640 1 -- 172.21.15.94:0/1383165693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694bfb60 msgr2=0x55f9694c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8f59af640 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694bfb60 0x55f9694c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8f59af640 1 -- 172.21.15.94:0/1383165693 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f969137210 con 0x55f9694c08d0 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8f69b1640 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 0x55f9694c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8f59af640 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694c5e20 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00ce5a0 tx=0x7ff8e009c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02d89749a830aab server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1383165693 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e009c7f0 con 0x55f9694c08d0 2024-04-21T08:42:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8fb008740 1 -- 172.21.15.94:0/1383165693 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8ec01a480 con 0x55f9694c08d0 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1383165693 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff8e00ce220 con 0x55f9694c08d0 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1383165693 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8e00a0ba0 con 0x55f9694c08d0 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8fb008740 10 client.16401 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8fb008740 1 -- 172.21.15.94:0/1383165693 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f969203620 con 0x55f9694c08d0 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1383165693 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8ec01a480 con 0x55f9694c08d0 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1383165693 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff8e009bc40 con 0x55f9694c08d0 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.788+0000 7ff8fb008740 10 client.16401 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.789+0000 7ff8fb008740 10 client.16401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.789+0000 7ff8fb008740 20 client.16401 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.789+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f9694d9010 0x55f9694db400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.789+0000 7ff8fb008740 4 client.16401 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.789+0000 7ff8fb008740 1 -- 172.21.15.94:0/1383165693 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f969438880 con 0x55f9694d9010 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.789+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f9694d9010 0x55f9694db400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.789+0000 7ff8d9ffb640 10 client.16401 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.789+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f9694d9010 0x55f9694db400 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=351629852fa80ad8 server_cookie=8005ea350891212f in_seq=0 out_seq=0 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1383165693 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f969438880 con 0x55f9694d9010 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8d9ffb640 10 client.16401 handle_command_reply: tid=0 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 client.16401 shutdown 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 -- 172.21.15.94:0/1383165693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 msgr2=0x55f9694c5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694c5e20 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00ce5a0 tx=0x7ff8e009c330 comp rx=0 tx=0).stop 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 -- 172.21.15.94:0/1383165693 shutdown_connections 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f9694d9010 0x55f9694db400 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694c90e0 0x55f9694c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8d9ffb640 0 client.16401 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694c5e20 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1383165693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694bfb60 0x55f9694c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 -- 172.21.15.94:0/1383165693 >> 172.21.15.94:0/1383165693 conn(0x55f969486c90 msgr2=0x55f9694bc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 -- 172.21.15.94:0/1383165693 shutdown_connections 2024-04-21T08:42:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 1 -- 172.21.15.94:0/1383165693 wait complete. 2024-04-21T08:42:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.790+0000 7ff8fb008740 20 client.16401 trim_cache size 0 max 16384 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8fb008740 1 Processor -- start 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8fb008740 1 -- start start 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 0x55f9694d4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694ba380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c90e0 0x55f9694c8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8fb008740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01b3c80 con 0x55f9694c08d0 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8fb008740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01b1550 con 0x55f9694c90e0 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8fb008740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01af3a0 con 0x55f9694bfb60 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f59af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694ba380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f59af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694ba380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39524/0 (socket says 172.21.15.94:39524) 2024-04-21T08:42:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f59af640 1 -- 172.21.15.94:0/3333599425 learned_addr learned my addr 172.21.15.94:0/3333599425 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/3333599425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 0x55f9694d4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f69b1640 1 --2- 172.21.15.94:0/3333599425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c90e0 0x55f9694c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f59af640 1 -- 172.21.15.94:0/3333599425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 msgr2=0x55f9694d4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f59af640 1 --2- 172.21.15.94:0/3333599425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 0x55f9694d4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f59af640 1 -- 172.21.15.94:0/3333599425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c90e0 msgr2=0x55f9694c8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f59af640 1 --2- 172.21.15.94:0/3333599425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c90e0 0x55f9694c8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f59af640 1 -- 172.21.15.94:0/3333599425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9694bee30 con 0x55f9694c08d0 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.805+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/3333599425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 0x55f9694d4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8f59af640 1 --2- 172.21.15.94:0/3333599425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694ba380 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00ce050 tx=0x7ff8e009c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34898d1f92faa2cb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8dbfff640 1 -- 172.21.15.94:0/3333599425 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e009c7f0 con 0x55f9694c08d0 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8dbfff640 1 -- 172.21.15.94:0/3333599425 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e00a5460 con 0x55f9694c08d0 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8dbfff640 1 -- 172.21.15.94:0/3333599425 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e00a4fd0 con 0x55f9694c08d0 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8fb008740 1 -- 172.21.15.94:0/3333599425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 msgr2=0x55f9694ba380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8fb008740 1 --2- 172.21.15.94:0/3333599425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694ba380 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00ce050 tx=0x7ff8e009c460 comp rx=0 tx=0).stop 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8fb008740 1 -- 172.21.15.94:0/3333599425 shutdown_connections 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8fb008740 1 --2- 172.21.15.94:0/3333599425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c90e0 0x55f9694c8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8fb008740 1 --2- 172.21.15.94:0/3333599425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c08d0 0x55f9694ba380 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8fb008740 1 --2- 172.21.15.94:0/3333599425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 0x55f9694d4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8fb008740 1 -- 172.21.15.94:0/3333599425 >> 172.21.15.94:0/3333599425 conn(0x55f969486c90 msgr2=0x55f9694c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8fb008740 1 -- 172.21.15.94:0/3333599425 shutdown_connections 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.806+0000 7ff8fb008740 1 -- 172.21.15.94:0/3333599425 wait complete. 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8fb008740 1 -- start start 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8fb008740 1 Processor -- start 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 0x55f9694c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c08d0 0x55f9694ca520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8fb008740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c90e0 0x55f9694cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8fb008740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01afa10 con 0x55f9694c90e0 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8fb008740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01b0e80 con 0x55f9694c08d0 2024-04-21T08:42:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8fb008740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01b00e0 con 0x55f9694bfb60 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8fb008740 10 client.0 _ll_register_callbacks cb 0x7ff8e000cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f69b1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c90e0 0x55f9694cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f69b1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c90e0 0x55f9694cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39526/0 (socket says 172.21.15.94:39526) 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f69b1640 1 -- 172.21.15.94:0/1087952485 learned_addr learned my addr 172.21.15.94:0/1087952485 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f59af640 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c08d0 0x55f9694ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 0x55f9694c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f69b1640 1 -- 172.21.15.94:0/1087952485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 msgr2=0x55f9694c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f69b1640 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 0x55f9694c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f69b1640 1 -- 172.21.15.94:0/1087952485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c08d0 msgr2=0x55f9694ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f69b1640 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c08d0 0x55f9694ca520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f69b1640 1 -- 172.21.15.94:0/1087952485 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9694cfc90 con 0x55f9694c90e0 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 0x55f9694c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8f69b1640 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c90e0 0x55f9694cf750 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009900 tx=0x7ff8e8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2223e3fc3c42689 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.807+0000 7ff8d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1087952485 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8004990 con 0x55f9694c90e0 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1087952485 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff8e8004b30 con 0x55f9694c90e0 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8fb008740 1 -- 172.21.15.94:0/1087952485 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8e00a5460 con 0x55f9694c90e0 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1087952485 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8e8004d30 con 0x55f9694c90e0 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8fb008740 10 client.16407 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8fb008740 1 -- 172.21.15.94:0/1087952485 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f96953fc10 con 0x55f9694c90e0 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1087952485 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e00a5460 con 0x55f9694c90e0 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1087952485 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff8e80504e0 con 0x55f9694c90e0 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8fb008740 10 client.16407 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8fb008740 10 client.16407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8fb008740 20 client.16407 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f9695403e0 0x55f969542800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8fb008740 4 client.16407 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8fb008740 1 -- 172.21.15.94:0/1087952485 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f9693c43d0 con 0x55f9695403e0 2024-04-21T08:42:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.808+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f9695403e0 0x55f969542800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.809+0000 7ff8d9ffb640 10 client.16407 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.809+0000 7ff8f61b0640 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f9695403e0 0x55f969542800 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaf1dd81ae460878 server_cookie=3d8124e3dd36184b in_seq=0 out_seq=0 2024-04-21T08:42:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8d9ffb640 1 -- 172.21.15.94:0/1087952485 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f9693c43d0 con 0x55f9695403e0 2024-04-21T08:42:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8d9ffb640 10 client.16407 handle_command_reply: tid=0 2024-04-21T08:42:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 client.16407 shutdown 2024-04-21T08:42:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 -- 172.21.15.94:0/1087952485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c90e0 msgr2=0x55f9694cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c90e0 0x55f9694cf750 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009900 tx=0x7ff8e8009260 comp rx=0 tx=0).stop 2024-04-21T08:42:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 -- 172.21.15.94:0/1087952485 shutdown_connections 2024-04-21T08:42:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f9695403e0 0x55f969542800 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f9694c90e0 0x55f9694cf750 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9694c08d0 0x55f9694ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 --2- 172.21.15.94:0/1087952485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f9694bfb60 0x55f9694c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8d9ffb640 0 client.16407 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 -- 172.21.15.94:0/1087952485 >> 172.21.15.94:0/1087952485 conn(0x55f969486c90 msgr2=0x55f9694bbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 -- 172.21.15.94:0/1087952485 shutdown_connections 2024-04-21T08:42:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 1 -- 172.21.15.94:0/1087952485 wait complete. 2024-04-21T08:42:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.817+0000 7ff8fb008740 20 client.16407 trim_cache size 0 max 16384 2024-04-21T08:42:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:{ 2024-04-21T08:42:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "message": "", 2024-04-21T08:42:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout: "return_code": 0 2024-04-21T08:42:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stdout:} 2024-04-21T08:42:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 -- 172.21.15.94:0/766921894 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8dc0a3e40 msgr2=0x7ff8dc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8dc0a3e40 0x7ff8dc0a6300 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8005ab0 tx=0x7ff8e8005fd0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 -- 172.21.15.94:0/766921894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f0102e70 msgr2=0x7ff8f019acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f0102e70 0x7ff8f019acd0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000c920 tx=0x7ff8e000cde0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 -- 172.21.15.94:0/766921894 shutdown_connections 2024-04-21T08:42:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8dc0a3e40 0x7ff8dc0a6300 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f0103250 0x7ff8f019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f0102e70 0x7ff8f019acd0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 --2- 172.21.15.94:0/766921894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f0100fb0 0x7ff8f019a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 -- 172.21.15.94:0/766921894 >> 172.21.15.94:0/766921894 conn(0x7ff8f0077390 msgr2=0x7ff8f00fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 -- 172.21.15.94:0/766921894 shutdown_connections 2024-04-21T08:42:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi094.front.sepia.ceph.com.smithi094.stderr:2024-04-21T08:42:05.818+0000 7ff8f8414640 1 -- 172.21.15.94:0/766921894 wait complete. 2024-04-21T08:42:06.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:05 smithi111 ceph-mon[31493]: pgmap v66: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 340 MiB/s wr, 96 op/s 2024-04-21T08:42:06.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:05 smithi094 ceph-mon[27988]: pgmap v66: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 340 MiB/s wr, 96 op/s 2024-04-21T08:42:06.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:05 smithi173 ceph-mon[31428]: pgmap v66: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 340 MiB/s wr, 96 op/s 2024-04-21T08:42:07.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:06 smithi111 ceph-mon[31493]: pgmap v67: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 113 MiB/s wr, 49 op/s 2024-04-21T08:42:07.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:06 smithi094 ceph-mon[27988]: pgmap v67: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 113 MiB/s wr, 49 op/s 2024-04-21T08:42:07.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:06 smithi173 ceph-mon[31428]: pgmap v67: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 113 MiB/s wr, 49 op/s 2024-04-21T08:42:09.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:09 smithi094 ceph-mon[27988]: pgmap v68: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:42:09.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:09 smithi173 ceph-mon[31428]: pgmap v68: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:42:09.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:09 smithi111 ceph-mon[31493]: pgmap v68: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:42:11.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:11 smithi111 ceph-mon[31493]: pgmap v69: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 59 op/s 2024-04-21T08:42:11.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:11 smithi094 ceph-mon[27988]: pgmap v69: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 59 op/s 2024-04-21T08:42:11.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:11 smithi173 ceph-mon[31428]: pgmap v69: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 59 op/s 2024-04-21T08:42:13.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:13 smithi111 ceph-mon[31493]: pgmap v70: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 56 op/s 2024-04-21T08:42:13.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:13 smithi094 ceph-mon[27988]: pgmap v70: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 56 op/s 2024-04-21T08:42:13.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:13 smithi173 ceph-mon[31428]: pgmap v70: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 56 op/s 2024-04-21T08:42:13.750 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:42:13.751 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:42:13.751 INFO:tasks.workunit.client.0.smithi094.stdout: 2024-04-21T08:42:13.751 INFO:tasks.workunit.client.0.smithi094.stdout:iozone test complete. 2024-04-21T08:42:13.752 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-21T08:42:13.752 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-21T08:42:13.789 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-04-21T08:42:13.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-21T08:42:13.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:42:13.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:42:13.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.945+0000 7f9d84ad2640 1 -- 172.21.15.94:0/832305508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d80101080 msgr2=0x7f9d80101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.945+0000 7f9d84ad2640 1 --2- 172.21.15.94:0/832305508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d80101080 0x7f9d80101460 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009900 tx=0x7f9d7002bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:13.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.945+0000 7f9d84ad2640 1 -- 172.21.15.94:0/832305508 shutdown_connections 2024-04-21T08:42:13.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.945+0000 7f9d84ad2640 1 --2- 172.21.15.94:0/832305508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d80101a30 0x7f9d80105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.945+0000 7f9d84ad2640 1 --2- 172.21.15.94:0/832305508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d80106150 0x7f9d8010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.945+0000 7f9d84ad2640 1 --2- 172.21.15.94:0/832305508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d80101080 0x7f9d80101460 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.945+0000 7f9d84ad2640 1 -- 172.21.15.94:0/832305508 >> 172.21.15.94:0/832305508 conn(0x7f9d80078000 msgr2=0x7f9d80078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:13.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.945+0000 7f9d84ad2640 1 -- 172.21.15.94:0/832305508 shutdown_connections 2024-04-21T08:42:13.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.945+0000 7f9d84ad2640 1 -- 172.21.15.94:0/832305508 wait complete. 2024-04-21T08:42:13.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.946+0000 7f9d84ad2640 1 Processor -- start 2024-04-21T08:42:13.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.946+0000 7f9d84ad2640 1 -- start start 2024-04-21T08:42:13.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.946+0000 7f9d84ad2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d80101080 0x7f9d80197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.946+0000 7f9d84ad2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d80106150 0x7f9d80197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.946+0000 7f9d84ad2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d8019bed0 0x7f9d8019c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.946+0000 7f9d84ad2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801110b0 con 0x7f9d80101080 2024-04-21T08:42:13.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.946+0000 7f9d84ad2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9d80110f30 con 0x7f9d8019bed0 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.946+0000 7f9d84ad2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9d80111230 con 0x7f9d80106150 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d80101080 0x7f9d80197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d80101080 0x7f9d80197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57542/0 (socket says 172.21.15.94:57542) 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7e59c640 1 -- 172.21.15.94:0/3777454229 learned_addr learned my addr 172.21.15.94:0/3777454229 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d80106150 0x7f9d80197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7ed9d640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d8019bed0 0x7f9d8019c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/3777454229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d8019bed0 msgr2=0x7f9d8019c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d8019bed0 0x7f9d8019c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/3777454229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d80101080 msgr2=0x7f9d80197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d80101080 0x7f9d80197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/3777454229 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8019c9c0 con 0x7f9d80106150 2024-04-21T08:42:13.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d80101080 0x7f9d80197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:13.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d80106150 0x7f9d80197b50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400e9c0 tx=0x7f9d7400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d6f7fe640 1 -- 172.21.15.94:0/3777454229 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7400ccc0 con 0x7f9d80106150 2024-04-21T08:42:13.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d84ad2640 1 -- 172.21.15.94:0/3777454229 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8019ccb0 con 0x7f9d80106150 2024-04-21T08:42:13.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d6f7fe640 1 -- 172.21.15.94:0/3777454229 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d74005020 con 0x7f9d80106150 2024-04-21T08:42:13.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d6f7fe640 1 -- 172.21.15.94:0/3777454229 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d74004a00 con 0x7f9d80106150 2024-04-21T08:42:13.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.947+0000 7f9d84ad2640 1 -- 172.21.15.94:0/3777454229 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d80069400 con 0x7f9d80106150 2024-04-21T08:42:13.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d8779f740 1 Processor -- start 2024-04-21T08:42:13.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d8779f740 1 -- start start 2024-04-21T08:42:13.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75f3580 0x564fd75f3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75fca80 0x564fd75fee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d8779f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564fd75581e0 con 0x564fd75f3ea0 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d8779f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564fd74f5390 con 0x564fd75fca80 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d8779f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564fd75541a0 con 0x564fd75f3580 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d7e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d7e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57558/0 (socket says 172.21.15.94:57558) 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.950+0000 7f9d7e59c640 1 -- 172.21.15.94:0/3036727792 learned_addr learned my addr 172.21.15.94:0/3036727792 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.951+0000 7f9d7e59c640 1 -- 172.21.15.94:0/3036727792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75f3580 msgr2=0x564fd75f3960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.951+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/3036727792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75f3580 0x564fd75f3960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.951+0000 7f9d7e59c640 1 -- 172.21.15.94:0/3036727792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75fca80 msgr2=0x564fd75fee70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:42:13.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.951+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/3036727792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75fca80 0x564fd75fee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.951+0000 7f9d7e59c640 1 -- 172.21.15.94:0/3036727792 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fd73347d0 con 0x564fd75f3ea0 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.951+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/3036727792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fc540 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9d70036040 tx=0x7f9d7002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cbd5f4a36810a2b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.951+0000 7f9d6cff9640 1 -- 172.21.15.94:0/3036727792 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d700040c0 con 0x564fd75f3ea0 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.951+0000 7f9d6cff9640 1 -- 172.21.15.94:0/3036727792 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d7002cc90 con 0x564fd75f3ea0 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.951+0000 7f9d6cff9640 1 -- 172.21.15.94:0/3036727792 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70003cb0 con 0x564fd75f3ea0 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.951+0000 7f9d6f7fe640 1 -- 172.21.15.94:0/3777454229 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9d74004ba0 con 0x7f9d80106150 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d8779f740 1 -- 172.21.15.94:0/3036727792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 msgr2=0x564fd75fc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d8779f740 1 --2- 172.21.15.94:0/3036727792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fc540 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9d70036040 tx=0x7f9d7002c5d0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d6f7fe640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9d680a3d20 0x7f9d680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d8779f740 1 -- 172.21.15.94:0/3036727792 shutdown_connections 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d8779f740 1 --2- 172.21.15.94:0/3036727792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75fca80 0x564fd75fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d8779f740 1 --2- 172.21.15.94:0/3036727792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fc540 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d8779f740 1 --2- 172.21.15.94:0/3036727792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75f3580 0x564fd75f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d8779f740 1 -- 172.21.15.94:0/3036727792 >> 172.21.15.94:0/3036727792 conn(0x564fd75edcb0 msgr2=0x564fd75ee090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d7ed9d640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9d680a3d20 0x7f9d680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d6f7fe640 1 -- 172.21.15.94:0/3777454229 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9d740d0c40 con 0x7f9d80106150 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d8779f740 1 -- 172.21.15.94:0/3036727792 shutdown_connections 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.952+0000 7f9d8779f740 1 -- 172.21.15.94:0/3036727792 wait complete. 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d7ed9d640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9d680a3d20 0x7f9d680a61e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9d78005fd0 tx=0x7f9d78009750 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d8779f740 1 -- start start 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d8779f740 1 Processor -- start 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75f3580 0x564fd75f9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75fca80 0x564fd75fa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d8779f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564fd75ff810 con 0x564fd75f3ea0 2024-04-21T08:42:13.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d8779f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564fd75112f0 con 0x564fd75f3580 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d8779f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564fd75ff990 con 0x564fd75fca80 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d8779f740 10 client.0 _ll_register_callbacks cb 0x564fd751e580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fa0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57570/0 (socket says 172.21.15.94:57570) 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/866583947 learned_addr learned my addr 172.21.15.94:0/866583947 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75f3580 0x564fd75f9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.953+0000 7f9d7ed9d640 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75fca80 0x564fd75fa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/866583947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75fca80 msgr2=0x564fd75fa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75fca80 0x564fd75fa600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/866583947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75f3580 msgr2=0x564fd75f9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75f3580 0x564fd75f9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/866583947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fd726a3f0 con 0x564fd75f3ea0 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fa0c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9d740ce5b0 tx=0x7f9d7409c220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a3d659f80e58a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d65ffb640 1 -- 172.21.15.94:0/866583947 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7401c040 con 0x564fd75f3ea0 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d65ffb640 1 -- 172.21.15.94:0/866583947 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9d74087070 con 0x564fd75f3ea0 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d8779f740 1 -- 172.21.15.94:0/866583947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d7002cc90 con 0x564fd75f3ea0 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d65ffb640 1 -- 172.21.15.94:0/866583947 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9d740a0bb0 con 0x564fd75f3ea0 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d8779f740 10 client.16419 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d8779f740 1 -- 172.21.15.94:0/866583947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564fd722a0f0 con 0x564fd75f3ea0 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.954+0000 7f9d65ffb640 1 -- 172.21.15.94:0/866583947 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d7002cc90 con 0x564fd75f3ea0 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.955+0000 7f9d65ffb640 1 -- 172.21.15.94:0/866583947 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9d740a5070 con 0x564fd75f3ea0 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.955+0000 7f9d8779f740 10 client.16419 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.955+0000 7f9d8779f740 10 client.16419 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.955+0000 7f9d8779f740 20 client.16419 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.955+0000 7f9d8779f740 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564fd760cba0 0x564fd760ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.955+0000 7f9d8779f740 4 client.16419 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.955+0000 7f9d8779f740 1 -- 172.21.15.94:0/866583947 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fd7565c00 con 0x564fd760cba0 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.955+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564fd760cba0 0x564fd760ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.955+0000 7f9d65ffb640 10 client.16419 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.955+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564fd760cba0 0x564fd760ef90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed254769205524bd server_cookie=556e3f771c599ba0 in_seq=0 out_seq=0 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d65ffb640 1 -- 172.21.15.94:0/866583947 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564fd7565c00 con 0x564fd760cba0 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d65ffb640 10 client.16419 handle_command_reply: tid=0 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 client.16419 shutdown 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 -- 172.21.15.94:0/866583947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 msgr2=0x564fd75fa0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fa0c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9d740ce5b0 tx=0x7f9d7409c220 comp rx=0 tx=0).stop 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 -- 172.21.15.94:0/866583947 shutdown_connections 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564fd760cba0 0x564fd760ef90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75fca80 0x564fd75fa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d65ffb640 0 client.16419 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:13.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd75fa0c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 --2- 172.21.15.94:0/866583947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75f3580 0x564fd75f9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 -- 172.21.15.94:0/866583947 >> 172.21.15.94:0/866583947 conn(0x564fd75edcb0 msgr2=0x564fd75efa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:13.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 -- 172.21.15.94:0/866583947 shutdown_connections 2024-04-21T08:42:13.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.956+0000 7f9d8779f740 1 -- 172.21.15.94:0/866583947 wait complete. 2024-04-21T08:42:13.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.957+0000 7f9d8779f740 20 client.16419 trim_cache size 0 max 16384 2024-04-21T08:42:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d8779f740 1 Processor -- start 2024-04-21T08:42:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d8779f740 1 -- start start 2024-04-21T08:42:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3580 0x564fd75fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75f3ea0 0x564fd76086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75fca80 0x564fd75ee090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d8779f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801af9d0 con 0x564fd75f3580 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d8779f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801ad2a0 con 0x564fd75f3ea0 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d8779f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801ab0f0 con 0x564fd75fca80 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d7e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3580 0x564fd75fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d7e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3580 0x564fd75fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57578/0 (socket says 172.21.15.94:57578) 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d7e59c640 1 -- 172.21.15.94:0/1940260263 learned_addr learned my addr 172.21.15.94:0/1940260263 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.974+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/1940260263 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75f3ea0 0x564fd76086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.975+0000 7f9d7e59c640 1 -- 172.21.15.94:0/1940260263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75fca80 msgr2=0x564fd75ee090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.975+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/1940260263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75fca80 0x564fd75ee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.975+0000 7f9d7e59c640 1 -- 172.21.15.94:0/1940260263 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75f3ea0 msgr2=0x564fd76086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.975+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/1940260263 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75f3ea0 0x564fd76086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.975+0000 7f9d7e59c640 1 -- 172.21.15.94:0/1940260263 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fd75f3cd0 con 0x564fd75f3580 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.975+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/1940260263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3580 0x564fd75fc2b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009430 tx=0x7f9d7003e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a77c2062ee92f3f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.975+0000 7f9d67fff640 1 -- 172.21.15.94:0/1940260263 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7002cb00 con 0x564fd75f3580 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.975+0000 7f9d67fff640 1 -- 172.21.15.94:0/1940260263 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d70055630 con 0x564fd75f3580 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.975+0000 7f9d67fff640 1 -- 172.21.15.94:0/1940260263 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7003c910 con 0x564fd75f3580 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 -- 172.21.15.94:0/1940260263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3580 msgr2=0x564fd75fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 --2- 172.21.15.94:0/1940260263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3580 0x564fd75fc2b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009430 tx=0x7f9d7003e040 comp rx=0 tx=0).stop 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 -- 172.21.15.94:0/1940260263 shutdown_connections 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 --2- 172.21.15.94:0/1940260263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75fca80 0x564fd75ee090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 --2- 172.21.15.94:0/1940260263 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75f3ea0 0x564fd76086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 --2- 172.21.15.94:0/1940260263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3580 0x564fd75fc2b0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 -- 172.21.15.94:0/1940260263 >> 172.21.15.94:0/1940260263 conn(0x564fd75edcb0 msgr2=0x564fd75ec4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 -- 172.21.15.94:0/1940260263 shutdown_connections 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 -- 172.21.15.94:0/1940260263 wait complete. 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 -- start start 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 Processor -- start 2024-04-21T08:42:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75f3580 0x564fd76007d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd7600d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75fca80 0x564fd7601250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801ab760 con 0x564fd75f3ea0 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801acbd0 con 0x564fd75fca80 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801abe30 con 0x564fd75f3580 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d8779f740 10 client.0 _ll_register_callbacks cb 0x7f9d7400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd7600d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.976+0000 7f9d7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd7600d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57580/0 (socket says 172.21.15.94:57580) 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/3982887174 learned_addr learned my addr 172.21.15.94:0/3982887174 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75f3580 0x564fd76007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/3982887174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75f3580 msgr2=0x564fd76007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75f3580 0x564fd76007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/3982887174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75fca80 msgr2=0x564fd7601250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d7ed9d640 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75fca80 0x564fd7601250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75fca80 0x564fd7601250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d7dd9b640 1 -- 172.21.15.94:0/3982887174 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fd7672260 con 0x564fd75f3ea0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75f3580 0x564fd76007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d7dd9b640 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd7600d10 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9d7409c790 tx=0x7f9d74005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ff1705ec8712d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d65ffb640 1 -- 172.21.15.94:0/3982887174 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7401c040 con 0x564fd75f3ea0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d65ffb640 1 -- 172.21.15.94:0/3982887174 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9d740985d0 con 0x564fd75f3ea0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d8779f740 1 -- 172.21.15.94:0/3982887174 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d70055630 con 0x564fd75f3ea0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d65ffb640 1 -- 172.21.15.94:0/3982887174 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9d7409c340 con 0x564fd75f3ea0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d8779f740 10 client.16425 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d8779f740 1 -- 172.21.15.94:0/3982887174 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564fd7673530 con 0x564fd75f3ea0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.977+0000 7f9d65ffb640 1 -- 172.21.15.94:0/3982887174 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d70055630 con 0x564fd75f3ea0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.978+0000 7f9d65ffb640 1 -- 172.21.15.94:0/3982887174 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9d7409b6c0 con 0x564fd75f3ea0 2024-04-21T08:42:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.978+0000 7f9d8779f740 10 client.16425 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:13.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.978+0000 7f9d8779f740 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:13.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.978+0000 7f9d8779f740 20 client.16425 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:13.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.978+0000 7f9d8779f740 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564fd7673bd0 0x564fd7675ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.978+0000 7f9d8779f740 4 client.16425 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:42:13.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.978+0000 7f9d8779f740 1 -- 172.21.15.94:0/3982887174 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564fd755fa70 con 0x564fd7673bd0 2024-04-21T08:42:13.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.978+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564fd7673bd0 0x564fd7675ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.978+0000 7f9d7e59c640 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564fd7673bd0 0x564fd7675ff0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36da13bef3252dc6 server_cookie=7d2560244d0240aa in_seq=0 out_seq=0 2024-04-21T08:42:13.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.978+0000 7f9d65ffb640 10 client.16425 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:13.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d65ffb640 1 -- 172.21.15.94:0/3982887174 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564fd755fa70 con 0x564fd7673bd0 2024-04-21T08:42:13.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d65ffb640 10 client.16425 handle_command_reply: tid=0 2024-04-21T08:42:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 client.16425 shutdown 2024-04-21T08:42:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 -- 172.21.15.94:0/3982887174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 msgr2=0x564fd7600d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd7600d10 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9d7409c790 tx=0x7f9d74005400 comp rx=0 tx=0).stop 2024-04-21T08:42:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 -- 172.21.15.94:0/3982887174 shutdown_connections 2024-04-21T08:42:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564fd7673bd0 0x564fd7675ff0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564fd75fca80 0x564fd7601250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fd75f3ea0 0x564fd7600d10 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 --2- 172.21.15.94:0/3982887174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564fd75f3580 0x564fd76007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d65ffb640 0 client.16425 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 -- 172.21.15.94:0/3982887174 >> 172.21.15.94:0/3982887174 conn(0x564fd75edcb0 msgr2=0x564fd75ecd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:13.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 -- 172.21.15.94:0/3982887174 shutdown_connections 2024-04-21T08:42:13.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 1 -- 172.21.15.94:0/3982887174 wait complete. 2024-04-21T08:42:13.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d8779f740 20 client.16425 trim_cache size 0 max 16384 2024-04-21T08:42:13.986 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:42:13.986 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:42:13.986 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:42:13.986 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:42:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d84ad2640 1 -- 172.21.15.94:0/3777454229 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9d680a3d20 msgr2=0x7f9d680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d84ad2640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9d680a3d20 0x7f9d680a61e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9d78005fd0 tx=0x7f9d78009750 comp rx=0 tx=0).stop 2024-04-21T08:42:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d84ad2640 1 -- 172.21.15.94:0/3777454229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d80106150 msgr2=0x7f9d80197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d84ad2640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d80106150 0x7f9d80197b50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400e9c0 tx=0x7f9d7400ee80 comp rx=0 tx=0).stop 2024-04-21T08:42:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d84ad2640 1 -- 172.21.15.94:0/3777454229 shutdown_connections 2024-04-21T08:42:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d84ad2640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9d680a3d20 0x7f9d680a61e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d84ad2640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d80106150 0x7f9d80197b50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d84ad2640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d8019bed0 0x7f9d8019c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d84ad2640 1 --2- 172.21.15.94:0/3777454229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d80101080 0x7f9d80197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.979+0000 7f9d84ad2640 1 -- 172.21.15.94:0/3777454229 >> 172.21.15.94:0/3777454229 conn(0x7f9d80078000 msgr2=0x7f9d800ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.980+0000 7f9d84ad2640 1 -- 172.21.15.94:0/3777454229 shutdown_connections 2024-04-21T08:42:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:13.980+0000 7f9d84ad2640 1 -- 172.21.15.94:0/3777454229 wait complete. 2024-04-21T08:42:13.990 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:42:13.990 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:14.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.091+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1064073437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a641011a0 msgr2=0x7f3a64101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.091+0000 7f3a6b5bd640 1 --2- 172.21.15.94:0/1064073437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a641011a0 0x7f3a64101620 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009930 tx=0x7f3a6002bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:14.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1064073437 shutdown_connections 2024-04-21T08:42:14.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 --2- 172.21.15.94:0/1064073437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a64101b60 0x7f3a6410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 --2- 172.21.15.94:0/1064073437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a641011a0 0x7f3a64101620 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 --2- 172.21.15.94:0/1064073437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a64105e30 0x7f3a64106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1064073437 >> 172.21.15.94:0/1064073437 conn(0x7f3a64077e30 msgr2=0x7f3a64078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1064073437 shutdown_connections 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1064073437 wait complete. 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 Processor -- start 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 -- start start 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a641011a0 0x7f3a64199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a64101b60 0x7f3a64199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64105e30 0x7f3a641a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3a641110c0 con 0x7f3a64105e30 2024-04-21T08:42:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3a64110f40 con 0x7f3a64101b60 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.092+0000 7f3a6b5bd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3a64111240 con 0x7f3a641011a0 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a69b5a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64105e30 0x7f3a641a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a69b5a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64105e30 0x7f3a641a45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57604/0 (socket says 172.21.15.94:57604) 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a69359640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a641011a0 0x7f3a64199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a69b5a640 1 -- 172.21.15.94:0/1342363858 learned_addr learned my addr 172.21.15.94:0/1342363858 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a68b58640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a64101b60 0x7f3a64199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a69b5a640 1 -- 172.21.15.94:0/1342363858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a641011a0 msgr2=0x7f3a64199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a69b5a640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a641011a0 0x7f3a64199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a69b5a640 1 -- 172.21.15.94:0/1342363858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a64101b60 msgr2=0x7f3a64199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a69b5a640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a64101b60 0x7f3a64199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a69b5a640 1 -- 172.21.15.94:0/1342363858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a641a4b00 con 0x7f3a64105e30 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a68b58640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a64101b60 0x7f3a64199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a69b5a640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64105e30 0x7f3a641a45c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00cce0 tx=0x7f3a5c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a5a7fc640 1 -- 172.21.15.94:0/1342363858 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c00f450 con 0x7f3a64105e30 2024-04-21T08:42:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a5a7fc640 1 -- 172.21.15.94:0/1342363858 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a5c00faa0 con 0x7f3a64105e30 2024-04-21T08:42:14.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1342363858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a641a4cd0 con 0x7f3a64105e30 2024-04-21T08:42:14.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a5a7fc640 1 -- 172.21.15.94:0/1342363858 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c018b70 con 0x7f3a64105e30 2024-04-21T08:42:14.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.093+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1342363858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a641a51a0 con 0x7f3a64105e30 2024-04-21T08:42:14.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.094+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1342363858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a64068de0 con 0x7f3a64105e30 2024-04-21T08:42:14.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.095+0000 7f3a5a7fc640 1 -- 172.21.15.94:0/1342363858 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f3a5c018d10 con 0x7f3a64105e30 2024-04-21T08:42:14.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.096+0000 7f3a5a7fc640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3a500a3d20 0x7f3a500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.096+0000 7f3a69359640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3a500a3d20 0x7f3a500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.097+0000 7f3a5a7fc640 1 -- 172.21.15.94:0/1342363858 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3a5c0d1060 con 0x7f3a64105e30 2024-04-21T08:42:14.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.097+0000 7f3a69359640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3a500a3d20 0x7f3a500a61e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3a54005d00 tx=0x7f3a5400b410 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.100+0000 7f3a5a7fc640 1 -- 172.21.15.94:0/1342363858 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a5c014030 con 0x7f3a64105e30 2024-04-21T08:42:14.127 DEBUG:teuthology.parallel:result is None 2024-04-21T08:42:14.128 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-21T08:42:14.160 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-21T08:42:14.160 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-21T08:42:14.171 INFO:tasks.quiescer:joining Quiescers 2024-04-21T08:42:14.172 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:14.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 -- 172.21.15.94:0/1458666477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c8069aa0 msgr2=0x7f84c810ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 --2- 172.21.15.94:0/1458666477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c8069aa0 0x7f84c810ede0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009ca0 tx=0x7f84c002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 -- 172.21.15.94:0/1458666477 shutdown_connections 2024-04-21T08:42:14.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 --2- 172.21.15.94:0/1458666477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84c80690e0 0x7f84c8069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 --2- 172.21.15.94:0/1458666477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c8069aa0 0x7f84c810ede0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 --2- 172.21.15.94:0/1458666477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84c8106f50 0x7f84c8107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 -- 172.21.15.94:0/1458666477 >> 172.21.15.94:0/1458666477 conn(0x7f84c8077e30 msgr2=0x7f84c8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 -- 172.21.15.94:0/1458666477 shutdown_connections 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 -- 172.21.15.94:0/1458666477 wait complete. 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 Processor -- start 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 -- start start 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84c80690e0 0x7f84c8101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c8069aa0 0x7f84c81017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84c8106f50 0x7f84c8101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f84c8112170 con 0x7f84c8069aa0 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f84c8111ff0 con 0x7f84c80690e0 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.301+0000 7f84cffa4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f84c81122f0 con 0x7f84c8106f50 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cd53f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c8069aa0 0x7f84c81017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cd53f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c8069aa0 0x7f84c81017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57620/0 (socket says 172.21.15.94:57620) 2024-04-21T08:42:14.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cd53f640 1 -- 172.21.15.94:0/1182517067 learned_addr learned my addr 172.21.15.94:0/1182517067 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84ce541640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84c8106f50 0x7f84c8101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cdd40640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84c80690e0 0x7f84c8101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cd53f640 1 -- 172.21.15.94:0/1182517067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84c8106f50 msgr2=0x7f84c8101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cd53f640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84c8106f50 0x7f84c8101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cd53f640 1 -- 172.21.15.94:0/1182517067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84c80690e0 msgr2=0x7f84c8101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cd53f640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84c80690e0 0x7f84c8101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cd53f640 1 -- 172.21.15.94:0/1182517067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c81a6ab0 con 0x7f84c8069aa0 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cd53f640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c8069aa0 0x7f84c81017b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f84b800e9f0 tx=0x7f84b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84beffd640 1 -- 172.21.15.94:0/1182517067 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b800cd80 con 0x7f84c8069aa0 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84beffd640 1 -- 172.21.15.94:0/1182517067 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84b80054d0 con 0x7f84c8069aa0 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cffa4640 1 -- 172.21.15.94:0/1182517067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c81a6da0 con 0x7f84c8069aa0 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84beffd640 1 -- 172.21.15.94:0/1182517067 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b8010690 con 0x7f84c8069aa0 2024-04-21T08:42:14.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.302+0000 7f84cffa4640 1 -- 172.21.15.94:0/1182517067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c81a7260 con 0x7f84c8069aa0 2024-04-21T08:42:14.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.303+0000 7f84cffa4640 1 -- 172.21.15.94:0/1182517067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c8068de0 con 0x7f84c8069aa0 2024-04-21T08:42:14.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.305+0000 7f84beffd640 1 -- 172.21.15.94:0/1182517067 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f84b8004e50 con 0x7f84c8069aa0 2024-04-21T08:42:14.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.306+0000 7f84beffd640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84b40a3dd0 0x7f84b40a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.306+0000 7f84beffd640 1 -- 172.21.15.94:0/1182517067 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f84b80d0f10 con 0x7f84c8069aa0 2024-04-21T08:42:14.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.306+0000 7f84cdd40640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84b40a3dd0 0x7f84b40a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.306+0000 7f84cdd40640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84b40a3dd0 0x7f84b40a6290 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f84c40058c0 tx=0x7f84c400a2d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.310+0000 7f84beffd640 1 -- 172.21.15.94:0/1182517067 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f84b809d1d0 con 0x7f84c8069aa0 2024-04-21T08:42:14.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.326+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1342363858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a640696d0 con 0x7f3a64105e30 2024-04-21T08:42:14.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.327+0000 7f3a5a7fc640 1 -- 172.21.15.94:0/1342363858 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f3a640696d0 con 0x7f3a64105e30 2024-04-21T08:42:14.327 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:42:14.327 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:42:14.327 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:42:14.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1342363858 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3a500a3d20 msgr2=0x7f3a500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3a500a3d20 0x7f3a500a61e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3a54005d00 tx=0x7f3a5400b410 comp rx=0 tx=0).stop 2024-04-21T08:42:14.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1342363858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64105e30 msgr2=0x7f3a641a45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64105e30 0x7f3a641a45c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00cce0 tx=0x7f3a5c00a620 comp rx=0 tx=0).stop 2024-04-21T08:42:14.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1342363858 shutdown_connections 2024-04-21T08:42:14.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3a500a3d20 0x7f3a500a61e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64105e30 0x7f3a641a45c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a64101b60 0x7f3a64199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 --2- 172.21.15.94:0/1342363858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a641011a0 0x7f3a64199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1342363858 >> 172.21.15.94:0/1342363858 conn(0x7f3a64077e30 msgr2=0x7f3a640fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1342363858 shutdown_connections 2024-04-21T08:42:14.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.329+0000 7f3a6b5bd640 1 -- 172.21.15.94:0/1342363858 wait complete. 2024-04-21T08:42:14.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:42:14.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.444+0000 7fb2d041e640 1 -- 172.21.15.94:0/511840729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2c8101a40 msgr2=0x7fb2c8105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.444+0000 7fb2d041e640 1 --2- 172.21.15.94:0/511840729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2c8101a40 0x7fb2c8105a80 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009900 tx=0x7fb2b802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:14.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.444+0000 7fb2d041e640 1 -- 172.21.15.94:0/511840729 shutdown_connections 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.444+0000 7fb2d041e640 1 --2- 172.21.15.94:0/511840729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8106160 0x7fb2c810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.444+0000 7fb2d041e640 1 --2- 172.21.15.94:0/511840729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2c8101a40 0x7fb2c8105a80 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.444+0000 7fb2d041e640 1 --2- 172.21.15.94:0/511840729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2c8101090 0x7fb2c8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.444+0000 7fb2d041e640 1 -- 172.21.15.94:0/511840729 >> 172.21.15.94:0/511840729 conn(0x7fb2c8077f50 msgr2=0x7fb2c8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.444+0000 7fb2d041e640 1 -- 172.21.15.94:0/511840729 shutdown_connections 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.444+0000 7fb2d041e640 1 -- 172.21.15.94:0/511840729 wait complete. 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2d041e640 1 Processor -- start 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2d041e640 1 -- start start 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2d041e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8101090 0x7fb2c8102b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2d041e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2c8106160 0x7fb2c8103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2d041e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2c81035a0 0x7fb2c81a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2d041e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81110a0 con 0x7fb2c8106160 2024-04-21T08:42:14.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2d041e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c8110f20 con 0x7fb2c8101090 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2d041e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c8111220 con 0x7fb2c81035a0 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2cd9b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2c8106160 0x7fb2c8103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2cd9b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2c8106160 0x7fb2c8103060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57650/0 (socket says 172.21.15.94:57650) 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2cd9b9640 1 -- 172.21.15.94:0/2513756485 learned_addr learned my addr 172.21.15.94:0/2513756485 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2ce1ba640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8101090 0x7fb2c8102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2c81035a0 0x7fb2c81a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2cd9b9640 1 -- 172.21.15.94:0/2513756485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2c81035a0 msgr2=0x7fb2c81a69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2cd9b9640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2c81035a0 0x7fb2c81a69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2cd9b9640 1 -- 172.21.15.94:0/2513756485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8101090 msgr2=0x7fb2c8102b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2cd9b9640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8101090 0x7fb2c8102b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2cd9b9640 1 -- 172.21.15.94:0/2513756485 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c81a7040 con 0x7fb2c8106160 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.445+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2c81035a0 0x7fb2c81a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.446+0000 7fb2cd9b9640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2c8106160 0x7fb2c8103060 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8036040 tx=0x7fb2b802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.446+0000 7fb2bf7fe640 1 -- 172.21.15.94:0/2513756485 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8043020 con 0x7fb2c8106160 2024-04-21T08:42:14.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.446+0000 7fb2bf7fe640 1 -- 172.21.15.94:0/2513756485 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2b802cbe0 con 0x7fb2c8106160 2024-04-21T08:42:14.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.446+0000 7fb2d041e640 1 -- 172.21.15.94:0/2513756485 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c81a72d0 con 0x7fb2c8106160 2024-04-21T08:42:14.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.446+0000 7fb2bf7fe640 1 -- 172.21.15.94:0/2513756485 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8003bf0 con 0x7fb2c8106160 2024-04-21T08:42:14.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.446+0000 7fb2d041e640 1 -- 172.21.15.94:0/2513756485 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c81a7820 con 0x7fb2c8106160 2024-04-21T08:42:14.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.448+0000 7fb2bf7fe640 1 -- 172.21.15.94:0/2513756485 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fb2b8046020 con 0x7fb2c8106160 2024-04-21T08:42:14.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.448+0000 7fb2d3005740 1 Processor -- start 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2d3005740 1 -- start start 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769128b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x55747691b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747691de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2d3005740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5574768771f0 con 0x55747691ba00 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2d3005740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5574768144b0 con 0x557476912e80 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2d3005740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5574768731b0 con 0x5574769124d0 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2cd9b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747691de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2cd9b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747691de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57666/0 (socket says 172.21.15.94:57666) 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2cd9b9640 1 -- 172.21.15.94:0/1173762497 learned_addr learned my addr 172.21.15.94:0/1173762497 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2bf7fe640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2b40a3d20 0x7fb2b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/1173762497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2ce1ba640 1 --2- 172.21.15.94:0/1173762497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x55747691b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2b40a3d20 0x7fb2b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2bf7fe640 1 -- 172.21.15.94:0/2513756485 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb2b80f2dd0 con 0x7fb2c8106160 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2cd9b9640 1 -- 172.21.15.94:0/1173762497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 msgr2=0x5574769128b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2cd9b9640 1 --2- 172.21.15.94:0/1173762497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769128b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2cd9b9640 1 -- 172.21.15.94:0/1173762497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 msgr2=0x55747691b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2cd9b9640 1 --2- 172.21.15.94:0/1173762497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x55747691b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.449+0000 7fb2cd9b9640 1 -- 172.21.15.94:0/1173762497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574768061b0 con 0x55747691ba00 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.450+0000 7fb2cd9b9640 1 --2- 172.21.15.94:0/1173762497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747691de50 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80c71e0 tx=0x7fb2b802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11aa63604c1b6f2b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.450+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2b40a3d20 0x7fb2b40a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0005c40 tx=0x7fb2c0009200 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.450+0000 7fb2bcbfe640 1 -- 172.21.15.94:0/1173762497 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b80be5e0 con 0x55747691ba00 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.450+0000 7fb2bcbfe640 1 -- 172.21.15.94:0/1173762497 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2b80bf1a0 con 0x55747691ba00 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.450+0000 7fb2bcbfe640 1 -- 172.21.15.94:0/1173762497 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b80be400 con 0x55747691ba00 2024-04-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 -- 172.21.15.94:0/1173762497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 msgr2=0x55747691de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 --2- 172.21.15.94:0/1173762497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747691de50 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80c71e0 tx=0x7fb2b802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 -- 172.21.15.94:0/1173762497 shutdown_connections 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 --2- 172.21.15.94:0/1173762497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747691de50 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 --2- 172.21.15.94:0/1173762497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x55747691b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 --2- 172.21.15.94:0/1173762497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769128b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 -- 172.21.15.94:0/1173762497 >> 172.21.15.94:0/1173762497 conn(0x55747690cba0 msgr2=0x55747690cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 -- 172.21.15.94:0/1173762497 shutdown_connections 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 -- 172.21.15.94:0/1173762497 wait complete. 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 -- start start 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 Processor -- start 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5574769124d0 0x55747691b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557476912e80 0x557476918650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x557476918bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55747691e730 con 0x55747691ba00 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557476830500 con 0x5574769124d0 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55747691e8b0 con 0x557476912e80 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2d3005740 10 client.0 _ll_register_callbacks cb 0x5574767feb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2ce9bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x557476918bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2ce9bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x557476918bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57680/0 (socket says 172.21.15.94:57680) 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/1239156454 learned_addr learned my addr 172.21.15.94:0/1239156454 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2cd9b9640 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557476912e80 0x557476918650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.451+0000 7fb2ce1ba640 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5574769124d0 0x55747691b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/1239156454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557476912e80 msgr2=0x557476918650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557476912e80 0x557476918650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/1239156454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5574769124d0 msgr2=0x55747691b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5574769124d0 0x55747691b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/1239156454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574765893f0 con 0x55747691ba00 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x557476918bb0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00142e0 tx=0x7fb2c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577e380f612d8353 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/1239156454 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0025070 con 0x55747691ba00 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/1239156454 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb2c000fba0 con 0x55747691ba00 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2d3005740 1 -- 172.21.15.94:0/1239156454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2b80bf1a0 con 0x55747691ba00 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/1239156454 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb2c0031050 con 0x55747691ba00 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2d3005740 10 client.16449 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2d3005740 1 -- 172.21.15.94:0/1239156454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5574766537d0 con 0x55747691ba00 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.452+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/1239156454 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2b80bf1a0 con 0x55747691ba00 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.453+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/1239156454 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb2c0021c70 con 0x55747691ba00 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.453+0000 7fb2d3005740 10 client.16449 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.453+0000 7fb2d3005740 10 client.16449 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:42:14.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.453+0000 7fb2d3005740 20 client.16449 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.453+0000 7fb2d3005740 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55747692b8c0 0x55747692dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.453+0000 7fb2d3005740 4 client.16449 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.453+0000 7fb2d3005740 1 -- 172.21.15.94:0/1239156454 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557476816930 con 0x55747692b8c0 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.453+0000 7fb2ce1ba640 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55747692b8c0 0x55747692dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.453+0000 7fb2b1ffb640 10 client.16449 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.453+0000 7fb2ce1ba640 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55747692b8c0 0x55747692dd30 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e98d12613332b4bd server_cookie=ba310e4fec83df4d in_seq=0 out_seq=0 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/1239156454 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557476816930 con 0x55747692b8c0 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2b1ffb640 10 client.16449 handle_command_reply: tid=0 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 client.16449 shutdown 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 -- 172.21.15.94:0/1239156454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 msgr2=0x557476918bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x557476918bb0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00142e0 tx=0x7fb2c0004a40 comp rx=0 tx=0).stop 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 -- 172.21.15.94:0/1239156454 shutdown_connections 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55747692b8c0 0x55747692dd30 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557476912e80 0x557476918650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x557476918bb0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 --2- 172.21.15.94:0/1239156454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5574769124d0 0x55747691b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 -- 172.21.15.94:0/1239156454 >> 172.21.15.94:0/1239156454 conn(0x55747690cba0 msgr2=0x55747690e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2b1ffb640 0 client.16449 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 -- 172.21.15.94:0/1239156454 shutdown_connections 2024-04-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 1 -- 172.21.15.94:0/1239156454 wait complete. 2024-04-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.454+0000 7fb2d3005740 20 client.16449 trim_cache size 0 max 16384 2024-04-21T08:42:14.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2d3005740 1 Processor -- start 2024-04-21T08:42:14.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2d3005740 1 -- start start 2024-04-21T08:42:14.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769274c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x5574769177b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747690ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2d3005740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81b3f00 con 0x55747691ba00 2024-04-21T08:42:14.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2d3005740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81b17d0 con 0x557476912e80 2024-04-21T08:42:14.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2d3005740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81af620 con 0x5574769124d0 2024-04-21T08:42:14.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2ce9bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747690ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2ce9bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747690ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57684/0 (socket says 172.21.15.94:57684) 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/2406766610 learned_addr learned my addr 172.21.15.94:0/2406766610 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2ce1ba640 1 --2- 172.21.15.94:0/2406766610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769274c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/2406766610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 msgr2=0x5574769274c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/2406766610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769274c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2cd9b9640 1 --2- 172.21.15.94:0/2406766610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x5574769177b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/2406766610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 msgr2=0x5574769177b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/2406766610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x5574769177b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/2406766610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557476911800 con 0x55747691ba00 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.468+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/2406766610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747690ba80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0000f80 tx=0x7fb2c0014a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83411b29c1ba68ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2b3fff640 1 -- 172.21.15.94:0/2406766610 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0025040 con 0x55747691ba00 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2b3fff640 1 -- 172.21.15.94:0/2406766610 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c003a070 con 0x55747691ba00 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2b3fff640 1 -- 172.21.15.94:0/2406766610 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0013920 con 0x55747691ba00 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2d3005740 1 -- 172.21.15.94:0/2406766610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 msgr2=0x55747690ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2d3005740 1 --2- 172.21.15.94:0/2406766610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747690ba80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0000f80 tx=0x7fb2c0014a10 comp rx=0 tx=0).stop 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2d3005740 1 -- 172.21.15.94:0/2406766610 shutdown_connections 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2d3005740 1 --2- 172.21.15.94:0/2406766610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x55747690ba80 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2d3005740 1 --2- 172.21.15.94:0/2406766610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x5574769177b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2d3005740 1 --2- 172.21.15.94:0/2406766610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769274c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2d3005740 1 -- 172.21.15.94:0/2406766610 >> 172.21.15.94:0/2406766610 conn(0x55747690cba0 msgr2=0x55747690b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2d3005740 1 -- 172.21.15.94:0/2406766610 shutdown_connections 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2d3005740 1 -- 172.21.15.94:0/2406766610 wait complete. 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.469+0000 7fb2d3005740 1 -- start start 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2d3005740 1 Processor -- start 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769239d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x557476923f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2d3005740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x5574769148c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2d3005740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81afc90 con 0x55747691ba00 2024-04-21T08:42:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2d3005740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81b1100 con 0x557476912e80 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2d3005740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81b0360 con 0x5574769124d0 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2d3005740 10 client.0 _ll_register_callbacks cb 0x7fb2c0009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2ce9bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x5574769148c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2ce9bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x5574769148c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57700/0 (socket says 172.21.15.94:57700) 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/2713701957 learned_addr learned my addr 172.21.15.94:0/2713701957 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2cd9b9640 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x557476923f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2ce1ba640 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769239d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/2713701957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 msgr2=0x5574769239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769239d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/2713701957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 msgr2=0x557476923f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x557476923f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2ce9bb640 1 -- 172.21.15.94:0/2713701957 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557476914e00 con 0x55747691ba00 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2cd9b9640 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x557476923f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:14.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2ce9bb640 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x5574769148c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0037040 tx=0x7fb2c0004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ccd27a50ebfe93 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.470+0000 7fb2b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/2713701957 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0025040 con 0x55747691ba00 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/2713701957 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb2c000fc00 con 0x55747691ba00 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2d3005740 1 -- 172.21.15.94:0/2713701957 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c003a070 con 0x55747691ba00 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/2713701957 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb2c004c050 con 0x55747691ba00 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2d3005740 10 client.16455 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2d3005740 1 -- 172.21.15.94:0/2713701957 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557476991610 con 0x55747691ba00 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/2713701957 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c003a070 con 0x55747691ba00 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/2713701957 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb2c0005080 con 0x55747691ba00 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2d3005740 10 client.16455 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2d3005740 10 client.16455 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2d3005740 20 client.16455 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2d3005740 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557476991c40 0x557476994060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2d3005740 4 client.16455 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2d3005740 1 -- 172.21.15.94:0/2713701957 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557476884c10 con 0x557476991c40 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.471+0000 7fb2ce1ba640 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557476991c40 0x557476994060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2b1ffb640 10 client.16455 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2ce1ba640 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557476991c40 0x557476994060 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6dd6c627277cf52f server_cookie=4f6db3f998d00ab1 in_seq=0 out_seq=0 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2b1ffb640 1 -- 172.21.15.94:0/2713701957 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557476884c10 con 0x557476991c40 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2b1ffb640 10 client.16455 handle_command_reply: tid=0 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 client.16455 shutdown 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 -- 172.21.15.94:0/2713701957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 msgr2=0x5574769148c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x5574769148c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0037040 tx=0x7fb2c0004d70 comp rx=0 tx=0).stop 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 -- 172.21.15.94:0/2713701957 shutdown_connections 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557476991c40 0x557476994060 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55747691ba00 0x5574769148c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2b1ffb640 0 client.16455 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:42:14.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557476912e80 0x557476923f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 --2- 172.21.15.94:0/2713701957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574769124d0 0x5574769239d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 -- 172.21.15.94:0/2713701957 >> 172.21.15.94:0/2713701957 conn(0x55747690cba0 msgr2=0x5574769132b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 -- 172.21.15.94:0/2713701957 shutdown_connections 2024-04-21T08:42:14.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 1 -- 172.21.15.94:0/2713701957 wait complete. 2024-04-21T08:42:14.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.472+0000 7fb2d3005740 20 client.16455 trim_cache size 0 max 16384 2024-04-21T08:42:14.479 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:42:14.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 -- 172.21.15.94:0/2513756485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2b40a3d20 msgr2=0x7fb2b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2b40a3d20 0x7fb2b40a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0005c40 tx=0x7fb2c0009200 comp rx=0 tx=0).stop 2024-04-21T08:42:14.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 -- 172.21.15.94:0/2513756485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2c8106160 msgr2=0x7fb2c8103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2c8106160 0x7fb2c8103060 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8036040 tx=0x7fb2b802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 -- 172.21.15.94:0/2513756485 shutdown_connections 2024-04-21T08:42:14.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2b40a3d20 0x7fb2b40a61e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2c81035a0 0x7fb2c81a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2c8106160 0x7fb2c8103060 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 --2- 172.21.15.94:0/2513756485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8101090 0x7fb2c8102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 -- 172.21.15.94:0/2513756485 >> 172.21.15.94:0/2513756485 conn(0x7fb2c8077f50 msgr2=0x7fb2c80695c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 -- 172.21.15.94:0/2513756485 shutdown_connections 2024-04-21T08:42:14.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.473+0000 7fb2d041e640 1 -- 172.21.15.94:0/2513756485 wait complete. 2024-04-21T08:42:14.484 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:42:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.542+0000 7f84cffa4640 1 -- 172.21.15.94:0/1182517067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84c8102750 con 0x7f84c8069aa0 2024-04-21T08:42:14.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.543+0000 7f84beffd640 1 -- 172.21.15.94:0/1182517067 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f84b809cad0 con 0x7f84c8069aa0 2024-04-21T08:42:14.543 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:42:14.543 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:42:14.543 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:42:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 -- 172.21.15.94:0/1182517067 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84b40a3dd0 msgr2=0x7f84b40a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84b40a3dd0 0x7f84b40a6290 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f84c40058c0 tx=0x7f84c400a2d0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 -- 172.21.15.94:0/1182517067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c8069aa0 msgr2=0x7f84c81017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c8069aa0 0x7f84c81017b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f84b800e9f0 tx=0x7f84b800edf0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 -- 172.21.15.94:0/1182517067 shutdown_connections 2024-04-21T08:42:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84b40a3dd0 0x7f84b40a6290 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84c8106f50 0x7f84c8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c8069aa0 0x7f84c81017b0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 --2- 172.21.15.94:0/1182517067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84c80690e0 0x7f84c8101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 -- 172.21.15.94:0/1182517067 >> 172.21.15.94:0/1182517067 conn(0x7f84c8077e30 msgr2=0x7f84c80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 -- 172.21.15.94:0/1182517067 shutdown_connections 2024-04-21T08:42:14.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.545+0000 7f84cffa4640 1 -- 172.21.15.94:0/1182517067 wait complete. 2024-04-21T08:42:14.557 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24488 quiesce db --cancel --all' 2024-04-21T08:42:14.557 DEBUG:teuthology.orchestra.run.smithi094:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24488 quiesce db --cancel --all 2024-04-21T08:42:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.586+0000 7fb28dcf2640 1 -- 172.21.15.94:0/1274070307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb28810c510 msgr2=0x7fb28810e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.586+0000 7fb28dcf2640 1 --2- 172.21.15.94:0/1274070307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb28810c510 0x7fb28810e9b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb280009ca0 tx=0x7fb28002c230 comp rx=0 tx=0).stop 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.586+0000 7fb28dcf2640 1 -- 172.21.15.94:0/1274070307 shutdown_connections 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.586+0000 7fb28dcf2640 1 --2- 172.21.15.94:0/1274070307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb28810c510 0x7fb28810e9b0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.586+0000 7fb28dcf2640 1 --2- 172.21.15.94:0/1274070307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb288103af0 0x7fb28810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.586+0000 7fb28dcf2640 1 --2- 172.21.15.94:0/1274070307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2881031d0 0x7fb2881035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.586+0000 7fb28dcf2640 1 -- 172.21.15.94:0/1274070307 >> 172.21.15.94:0/1274070307 conn(0x7fb288077f50 msgr2=0x7fb288078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.587+0000 7fb28dcf2640 1 -- 172.21.15.94:0/1274070307 shutdown_connections 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.587+0000 7fb28dcf2640 1 -- 172.21.15.94:0/1274070307 wait complete. 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.587+0000 7fb28dcf2640 1 Processor -- start 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.587+0000 7fb28dcf2640 1 -- start start 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.587+0000 7fb28dcf2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2881031d0 0x7fb288198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.587+0000 7fb28dcf2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb288103af0 0x7fb288198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.587+0000 7fb28dcf2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb28810c510 0x7fb28819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.587+0000 7fb28dcf2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881121a0 con 0x7fb288103af0 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb28dcf2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb288112020 con 0x7fb28810c510 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb28dcf2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb288112320 con 0x7fb2881031d0 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb286ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb288103af0 0x7fb288198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb286ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb288103af0 0x7fb288198cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57716/0 (socket says 172.21.15.94:57716) 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb286ffd640 1 -- 172.21.15.94:0/41686949 learned_addr learned my addr 172.21.15.94:0/41686949 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb2877fe640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2881031d0 0x7fb288198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb287fff640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb28810c510 0x7fb28819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb286ffd640 1 -- 172.21.15.94:0/41686949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2881031d0 msgr2=0x7fb288198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb286ffd640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2881031d0 0x7fb288198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb286ffd640 1 -- 172.21.15.94:0/41686949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb28810c510 msgr2=0x7fb28819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb286ffd640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb28810c510 0x7fb28819cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb286ffd640 1 -- 172.21.15.94:0/41686949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28819d720 con 0x7fb288103af0 2024-04-21T08:42:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb286ffd640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb288103af0 0x7fb288198cc0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb27800c890 tx=0x7fb27800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb284ff9640 1 -- 172.21.15.94:0/41686949 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278004220 con 0x7fb288103af0 2024-04-21T08:42:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb284ff9640 1 -- 172.21.15.94:0/41686949 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2780043c0 con 0x7fb288103af0 2024-04-21T08:42:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb28dcf2640 1 -- 172.21.15.94:0/41686949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb28819da10 con 0x7fb288103af0 2024-04-21T08:42:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb284ff9640 1 -- 172.21.15.94:0/41686949 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278004a00 con 0x7fb288103af0 2024-04-21T08:42:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.588+0000 7fb28dcf2640 1 -- 172.21.15.94:0/41686949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2881a52a0 con 0x7fb288103af0 2024-04-21T08:42:14.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.590+0000 7fb284ff9640 1 -- 172.21.15.94:0/41686949 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fb2780079b0 con 0x7fb288103af0 2024-04-21T08:42:14.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb2909c2740 1 Processor -- start 2024-04-21T08:42:14.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb2909c2740 1 -- start start 2024-04-21T08:42:14.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 0x5559bf40e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf417430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf419e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb2909c2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5559bf3731f0 con 0x5559bf417a00 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb2909c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5559bf3104b0 con 0x5559bf40e4d0 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb2909c2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5559bf36f1b0 con 0x5559bf40ee80 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb284ff9640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2700a3d20 0x7fb2700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb286ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf419e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb286ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf419e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57730/0 (socket says 172.21.15.94:57730) 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb286ffd640 1 -- 172.21.15.94:0/991177186 learned_addr learned my addr 172.21.15.94:0/991177186 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb287fff640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2700a3d20 0x7fb2700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb2877fe640 1 --2- 172.21.15.94:0/991177186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf417430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb284ff9640 1 -- 172.21.15.94:0/41686949 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb2780d0970 con 0x7fb288103af0 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb286ffd640 1 -- 172.21.15.94:0/991177186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 msgr2=0x5559bf417430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb286ffd640 1 --2- 172.21.15.94:0/991177186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf417430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb286ffd640 1 -- 172.21.15.94:0/991177186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 msgr2=0x5559bf40e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb287fff640 1 --2- 172.21.15.94:0/991177186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 0x5559bf40e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb286ffd640 1 --2- 172.21.15.94:0/991177186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 0x5559bf40e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb286ffd640 1 -- 172.21.15.94:0/991177186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559bf3021b0 con 0x5559bf417a00 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.591+0000 7fb287fff640 1 --2- 172.21.15.94:0/991177186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 0x5559bf40e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.592+0000 7fb286ffd640 1 --2- 172.21.15.94:0/991177186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf419e50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb2780cdd90 tx=0x7fb278000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bece27bca6050bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.592+0000 7fb287fff640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2700a3d20 0x7fb2700a61e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb280005e30 tx=0x7fb280005da0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.592+0000 7fb26dffb640 1 -- 172.21.15.94:0/991177186 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27801d070 con 0x5559bf417a00 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.592+0000 7fb26dffb640 1 -- 172.21.15.94:0/991177186 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb27809cc30 con 0x5559bf417a00 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.592+0000 7fb26dffb640 1 -- 172.21.15.94:0/991177186 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278098ba0 con 0x5559bf417a00 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.593+0000 7fb2909c2740 1 -- 172.21.15.94:0/991177186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 msgr2=0x5559bf419e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.593+0000 7fb2909c2740 1 --2- 172.21.15.94:0/991177186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf419e50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb2780cdd90 tx=0x7fb278000bc0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.593+0000 7fb2909c2740 1 -- 172.21.15.94:0/991177186 shutdown_connections 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.593+0000 7fb2909c2740 1 --2- 172.21.15.94:0/991177186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf417430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.593+0000 7fb2909c2740 1 --2- 172.21.15.94:0/991177186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf419e50 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.593+0000 7fb2909c2740 1 --2- 172.21.15.94:0/991177186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 0x5559bf40e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.593+0000 7fb2909c2740 1 -- 172.21.15.94:0/991177186 >> 172.21.15.94:0/991177186 conn(0x5559bf408ba0 msgr2=0x5559bf408f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.593+0000 7fb2909c2740 1 -- 172.21.15.94:0/991177186 shutdown_connections 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.593+0000 7fb2909c2740 1 -- 172.21.15.94:0/991177186 wait complete. 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.593+0000 7fb2909c2740 1 -- start start 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb2909c2740 1 Processor -- start 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40e4d0 0x5559bf417380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40ee80 0x5559bf414710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf414d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb2909c2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5559bf41a910 con 0x5559bf417a00 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb2909c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5559bf32c500 con 0x5559bf40ee80 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb2909c2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5559bf41aa90 con 0x5559bf40e4d0 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb2909c2740 10 client.0 _ll_register_callbacks cb 0x5559bf2fab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb287fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf414d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb287fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf414d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57746/0 (socket says 172.21.15.94:57746) 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb287fff640 1 -- 172.21.15.94:0/2629145023 learned_addr learned my addr 172.21.15.94:0/2629145023 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb286ffd640 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40ee80 0x5559bf414710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb2877fe640 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40e4d0 0x5559bf417380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb287fff640 1 -- 172.21.15.94:0/2629145023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40e4d0 msgr2=0x5559bf417380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb287fff640 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40e4d0 0x5559bf417380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb287fff640 1 -- 172.21.15.94:0/2629145023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40ee80 msgr2=0x5559bf414710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb287fff640 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40ee80 0x5559bf414710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb287fff640 1 -- 172.21.15.94:0/2629145023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559bf0853f0 con 0x5559bf417a00 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb286ffd640 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40ee80 0x5559bf414710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb287fff640 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf414d20 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb280036000 tx=0x7fb28002c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df6363606eaa42d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.594+0000 7fb266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:14.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb266ffd640 1 -- 172.21.15.94:0/2629145023 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28004e070 con 0x5559bf417a00 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb266ffd640 1 -- 172.21.15.94:0/2629145023 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb280032850 con 0x5559bf417a00 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb2909c2740 1 -- 172.21.15.94:0/2629145023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb27809cc30 con 0x5559bf417a00 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb266ffd640 1 -- 172.21.15.94:0/2629145023 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb280039070 con 0x5559bf417a00 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb2909c2740 10 client.16467 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb2909c2740 1 -- 172.21.15.94:0/2629145023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5559bf14f7d0 con 0x5559bf417a00 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb266ffd640 1 -- 172.21.15.94:0/2629145023 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb27809cc30 con 0x5559bf417a00 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb266ffd640 1 -- 172.21.15.94:0/2629145023 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb280049070 con 0x5559bf417a00 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb2909c2740 10 client.16467 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb2909c2740 10 client.16467 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb2909c2740 20 client.16467 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb2909c2740 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5559bf427af0 0x5559bf429f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb2909c2740 4 client.16467 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.595+0000 7fb2909c2740 1 -- 172.21.15.94:0/2629145023 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559bf312930 con 0x5559bf427af0 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.596+0000 7fb2877fe640 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5559bf427af0 0x5559bf429f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.596+0000 7fb2877fe640 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5559bf427af0 0x5559bf429f40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fe63cd0c9acac3d6 server_cookie=976507cb7633ca29 in_seq=0 out_seq=0 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.596+0000 7fb266ffd640 10 client.16467 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb266ffd640 1 -- 172.21.15.94:0/2629145023 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5559bf312930 con 0x5559bf427af0 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb266ffd640 10 client.16467 handle_command_reply: tid=0 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 client.16467 shutdown 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 -- 172.21.15.94:0/2629145023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 msgr2=0x5559bf414d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf414d20 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb280036000 tx=0x7fb28002c230 comp rx=0 tx=0).stop 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 -- 172.21.15.94:0/2629145023 shutdown_connections 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5559bf427af0 0x5559bf429f40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf414d20 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb266ffd640 0 client.16467 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40ee80 0x5559bf414710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 --2- 172.21.15.94:0/2629145023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40e4d0 0x5559bf417380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 -- 172.21.15.94:0/2629145023 >> 172.21.15.94:0/2629145023 conn(0x5559bf408ba0 msgr2=0x5559bf40a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 -- 172.21.15.94:0/2629145023 shutdown_connections 2024-04-21T08:42:14.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 1 -- 172.21.15.94:0/2629145023 wait complete. 2024-04-21T08:42:14.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.597+0000 7fb2909c2740 20 client.16467 trim_cache size 0 max 16384 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb2909c2740 1 Processor -- start 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb2909c2740 1 -- start start 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 0x5559bf4237d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf416e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf417380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb2909c2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881b1d20 con 0x5559bf417a00 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb2909c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881af5f0 con 0x5559bf40e4d0 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb2909c2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881ad440 con 0x5559bf40ee80 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb287fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf417380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb287fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf417380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57750/0 (socket says 172.21.15.94:57750) 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb287fff640 1 -- 172.21.15.94:0/2006967528 learned_addr learned my addr 172.21.15.94:0/2006967528 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.621+0000 7fb286ffd640 1 --2- 172.21.15.94:0/2006967528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf416e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb2877fe640 1 --2- 172.21.15.94:0/2006967528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 0x5559bf4237d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb287fff640 1 -- 172.21.15.94:0/2006967528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 msgr2=0x5559bf416e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb287fff640 1 --2- 172.21.15.94:0/2006967528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf416e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb287fff640 1 -- 172.21.15.94:0/2006967528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 msgr2=0x5559bf4237d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb287fff640 1 --2- 172.21.15.94:0/2006967528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 0x5559bf4237d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb287fff640 1 -- 172.21.15.94:0/2006967528 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559bf4083f0 con 0x5559bf417a00 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb2877fe640 1 --2- 172.21.15.94:0/2006967528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 0x5559bf4237d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb287fff640 1 --2- 172.21.15.94:0/2006967528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf417380 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb280009ca0 tx=0x7fb280009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e81467d5872734 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb26d7fa640 1 -- 172.21.15.94:0/2006967528 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280063070 con 0x5559bf417a00 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb26d7fa640 1 -- 172.21.15.94:0/2006967528 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb280044070 con 0x5559bf417a00 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.622+0000 7fb26d7fa640 1 -- 172.21.15.94:0/2006967528 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28004e040 con 0x5559bf417a00 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 -- 172.21.15.94:0/2006967528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 msgr2=0x5559bf417380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 --2- 172.21.15.94:0/2006967528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf417380 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb280009ca0 tx=0x7fb280009ba0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 -- 172.21.15.94:0/2006967528 shutdown_connections 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 --2- 172.21.15.94:0/2006967528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf416e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 --2- 172.21.15.94:0/2006967528 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf417a00 0x5559bf417380 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 --2- 172.21.15.94:0/2006967528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf40e4d0 0x5559bf4237d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 -- 172.21.15.94:0/2006967528 >> 172.21.15.94:0/2006967528 conn(0x5559bf408ba0 msgr2=0x5559bf407c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 -- 172.21.15.94:0/2006967528 shutdown_connections 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 -- 172.21.15.94:0/2006967528 wait complete. 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 -- start start 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.623+0000 7fb2909c2740 1 Processor -- start 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf40e4d0 0x5559bf411480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf4119c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2909c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf417a00 0x5559bf411f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2909c2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881adab0 con 0x5559bf40e4d0 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2909c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881aef20 con 0x5559bf417a00 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2909c2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881ae180 con 0x5559bf40ee80 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2909c2740 10 client.0 _ll_register_callbacks cb 0x7fb280003d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2877fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf40e4d0 0x5559bf411480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2877fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf40e4d0 0x5559bf411480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57764/0 (socket says 172.21.15.94:57764) 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2877fe640 1 -- 172.21.15.94:0/682364713 learned_addr learned my addr 172.21.15.94:0/682364713 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb287fff640 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf417a00 0x5559bf411f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb286ffd640 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf4119c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2877fe640 1 -- 172.21.15.94:0/682364713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 msgr2=0x5559bf4119c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2877fe640 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf4119c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2877fe640 1 -- 172.21.15.94:0/682364713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf417a00 msgr2=0x5559bf411f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2877fe640 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf417a00 0x5559bf411f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2877fe640 1 -- 172.21.15.94:0/682364713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559bf48d180 con 0x5559bf40e4d0 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb286ffd640 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf4119c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb287fff640 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf417a00 0x5559bf411f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb2877fe640 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf40e4d0 0x5559bf411480 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb27c005780 tx=0x7fb27c01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa59b99cc7eed151 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb266ffd640 1 -- 172.21.15.94:0/682364713 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c012e50 con 0x5559bf40e4d0 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.624+0000 7fb266ffd640 1 -- 172.21.15.94:0/682364713 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb27c01b2c0 con 0x5559bf40e4d0 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb2909c2740 1 -- 172.21.15.94:0/682364713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb280044070 con 0x5559bf40e4d0 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb266ffd640 1 -- 172.21.15.94:0/682364713 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb27c02c050 con 0x5559bf40e4d0 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb2909c2740 10 client.16473 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb2909c2740 1 -- 172.21.15.94:0/682364713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5559bf48e430 con 0x5559bf40e4d0 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb266ffd640 1 -- 172.21.15.94:0/682364713 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb280044070 con 0x5559bf40e4d0 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb266ffd640 1 -- 172.21.15.94:0/682364713 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb27c024ad0 con 0x5559bf40e4d0 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb2909c2740 10 client.16473 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb2909c2740 10 client.16473 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:42:14.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb2909c2740 20 client.16473 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb2909c2740 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5559bf427af0 0x5559bf490c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb2909c2740 4 client.16473 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb2909c2740 1 -- 172.21.15.94:0/682364713 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5559bf380c10 con 0x5559bf427af0 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.625+0000 7fb286ffd640 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5559bf427af0 0x5559bf490c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb266ffd640 10 client.16473 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb286ffd640 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5559bf427af0 0x5559bf490c50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b61ccc25e0cd3cfa server_cookie=252a69fc727f81e9 in_seq=0 out_seq=0 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb266ffd640 1 -- 172.21.15.94:0/682364713 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5559bf380c10 con 0x5559bf427af0 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb266ffd640 10 client.16473 handle_command_reply: tid=0 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 client.16473 shutdown 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 -- 172.21.15.94:0/682364713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf40e4d0 msgr2=0x5559bf411480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf40e4d0 0x5559bf411480 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb27c005780 tx=0x7fb27c01bf90 comp rx=0 tx=0).stop 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 -- 172.21.15.94:0/682364713 shutdown_connections 2024-04-21T08:42:14.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559bf40ee80 0x5559bf4119c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5559bf427af0 0x5559bf490c50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559bf417a00 0x5559bf411f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 --2- 172.21.15.94:0/682364713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5559bf40e4d0 0x5559bf411480 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb266ffd640 0 client.16473 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:42:14.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 -- 172.21.15.94:0/682364713 >> 172.21.15.94:0/682364713 conn(0x5559bf408ba0 msgr2=0x5559bf409f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 -- 172.21.15.94:0/682364713 shutdown_connections 2024-04-21T08:42:14.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 1 -- 172.21.15.94:0/682364713 wait complete. 2024-04-21T08:42:14.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.626+0000 7fb2909c2740 20 client.16473 trim_cache size 0 max 16384 2024-04-21T08:42:14.638 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:42:14.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 -- 172.21.15.94:0/41686949 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2700a3d20 msgr2=0x7fb2700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2700a3d20 0x7fb2700a61e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb280005e30 tx=0x7fb280005da0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 -- 172.21.15.94:0/41686949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb288103af0 msgr2=0x7fb288198cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb288103af0 0x7fb288198cc0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb27800c890 tx=0x7fb27800cd50 comp rx=0 tx=0).stop 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 -- 172.21.15.94:0/41686949 shutdown_connections 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb2700a3d20 0x7fb2700a61e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb28810c510 0x7fb28819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb288103af0 0x7fb288198cc0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 --2- 172.21.15.94:0/41686949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2881031d0 0x7fb288198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 -- 172.21.15.94:0/41686949 >> 172.21.15.94:0/41686949 conn(0x7fb288077f50 msgr2=0x7fb2880fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 -- 172.21.15.94:0/41686949 shutdown_connections 2024-04-21T08:42:14.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.627+0000 7fb28dcf2640 1 -- 172.21.15.94:0/41686949 wait complete. 2024-04-21T08:42:14.642 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:42:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.678+0000 7ff2f97f4640 1 -- 172.21.15.94:0/2292245546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2f4106680 msgr2=0x7ff2f4106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.678+0000 7ff2f97f4640 1 --2- 172.21.15.94:0/2292245546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2f4106680 0x7ff2f4106a60 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4009900 tx=0x7ff2e402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.678+0000 7ff2f97f4640 1 -- 172.21.15.94:0/2292245546 shutdown_connections 2024-04-21T08:42:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.678+0000 7ff2f97f4640 1 --2- 172.21.15.94:0/2292245546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f4100f80 0x7ff2f410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.678+0000 7ff2f97f4640 1 --2- 172.21.15.94:0/2292245546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2f41005c0 0x7ff2f4100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.678+0000 7ff2f97f4640 1 --2- 172.21.15.94:0/2292245546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2f4106680 0x7ff2f4106a60 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.678+0000 7ff2f97f4640 1 -- 172.21.15.94:0/2292245546 >> 172.21.15.94:0/2292245546 conn(0x7ff2f40773e0 msgr2=0x7ff2f40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.678+0000 7ff2f97f4640 1 -- 172.21.15.94:0/2292245546 shutdown_connections 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.678+0000 7ff2f97f4640 1 -- 172.21.15.94:0/2292245546 wait complete. 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f97f4640 1 Processor -- start 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f97f4640 1 -- start start 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f97f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f41005c0 0x7ff2f4199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f97f4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2f4100f80 0x7ff2f4199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f97f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2f4106680 0x7ff2f41a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f97f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f4110830 con 0x7ff2f4106680 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f97f4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41106b0 con 0x7ff2f4100f80 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f97f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41109b0 con 0x7ff2f41005c0 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2f4106680 0x7ff2f41a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2f4106680 0x7ff2f41a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57794/0 (socket says 172.21.15.94:57794) 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f37fe640 1 -- 172.21.15.94:0/3821246669 learned_addr learned my addr 172.21.15.94:0/3821246669 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f2ffd640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f41005c0 0x7ff2f4199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2f4100f80 0x7ff2f4199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f37fe640 1 -- 172.21.15.94:0/3821246669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f41005c0 msgr2=0x7ff2f4199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f37fe640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f41005c0 0x7ff2f4199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f37fe640 1 -- 172.21.15.94:0/3821246669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2f4100f80 msgr2=0x7ff2f4199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f37fe640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2f4100f80 0x7ff2f4199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.679+0000 7ff2f37fe640 1 -- 172.21.15.94:0/3821246669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f41a4ad0 con 0x7ff2f4106680 2024-04-21T08:42:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.680+0000 7ff2f37fe640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2f4106680 0x7ff2f41a4590 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec00cce0 tx=0x7ff2ec00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.680+0000 7ff2f0ff9640 1 -- 172.21.15.94:0/3821246669 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec00f450 con 0x7ff2f4106680 2024-04-21T08:42:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.680+0000 7ff2f97f4640 1 -- 172.21.15.94:0/3821246669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f41a4ca0 con 0x7ff2f4106680 2024-04-21T08:42:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.680+0000 7ff2f0ff9640 1 -- 172.21.15.94:0/3821246669 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2ec00faa0 con 0x7ff2f4106680 2024-04-21T08:42:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.680+0000 7ff2f0ff9640 1 -- 172.21.15.94:0/3821246669 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec018b70 con 0x7ff2f4106680 2024-04-21T08:42:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.680+0000 7ff2f97f4640 1 -- 172.21.15.94:0/3821246669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f41a51f0 con 0x7ff2f4106680 2024-04-21T08:42:14.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.682+0000 7ff2f0ff9640 1 -- 172.21.15.94:0/3821246669 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff2ec018d10 con 0x7ff2f4106680 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.682+0000 7ff2fc3e5740 1 Processor -- start 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.682+0000 7ff2fc3e5740 1 -- start start 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67acc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67af690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5619d6709000 con 0x5619d67a3d10 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5619d66a5b20 con 0x5619d67ad240 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5619d6704fc0 con 0x5619d67a46c0 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2ebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2ebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67a40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57808/0 (socket says 172.21.15.94:57808) 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2ebfff640 1 -- 172.21.15.94:0/3691608124 learned_addr learned my addr 172.21.15.94:0/3691608124 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2f2ffd640 1 --2- 172.21.15.94:0/3691608124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/3691608124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2ebfff640 1 -- 172.21.15.94:0/3691608124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 msgr2=0x5619d67acc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/3691608124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67acc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2f0ff9640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2e00a3d20 0x7ff2e00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2ebfff640 1 -- 172.21.15.94:0/3691608124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 msgr2=0x5619d67af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/3691608124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2ebfff640 1 -- 172.21.15.94:0/3691608124 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619d66bfea0 con 0x5619d67a3d10 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2f37fe640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2e00a3d20 0x7ff2e00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.683+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/3691608124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67a40f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00bf90 tx=0x7ff2dc00b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881362c841293cef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.684+0000 7ff2f0ff9640 1 -- 172.21.15.94:0/3821246669 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff2ec0d1090 con 0x7ff2f4106680 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.684+0000 7ff2e97fa640 1 -- 172.21.15.94:0/3691608124 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc00e670 con 0x5619d67a3d10 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.684+0000 7ff2e97fa640 1 -- 172.21.15.94:0/3691608124 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2dc01a070 con 0x5619d67a3d10 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.684+0000 7ff2e97fa640 1 -- 172.21.15.94:0/3691608124 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc0128e0 con 0x5619d67a3d10 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.684+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3691608124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 msgr2=0x5619d67a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.684+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/3691608124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67a40f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00bf90 tx=0x7ff2dc00b470 comp rx=0 tx=0).stop 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.684+0000 7ff2f37fe640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2e00a3d20 0x7ff2e00a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec0ce4b0 tx=0x7ff2ec0920e0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3691608124 shutdown_connections 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/3691608124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67acc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/3691608124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/3691608124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67a40f0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3691608124 >> 172.21.15.94:0/3691608124 conn(0x5619d679e3e0 msgr2=0x5619d679e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3691608124 shutdown_connections 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3691608124 wait complete. 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 -- start start 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 Processor -- start 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67aa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5619d67b00c0 con 0x5619d67a3d10 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5619d66a3510 con 0x5619d67ad240 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5619d67b0240 con 0x5619d67a46c0 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2fc3e5740 10 client.0 _ll_register_callbacks cb 0x5619d66a2f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2ebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2ebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67aa170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57822/0 (socket says 172.21.15.94:57822) 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2ebfff640 1 -- 172.21.15.94:0/3620670774 learned_addr learned my addr 172.21.15.94:0/3620670774 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2f2ffd640 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.685+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2ebfff640 1 -- 172.21.15.94:0/3620670774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 msgr2=0x5619d67aa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67aa6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2ebfff640 1 -- 172.21.15.94:0/3620670774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 msgr2=0x5619d67aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67aabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2ebfff640 1 -- 172.21.15.94:0/3620670774 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619d64e7a20 con 0x5619d67a3d10 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67aa170 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc005c00 tx=0x7ff2dc005a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f31182d23c5ff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2da7fc640 1 -- 172.21.15.94:0/3620670774 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc005230 con 0x5619d67a3d10 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2da7fc640 1 -- 172.21.15.94:0/3620670774 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff2dc0053d0 con 0x5619d67a3d10 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3620670774 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2dc01a070 con 0x5619d67a3d10 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2da7fc640 1 -- 172.21.15.94:0/3620670774 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff2dc032050 con 0x5619d67a3d10 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2fc3e5740 10 client.16485 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3620670774 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5619d65481b0 con 0x5619d67a3d10 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2da7fc640 1 -- 172.21.15.94:0/3620670774 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2dc01a070 con 0x5619d67a3d10 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2da7fc640 1 -- 172.21.15.94:0/3620670774 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff2dc004bd0 con 0x5619d67a3d10 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2fc3e5740 10 client.16485 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2fc3e5740 10 client.16485 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2fc3e5740 20 client.16485 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5619d67bd580 0x5619d67bf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2fc3e5740 4 client.16485 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:14.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.686+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3620670774 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619d6716a20 con 0x5619d67bd580 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.687+0000 7ff2f2ffd640 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5619d67bd580 0x5619d67bf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.687+0000 7ff2f2ffd640 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5619d67bd580 0x5619d67bf970 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=813a8c6dd0caf040 server_cookie=ebb951937853d311 in_seq=0 out_seq=0 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.687+0000 7ff2da7fc640 10 client.16485 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2da7fc640 1 -- 172.21.15.94:0/3620670774 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5619d6716a20 con 0x5619d67bd580 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2da7fc640 10 client.16485 handle_command_reply: tid=0 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 client.16485 shutdown 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3620670774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 msgr2=0x5619d67aa170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67aa170 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc005c00 tx=0x7ff2dc005a30 comp rx=0 tx=0).stop 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3620670774 shutdown_connections 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5619d67bd580 0x5619d67bf970 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67aa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67aabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2da7fc640 0 client.16485 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:14.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/3620670774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a3d10 0x5619d67aa170 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3620670774 >> 172.21.15.94:0/3620670774 conn(0x5619d679e3e0 msgr2=0x5619d679d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3620670774 shutdown_connections 2024-04-21T08:42:14.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/3620670774 wait complete. 2024-04-21T08:42:14.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.688+0000 7ff2fc3e5740 20 client.16485 trim_cache size 0 max 16384 2024-04-21T08:42:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2fc3e5740 1 Processor -- start 2024-04-21T08:42:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2fc3e5740 1 -- start start 2024-04-21T08:42:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a3d10 0x5619d67b9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a46c0 0x5619d67a9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41b1cb0 con 0x5619d67a46c0 2024-04-21T08:42:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41af580 con 0x5619d67ad240 2024-04-21T08:42:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41ad3d0 con 0x5619d67a3d10 2024-04-21T08:42:14.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2f2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a46c0 0x5619d67a9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2f2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a46c0 0x5619d67a9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57830/0 (socket says 172.21.15.94:57830) 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2f2ffd640 1 -- 172.21.15.94:0/894062951 learned_addr learned my addr 172.21.15.94:0/894062951 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/894062951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.706+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/894062951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a3d10 0x5619d67b9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.707+0000 7ff2f2ffd640 1 -- 172.21.15.94:0/894062951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a3d10 msgr2=0x5619d67b9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.707+0000 7ff2f2ffd640 1 --2- 172.21.15.94:0/894062951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a3d10 0x5619d67b9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.707+0000 7ff2f2ffd640 1 -- 172.21.15.94:0/894062951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 msgr2=0x5619d67c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.707+0000 7ff2f2ffd640 1 --2- 172.21.15.94:0/894062951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67c4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.707+0000 7ff2f2ffd640 1 -- 172.21.15.94:0/894062951 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619d63db310 con 0x5619d67a46c0 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.707+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/894062951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a3d10 0x5619d67b9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.707+0000 7ff2f2ffd640 1 --2- 172.21.15.94:0/894062951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a46c0 0x5619d67a9920 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff2e402c720 tx=0x7ff2e402cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d6e44906f32668 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.707+0000 7ff2e8ff9640 1 -- 172.21.15.94:0/894062951 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e404e070 con 0x5619d67a46c0 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.707+0000 7ff2e8ff9640 1 -- 172.21.15.94:0/894062951 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2e4005040 con 0x5619d67a46c0 2024-04-21T08:42:14.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.707+0000 7ff2e8ff9640 1 -- 172.21.15.94:0/894062951 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40049b0 con 0x5619d67a46c0 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.708+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/894062951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a46c0 msgr2=0x5619d67a9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.708+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/894062951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a46c0 0x5619d67a9920 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff2e402c720 tx=0x7ff2e402cb00 comp rx=0 tx=0).stop 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.708+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/894062951 shutdown_connections 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.708+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/894062951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67ad240 0x5619d67c4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.708+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/894062951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67a46c0 0x5619d67a9920 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.708+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/894062951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a3d10 0x5619d67b9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.708+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/894062951 >> 172.21.15.94:0/894062951 conn(0x5619d679e3e0 msgr2=0x5619d679f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.708+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/894062951 shutdown_connections 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.708+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/894062951 wait complete. 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.708+0000 7ff2fc3e5740 1 -- start start 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2fc3e5740 1 Processor -- start 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67a3d10 0x5619d67b1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2fc3e5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67ad240 0x5619d6822780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41ada40 con 0x5619d67ad240 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41aeeb0 con 0x5619d67a3d10 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2fc3e5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41ae110 con 0x5619d67a46c0 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2fc3e5740 10 client.0 _ll_register_callbacks cb 0x7ff2dc012af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2ebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67ad240 0x5619d6822780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2ebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67ad240 0x5619d6822780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57832/0 (socket says 172.21.15.94:57832) 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2ebfff640 1 -- 172.21.15.94:0/757093880 learned_addr learned my addr 172.21.15.94:0/757093880 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67a3d10 0x5619d67b1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2ebfff640 1 -- 172.21.15.94:0/757093880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 msgr2=0x5619d67b1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2ebfff640 1 -- 172.21.15.94:0/757093880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67a3d10 msgr2=0x5619d67b1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67a3d10 0x5619d67b1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2ebfff640 1 -- 172.21.15.94:0/757093880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619d6822d90 con 0x5619d67ad240 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2ebfff640 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67ad240 0x5619d6822780 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00e770 tx=0x7ff2dc00e870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffcef34ed870fd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.709+0000 7ff2da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2da7fc640 1 -- 172.21.15.94:0/757093880 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc03a020 con 0x5619d67ad240 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2da7fc640 1 -- 172.21.15.94:0/757093880 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff2dc00b380 con 0x5619d67ad240 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/757093880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2e4005040 con 0x5619d67ad240 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2da7fc640 1 -- 172.21.15.94:0/757093880 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff2dc045050 con 0x5619d67ad240 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2fc3e5740 10 client.16491 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/757093880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5619d6824070 con 0x5619d67ad240 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2da7fc640 1 -- 172.21.15.94:0/757093880 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2e4005040 con 0x5619d67ad240 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2da7fc640 1 -- 172.21.15.94:0/757093880 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff2dc0046f0 con 0x5619d67ad240 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2fc3e5740 10 client.16491 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2fc3e5740 10 client.16491 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2fc3e5740 20 client.16491 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:14.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5619d6824650 0x5619d6826a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2fc3e5740 4 client.16491 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/757093880 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x5619d6710890 con 0x5619d6824650 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.710+0000 7ff2f2ffd640 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5619d6824650 0x5619d6826a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2f2ffd640 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5619d6824650 0x5619d6826a70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=226bb4f990bea33a server_cookie=39653526e1751072 in_seq=0 out_seq=0 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2da7fc640 10 client.16491 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2da7fc640 1 -- 172.21.15.94:0/757093880 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+79 (crc 0 0 0) 0x5619d6710890 con 0x5619d6824650 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2da7fc640 10 client.16491 handle_command_reply: tid=0 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 client.16491 shutdown 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/757093880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67ad240 msgr2=0x5619d6822780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67ad240 0x5619d6822780 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00e770 tx=0x7ff2dc00e870 comp rx=0 tx=0).stop 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/757093880 shutdown_connections 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5619d6824650 0x5619d6826a70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619d67a46c0 0x5619d67b1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5619d67ad240 0x5619d6822780 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2da7fc640 0 client.16491 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:14.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 --2- 172.21.15.94:0/757093880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5619d67a3d10 0x5619d67b1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/757093880 >> 172.21.15.94:0/757093880 conn(0x5619d679e3e0 msgr2=0x5619d679fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/757093880 shutdown_connections 2024-04-21T08:42:14.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 1 -- 172.21.15.94:0/757093880 wait complete. 2024-04-21T08:42:14.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.711+0000 7ff2fc3e5740 20 client.16491 trim_cache size 0 max 16384 2024-04-21T08:42:14.718 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:42:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 24, 2024-04-21T08:42:14.718 INFO:teuthology.orchestra.run.smithi094.stdout: "leader": 24488, 2024-04-21T08:42:14.719 INFO:teuthology.orchestra.run.smithi094.stdout: "set_version": 0, 2024-04-21T08:42:14.719 INFO:teuthology.orchestra.run.smithi094.stdout: "sets": {} 2024-04-21T08:42:14.719 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:42:14.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 -- 172.21.15.94:0/3821246669 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2e00a3d20 msgr2=0x7ff2e00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2e00a3d20 0x7ff2e00a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec0ce4b0 tx=0x7ff2ec0920e0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 -- 172.21.15.94:0/3821246669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2f4106680 msgr2=0x7ff2f41a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2f4106680 0x7ff2f41a4590 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec00cce0 tx=0x7ff2ec00a620 comp rx=0 tx=0).stop 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 -- 172.21.15.94:0/3821246669 shutdown_connections 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2e00a3d20 0x7ff2e00a61e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2f4106680 0x7ff2f41a4590 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2f4100f80 0x7ff2f4199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 --2- 172.21.15.94:0/3821246669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f41005c0 0x7ff2f4199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 -- 172.21.15.94:0/3821246669 >> 172.21.15.94:0/3821246669 conn(0x7ff2f40773e0 msgr2=0x7ff2f4102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 -- 172.21.15.94:0/3821246669 shutdown_connections 2024-04-21T08:42:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.712+0000 7ff2f97f4640 1 -- 172.21.15.94:0/3821246669 wait complete. 2024-04-21T08:42:14.723 INFO:tasks.quiescer:done joining Quiescers 2024-04-21T08:42:14.723 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-21T08:42:14.734 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:42:14.772 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-21T08:42:14.784 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-21T08:42:14.784 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:42:14.784 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:42:14.784 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-21T08:42:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.783+0000 7f5b22ba4640 1 -- 172.21.15.94:0/4216704257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b1c1014a0 msgr2=0x7f5b1c106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.783+0000 7f5b22ba4640 1 --2- 172.21.15.94:0/4216704257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b1c1014a0 0x7f5b1c106bd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009900 tx=0x7f5b0802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.783+0000 7f5b22ba4640 1 -- 172.21.15.94:0/4216704257 shutdown_connections 2024-04-21T08:42:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.783+0000 7f5b22ba4640 1 --2- 172.21.15.94:0/4216704257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b1c1072b0 0x7f5b1c10eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.783+0000 7f5b22ba4640 1 --2- 172.21.15.94:0/4216704257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b1c1014a0 0x7f5b1c106bd0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.783+0000 7f5b22ba4640 1 --2- 172.21.15.94:0/4216704257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b1c100af0 0x7f5b1c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.783+0000 7f5b22ba4640 1 -- 172.21.15.94:0/4216704257 >> 172.21.15.94:0/4216704257 conn(0x7f5b1c0760e0 msgr2=0x7f5b1c0764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.783+0000 7f5b22ba4640 1 -- 172.21.15.94:0/4216704257 shutdown_connections 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.783+0000 7f5b22ba4640 1 -- 172.21.15.94:0/4216704257 wait complete. 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.783+0000 7f5b22ba4640 1 Processor -- start 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b22ba4640 1 -- start start 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b22ba4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b1c100af0 0x7f5b1c19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b22ba4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b1c1072b0 0x7f5b1c19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b22ba4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b1c1985e0 0x7f5b1c198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b22ba4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c112280 con 0x7f5b1c1985e0 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b22ba4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c112100 con 0x7f5b1c100af0 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b22ba4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c112400 con 0x7f5b1c1072b0 2024-04-21T08:42:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b21141640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b1c1985e0 0x7f5b1c198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b21141640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b1c1985e0 0x7f5b1c198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57850/0 (socket says 172.21.15.94:57850) 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b21141640 1 -- 172.21.15.94:0/2135704831 learned_addr learned my addr 172.21.15.94:0/2135704831 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b21141640 1 -- 172.21.15.94:0/2135704831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b1c1072b0 msgr2=0x7f5b1c19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b21141640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b1c1072b0 0x7f5b1c19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b20940640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b1c100af0 0x7f5b1c19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b21141640 1 -- 172.21.15.94:0/2135704831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b1c100af0 msgr2=0x7f5b1c19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b21141640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b1c100af0 0x7f5b1c19e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b21141640 1 -- 172.21.15.94:0/2135704831 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1c199100 con 0x7f5b1c1985e0 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b21141640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b1c1985e0 0x7f5b1c198a90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400eaa0 tx=0x7f5b1400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b19ffb640 1 -- 172.21.15.94:0/2135704831 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1400cdb0 con 0x7f5b1c1985e0 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b19ffb640 1 -- 172.21.15.94:0/2135704831 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b14005020 con 0x7f5b1c1985e0 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b19ffb640 1 -- 172.21.15.94:0/2135704831 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b140116d0 con 0x7f5b1c1985e0 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.784+0000 7f5b22ba4640 1 -- 172.21.15.94:0/2135704831 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b1c1993f0 con 0x7f5b1c1985e0 2024-04-21T08:42:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.785+0000 7f5b22ba4640 1 -- 172.21.15.94:0/2135704831 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b1c1a55f0 con 0x7f5b1c1985e0 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.787+0000 7f5b19ffb640 1 -- 172.21.15.94:0/2135704831 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5b140051d0 con 0x7f5b1c1985e0 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b25793740 1 Processor -- start 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b25793740 1 -- start start 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b19ffb640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b0c0a3d20 0x7f5b0c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b25793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 0x5645e9a888b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b25793740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a88e80 0x5645e9a91430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b25793740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a91a00 0x5645e9a93e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b25793740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5645e99ed1f0 con 0x5645e9a88e80 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b25793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5645e998a4b0 con 0x5645e9a884d0 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b25793740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5645e99e91b0 con 0x5645e9a91a00 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b0c0a3d20 0x7f5b0c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b19ffb640 1 -- 172.21.15.94:0/2135704831 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5b140d0dd0 con 0x7f5b1c1985e0 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b1bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a88e80 0x5645e9a91430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b1bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a88e80 0x5645e9a91430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57858/0 (socket says 172.21.15.94:57858) 2024-04-21T08:42:14.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b1bfff640 1 -- 172.21.15.94:0/1941625951 learned_addr learned my addr 172.21.15.94:0/1941625951 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b20940640 1 --2- 172.21.15.94:0/1941625951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 0x5645e9a888b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b21141640 1 --2- 172.21.15.94:0/1941625951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a91a00 0x5645e9a93e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b1bfff640 1 -- 172.21.15.94:0/1941625951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a91a00 msgr2=0x5645e9a93e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/1941625951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a91a00 0x5645e9a93e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b1bfff640 1 -- 172.21.15.94:0/1941625951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 msgr2=0x5645e9a888b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/1941625951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 0x5645e9a888b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.788+0000 7f5b1bfff640 1 -- 172.21.15.94:0/1941625951 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645e997c1b0 con 0x5645e9a88e80 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.789+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b0c0a3d20 0x7f5b0c0a61e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009900 tx=0x7f5b08005e20 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.789+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/1941625951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a88e80 0x5645e9a91430 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009230 tx=0x7f5b08044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98bf9c23b44bf6a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.789+0000 7f5b06ffd640 1 -- 172.21.15.94:0/1941625951 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08051070 con 0x5645e9a88e80 2024-04-21T08:42:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.789+0000 7f5b06ffd640 1 -- 172.21.15.94:0/1941625951 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b08004b70 con 0x5645e9a88e80 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.789+0000 7f5b06ffd640 1 -- 172.21.15.94:0/1941625951 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08042bd0 con 0x5645e9a88e80 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.789+0000 7f5b25793740 1 -- 172.21.15.94:0/1941625951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a88e80 msgr2=0x5645e9a91430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.789+0000 7f5b25793740 1 --2- 172.21.15.94:0/1941625951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a88e80 0x5645e9a91430 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009230 tx=0x7f5b08044000 comp rx=0 tx=0).stop 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.789+0000 7f5b25793740 1 -- 172.21.15.94:0/1941625951 shutdown_connections 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.789+0000 7f5b25793740 1 --2- 172.21.15.94:0/1941625951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a91a00 0x5645e9a93e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.789+0000 7f5b25793740 1 --2- 172.21.15.94:0/1941625951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a88e80 0x5645e9a91430 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 --2- 172.21.15.94:0/1941625951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 0x5645e9a888b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 -- 172.21.15.94:0/1941625951 >> 172.21.15.94:0/1941625951 conn(0x5645e9a82ba0 msgr2=0x5645e9a82f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 -- 172.21.15.94:0/1941625951 shutdown_connections 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 -- 172.21.15.94:0/1941625951 wait complete. 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 -- start start 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 Processor -- start 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a884d0 0x5645e9a91340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 0x5645e9a8e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a8ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5645e9a94910 con 0x5645e9a91a00 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5645e99a6500 con 0x5645e9a88e80 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5645e9a94a90 con 0x5645e9a884d0 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b25793740 10 client.0 _ll_register_callbacks cb 0x5645e9974b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b21141640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a8ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b21141640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a8ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57874/0 (socket says 172.21.15.94:57874) 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b21141640 1 -- 172.21.15.94:0/3595689656 learned_addr learned my addr 172.21.15.94:0/3595689656 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.790+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 0x5645e9a8e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5b20940640 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a884d0 0x5645e9a91340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5b21141640 1 -- 172.21.15.94:0/3595689656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a884d0 msgr2=0x5645e9a91340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5b21141640 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a884d0 0x5645e9a91340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5b21141640 1 -- 172.21.15.94:0/3595689656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 msgr2=0x5645e9a8e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5b21141640 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 0x5645e9a8e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5b21141640 1 -- 172.21.15.94:0/3595689656 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645e96ff3f0 con 0x5645e9a91a00 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5b21141640 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a8ece0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f5b140ce1a0 tx=0x7f5b1409c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7a3e49dd395416 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5afbfff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5afbfff640 1 -- 172.21.15.94:0/3595689656 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1401f040 con 0x5645e9a91a00 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5afbfff640 1 -- 172.21.15.94:0/3595689656 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5b140054f0 con 0x5645e9a91a00 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5b25793740 1 -- 172.21.15.94:0/3595689656 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b08004b70 con 0x5645e9a91a00 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5afbfff640 1 -- 172.21.15.94:0/3595689656 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5b14013ae0 con 0x5645e9a91a00 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5b25793740 10 client.16503 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5b25793740 1 -- 172.21.15.94:0/3595689656 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5645e97c97d0 con 0x5645e9a91a00 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.791+0000 7f5afbfff640 1 -- 172.21.15.94:0/3595689656 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b08004b70 con 0x5645e9a91a00 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.792+0000 7f5afbfff640 1 -- 172.21.15.94:0/3595689656 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5b1400f040 con 0x5645e9a91a00 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.792+0000 7f5b25793740 10 client.16503 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.792+0000 7f5b25793740 10 client.16503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:14.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.792+0000 7f5b25793740 20 client.16503 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.792+0000 7f5b25793740 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5645e9aa1c80 0x5645e9aa4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.792+0000 7f5b25793740 4 client.16503 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.792+0000 7f5b25793740 1 -- 172.21.15.94:0/3595689656 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645e998c930 con 0x5645e9aa1c80 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.792+0000 7f5b20940640 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5645e9aa1c80 0x5645e9aa4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.792+0000 7f5b20940640 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5645e9aa1c80 0x5645e9aa4070 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff43dec584a18f36 server_cookie=d17732f30b8a7a97 in_seq=0 out_seq=0 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.792+0000 7f5afbfff640 10 client.16503 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5afbfff640 1 -- 172.21.15.94:0/3595689656 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5645e998c930 con 0x5645e9aa1c80 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5afbfff640 10 client.16503 handle_command_reply: tid=0 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 client.16503 shutdown 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 -- 172.21.15.94:0/3595689656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 msgr2=0x5645e9a8ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a8ece0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f5b140ce1a0 tx=0x7f5b1409c5f0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 -- 172.21.15.94:0/3595689656 shutdown_connections 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5645e9aa1c80 0x5645e9aa4070 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a8ece0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 0x5645e9a8e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5afbfff640 0 client.16503 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:14.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 --2- 172.21.15.94:0/3595689656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a884d0 0x5645e9a91340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 -- 172.21.15.94:0/3595689656 >> 172.21.15.94:0/3595689656 conn(0x5645e9a82ba0 msgr2=0x5645e9a848f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 -- 172.21.15.94:0/3595689656 shutdown_connections 2024-04-21T08:42:14.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 1 -- 172.21.15.94:0/3595689656 wait complete. 2024-04-21T08:42:14.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.793+0000 7f5b25793740 20 client.16503 trim_cache size 0 max 16384 2024-04-21T08:42:14.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b25793740 1 Processor -- start 2024-04-21T08:42:14.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b25793740 1 -- start start 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b25793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 0x5645e9a9d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b25793740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a88e80 0x5645e9a8d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b25793740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a90e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b25793740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c1b1d70 con 0x5645e9a91a00 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b25793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c1af640 con 0x5645e9a884d0 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b25793740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c1ad490 con 0x5645e9a88e80 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b21141640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a90e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b21141640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a90e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57880/0 (socket says 172.21.15.94:57880) 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b21141640 1 -- 172.21.15.94:0/1505704640 learned_addr learned my addr 172.21.15.94:0/1505704640 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/1505704640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a88e80 0x5645e9a8d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b20940640 1 --2- 172.21.15.94:0/1505704640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 0x5645e9a9d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b21141640 1 -- 172.21.15.94:0/1505704640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a88e80 msgr2=0x5645e9a8d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b21141640 1 --2- 172.21.15.94:0/1505704640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a88e80 0x5645e9a8d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b21141640 1 -- 172.21.15.94:0/1505704640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 msgr2=0x5645e9a9d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b21141640 1 --2- 172.21.15.94:0/1505704640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 0x5645e9a9d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b21141640 1 -- 172.21.15.94:0/1505704640 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645e9aa0820 con 0x5645e9a91a00 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.807+0000 7f5b20940640 1 --2- 172.21.15.94:0/1505704640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 0x5645e9a9d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.808+0000 7f5b21141640 1 --2- 172.21.15.94:0/1505704640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a90e00 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f5b14087330 tx=0x7f5b14005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72dd34cc661f249 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.808+0000 7f5b067fc640 1 -- 172.21.15.94:0/1505704640 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b14098c30 con 0x5645e9a91a00 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.808+0000 7f5b067fc640 1 -- 172.21.15.94:0/1505704640 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b1409c990 con 0x5645e9a91a00 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.808+0000 7f5b067fc640 1 -- 172.21.15.94:0/1505704640 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b140054f0 con 0x5645e9a91a00 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.808+0000 7f5b25793740 1 -- 172.21.15.94:0/1505704640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 msgr2=0x5645e9a90e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.808+0000 7f5b25793740 1 --2- 172.21.15.94:0/1505704640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a90e00 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f5b14087330 tx=0x7f5b14005890 comp rx=0 tx=0).stop 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 -- 172.21.15.94:0/1505704640 shutdown_connections 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 --2- 172.21.15.94:0/1505704640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a88e80 0x5645e9a8d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 --2- 172.21.15.94:0/1505704640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a91a00 0x5645e9a90e00 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 --2- 172.21.15.94:0/1505704640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a884d0 0x5645e9a9d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 -- 172.21.15.94:0/1505704640 >> 172.21.15.94:0/1505704640 conn(0x5645e9a82ba0 msgr2=0x5645e9a84880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 -- 172.21.15.94:0/1505704640 shutdown_connections 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 -- 172.21.15.94:0/1505704640 wait complete. 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 -- start start 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 Processor -- start 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a884d0 0x5645e9a956c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 0x5645e9a95c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a91a00 0x5645e9a96140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c1adb00 con 0x5645e9a884d0 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c1aef70 con 0x5645e9a88e80 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c1ae1d0 con 0x5645e9a91a00 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b25793740 10 client.0 _ll_register_callbacks cb 0x7f5b1409c4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.809+0000 7f5b20940640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a884d0 0x5645e9a956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b20940640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a884d0 0x5645e9a956c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57892/0 (socket says 172.21.15.94:57892) 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b20940640 1 -- 172.21.15.94:0/2322940147 learned_addr learned my addr 172.21.15.94:0/2322940147 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b21141640 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a91a00 0x5645e9a96140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 0x5645e9a95c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b20940640 1 -- 172.21.15.94:0/2322940147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a91a00 msgr2=0x5645e9a96140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b20940640 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a91a00 0x5645e9a96140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b20940640 1 -- 172.21.15.94:0/2322940147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 msgr2=0x5645e9a95c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b20940640 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 0x5645e9a95c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b20940640 1 -- 172.21.15.94:0/2322940147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645e9b070a0 con 0x5645e9a884d0 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 0x5645e9a95c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b20940640 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a884d0 0x5645e9a956c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x5645e9aa4770 tx=0x7f5b10005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab68de30b605c888 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5afbfff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5afbfff640 1 -- 172.21.15.94:0/2322940147 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b10004de0 con 0x5645e9a884d0 2024-04-21T08:42:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b25793740 1 -- 172.21.15.94:0/2322940147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b1409c990 con 0x5645e9a884d0 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5afbfff640 1 -- 172.21.15.94:0/2322940147 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5b1001b290 con 0x5645e9a884d0 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5afbfff640 1 -- 172.21.15.94:0/2322940147 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5b1002d050 con 0x5645e9a884d0 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b25793740 10 client.16509 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.810+0000 7f5b25793740 1 -- 172.21.15.94:0/2322940147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5645e9b083b0 con 0x5645e9a884d0 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5afbfff640 1 -- 172.21.15.94:0/2322940147 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b1409c990 con 0x5645e9a884d0 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5afbfff640 1 -- 172.21.15.94:0/2322940147 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5b1000f040 con 0x5645e9a884d0 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5b25793740 10 client.16509 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5b25793740 10 client.16509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5b25793740 20 client.16509 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5b25793740 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5645e9b08ae0 0x5645e9b0af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5b25793740 4 client.16509 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5b25793740 1 -- 172.21.15.94:0/2322940147 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5645e99fac10 con 0x5645e9b08ae0 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5645e9b08ae0 0x5645e9b0af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5afbfff640 10 client.16509 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:14.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.811+0000 7f5b1bfff640 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5645e9b08ae0 0x5645e9b0af00 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abf57c3dd38c6180 server_cookie=f1d6cdb1d35d55a8 in_seq=0 out_seq=0 2024-04-21T08:42:14.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5afbfff640 1 -- 172.21.15.94:0/2322940147 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5645e99fac10 con 0x5645e9b08ae0 2024-04-21T08:42:14.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5afbfff640 10 client.16509 handle_command_reply: tid=0 2024-04-21T08:42:14.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 client.16509 shutdown 2024-04-21T08:42:14.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 -- 172.21.15.94:0/2322940147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a884d0 msgr2=0x5645e9a956c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a884d0 0x5645e9a956c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x5645e9aa4770 tx=0x7f5b10005410 comp rx=0 tx=0).stop 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 -- 172.21.15.94:0/2322940147 shutdown_connections 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5645e9b08ae0 0x5645e9b0af00 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645e9a91a00 0x5645e9a96140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645e9a88e80 0x5645e9a95c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5afbfff640 0 client.16509 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 --2- 172.21.15.94:0/2322940147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5645e9a884d0 0x5645e9a956c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 -- 172.21.15.94:0/2322940147 >> 172.21.15.94:0/2322940147 conn(0x5645e9a82ba0 msgr2=0x5645e9a83ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 -- 172.21.15.94:0/2322940147 shutdown_connections 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 1 -- 172.21.15.94:0/2322940147 wait complete. 2024-04-21T08:42:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b25793740 20 client.16509 trim_cache size 0 max 16384 2024-04-21T08:42:14.820 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.812+0000 7f5b22ba4640 1 -- 172.21.15.94:0/2135704831 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b0c0a3d20 msgr2=0x7f5b0c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b0c0a3d20 0x7f5b0c0a61e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009900 tx=0x7f5b08005e20 comp rx=0 tx=0).stop 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 -- 172.21.15.94:0/2135704831 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b1c1985e0 msgr2=0x7f5b1c198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b1c1985e0 0x7f5b1c198a90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400eaa0 tx=0x7f5b1400ef60 comp rx=0 tx=0).stop 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 -- 172.21.15.94:0/2135704831 shutdown_connections 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b0c0a3d20 0x7f5b0c0a61e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b1c1072b0 0x7f5b1c19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b1c1985e0 0x7f5b1c198a90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 --2- 172.21.15.94:0/2135704831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b1c100af0 0x7f5b1c19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 -- 172.21.15.94:0/2135704831 >> 172.21.15.94:0/2135704831 conn(0x7f5b1c0760e0 msgr2=0x7f5b1c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 -- 172.21.15.94:0/2135704831 shutdown_connections 2024-04-21T08:42:14.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:14.813+0000 7f5b22ba4640 1 -- 172.21.15.94:0/2135704831 wait complete. 2024-04-21T08:42:15.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:15 smithi094 ceph-mon[27988]: pgmap v71: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 185 op/s 2024-04-21T08:42:15.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:15 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1342363858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:15.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:15 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1182517067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:15.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:15 smithi111 ceph-mon[31493]: pgmap v71: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 185 op/s 2024-04-21T08:42:15.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:15 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1342363858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:15.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:15 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1182517067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:15.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:15 smithi173 ceph-mon[31428]: pgmap v71: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 185 op/s 2024-04-21T08:42:15.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:15 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1342363858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:15.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:15 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1182517067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:15.824 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:42:15.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:15.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:42:15.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.938+0000 7f46470ce640 1 -- 172.21.15.94:0/2071107055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46401031e0 msgr2=0x7f46401035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.938+0000 7f46470ce640 1 --2- 172.21.15.94:0/2071107055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46401031e0 0x7f46401035c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4634009900 tx=0x7f463402bd60 comp rx=0 tx=0).stop 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.938+0000 7f46470ce640 1 -- 172.21.15.94:0/2071107055 shutdown_connections 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.938+0000 7f46470ce640 1 --2- 172.21.15.94:0/2071107055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4640103b00 0x7f464010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.938+0000 7f46470ce640 1 --2- 172.21.15.94:0/2071107055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464010c520 0x7f464010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.938+0000 7f46470ce640 1 --2- 172.21.15.94:0/2071107055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46401031e0 0x7f46401035c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.938+0000 7f46470ce640 1 -- 172.21.15.94:0/2071107055 >> 172.21.15.94:0/2071107055 conn(0x7f46400771d0 msgr2=0x7f46400775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.938+0000 7f46470ce640 1 -- 172.21.15.94:0/2071107055 shutdown_connections 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.938+0000 7f46470ce640 1 -- 172.21.15.94:0/2071107055 wait complete. 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f46470ce640 1 Processor -- start 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f46470ce640 1 -- start start 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f46470ce640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46401031e0 0x7f464019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f46470ce640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640103b00 0x7f464019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f46470ce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f464010c520 0x7f46401985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f46470ce640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4640112230 con 0x7f46401031e0 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f46470ce640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f46401120b0 con 0x7f4640103b00 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f46470ce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f46401123b0 con 0x7f464010c520 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f4644e6a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46401031e0 0x7f464019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f4644e6a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46401031e0 0x7f464019e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57908/0 (socket says 172.21.15.94:57908) 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f4644e6a640 1 -- 172.21.15.94:0/556889570 learned_addr learned my addr 172.21.15.94:0/556889570 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.939+0000 7f463ffff640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640103b00 0x7f464019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f4644e6a640 1 -- 172.21.15.94:0/556889570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f464010c520 msgr2=0x7f46401985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f4644e6a640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f464010c520 0x7f46401985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f4644e6a640 1 -- 172.21.15.94:0/556889570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640103b00 msgr2=0x7f464019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f4644e6a640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640103b00 0x7f464019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f4644e6a640 1 -- 172.21.15.94:0/556889570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4640198d50 con 0x7f46401031e0 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f463ffff640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640103b00 0x7f464019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f4644e6a640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46401031e0 0x7f464019e420 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4634005940 tx=0x7f463402d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f463dffb640 1 -- 172.21.15.94:0/556889570 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634043020 con 0x7f46401031e0 2024-04-21T08:42:15.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f463dffb640 1 -- 172.21.15.94:0/556889570 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f463402dd20 con 0x7f46401031e0 2024-04-21T08:42:15.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f463dffb640 1 -- 172.21.15.94:0/556889570 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634003b80 con 0x7f46401031e0 2024-04-21T08:42:15.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f46470ce640 1 -- 172.21.15.94:0/556889570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4640198fe0 con 0x7f46401031e0 2024-04-21T08:42:15.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.940+0000 7f46470ce640 1 -- 172.21.15.94:0/556889570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46401a52e0 con 0x7f46401031e0 2024-04-21T08:42:15.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.942+0000 7f463dffb640 1 -- 172.21.15.94:0/556889570 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4634046020 con 0x7f46401031e0 2024-04-21T08:42:15.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4649d95740 1 Processor -- start 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4649d95740 1 -- start start 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4649d95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 0x564dc07a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4649d95740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a2290 0x564dc07aa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4649d95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07aae10 0x564dc07ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4649d95740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564dc0705f00 con 0x564dc07a2290 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4649d95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564dc06a3590 con 0x564dc07aae10 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4649d95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564dc0701ec0 con 0x564dc07a18e0 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f463dffb640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f46300a3d20 0x7f46300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4644e6a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a2290 0x564dc07aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4644e6a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a2290 0x564dc07aa840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57912/0 (socket says 172.21.15.94:57912) 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f4644e6a640 1 -- 172.21.15.94:0/496825077 learned_addr learned my addr 172.21.15.94:0/496825077 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.943+0000 7f464566b640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f46300a3d20 0x7f46300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f464566b640 1 --2- 172.21.15.94:0/496825077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 0x564dc07a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f463dffb640 1 -- 172.21.15.94:0/556889570 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f46340f2e20 con 0x7f46401031e0 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f463ffff640 1 --2- 172.21.15.94:0/496825077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07aae10 0x564dc07ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f4644e6a640 1 -- 172.21.15.94:0/496825077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 msgr2=0x564dc07a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f4644e6a640 1 --2- 172.21.15.94:0/496825077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 0x564dc07a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f4644e6a640 1 -- 172.21.15.94:0/496825077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07aae10 msgr2=0x564dc07ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f4644e6a640 1 --2- 172.21.15.94:0/496825077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07aae10 0x564dc07ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f4644e6a640 1 -- 172.21.15.94:0/496825077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dc04e07e0 con 0x564dc07a2290 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f464566b640 1 --2- 172.21.15.94:0/496825077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 0x564dc07a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f4644e6a640 1 --2- 172.21.15.94:0/496825077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a2290 0x564dc07aa840 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f463402dfd0 tx=0x7f4634000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c3bd4308706595 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f464566b640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f46300a3d20 0x7f46300a61e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4638008fd0 tx=0x7f463800b250 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f462affd640 1 -- 172.21.15.94:0/496825077 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46340c2e40 con 0x564dc07a2290 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f462affd640 1 -- 172.21.15.94:0/496825077 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46340c7700 con 0x564dc07a2290 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.944+0000 7f462affd640 1 -- 172.21.15.94:0/496825077 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46340f03b0 con 0x564dc07a2290 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 -- 172.21.15.94:0/496825077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a2290 msgr2=0x564dc07aa840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 --2- 172.21.15.94:0/496825077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a2290 0x564dc07aa840 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f463402dfd0 tx=0x7f4634000bc0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 -- 172.21.15.94:0/496825077 shutdown_connections 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 --2- 172.21.15.94:0/496825077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07aae10 0x564dc07ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 --2- 172.21.15.94:0/496825077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a2290 0x564dc07aa840 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 --2- 172.21.15.94:0/496825077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 0x564dc07a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 -- 172.21.15.94:0/496825077 >> 172.21.15.94:0/496825077 conn(0x564dc079bfb0 msgr2=0x564dc079c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 -- 172.21.15.94:0/496825077 shutdown_connections 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 -- 172.21.15.94:0/496825077 wait complete. 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 -- start start 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.945+0000 7f4649d95740 1 Processor -- start 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f4649d95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 0x564dc07a7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f4649d95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a2290 0x564dc07a8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f4649d95740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc07a88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f4649d95740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564dc07adb40 con 0x564dc07aae10 2024-04-21T08:42:15.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f4649d95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564dc06bf580 con 0x564dc07a2290 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f4649d95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564dc07adcc0 con 0x564dc07a18e0 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f4649d95740 10 client.0 _ll_register_callbacks cb 0x564dc06c5650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f464566b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc07a88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f464566b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc07a88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57922/0 (socket says 172.21.15.94:57922) 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f464566b640 1 -- 172.21.15.94:0/1343362414 learned_addr learned my addr 172.21.15.94:0/1343362414 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f4644e6a640 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 0x564dc07a7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f464566b640 1 -- 172.21.15.94:0/1343362414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 msgr2=0x564dc07a7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f464566b640 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 0x564dc07a7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f463ffff640 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a2290 0x564dc07a8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f464566b640 1 -- 172.21.15.94:0/1343362414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a2290 msgr2=0x564dc07a8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f464566b640 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a2290 0x564dc07a8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f464566b640 1 -- 172.21.15.94:0/1343362414 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dc0540f70 con 0x564dc07aae10 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f464566b640 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc07a88a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f4638006630 tx=0x7f4638004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebcabd38e7aaca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:15.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f461ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f461ffff640 1 -- 172.21.15.94:0/1343362414 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638021070 con 0x564dc07aae10 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f461ffff640 1 -- 172.21.15.94:0/1343362414 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f463800f6f0 con 0x564dc07aae10 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.946+0000 7f4649d95740 1 -- 172.21.15.94:0/1343362414 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46340c7700 con 0x564dc07aae10 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f461ffff640 1 -- 172.21.15.94:0/1343362414 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f463802d050 con 0x564dc07aae10 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f4649d95740 10 client.16521 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f4649d95740 1 -- 172.21.15.94:0/1343362414 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564dc03d8170 con 0x564dc07aae10 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f461ffff640 1 -- 172.21.15.94:0/1343362414 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46340c7700 con 0x564dc07aae10 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f461ffff640 1 -- 172.21.15.94:0/1343362414 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f463800ea60 con 0x564dc07aae10 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f4649d95740 10 client.16521 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f4649d95740 10 client.16521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f4649d95740 20 client.16521 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f4649d95740 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564dc07bb0d0 0x564dc07bd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f4649d95740 4 client.16521 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f4649d95740 1 -- 172.21.15.94:0/1343362414 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564dc0713920 con 0x564dc07bb0d0 2024-04-21T08:42:15.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.947+0000 7f4644e6a640 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564dc07bb0d0 0x564dc07bd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.948+0000 7f4644e6a640 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564dc07bb0d0 0x564dc07bd550 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d828e37a683cdf20 server_cookie=a6344c7e4e454e18 in_seq=0 out_seq=0 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.948+0000 7f461ffff640 10 client.16521 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.948+0000 7f461ffff640 1 -- 172.21.15.94:0/1343362414 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564dc0713920 con 0x564dc07bb0d0 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.948+0000 7f461ffff640 10 client.16521 handle_command_reply: tid=0 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.948+0000 7f4649d95740 1 client.16521 shutdown 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.948+0000 7f4649d95740 1 -- 172.21.15.94:0/1343362414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 msgr2=0x564dc07a88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.948+0000 7f4649d95740 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc07a88a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f4638006630 tx=0x7f4638004b50 comp rx=0 tx=0).stop 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.949+0000 7f4649d95740 1 -- 172.21.15.94:0/1343362414 shutdown_connections 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.949+0000 7f4649d95740 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564dc07bb0d0 0x564dc07bd550 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.949+0000 7f4649d95740 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc07a88a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.949+0000 7f4649d95740 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a2290 0x564dc07a8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.949+0000 7f461ffff640 0 client.16521 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.949+0000 7f4649d95740 1 --2- 172.21.15.94:0/1343362414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a18e0 0x564dc07a7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.949+0000 7f4649d95740 1 -- 172.21.15.94:0/1343362414 >> 172.21.15.94:0/1343362414 conn(0x564dc079bfb0 msgr2=0x564dc079dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:15.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.949+0000 7f4649d95740 1 -- 172.21.15.94:0/1343362414 shutdown_connections 2024-04-21T08:42:15.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.949+0000 7f4649d95740 1 -- 172.21.15.94:0/1343362414 wait complete. 2024-04-21T08:42:15.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.949+0000 7f4649d95740 20 client.16521 trim_cache size 0 max 16384 2024-04-21T08:42:15.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.966+0000 7f4649d95740 1 Processor -- start 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.966+0000 7f4649d95740 1 -- start start 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4649d95740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a18e0 0x564dc07aa550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4649d95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a2290 0x564dc07b6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4649d95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07aae10 0x564dc07b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4649d95740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f46401b1e10 con 0x564dc07a18e0 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4649d95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f46401af6e0 con 0x564dc07a2290 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4649d95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f46401ad530 con 0x564dc07aae10 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4644e6a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a18e0 0x564dc07aa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4644e6a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a18e0 0x564dc07aa550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57924/0 (socket says 172.21.15.94:57924) 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4644e6a640 1 -- 172.21.15.94:0/833978828 learned_addr learned my addr 172.21.15.94:0/833978828 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4644e6a640 1 -- 172.21.15.94:0/833978828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07aae10 msgr2=0x564dc07b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:15.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4644e6a640 1 --2- 172.21.15.94:0/833978828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07aae10 0x564dc07b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4644e6a640 1 -- 172.21.15.94:0/833978828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a2290 msgr2=0x564dc07b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4644e6a640 1 --2- 172.21.15.94:0/833978828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a2290 0x564dc07b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4644e6a640 1 -- 172.21.15.94:0/833978828 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dc07a8d80 con 0x564dc07a18e0 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f4644e6a640 1 --2- 172.21.15.94:0/833978828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a18e0 0x564dc07aa550 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f463402bda0 tx=0x7f46340df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7fdd3bcf2bba524 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f462a7fc640 1 -- 172.21.15.94:0/833978828 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46340d7020 con 0x564dc07a18e0 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f462a7fc640 1 -- 172.21.15.94:0/833978828 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46340c7700 con 0x564dc07a18e0 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.967+0000 7f462a7fc640 1 -- 172.21.15.94:0/833978828 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46340bb100 con 0x564dc07a18e0 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 -- 172.21.15.94:0/833978828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a18e0 msgr2=0x564dc07aa550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 --2- 172.21.15.94:0/833978828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a18e0 0x564dc07aa550 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f463402bda0 tx=0x7f46340df040 comp rx=0 tx=0).stop 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 -- 172.21.15.94:0/833978828 shutdown_connections 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 --2- 172.21.15.94:0/833978828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07aae10 0x564dc07b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 --2- 172.21.15.94:0/833978828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a2290 0x564dc07b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 --2- 172.21.15.94:0/833978828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07a18e0 0x564dc07aa550 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 -- 172.21.15.94:0/833978828 >> 172.21.15.94:0/833978828 conn(0x564dc079bfb0 msgr2=0x564dc07a8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 -- 172.21.15.94:0/833978828 shutdown_connections 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 -- 172.21.15.94:0/833978828 wait complete. 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 -- start start 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.968+0000 7f4649d95740 1 Processor -- start 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f4649d95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a18e0 0x564dc081f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f4649d95740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a2290 0x564dc081f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f4649d95740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc0819650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f4649d95740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f46401adba0 con 0x564dc07aae10 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f4649d95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f46401af010 con 0x564dc07a18e0 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f4649d95740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f46401ae270 con 0x564dc07a2290 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f4649d95740 10 client.0 _ll_register_callbacks cb 0x7f4638006300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f464566b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc0819650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f464566b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc0819650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57934/0 (socket says 172.21.15.94:57934) 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f464566b640 1 -- 172.21.15.94:0/3127384070 learned_addr learned my addr 172.21.15.94:0/3127384070 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:15.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f4644e6a640 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a18e0 0x564dc081f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f463ffff640 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a2290 0x564dc081f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f464566b640 1 -- 172.21.15.94:0/3127384070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a2290 msgr2=0x564dc081f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f464566b640 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a2290 0x564dc081f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f464566b640 1 -- 172.21.15.94:0/3127384070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a18e0 msgr2=0x564dc081f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f464566b640 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a18e0 0x564dc081f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f464566b640 1 -- 172.21.15.94:0/3127384070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dc0819d40 con 0x564dc07aae10 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f464566b640 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc0819650 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4638006600 tx=0x7f4638006c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5191a011192bdd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f461ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f461ffff640 1 -- 172.21.15.94:0/3127384070 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638036060 con 0x564dc07aae10 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f4649d95740 1 -- 172.21.15.94:0/3127384070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46340c7700 con 0x564dc07aae10 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.969+0000 7f461ffff640 1 -- 172.21.15.94:0/3127384070 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f463801d8e0 con 0x564dc07aae10 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f461ffff640 1 -- 172.21.15.94:0/3127384070 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f463803d050 con 0x564dc07aae10 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f4649d95740 10 client.16527 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:15.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f4649d95740 1 -- 172.21.15.94:0/3127384070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564dc0826920 con 0x564dc07aae10 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f461ffff640 1 -- 172.21.15.94:0/3127384070 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46340c7700 con 0x564dc07aae10 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f461ffff640 1 -- 172.21.15.94:0/3127384070 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f463800fc30 con 0x564dc07aae10 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f4649d95740 10 client.16527 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f4649d95740 10 client.16527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f4649d95740 20 client.16527 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f4649d95740 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564dc0826f80 0x564dc08293a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f4649d95740 4 client.16527 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f4649d95740 1 -- 172.21.15.94:0/3127384070 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564dc070d790 con 0x564dc0826f80 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.970+0000 7f4644e6a640 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564dc0826f80 0x564dc08293a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.971+0000 7f461ffff640 10 client.16527 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:15.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.971+0000 7f4644e6a640 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564dc0826f80 0x564dc08293a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbee4e1c84de9bd2 server_cookie=4eae1ef3d82d25d6 in_seq=0 out_seq=0 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f461ffff640 1 -- 172.21.15.94:0/3127384070 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564dc070d790 con 0x564dc0826f80 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f461ffff640 10 client.16527 handle_command_reply: tid=0 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 client.16527 shutdown 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 -- 172.21.15.94:0/3127384070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 msgr2=0x564dc0819650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc0819650 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4638006600 tx=0x7f4638006c30 comp rx=0 tx=0).stop 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 -- 172.21.15.94:0/3127384070 shutdown_connections 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564dc0826f80 0x564dc08293a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dc07a2290 0x564dc081f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564dc07aae10 0x564dc0819650 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 --2- 172.21.15.94:0/3127384070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564dc07a18e0 0x564dc081f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 -- 172.21.15.94:0/3127384070 >> 172.21.15.94:0/3127384070 conn(0x564dc079bfb0 msgr2=0x564dc079a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f461ffff640 0 client.16527 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 -- 172.21.15.94:0/3127384070 shutdown_connections 2024-04-21T08:42:15.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 1 -- 172.21.15.94:0/3127384070 wait complete. 2024-04-21T08:42:15.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.972+0000 7f4649d95740 20 client.16527 trim_cache size 0 max 16384 2024-04-21T08:42:15.978 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:42:15.978 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:42:15.978 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "91b1146c-16d8-496d-b415-cd0f9b678ec9", 2024-04-21T08:42:15.978 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:42:15.978 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:42:15.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 -- 172.21.15.94:0/556889570 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f46300a3d20 msgr2=0x7f46300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f46300a3d20 0x7f46300a61e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4638008fd0 tx=0x7f463800b250 comp rx=0 tx=0).stop 2024-04-21T08:42:15.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 -- 172.21.15.94:0/556889570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46401031e0 msgr2=0x7f464019e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46401031e0 0x7f464019e420 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4634005940 tx=0x7f463402d920 comp rx=0 tx=0).stop 2024-04-21T08:42:15.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 -- 172.21.15.94:0/556889570 shutdown_connections 2024-04-21T08:42:15.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f46300a3d20 0x7f46300a61e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f464010c520 0x7f46401985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640103b00 0x7f464019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 --2- 172.21.15.94:0/556889570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46401031e0 0x7f464019e420 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 -- 172.21.15.94:0/556889570 >> 172.21.15.94:0/556889570 conn(0x7f46400771d0 msgr2=0x7f46400fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:15.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 -- 172.21.15.94:0/556889570 shutdown_connections 2024-04-21T08:42:15.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:15.973+0000 7f46470ce640 1 -- 172.21.15.94:0/556889570 wait complete. 2024-04-21T08:42:15.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:42:15.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:42:16.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.085+0000 7f931b796640 1 -- 172.21.15.94:0/1354169839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9314103ae0 msgr2=0x7f931410bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.085+0000 7f931b796640 1 --2- 172.21.15.94:0/1354169839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9314103ae0 0x7f931410bfc0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9304009900 tx=0x7f930402bd60 comp rx=0 tx=0).stop 2024-04-21T08:42:16.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.085+0000 7f931b796640 1 -- 172.21.15.94:0/1354169839 shutdown_connections 2024-04-21T08:42:16.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.085+0000 7f931b796640 1 --2- 172.21.15.94:0/1354169839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f931410c500 0x7f931410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.085+0000 7f931b796640 1 --2- 172.21.15.94:0/1354169839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9314103ae0 0x7f931410bfc0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.085+0000 7f931b796640 1 --2- 172.21.15.94:0/1354169839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93141031c0 0x7f93141035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.085+0000 7f931b796640 1 -- 172.21.15.94:0/1354169839 >> 172.21.15.94:0/1354169839 conn(0x7f9314078000 msgr2=0x7f9314078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.085+0000 7f931b796640 1 -- 172.21.15.94:0/1354169839 shutdown_connections 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.085+0000 7f931b796640 1 -- 172.21.15.94:0/1354169839 wait complete. 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f931b796640 1 Processor -- start 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f931b796640 1 -- start start 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f931b796640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93141031c0 0x7f9314198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f931b796640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9314103ae0 0x7f9314198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f931b796640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f931410c500 0x7f931419cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f931b796640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9314112190 con 0x7f93141031c0 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f931b796640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9314112010 con 0x7f9314103ae0 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f931b796640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9314112310 con 0x7f931410c500 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9319532640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93141031c0 0x7f9314198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9319532640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93141031c0 0x7f9314198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57946/0 (socket says 172.21.15.94:57946) 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9319532640 1 -- 172.21.15.94:0/413890394 learned_addr learned my addr 172.21.15.94:0/413890394 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:16.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9318d31640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9314103ae0 0x7f9314198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9319d33640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f931410c500 0x7f931419cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9319532640 1 -- 172.21.15.94:0/413890394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f931410c500 msgr2=0x7f931419cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9319532640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f931410c500 0x7f931419cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9319532640 1 -- 172.21.15.94:0/413890394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9314103ae0 msgr2=0x7f9314198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9319532640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9314103ae0 0x7f9314198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9319532640 1 -- 172.21.15.94:0/413890394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931419d710 con 0x7f93141031c0 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f9319532640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93141031c0 0x7f9314198770 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f931000d950 tx=0x7f931000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f930a7fc640 1 -- 172.21.15.94:0/413890394 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9310017070 con 0x7f93141031c0 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.086+0000 7f931b796640 1 -- 172.21.15.94:0/413890394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f931419da00 con 0x7f93141031c0 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.087+0000 7f930a7fc640 1 -- 172.21.15.94:0/413890394 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9310009c30 con 0x7f93141031c0 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.087+0000 7f930a7fc640 1 -- 172.21.15.94:0/413890394 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931000f620 con 0x7f93141031c0 2024-04-21T08:42:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.087+0000 7f931b796640 1 -- 172.21.15.94:0/413890394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93141a5290 con 0x7f93141031c0 2024-04-21T08:42:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.089+0000 7f930a7fc640 1 -- 172.21.15.94:0/413890394 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9310010460 con 0x7f93141031c0 2024-04-21T08:42:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.089+0000 7f931e383740 1 Processor -- start 2024-04-21T08:42:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.089+0000 7f931e383740 1 -- start start 2024-04-21T08:42:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.089+0000 7f931e383740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629769960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.089+0000 7f931e383740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629769ea0 0x563629772540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.089+0000 7f931e383740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 0x563629774e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.089+0000 7f931e383740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5636296ce1e0 con 0x563629769580 2024-04-21T08:42:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.089+0000 7f931e383740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56362966b390 con 0x563629772a80 2024-04-21T08:42:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.089+0000 7f931e383740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636296ca1a0 con 0x563629769ea0 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.089+0000 7f930a7fc640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93000a3d20 0x7f93000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629769960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629769960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57962/0 (socket says 172.21.15.94:57962) 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 -- 172.21.15.94:0/3967093731 learned_addr learned my addr 172.21.15.94:0/3967093731 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93000a3d20 0x7f93000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f930a7fc640 1 -- 172.21.15.94:0/413890394 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f93100d0840 con 0x7f93141031c0 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319532640 1 --2- 172.21.15.94:0/3967093731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629769ea0 0x563629772540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9318d31640 1 --2- 172.21.15.94:0/3967093731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 0x563629774e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 -- 172.21.15.94:0/3967093731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629769ea0 msgr2=0x563629772540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 --2- 172.21.15.94:0/3967093731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629769ea0 0x563629772540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 -- 172.21.15.94:0/3967093731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 msgr2=0x563629774e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 --2- 172.21.15.94:0/3967093731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 0x563629774e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 -- 172.21.15.94:0/3967093731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636294aa7d0 con 0x563629769580 2024-04-21T08:42:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 --2- 172.21.15.94:0/3967093731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629769960 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f930c013e70 tx=0x7f930c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e32278384ba767 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f9319d33640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93000a3d20 0x7f93000a61e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f930c010ca0 tx=0x7f930c015000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f92ff7fe640 1 -- 172.21.15.94:0/3967093731 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c00ae10 con 0x563629769580 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f92ff7fe640 1 -- 172.21.15.94:0/3967093731 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f930c00bb70 con 0x563629769580 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.090+0000 7f92ff7fe640 1 -- 172.21.15.94:0/3967093731 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c00e440 con 0x563629769580 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 -- 172.21.15.94:0/3967093731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 msgr2=0x563629769960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 --2- 172.21.15.94:0/3967093731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629769960 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f930c013e70 tx=0x7f930c0107a0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 -- 172.21.15.94:0/3967093731 shutdown_connections 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 --2- 172.21.15.94:0/3967093731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629769ea0 0x563629772540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 --2- 172.21.15.94:0/3967093731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 0x563629774e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 --2- 172.21.15.94:0/3967093731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629769960 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 -- 172.21.15.94:0/3967093731 >> 172.21.15.94:0/3967093731 conn(0x563629763cb0 msgr2=0x563629764090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 -- 172.21.15.94:0/3967093731 shutdown_connections 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 -- 172.21.15.94:0/3967093731 wait complete. 2024-04-21T08:42:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 -- start start 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.091+0000 7f931e383740 1 Processor -- start 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f931e383740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x56362976fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f931e383740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629769ea0 0x563629770060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f931e383740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 0x5636297705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f931e383740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563629775810 con 0x563629769580 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f931e383740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5636296872f0 con 0x563629772a80 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f931e383740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563629775990 con 0x563629769ea0 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f931e383740 10 client.0 _ll_register_callbacks cb 0x563629694580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319532640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x56362976fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319532640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x56362976fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57976/0 (socket says 172.21.15.94:57976) 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319532640 1 -- 172.21.15.94:0/1648688034 learned_addr learned my addr 172.21.15.94:0/1648688034 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319d33640 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 0x5636297705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9318d31640 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629769ea0 0x563629770060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319532640 1 -- 172.21.15.94:0/1648688034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629769ea0 msgr2=0x563629770060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319532640 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629769ea0 0x563629770060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319532640 1 -- 172.21.15.94:0/1648688034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 msgr2=0x5636297705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319532640 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 0x5636297705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319532640 1 -- 172.21.15.94:0/1648688034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636293e03f0 con 0x563629769580 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319d33640 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 0x5636297705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f9319532640 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x56362976fb20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f931009c100 tx=0x7f931009be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a29eca68437f81e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f92fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.092+0000 7f92fcff9640 1 -- 172.21.15.94:0/1648688034 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93100b5060 con 0x563629769580 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f92fcff9640 1 -- 172.21.15.94:0/1648688034 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f93100d0380 con 0x563629769580 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f931e383740 1 -- 172.21.15.94:0/1648688034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5636296dbc00 con 0x563629769580 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f92fcff9640 1 -- 172.21.15.94:0/1648688034 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f93100bb070 con 0x563629769580 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f931e383740 10 client.16539 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f931e383740 1 -- 172.21.15.94:0/1648688034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5636293a00f0 con 0x563629769580 2024-04-21T08:42:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f92fcff9640 1 -- 172.21.15.94:0/1648688034 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5636296dbc00 con 0x563629769580 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f92fcff9640 1 -- 172.21.15.94:0/1648688034 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f931007e020 con 0x563629769580 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f931e383740 10 client.16539 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f931e383740 10 client.16539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f931e383740 20 client.16539 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f931e383740 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5636297829c0 0x563629784db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f931e383740 4 client.16539 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f931e383740 1 -- 172.21.15.94:0/1648688034 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636296d5a70 con 0x5636297829c0 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.093+0000 7f9318d31640 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5636297829c0 0x563629784db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f92fcff9640 10 client.16539 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f9318d31640 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5636297829c0 0x563629784db0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac62a2e9cde35202 server_cookie=4bb304ebfd458477 in_seq=0 out_seq=0 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f92fcff9640 1 -- 172.21.15.94:0/1648688034 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636296d5a70 con 0x5636297829c0 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f92fcff9640 10 client.16539 handle_command_reply: tid=0 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f931e383740 1 client.16539 shutdown 2024-04-21T08:42:16.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f931e383740 1 -- 172.21.15.94:0/1648688034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 msgr2=0x56362976fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f931e383740 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x56362976fb20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f931009c100 tx=0x7f931009be90 comp rx=0 tx=0).stop 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f931e383740 1 -- 172.21.15.94:0/1648688034 shutdown_connections 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f931e383740 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5636297829c0 0x563629784db0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f931e383740 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629769ea0 0x563629770060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f92fcff9640 0 client.16539 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f931e383740 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629772a80 0x5636297705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.094+0000 7f931e383740 1 --2- 172.21.15.94:0/1648688034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x56362976fb20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.095+0000 7f931e383740 1 -- 172.21.15.94:0/1648688034 >> 172.21.15.94:0/1648688034 conn(0x563629763cb0 msgr2=0x563629765a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.095+0000 7f931e383740 1 -- 172.21.15.94:0/1648688034 shutdown_connections 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.095+0000 7f931e383740 1 -- 172.21.15.94:0/1648688034 wait complete. 2024-04-21T08:42:16.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.095+0000 7f931e383740 20 client.16539 trim_cache size 0 max 16384 2024-04-21T08:42:16.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f931e383740 1 Processor -- start 2024-04-21T08:42:16.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f931e383740 1 -- start start 2024-04-21T08:42:16.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f931e383740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769580 0x563629772250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f931e383740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769ea0 0x56362977e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f931e383740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629772a80 0x5636297657f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f931e383740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f93141b1df0 con 0x563629769ea0 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f931e383740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f93141af6c0 con 0x563629769580 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f931e383740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f93141ad510 con 0x563629772a80 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9318d31640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769ea0 0x56362977e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9318d31640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769ea0 0x56362977e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57986/0 (socket says 172.21.15.94:57986) 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9318d31640 1 -- 172.21.15.94:0/3331158149 learned_addr learned my addr 172.21.15.94:0/3331158149 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9319532640 1 --2- 172.21.15.94:0/3331158149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769580 0x563629772250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9318d31640 1 -- 172.21.15.94:0/3331158149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629772a80 msgr2=0x5636297657f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9318d31640 1 --2- 172.21.15.94:0/3331158149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629772a80 0x5636297657f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9318d31640 1 -- 172.21.15.94:0/3331158149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769580 msgr2=0x563629772250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9318d31640 1 --2- 172.21.15.94:0/3331158149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769580 0x563629772250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9318d31640 1 -- 172.21.15.94:0/3331158149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563629785120 con 0x563629769ea0 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9319532640 1 --2- 172.21.15.94:0/3331158149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769580 0x563629772250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f9318d31640 1 --2- 172.21.15.94:0/3331158149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769ea0 0x56362977e6d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f930402c4d0 tx=0x7f930402c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1d576e6e3c44bd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.112+0000 7f92feffd640 1 -- 172.21.15.94:0/3331158149 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930404c070 con 0x563629769ea0 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f92feffd640 1 -- 172.21.15.94:0/3331158149 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9304004450 con 0x563629769ea0 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f92feffd640 1 -- 172.21.15.94:0/3331158149 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304004920 con 0x563629769ea0 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f931e383740 1 -- 172.21.15.94:0/3331158149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769ea0 msgr2=0x56362977e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f931e383740 1 --2- 172.21.15.94:0/3331158149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769ea0 0x56362977e6d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f930402c4d0 tx=0x7f930402c460 comp rx=0 tx=0).stop 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f931e383740 1 -- 172.21.15.94:0/3331158149 shutdown_connections 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f931e383740 1 --2- 172.21.15.94:0/3331158149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629772a80 0x5636297657f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f931e383740 1 --2- 172.21.15.94:0/3331158149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769ea0 0x56362977e6d0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f931e383740 1 --2- 172.21.15.94:0/3331158149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769580 0x563629772250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f931e383740 1 -- 172.21.15.94:0/3331158149 >> 172.21.15.94:0/3331158149 conn(0x563629763cb0 msgr2=0x5636297650e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f931e383740 1 -- 172.21.15.94:0/3331158149 shutdown_connections 2024-04-21T08:42:16.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f931e383740 1 -- 172.21.15.94:0/3331158149 wait complete. 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.113+0000 7f931e383740 1 -- start start 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f931e383740 1 Processor -- start 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f931e383740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629776ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f931e383740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769ea0 0x563629777220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f931e383740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629772a80 0x5636297e7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f931e383740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f93141adb80 con 0x563629769580 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f931e383740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f93141aeff0 con 0x563629769ea0 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f931e383740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f93141ae250 con 0x563629772a80 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f931e383740 10 client.0 _ll_register_callbacks cb 0x7f9310098770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9319532640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629776ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9319532640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629776ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58000/0 (socket says 172.21.15.94:58000) 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9319532640 1 -- 172.21.15.94:0/2937908713 learned_addr learned my addr 172.21.15.94:0/2937908713 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9318d31640 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769ea0 0x563629777220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9319d33640 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629772a80 0x5636297e7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9319532640 1 -- 172.21.15.94:0/2937908713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629772a80 msgr2=0x5636297e7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9319532640 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629772a80 0x5636297e7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9319532640 1 -- 172.21.15.94:0/2937908713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769ea0 msgr2=0x563629777220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9319532640 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769ea0 0x563629777220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9319532640 1 -- 172.21.15.94:0/2937908713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636297e8430 con 0x563629769580 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f9319532640 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629776ce0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f93100d06f0 tx=0x7f93100d0bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c10a8ba032a4efb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.114+0000 7f92fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f92fcff9640 1 -- 172.21.15.94:0/2937908713 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93100b5050 con 0x563629769580 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f92fcff9640 1 -- 172.21.15.94:0/2937908713 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f93100b5b80 con 0x563629769580 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f931e383740 1 -- 172.21.15.94:0/2937908713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9304004450 con 0x563629769580 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f92fcff9640 1 -- 172.21.15.94:0/2937908713 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9310020040 con 0x563629769580 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f931e383740 10 client.16545 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f931e383740 1 -- 172.21.15.94:0/2937908713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5636297e96f0 con 0x563629769580 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f92fcff9640 1 -- 172.21.15.94:0/2937908713 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9304004450 con 0x563629769580 2024-04-21T08:42:16.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f92fcff9640 1 -- 172.21.15.94:0/2937908713 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f931007e040 con 0x563629769580 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f931e383740 10 client.16545 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f931e383740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f931e383740 20 client.16545 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f931e383740 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5636297e9e40 0x5636297ec260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f931e383740 4 client.16545 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f931e383740 1 -- 172.21.15.94:0/2937908713 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563629651590 con 0x5636297e9e40 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.115+0000 7f9318d31640 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5636297e9e40 0x5636297ec260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f92fcff9640 10 client.16545 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f9318d31640 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5636297e9e40 0x5636297ec260 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89cb29cacb20139 server_cookie=b163f9956fa068dd in_seq=0 out_seq=0 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f92fcff9640 1 -- 172.21.15.94:0/2937908713 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563629651590 con 0x5636297e9e40 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f92fcff9640 10 client.16545 handle_command_reply: tid=0 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 client.16545 shutdown 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 -- 172.21.15.94:0/2937908713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 msgr2=0x563629776ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629776ce0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f93100d06f0 tx=0x7f93100d0bb0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 -- 172.21.15.94:0/2937908713 shutdown_connections 2024-04-21T08:42:16.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5636297e9e40 0x5636297ec260 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563629772a80 0x5636297e7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563629769ea0 0x563629777220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f92fcff9640 0 client.16545 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:16.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 --2- 172.21.15.94:0/2937908713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563629769580 0x563629776ce0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 -- 172.21.15.94:0/2937908713 >> 172.21.15.94:0/2937908713 conn(0x563629763cb0 msgr2=0x563629765970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:16.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 -- 172.21.15.94:0/2937908713 shutdown_connections 2024-04-21T08:42:16.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 1 -- 172.21.15.94:0/2937908713 wait complete. 2024-04-21T08:42:16.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.116+0000 7f931e383740 20 client.16545 trim_cache size 0 max 16384 2024-04-21T08:42:16.124 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:42:16.124 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:42:16.124 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:42:16.124 INFO:teuthology.orchestra.run.smithi094.stdout: "91b1146c-16d8-496d-b415-cd0f9b678ec9": { 2024-04-21T08:42:16.124 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:42:16.124 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "91b1146c-16d8-496d-b415-cd0f9b678ec9", 2024-04-21T08:42:16.124 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:42:16.124 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:42:16.124 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:42:16.124 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:42:16.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 -- 172.21.15.94:0/413890394 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93000a3d20 msgr2=0x7f93000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93000a3d20 0x7f93000a61e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f930c010ca0 tx=0x7f930c015000 comp rx=0 tx=0).stop 2024-04-21T08:42:16.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 -- 172.21.15.94:0/413890394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93141031c0 msgr2=0x7f9314198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93141031c0 0x7f9314198770 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f931000d950 tx=0x7f931000de10 comp rx=0 tx=0).stop 2024-04-21T08:42:16.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 -- 172.21.15.94:0/413890394 shutdown_connections 2024-04-21T08:42:16.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93000a3d20 0x7f93000a61e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f931410c500 0x7f931419cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9314103ae0 0x7f9314198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 --2- 172.21.15.94:0/413890394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93141031c0 0x7f9314198770 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 -- 172.21.15.94:0/413890394 >> 172.21.15.94:0/413890394 conn(0x7f9314078000 msgr2=0x7f93140fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:16.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 -- 172.21.15.94:0/413890394 shutdown_connections 2024-04-21T08:42:16.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:16.117+0000 7f931b796640 1 -- 172.21.15.94:0/413890394 wait complete. 2024-04-21T08:42:16.127 INFO:tasks.cephfs.filesystem:scrub status for tag:91b1146c-16d8-496d-b415-cd0f9b678ec9 - {'path': '/', 'tag': '91b1146c-16d8-496d-b415-cd0f9b678ec9', 'options': 'recursive,force'} 2024-04-21T08:42:17.312 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:17 smithi173 ceph-mon[31428]: pgmap v72: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 233 op/s 2024-04-21T08:42:17.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:17 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:42:17.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:17 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:17.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:17 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:42:17.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:17 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:17.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:17 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:42:17.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:17 smithi111 ceph-mon[31493]: pgmap v72: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 233 op/s 2024-04-21T08:42:17.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:17 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:42:17.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:17 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:17.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:17 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:42:17.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:17 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:17.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:17 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:42:17.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:17 smithi094 ceph-mon[27988]: pgmap v72: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 233 op/s 2024-04-21T08:42:17.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:17 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:42:17.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:17 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:17.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:17 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:42:17.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:17 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:17.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:17 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:42:19.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:19 smithi111 ceph-mon[31493]: pgmap v73: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 220 op/s 2024-04-21T08:42:19.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:19 smithi094 ceph-mon[27988]: pgmap v73: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 220 op/s 2024-04-21T08:42:19.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:19 smithi173 ceph-mon[31428]: pgmap v73: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 220 op/s 2024-04-21T08:42:21.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:21 smithi111 ceph-mon[31493]: pgmap v74: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.5 KiB/s wr, 224 op/s 2024-04-21T08:42:21.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:21 smithi094 ceph-mon[27988]: pgmap v74: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.5 KiB/s wr, 224 op/s 2024-04-21T08:42:21.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:21 smithi173 ceph-mon[31428]: pgmap v74: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.5 KiB/s wr, 224 op/s 2024-04-21T08:42:23.312 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:23 smithi094 ceph-mon[27988]: pgmap v75: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.2 KiB/s wr, 185 op/s 2024-04-21T08:42:23.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:23 smithi111 ceph-mon[31493]: pgmap v75: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.2 KiB/s wr, 185 op/s 2024-04-21T08:42:23.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:23 smithi173 ceph-mon[31428]: pgmap v75: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.2 KiB/s wr, 185 op/s 2024-04-21T08:42:25.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:25 smithi111 ceph-mon[31493]: pgmap v76: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.1 KiB/s wr, 187 op/s 2024-04-21T08:42:25.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:25 smithi173 ceph-mon[31428]: pgmap v76: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.1 KiB/s wr, 187 op/s 2024-04-21T08:42:25.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:25 smithi094 ceph-mon[27988]: pgmap v76: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.1 KiB/s wr, 187 op/s 2024-04-21T08:42:27.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:27 smithi111 ceph-mon[31493]: pgmap v77: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.9 KiB/s wr, 59 op/s 2024-04-21T08:42:27.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:27 smithi094 ceph-mon[27988]: pgmap v77: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.9 KiB/s wr, 59 op/s 2024-04-21T08:42:27.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:27 smithi173 ceph-mon[31428]: pgmap v77: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.9 KiB/s wr, 59 op/s 2024-04-21T08:42:29.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:29 smithi111 ceph-mon[31493]: pgmap v78: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-04-21T08:42:29.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:29 smithi094 ceph-mon[27988]: pgmap v78: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-04-21T08:42:29.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:29 smithi173 ceph-mon[31428]: pgmap v78: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-04-21T08:42:31.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:31 smithi111 ceph-mon[31493]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.9 KiB/s wr, 9 op/s 2024-04-21T08:42:31.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:31 smithi094 ceph-mon[27988]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.9 KiB/s wr, 9 op/s 2024-04-21T08:42:31.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:31 smithi173 ceph-mon[31428]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.9 KiB/s wr, 9 op/s 2024-04-21T08:42:33.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:33 smithi111 ceph-mon[31493]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-21T08:42:33.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:33 smithi094 ceph-mon[27988]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-21T08:42:33.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:33 smithi173 ceph-mon[31428]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-21T08:42:35.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:35 smithi111 ceph-mon[31493]: pgmap v81: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-21T08:42:35.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:35 smithi094 ceph-mon[27988]: pgmap v81: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-21T08:42:35.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:35 smithi173 ceph-mon[31428]: pgmap v81: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-21T08:42:37.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:37 smithi111 ceph-mon[31493]: pgmap v82: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-21T08:42:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:37 smithi094 ceph-mon[27988]: pgmap v82: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-21T08:42:37.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:37 smithi173 ceph-mon[31428]: pgmap v82: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-21T08:42:39.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:39 smithi111 ceph-mon[31493]: pgmap v83: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:42:39.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:39 smithi094 ceph-mon[27988]: pgmap v83: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:42:39.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:39 smithi173 ceph-mon[31428]: pgmap v83: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:42:41.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:41 smithi111 ceph-mon[31493]: pgmap v84: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:42:41.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:41 smithi094 ceph-mon[27988]: pgmap v84: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:42:41.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:41 smithi173 ceph-mon[31428]: pgmap v84: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:42:43.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:43 smithi111 ceph-mon[31493]: pgmap v85: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:42:43.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:43 smithi094 ceph-mon[27988]: pgmap v85: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:42:43.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:43 smithi173 ceph-mon[31428]: pgmap v85: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:42:45.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:45 smithi111 ceph-mon[31493]: pgmap v86: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:42:45.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:45 smithi094 ceph-mon[27988]: pgmap v86: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:42:45.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:45 smithi173 ceph-mon[31428]: pgmap v86: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:42:46.129 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:42:46.129 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:42:46.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.251+0000 7f932ec6d640 1 -- 172.21.15.94:0/732873630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93281031b0 msgr2=0x7f9328103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.251+0000 7f932ec6d640 1 --2- 172.21.15.94:0/732873630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93281031b0 0x7f9328103590 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f931c009900 tx=0x7f931c02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.251+0000 7f932ec6d640 1 -- 172.21.15.94:0/732873630 shutdown_connections 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.251+0000 7f932ec6d640 1 --2- 172.21.15.94:0/732873630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f932810c580 0x7f932810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.251+0000 7f932ec6d640 1 --2- 172.21.15.94:0/732873630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9328103ad0 0x7f932810bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.251+0000 7f932ec6d640 1 --2- 172.21.15.94:0/732873630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93281031b0 0x7f9328103590 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.251+0000 7f932ec6d640 1 -- 172.21.15.94:0/732873630 >> 172.21.15.94:0/732873630 conn(0x7f93280771d0 msgr2=0x7f93280775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.252+0000 7f932ec6d640 1 -- 172.21.15.94:0/732873630 shutdown_connections 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.252+0000 7f932ec6d640 1 -- 172.21.15.94:0/732873630 wait complete. 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.252+0000 7f932ec6d640 1 Processor -- start 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.252+0000 7f932ec6d640 1 -- start start 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.252+0000 7f932ec6d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93281031b0 0x7f9328100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.252+0000 7f932ec6d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103ad0 0x7f9328101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.252+0000 7f932ec6d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f932810c580 0x7f9328101880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.252+0000 7f932ec6d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f93281121d0 con 0x7f9328103ad0 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.252+0000 7f932ec6d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9328112050 con 0x7f932810c580 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.252+0000 7f932ec6d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9328112350 con 0x7f93281031b0 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9327fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103ad0 0x7f9328101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9327fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103ad0 0x7f9328101340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35588/0 (socket says 172.21.15.94:35588) 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9327fff640 1 -- 172.21.15.94:0/1173314123 learned_addr learned my addr 172.21.15.94:0/1173314123 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f932d20a640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f932810c580 0x7f9328101880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9327fff640 1 -- 172.21.15.94:0/1173314123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93281031b0 msgr2=0x7f9328100e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9327fff640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93281031b0 0x7f9328100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9327fff640 1 -- 172.21.15.94:0/1173314123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f932810c580 msgr2=0x7f9328101880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9327fff640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f932810c580 0x7f9328101880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9327fff640 1 -- 172.21.15.94:0/1173314123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93281a6c30 con 0x7f9328103ad0 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9327fff640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103ad0 0x7f9328101340 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f931400d950 tx=0x7f931400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9325ffb640 1 -- 172.21.15.94:0/1173314123 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314017030 con 0x7f9328103ad0 2024-04-21T08:42:46.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9325ffb640 1 -- 172.21.15.94:0/1173314123 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9314004020 con 0x7f9328103ad0 2024-04-21T08:42:46.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f932ec6d640 1 -- 172.21.15.94:0/1173314123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93281a6f20 con 0x7f9328103ad0 2024-04-21T08:42:46.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.253+0000 7f9325ffb640 1 -- 172.21.15.94:0/1173314123 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93140047a0 con 0x7f9328103ad0 2024-04-21T08:42:46.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.254+0000 7f932ec6d640 1 -- 172.21.15.94:0/1173314123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93281a73e0 con 0x7f9328103ad0 2024-04-21T08:42:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.256+0000 7f9325ffb640 1 -- 172.21.15.94:0/1173314123 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9314009a30 con 0x7f9328103ad0 2024-04-21T08:42:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f9331863740 1 Processor -- start 2024-04-21T08:42:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f9331863740 1 -- start start 2024-04-21T08:42:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f9325ffb640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93180a3d20 0x7f93180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f9331863740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b29d580 0x55b68b29d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f9331863740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29dea0 0x55b68b2a6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f9331863740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b2a6a80 0x55b68b2a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f9331863740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b68b2021e0 con 0x55b68b29d580 2024-04-21T08:42:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f9331863740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b68b19f390 con 0x55b68b2a6a80 2024-04-21T08:42:46.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f9331863740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b68b1fe1a0 con 0x55b68b29dea0 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932ca09640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b29d580 0x55b68b29d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932d20a640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93180a3d20 0x7f93180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932ca09640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b29d580 0x55b68b29d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35596/0 (socket says 172.21.15.94:35596) 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932ca09640 1 -- 172.21.15.94:0/3928077787 learned_addr learned my addr 172.21.15.94:0/3928077787 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f9325ffb640 1 -- 172.21.15.94:0/1173314123 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f93140d06f0 con 0x7f9328103ad0 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932ca09640 1 -- 172.21.15.94:0/3928077787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29dea0 msgr2=0x55b68b2a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932ca09640 1 --2- 172.21.15.94:0/3928077787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29dea0 0x55b68b2a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932ca09640 1 -- 172.21.15.94:0/3928077787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b2a6a80 msgr2=0x55b68b2a8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932d20a640 1 --2- 172.21.15.94:0/3928077787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b2a6a80 0x55b68b2a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932ca09640 1 --2- 172.21.15.94:0/3928077787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b2a6a80 0x55b68b2a8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932ca09640 1 -- 172.21.15.94:0/3928077787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b68afde7d0 con 0x55b68b29d580 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.257+0000 7f932d20a640 1 --2- 172.21.15.94:0/3928077787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b2a6a80 0x55b68b2a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:46.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f932ca09640 1 --2- 172.21.15.94:0/3928077787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b29d580 0x55b68b29d960 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f931c036040 tx=0x7f931c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f044b3d69f7254 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f932d20a640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93180a3d20 0x7f93180a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f93200092f0 tx=0x7f9320014910 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f9312ffd640 1 -- 172.21.15.94:0/3928077787 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c042020 con 0x55b68b29d580 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f9312ffd640 1 -- 172.21.15.94:0/3928077787 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931c02cd30 con 0x55b68b29d580 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f9312ffd640 1 -- 172.21.15.94:0/3928077787 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c003a50 con 0x55b68b29d580 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f9331863740 1 -- 172.21.15.94:0/3928077787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b29d580 msgr2=0x55b68b29d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f9331863740 1 --2- 172.21.15.94:0/3928077787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b29d580 0x55b68b29d960 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f931c036040 tx=0x7f931c005d50 comp rx=0 tx=0).stop 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f9331863740 1 -- 172.21.15.94:0/3928077787 shutdown_connections 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f9331863740 1 --2- 172.21.15.94:0/3928077787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29dea0 0x55b68b2a6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f9331863740 1 --2- 172.21.15.94:0/3928077787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b2a6a80 0x55b68b2a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f9331863740 1 --2- 172.21.15.94:0/3928077787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b29d580 0x55b68b29d960 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.258+0000 7f9331863740 1 -- 172.21.15.94:0/3928077787 >> 172.21.15.94:0/3928077787 conn(0x55b68b297cb0 msgr2=0x55b68b298090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 1 -- 172.21.15.94:0/3928077787 shutdown_connections 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 1 -- 172.21.15.94:0/3928077787 wait complete. 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 1 -- start start 2024-04-21T08:42:46.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 1 Processor -- start 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29d580 0x55b68b2a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29dea0 0x55b68b2a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b2a6a80 0x55b68b2a4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b68b2a9810 con 0x55b68b2a6a80 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b68b1bb2f0 con 0x55b68b29d580 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b68b2a9990 con 0x55b68b29dea0 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.259+0000 7f9331863740 10 client.0 _ll_register_callbacks cb 0x55b68b1c8580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932d20a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b2a6a80 0x55b68b2a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932d20a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b2a6a80 0x55b68b2a4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35604/0 (socket says 172.21.15.94:35604) 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932d20a640 1 -- 172.21.15.94:0/1259439333 learned_addr learned my addr 172.21.15.94:0/1259439333 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932ca09640 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29d580 0x55b68b2a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932d20a640 1 -- 172.21.15.94:0/1259439333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29dea0 msgr2=0x55b68b2a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932d20a640 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29dea0 0x55b68b2a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932d20a640 1 -- 172.21.15.94:0/1259439333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29d580 msgr2=0x55b68b2a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932d20a640 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29d580 0x55b68b2a3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932d20a640 1 -- 172.21.15.94:0/1259439333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b68af143f0 con 0x55b68b2a6a80 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932ca09640 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29d580 0x55b68b2a3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f932d20a640 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b2a6a80 0x55b68b2a4500 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9320012a50 tx=0x7f9320004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd141ae331aaec6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f9307fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f9307fff640 1 -- 172.21.15.94:0/1259439333 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932000fdb0 con 0x55b68b2a6a80 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f9307fff640 1 -- 172.21.15.94:0/1259439333 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f932001f460 con 0x55b68b2a6a80 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f9331863740 1 -- 172.21.15.94:0/1259439333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f931c02cd30 con 0x55b68b2a6a80 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f9307fff640 1 -- 172.21.15.94:0/1259439333 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9320031050 con 0x55b68b2a6a80 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f9331863740 10 client.16557 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.260+0000 7f9331863740 1 -- 172.21.15.94:0/1259439333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b68aed40f0 con 0x55b68b2a6a80 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.261+0000 7f9307fff640 1 -- 172.21.15.94:0/1259439333 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931c02cd30 con 0x55b68b2a6a80 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.261+0000 7f9307fff640 1 -- 172.21.15.94:0/1259439333 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f932000f040 con 0x55b68b2a6a80 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.261+0000 7f9331863740 10 client.16557 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.261+0000 7f9331863740 10 client.16557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.261+0000 7f9331863740 20 client.16557 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.261+0000 7f9331863740 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b68b2b69c0 0x55b68b2b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.261+0000 7f9331863740 4 client.16557 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.261+0000 7f9331863740 1 -- 172.21.15.94:0/1259439333 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b68b20fc00 con 0x55b68b2b69c0 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.261+0000 7f932ca09640 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b68b2b69c0 0x55b68b2b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.262+0000 7f932ca09640 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b68b2b69c0 0x55b68b2b8e10 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9150d5a7deb51b2a server_cookie=b714fd9e29dc344e in_seq=0 out_seq=0 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.262+0000 7f9307fff640 10 client.16557 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.262+0000 7f9307fff640 1 -- 172.21.15.94:0/1259439333 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b68b20fc00 con 0x55b68b2b69c0 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.262+0000 7f9307fff640 10 client.16557 handle_command_reply: tid=0 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.262+0000 7f9331863740 1 client.16557 shutdown 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 1 -- 172.21.15.94:0/1259439333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b2a6a80 msgr2=0x55b68b2a4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b2a6a80 0x55b68b2a4500 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9320012a50 tx=0x7f9320004ab0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 1 -- 172.21.15.94:0/1259439333 shutdown_connections 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b68b2b69c0 0x55b68b2b8e10 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29dea0 0x55b68b2a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b2a6a80 0x55b68b2a4500 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 1 --2- 172.21.15.94:0/1259439333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29d580 0x55b68b2a3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9307fff640 0 client.16557 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:46.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 1 -- 172.21.15.94:0/1259439333 >> 172.21.15.94:0/1259439333 conn(0x55b68b297cb0 msgr2=0x55b68b299a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 1 -- 172.21.15.94:0/1259439333 shutdown_connections 2024-04-21T08:42:46.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 1 -- 172.21.15.94:0/1259439333 wait complete. 2024-04-21T08:42:46.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.263+0000 7f9331863740 20 client.16557 trim_cache size 0 max 16384 2024-04-21T08:42:46.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f9331863740 1 Processor -- start 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f9331863740 1 -- start start 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f9331863740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f9331863740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29d580 0x55b68b2b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f9331863740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29dea0 0x55b68b2be6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f9331863740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f93281b3fd0 con 0x55b68b297cb0 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f9331863740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f93281b18a0 con 0x55b68b29dea0 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f9331863740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f93281af6f0 con 0x55b68b29d580 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f932ca09640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f932ca09640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2a61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35610/0 (socket says 172.21.15.94:35610) 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f932ca09640 1 -- 172.21.15.94:0/897837653 learned_addr learned my addr 172.21.15.94:0/897837653 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.280+0000 7f9327fff640 1 --2- 172.21.15.94:0/897837653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29d580 0x55b68b2b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.281+0000 7f932d20a640 1 --2- 172.21.15.94:0/897837653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29dea0 0x55b68b2be6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.281+0000 7f932ca09640 1 -- 172.21.15.94:0/897837653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29d580 msgr2=0x55b68b2b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.281+0000 7f932ca09640 1 --2- 172.21.15.94:0/897837653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29d580 0x55b68b2b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.281+0000 7f932ca09640 1 -- 172.21.15.94:0/897837653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29dea0 msgr2=0x55b68b2be6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.281+0000 7f932ca09640 1 --2- 172.21.15.94:0/897837653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29dea0 0x55b68b2be6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.281+0000 7f932ca09640 1 -- 172.21.15.94:0/897837653 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b68b2a2350 con 0x55b68b297cb0 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.281+0000 7f932ca09640 1 --2- 172.21.15.94:0/897837653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2a61b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f931c009230 tx=0x7f931c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f16fe2520c468a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.281+0000 7f93127fc640 1 -- 172.21.15.94:0/897837653 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c042040 con 0x55b68b297cb0 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.281+0000 7f93127fc640 1 -- 172.21.15.94:0/897837653 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931c050070 con 0x55b68b297cb0 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.281+0000 7f93127fc640 1 -- 172.21.15.94:0/897837653 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c034900 con 0x55b68b297cb0 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 -- 172.21.15.94:0/897837653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 msgr2=0x55b68b2a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 --2- 172.21.15.94:0/897837653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2a61b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f931c009230 tx=0x7f931c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 -- 172.21.15.94:0/897837653 shutdown_connections 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 --2- 172.21.15.94:0/897837653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29d580 0x55b68b2b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 --2- 172.21.15.94:0/897837653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29dea0 0x55b68b2be6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 --2- 172.21.15.94:0/897837653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2a61b0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 -- 172.21.15.94:0/897837653 >> 172.21.15.94:0/897837653 conn(0x55b68b2a26c0 msgr2=0x55b68b296d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 -- 172.21.15.94:0/897837653 shutdown_connections 2024-04-21T08:42:46.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 -- 172.21.15.94:0/897837653 wait complete. 2024-04-21T08:42:46.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 -- start start 2024-04-21T08:42:46.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 Processor -- start 2024-04-21T08:42:46.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2aee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29d580 0x55b68b2ab230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29dea0 0x55b68b2ab770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f93281afd60 con 0x55b68b297cb0 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f93281b11d0 con 0x55b68b29dea0 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f93281b0430 con 0x55b68b29d580 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f9331863740 10 client.0 _ll_register_callbacks cb 0x7f9320009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.282+0000 7f932ca09640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f932ca09640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2aee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35618/0 (socket says 172.21.15.94:35618) 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f932ca09640 1 -- 172.21.15.94:0/1112062007 learned_addr learned my addr 172.21.15.94:0/1112062007 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f932d20a640 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29dea0 0x55b68b2ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f9327fff640 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29d580 0x55b68b2ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f932ca09640 1 -- 172.21.15.94:0/1112062007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29d580 msgr2=0x55b68b2ab230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f932ca09640 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29d580 0x55b68b2ab230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f932ca09640 1 -- 172.21.15.94:0/1112062007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29dea0 msgr2=0x55b68b2ab770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f932ca09640 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29dea0 0x55b68b2ab770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f932ca09640 1 -- 172.21.15.94:0/1112062007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b68b2abde0 con 0x55b68b297cb0 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f932ca09640 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2aee60 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f931c009900 tx=0x7f931c005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f377dce436b2936d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f9307fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f9307fff640 1 -- 172.21.15.94:0/1112062007 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c004240 con 0x55b68b297cb0 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f9307fff640 1 -- 172.21.15.94:0/1112062007 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f931c0043e0 con 0x55b68b297cb0 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f9331863740 1 -- 172.21.15.94:0/1112062007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f931c050070 con 0x55b68b297cb0 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f9307fff640 1 -- 172.21.15.94:0/1112062007 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f931c05d050 con 0x55b68b297cb0 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f9331863740 10 client.16563 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f9331863740 1 -- 172.21.15.94:0/1112062007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b68b31de00 con 0x55b68b297cb0 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.283+0000 7f9307fff640 1 -- 172.21.15.94:0/1112062007 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931c050070 con 0x55b68b297cb0 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.284+0000 7f9307fff640 1 -- 172.21.15.94:0/1112062007 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f931c003a00 con 0x55b68b297cb0 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.284+0000 7f9331863740 10 client.16563 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.284+0000 7f9331863740 10 client.16563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.284+0000 7f9331863740 20 client.16563 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.284+0000 7f9331863740 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b68b2a6a80 0x55b68b320090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.284+0000 7f9331863740 4 client.16563 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.284+0000 7f9331863740 1 -- 172.21.15.94:0/1112062007 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b68b209a70 con 0x55b68b2a6a80 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.284+0000 7f9327fff640 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b68b2a6a80 0x55b68b320090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.284+0000 7f9307fff640 10 client.16563 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.284+0000 7f9327fff640 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b68b2a6a80 0x55b68b320090 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=292300594d4c5238 server_cookie=94b422322993dc87 in_seq=0 out_seq=0 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9307fff640 1 -- 172.21.15.94:0/1112062007 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b68b209a70 con 0x55b68b2a6a80 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9307fff640 10 client.16563 handle_command_reply: tid=0 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 client.16563 shutdown 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 -- 172.21.15.94:0/1112062007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 msgr2=0x55b68b2aee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2aee60 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f931c009900 tx=0x7f931c005e70 comp rx=0 tx=0).stop 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 -- 172.21.15.94:0/1112062007 shutdown_connections 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b68b2a6a80 0x55b68b320090 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b68b29d580 0x55b68b2ab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b68b29dea0 0x55b68b2ab770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 --2- 172.21.15.94:0/1112062007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b68b297cb0 0x55b68b2aee60 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9307fff640 0 client.16563 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 -- 172.21.15.94:0/1112062007 >> 172.21.15.94:0/1112062007 conn(0x55b68b2a26c0 msgr2=0x55b68b2964e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 -- 172.21.15.94:0/1112062007 shutdown_connections 2024-04-21T08:42:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 1 -- 172.21.15.94:0/1112062007 wait complete. 2024-04-21T08:42:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f9331863740 20 client.16563 trim_cache size 0 max 16384 2024-04-21T08:42:46.290 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:42:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:42:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:42:46.291 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:42:46.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f932ec6d640 1 -- 172.21.15.94:0/1173314123 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93180a3d20 msgr2=0x7f93180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f932ec6d640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93180a3d20 0x7f93180a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f93200092f0 tx=0x7f9320014910 comp rx=0 tx=0).stop 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.285+0000 7f932ec6d640 1 -- 172.21.15.94:0/1173314123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103ad0 msgr2=0x7f9328101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.286+0000 7f932ec6d640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103ad0 0x7f9328101340 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f931400d950 tx=0x7f931400de10 comp rx=0 tx=0).stop 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.286+0000 7f932ec6d640 1 -- 172.21.15.94:0/1173314123 shutdown_connections 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.286+0000 7f932ec6d640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93180a3d20 0x7f93180a61e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.286+0000 7f932ec6d640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f932810c580 0x7f9328101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.286+0000 7f932ec6d640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103ad0 0x7f9328101340 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.286+0000 7f932ec6d640 1 --2- 172.21.15.94:0/1173314123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93281031b0 0x7f9328100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.286+0000 7f932ec6d640 1 -- 172.21.15.94:0/1173314123 >> 172.21.15.94:0/1173314123 conn(0x7f93280771d0 msgr2=0x7f93280fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.286+0000 7f932ec6d640 1 -- 172.21.15.94:0/1173314123 shutdown_connections 2024-04-21T08:42:46.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.286+0000 7f932ec6d640 1 -- 172.21.15.94:0/1173314123 wait complete. 2024-04-21T08:42:46.296 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:42:46.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.397+0000 7ff114006640 1 -- 172.21.15.94:0/391543366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff10c100e80 msgr2=0x7ff10c10ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.397+0000 7ff114006640 1 --2- 172.21.15.94:0/391543366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff10c100e80 0x7ff10c10ede0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7ff104009930 tx=0x7ff10402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.397+0000 7ff114006640 1 -- 172.21.15.94:0/391543366 shutdown_connections 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.397+0000 7ff114006640 1 --2- 172.21.15.94:0/391543366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c1004c0 0x7ff10c100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.397+0000 7ff114006640 1 --2- 172.21.15.94:0/391543366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff10c100e80 0x7ff10c10ede0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.397+0000 7ff114006640 1 --2- 172.21.15.94:0/391543366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff10c106f50 0x7ff10c107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.397+0000 7ff114006640 1 -- 172.21.15.94:0/391543366 >> 172.21.15.94:0/391543366 conn(0x7ff10c077450 msgr2=0x7ff10c0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.397+0000 7ff114006640 1 -- 172.21.15.94:0/391543366 shutdown_connections 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.397+0000 7ff114006640 1 -- 172.21.15.94:0/391543366 wait complete. 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff114006640 1 Processor -- start 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff114006640 1 -- start start 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff114006640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff10c1004c0 0x7ff10c19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff114006640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff10c100e80 0x7ff10c19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff114006640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c106f50 0x7ff10c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff114006640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c1122e0 con 0x7ff10c100e80 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff114006640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c112160 con 0x7ff10c1004c0 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff114006640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c112460 con 0x7ff10c106f50 2024-04-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff1115a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff10c100e80 0x7ff10c19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff1115a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff10c100e80 0x7ff10c19e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35636/0 (socket says 172.21.15.94:35636) 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff1115a1640 1 -- 172.21.15.94:0/1425294778 learned_addr learned my addr 172.21.15.94:0/1425294778 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff1125a3640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c106f50 0x7ff10c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.398+0000 7ff111da2640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff10c1004c0 0x7ff10c19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff1115a1640 1 -- 172.21.15.94:0/1425294778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c106f50 msgr2=0x7ff10c1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff1115a1640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c106f50 0x7ff10c1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff1115a1640 1 -- 172.21.15.94:0/1425294778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff10c1004c0 msgr2=0x7ff10c19e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff1115a1640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff10c1004c0 0x7ff10c19e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff1115a1640 1 -- 172.21.15.94:0/1425294778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10c198d40 con 0x7ff10c100e80 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff1115a1640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff10c100e80 0x7ff10c19e950 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00b4a0 tx=0x7ff0fc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff102ffd640 1 -- 172.21.15.94:0/1425294778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc014070 con 0x7ff10c100e80 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff114006640 1 -- 172.21.15.94:0/1425294778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10c199030 con 0x7ff10c100e80 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff102ffd640 1 -- 172.21.15.94:0/1425294778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0fc010ac0 con 0x7ff10c100e80 2024-04-21T08:42:46.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff102ffd640 1 -- 172.21.15.94:0/1425294778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc017c30 con 0x7ff10c100e80 2024-04-21T08:42:46.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.399+0000 7ff114006640 1 -- 172.21.15.94:0/1425294778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff10c1a52d0 con 0x7ff10c100e80 2024-04-21T08:42:46.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.400+0000 7ff114006640 1 -- 172.21.15.94:0/1425294778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff10c069230 con 0x7ff10c100e80 2024-04-21T08:42:46.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.401+0000 7ff102ffd640 1 -- 172.21.15.94:0/1425294778 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff0fc017070 con 0x7ff10c100e80 2024-04-21T08:42:46.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.401+0000 7ff102ffd640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff0f80a3d20 0x7ff0f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.402+0000 7ff111da2640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff0f80a3d20 0x7ff0f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.402+0000 7ff102ffd640 1 -- 172.21.15.94:0/1425294778 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff0fc0cff10 con 0x7ff10c100e80 2024-04-21T08:42:46.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.402+0000 7ff111da2640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff0f80a3d20 0x7ff0f80a61e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff108005c40 tx=0x7ff10800a780 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.406+0000 7ff102ffd640 1 -- 172.21.15.94:0/1425294778 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0fc09c2e0 con 0x7ff10c100e80 2024-04-21T08:42:46.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.622+0000 7ff114006640 1 -- 172.21.15.94:0/1425294778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff10c199af0 con 0x7ff10c100e80 2024-04-21T08:42:46.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.623+0000 7ff102ffd640 1 -- 172.21.15.94:0/1425294778 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7ff0fc09bbe0 con 0x7ff10c100e80 2024-04-21T08:42:46.622 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:42:46.622 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:42:46.623 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:42:46.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 -- 172.21.15.94:0/1425294778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff0f80a3d20 msgr2=0x7ff0f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff0f80a3d20 0x7ff0f80a61e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff108005c40 tx=0x7ff10800a780 comp rx=0 tx=0).stop 2024-04-21T08:42:46.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 -- 172.21.15.94:0/1425294778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff10c100e80 msgr2=0x7ff10c19e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff10c100e80 0x7ff10c19e950 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00b4a0 tx=0x7ff0fc00b960 comp rx=0 tx=0).stop 2024-04-21T08:42:46.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 -- 172.21.15.94:0/1425294778 shutdown_connections 2024-04-21T08:42:46.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff0f80a3d20 0x7ff0f80a61e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff10c106f50 0x7ff10c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff10c100e80 0x7ff10c19e950 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 --2- 172.21.15.94:0/1425294778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff10c1004c0 0x7ff10c19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 -- 172.21.15.94:0/1425294778 >> 172.21.15.94:0/1425294778 conn(0x7ff10c077450 msgr2=0x7ff10c104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 -- 172.21.15.94:0/1425294778 shutdown_connections 2024-04-21T08:42:46.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.625+0000 7ff114006640 1 -- 172.21.15.94:0/1425294778 wait complete. 2024-04-21T08:42:46.635 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:42:46.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.737+0000 7f6f3518f640 1 -- 172.21.15.94:0/2928950734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f30103af0 msgr2=0x7f6f3010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.737+0000 7f6f3518f640 1 --2- 172.21.15.94:0/2928950734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f30103af0 0x7f6f3010bfd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009900 tx=0x7f6f2402bd60 comp rx=0 tx=0).stop 2024-04-21T08:42:46.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.738+0000 7f6f3518f640 1 -- 172.21.15.94:0/2928950734 shutdown_connections 2024-04-21T08:42:46.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.738+0000 7f6f3518f640 1 --2- 172.21.15.94:0/2928950734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f3010c510 0x7f6f3010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.738+0000 7f6f3518f640 1 --2- 172.21.15.94:0/2928950734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f30103af0 0x7f6f3010bfd0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.738+0000 7f6f3518f640 1 --2- 172.21.15.94:0/2928950734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f301031d0 0x7f6f301035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.738+0000 7f6f3518f640 1 -- 172.21.15.94:0/2928950734 >> 172.21.15.94:0/2928950734 conn(0x7f6f30077f50 msgr2=0x7f6f30078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.738+0000 7f6f3518f640 1 -- 172.21.15.94:0/2928950734 shutdown_connections 2024-04-21T08:42:46.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.738+0000 7f6f3518f640 1 -- 172.21.15.94:0/2928950734 wait complete. 2024-04-21T08:42:46.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.738+0000 7f6f3518f640 1 Processor -- start 2024-04-21T08:42:46.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.738+0000 7f6f3518f640 1 -- start start 2024-04-21T08:42:46.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f3518f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f301031d0 0x7f6f3019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f3518f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f30103af0 0x7f6f3019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f3518f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f3010c510 0x7f6f30198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f3518f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6f30112200 con 0x7f6f301031d0 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f3518f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6f30112080 con 0x7f6f30103af0 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f3518f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f30112380 con 0x7f6f3010c510 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f2ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f301031d0 0x7f6f3019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f2ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f301031d0 0x7f6f3019e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35642/0 (socket says 172.21.15.94:35642) 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/3700907245 learned_addr learned my addr 172.21.15.94:0/3700907245 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f2e59c640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f30103af0 0x7f6f3019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/3700907245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f3010c510 msgr2=0x7f6f30198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f3010c510 0x7f6f30198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/3700907245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f30103af0 msgr2=0x7f6f3019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f30103af0 0x7f6f3019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/3700907245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f30198f10 con 0x7f6f301031d0 2024-04-21T08:42:46.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f301031d0 0x7f6f3019e590 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000b4a0 tx=0x7f6f2000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f17fff640 1 -- 172.21.15.94:0/3700907245 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20014070 con 0x7f6f301031d0 2024-04-21T08:42:46.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f3518f640 1 -- 172.21.15.94:0/3700907245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f30199200 con 0x7f6f301031d0 2024-04-21T08:42:46.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f17fff640 1 -- 172.21.15.94:0/3700907245 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f20010ac0 con 0x7f6f301031d0 2024-04-21T08:42:46.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f17fff640 1 -- 172.21.15.94:0/3700907245 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20017c30 con 0x7f6f301031d0 2024-04-21T08:42:46.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.739+0000 7f6f3518f640 1 -- 172.21.15.94:0/3700907245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f301a5510 con 0x7f6f301031d0 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f17fff640 1 -- 172.21.15.94:0/3700907245 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f6f20017070 con 0x7f6f301031d0 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f37d7d740 1 Processor -- start 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f37d7d740 1 -- start start 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 0x55c7149b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149b1a00 0x55c7149b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c71490d1f0 con 0x55c7149a84d0 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c7148aa4b0 con 0x55c7149b1a00 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c7149091b0 con 0x55c7149a8e80 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f2f59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f2f59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149a88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35656/0 (socket says 172.21.15.94:35656) 2024-04-21T08:42:46.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.742+0000 7f6f2f59e640 1 -- 172.21.15.94:0/486447346 learned_addr learned my addr 172.21.15.94:0/486447346 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f17fff640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f180a3d20 0x7f6f180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/486447346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 0x55c7149b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f2e59c640 1 --2- 172.21.15.94:0/486447346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149b1a00 0x55c7149b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f2f59e640 1 -- 172.21.15.94:0/486447346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 msgr2=0x55c7149b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f2f59e640 1 --2- 172.21.15.94:0/486447346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 0x55c7149b1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f2f59e640 1 -- 172.21.15.94:0/486447346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149b1a00 msgr2=0x55c7149b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f2f59e640 1 --2- 172.21.15.94:0/486447346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149b1a00 0x55c7149b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f2f59e640 1 -- 172.21.15.94:0/486447346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c71489c1b0 con 0x55c7149a84d0 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f17fff640 1 -- 172.21.15.94:0/3700907245 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f6f200cff10 con 0x7f6f301031d0 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f2f59e640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f180a3d20 0x7f6f180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f2f59e640 1 --2- 172.21.15.94:0/486447346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149a88b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009f80 tx=0x7f6f2800d770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1883a34e72c638e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f157fa640 1 -- 172.21.15.94:0/486447346 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f28018070 con 0x55c7149a84d0 2024-04-21T08:42:46.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f157fa640 1 -- 172.21.15.94:0/486447346 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f2801edc0 con 0x55c7149a84d0 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.743+0000 7f6f157fa640 1 -- 172.21.15.94:0/486447346 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f28010c20 con 0x55c7149a84d0 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 -- 172.21.15.94:0/486447346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 msgr2=0x55c7149a88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f2f59e640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f180a3d20 0x7f6f180a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6f2801e000 tx=0x7f6f2801f000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/486447346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149a88b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009f80 tx=0x7f6f2800d770 comp rx=0 tx=0).stop 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 -- 172.21.15.94:0/486447346 shutdown_connections 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/486447346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 0x55c7149b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/486447346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149b1a00 0x55c7149b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/486447346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149a88b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 -- 172.21.15.94:0/486447346 >> 172.21.15.94:0/486447346 conn(0x55c7149a2ba0 msgr2=0x55c7149a2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 -- 172.21.15.94:0/486447346 shutdown_connections 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 -- 172.21.15.94:0/486447346 wait complete. 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 -- start start 2024-04-21T08:42:46.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 Processor -- start 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 0x55c7149ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149b1a00 0x55c7149aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c7149b4970 con 0x55c7149a84d0 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c7148c6500 con 0x55c7149b1a00 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.744+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c7149b4af0 con 0x55c7149a8e80 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f37d7d740 10 client.0 _ll_register_callbacks cb 0x55c714894b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f2ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f2ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35658/0 (socket says 172.21.15.94:35658) 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/1728296792 learned_addr learned my addr 172.21.15.94:0/1728296792 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/1728296792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 msgr2=0x55c7149ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 0x55c7149ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/1728296792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149b1a00 msgr2=0x55c7149aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149b1a00 0x55c7149aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/1728296792 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c71461f3f0 con 0x55c7149a84d0 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149b12c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6f2009bb70 tx=0x7f6f200a2880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8d3c54d0b45621 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f12ff5640 1 -- 172.21.15.94:0/1728296792 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2001e030 con 0x55c7149a84d0 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f12ff5640 1 -- 172.21.15.94:0/1728296792 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f6f200bc0f0 con 0x55c7149a84d0 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f37d7d740 1 -- 172.21.15.94:0/1728296792 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f2801edc0 con 0x55c7149a84d0 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f12ff5640 1 -- 172.21.15.94:0/1728296792 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6f2009bbe0 con 0x55c7149a84d0 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f37d7d740 10 client.16581 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f37d7d740 1 -- 172.21.15.94:0/1728296792 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c7146e97d0 con 0x55c7149a84d0 2024-04-21T08:42:46.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f12ff5640 1 -- 172.21.15.94:0/1728296792 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f2801edc0 con 0x55c7149a84d0 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.745+0000 7f6f12ff5640 1 -- 172.21.15.94:0/1728296792 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6f20098250 con 0x55c7149a84d0 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.746+0000 7f6f37d7d740 10 client.16581 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.746+0000 7f6f37d7d740 10 client.16581 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.746+0000 7f6f37d7d740 20 client.16581 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.746+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c7149c1c30 0x55c7149c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.746+0000 7f6f37d7d740 4 client.16581 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.746+0000 7f6f37d7d740 1 -- 172.21.15.94:0/1728296792 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7148ac930 con 0x55c7149c1c30 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.746+0000 7f6f2e59c640 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c7149c1c30 0x55c7149c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.746+0000 7f6f12ff5640 10 client.16581 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.746+0000 7f6f2e59c640 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c7149c1c30 0x55c7149c4080 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=22e3035d405836f7 server_cookie=7db74b42228091ec in_seq=0 out_seq=0 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f12ff5640 1 -- 172.21.15.94:0/1728296792 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7148ac930 con 0x55c7149c1c30 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f12ff5640 10 client.16581 handle_command_reply: tid=0 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f37d7d740 1 client.16581 shutdown 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f37d7d740 1 -- 172.21.15.94:0/1728296792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 msgr2=0x55c7149b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149b12c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6f2009bb70 tx=0x7f6f200a2880 comp rx=0 tx=0).stop 2024-04-21T08:42:46.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f37d7d740 1 -- 172.21.15.94:0/1728296792 shutdown_connections 2024-04-21T08:42:46.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c7149c1c30 0x55c7149c4080 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 0x55c7149ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149b1a00 0x55c7149aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f12ff5640 0 client.16581 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:42:46.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/1728296792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149b12c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.747+0000 7f6f37d7d740 1 -- 172.21.15.94:0/1728296792 >> 172.21.15.94:0/1728296792 conn(0x55c7149a2ba0 msgr2=0x55c7149a48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.748+0000 7f6f37d7d740 1 -- 172.21.15.94:0/1728296792 shutdown_connections 2024-04-21T08:42:46.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.748+0000 7f6f37d7d740 1 -- 172.21.15.94:0/1728296792 wait complete. 2024-04-21T08:42:46.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.748+0000 7f6f37d7d740 20 client.16581 trim_cache size 0 max 16384 2024-04-21T08:42:46.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f37d7d740 1 Processor -- start 2024-04-21T08:42:46.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f37d7d740 1 -- start start 2024-04-21T08:42:46.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149bd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a8e80 0x55c7149b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149b1a00 0x55c7149b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6f301b1df0 con 0x55c7149a84d0 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6f301af6c0 con 0x55c7149a8e80 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f301ad510 con 0x55c7149b1a00 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f2ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f2ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149bd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35664/0 (socket says 172.21.15.94:35664) 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/216866563 learned_addr learned my addr 172.21.15.94:0/216866563 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.761+0000 7f6f2f59e640 1 --2- 172.21.15.94:0/216866563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149b1a00 0x55c7149b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f2e59c640 1 --2- 172.21.15.94:0/216866563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a8e80 0x55c7149b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/216866563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149b1a00 msgr2=0x55c7149b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/216866563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149b1a00 0x55c7149b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/216866563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a8e80 msgr2=0x55c7149b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/216866563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a8e80 0x55c7149b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f2ed9d640 1 -- 172.21.15.94:0/216866563 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7149aef20 con 0x55c7149a84d0 2024-04-21T08:42:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f2e59c640 1 --2- 172.21.15.94:0/216866563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a8e80 0x55c7149b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/216866563 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149bd830 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f6f20098040 tx=0x7f6f20017650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9637d6e1ffea98 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f14ff9640 1 -- 172.21.15.94:0/216866563 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2001e070 con 0x55c7149a84d0 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f14ff9640 1 -- 172.21.15.94:0/216866563 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f200a4850 con 0x55c7149a84d0 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f14ff9640 1 -- 172.21.15.94:0/216866563 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2009b080 con 0x55c7149a84d0 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f37d7d740 1 -- 172.21.15.94:0/216866563 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 msgr2=0x55c7149bd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/216866563 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149bd830 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f6f20098040 tx=0x7f6f20017650 comp rx=0 tx=0).stop 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.762+0000 7f6f37d7d740 1 -- 172.21.15.94:0/216866563 shutdown_connections 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/216866563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149b1a00 0x55c7149b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/216866563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a8e80 0x55c7149b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/216866563 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149a84d0 0x55c7149bd830 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 -- 172.21.15.94:0/216866563 >> 172.21.15.94:0/216866563 conn(0x55c7149a2ba0 msgr2=0x55c7149a1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 -- 172.21.15.94:0/216866563 shutdown_connections 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 -- 172.21.15.94:0/216866563 wait complete. 2024-04-21T08:42:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 -- start start 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 Processor -- start 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a84d0 0x55c7149ba030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 0x55c7149b6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149b1a00 0x55c7149b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6f301adb80 con 0x55c7149b1a00 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6f301aeff0 con 0x55c7149a84d0 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f301ae250 con 0x55c7149a8e80 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f37d7d740 10 client.0 _ll_register_callbacks cb 0x7f6f2009b540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f2f59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149b1a00 0x55c7149b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f2f59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149b1a00 0x55c7149b7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35666/0 (socket says 172.21.15.94:35666) 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f2f59e640 1 -- 172.21.15.94:0/3549556941 learned_addr learned my addr 172.21.15.94:0/3549556941 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a84d0 0x55c7149ba030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f2f59e640 1 -- 172.21.15.94:0/3549556941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 msgr2=0x55c7149b6d10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f2f59e640 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 0x55c7149b6d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f2f59e640 1 -- 172.21.15.94:0/3549556941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a84d0 msgr2=0x55c7149ba030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.763+0000 7f6f2f59e640 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a84d0 0x55c7149ba030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f2f59e640 1 -- 172.21.15.94:0/3549556941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7149b7910 con 0x55c7149b1a00 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f2f59e640 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149b1a00 0x55c7149b7250 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x55c7149a2510 tx=0x7f6f28004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c68d1ffb9c3c754 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f12ff5640 1 -- 172.21.15.94:0/3549556941 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2802b070 con 0x55c7149b1a00 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f12ff5640 1 -- 172.21.15.94:0/3549556941 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f6f2801eca0 con 0x55c7149b1a00 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f37d7d740 1 -- 172.21.15.94:0/3549556941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f200a4850 con 0x55c7149b1a00 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f12ff5640 1 -- 172.21.15.94:0/3549556941 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6f28039050 con 0x55c7149b1a00 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f37d7d740 10 client.16587 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f37d7d740 1 -- 172.21.15.94:0/3549556941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c714a29020 con 0x55c7149b1a00 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f12ff5640 1 -- 172.21.15.94:0/3549556941 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f200a4850 con 0x55c7149b1a00 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f12ff5640 1 -- 172.21.15.94:0/3549556941 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6f28010990 con 0x55c7149b1a00 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f37d7d740 10 client.16587 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f37d7d740 10 client.16587 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:42:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f37d7d740 20 client.16587 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c714a29460 0x55c714a2b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f37d7d740 4 client.16587 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.764+0000 7f6f37d7d740 1 -- 172.21.15.94:0/3549556941 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c71491ac10 con 0x55c714a29460 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c714a29460 0x55c714a2b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f2ed9d640 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c714a29460 0x55c714a2b850 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d3dab8cc29e8678 server_cookie=c44207bc050eca25 in_seq=0 out_seq=0 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f12ff5640 10 client.16587 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f12ff5640 1 -- 172.21.15.94:0/3549556941 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c71491ac10 con 0x55c714a29460 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f12ff5640 10 client.16587 handle_command_reply: tid=0 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 client.16587 shutdown 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 -- 172.21.15.94:0/3549556941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149b1a00 msgr2=0x55c7149b7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149b1a00 0x55c7149b7250 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x55c7149a2510 tx=0x7f6f28004d70 comp rx=0 tx=0).stop 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 -- 172.21.15.94:0/3549556941 shutdown_connections 2024-04-21T08:42:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c714a29460 0x55c714a2b850 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7149a8e80 0x55c7149b6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f12ff5640 0 client.16587 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:42:46.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c7149b1a00 0x55c7149b7250 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 --2- 172.21.15.94:0/3549556941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c7149a84d0 0x55c7149ba030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 -- 172.21.15.94:0/3549556941 >> 172.21.15.94:0/3549556941 conn(0x55c7149a2ba0 msgr2=0x55c7149a1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 -- 172.21.15.94:0/3549556941 shutdown_connections 2024-04-21T08:42:46.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 1 -- 172.21.15.94:0/3549556941 wait complete. 2024-04-21T08:42:46.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.765+0000 7f6f37d7d740 20 client.16587 trim_cache size 0 max 16384 2024-04-21T08:42:46.771 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 -- 172.21.15.94:0/3700907245 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f180a3d20 msgr2=0x7f6f180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f180a3d20 0x7f6f180a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6f2801e000 tx=0x7f6f2801f000 comp rx=0 tx=0).stop 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 -- 172.21.15.94:0/3700907245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f301031d0 msgr2=0x7f6f3019e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f301031d0 0x7f6f3019e590 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000b4a0 tx=0x7f6f2000b960 comp rx=0 tx=0).stop 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 -- 172.21.15.94:0/3700907245 shutdown_connections 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6f180a3d20 0x7f6f180a61e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f3010c510 0x7f6f30198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f30103af0 0x7f6f3019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 --2- 172.21.15.94:0/3700907245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f301031d0 0x7f6f3019e590 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 -- 172.21.15.94:0/3700907245 >> 172.21.15.94:0/3700907245 conn(0x7f6f30077f50 msgr2=0x7f6f300fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 -- 172.21.15.94:0/3700907245 shutdown_connections 2024-04-21T08:42:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.766+0000 7f6f3518f640 1 -- 172.21.15.94:0/3700907245 wait complete. 2024-04-21T08:42:46.776 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:42:46.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.877+0000 7f8435893640 1 -- 172.21.15.94:0/3232397019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84301031e0 msgr2=0x7f84301035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.877+0000 7f8435893640 1 --2- 172.21.15.94:0/3232397019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84301031e0 0x7f84301035c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f8424009900 tx=0x7f842402bd60 comp rx=0 tx=0).stop 2024-04-21T08:42:46.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.877+0000 7f8435893640 1 -- 172.21.15.94:0/3232397019 shutdown_connections 2024-04-21T08:42:46.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.877+0000 7f8435893640 1 --2- 172.21.15.94:0/3232397019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8430103b00 0x7f843010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.877+0000 7f8435893640 1 --2- 172.21.15.94:0/3232397019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f843010c520 0x7f843010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.877+0000 7f8435893640 1 --2- 172.21.15.94:0/3232397019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84301031e0 0x7f84301035c0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.877+0000 7f8435893640 1 -- 172.21.15.94:0/3232397019 >> 172.21.15.94:0/3232397019 conn(0x7f84300771d0 msgr2=0x7f84300775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.877+0000 7f8435893640 1 -- 172.21.15.94:0/3232397019 shutdown_connections 2024-04-21T08:42:46.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.877+0000 7f8435893640 1 -- 172.21.15.94:0/3232397019 wait complete. 2024-04-21T08:42:46.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.877+0000 7f8435893640 1 Processor -- start 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f8435893640 1 -- start start 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f8435893640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84301031e0 0x7f8430198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f8435893640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8430103b00 0x7f8430198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f8435893640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f843010c520 0x7f843019cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f8435893640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f84301121b0 con 0x7f843010c520 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f8435893640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8430112030 con 0x7f8430103b00 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f8435893640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8430112330 con 0x7f84301031e0 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f842f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f843010c520 0x7f843019cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f842f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f843010c520 0x7f843019cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35686/0 (socket says 172.21.15.94:35686) 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f842f7fe640 1 -- 172.21.15.94:0/3633760084 learned_addr learned my addr 172.21.15.94:0/3633760084 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f842e7fc640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8430103b00 0x7f8430198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f842f7fe640 1 -- 172.21.15.94:0/3633760084 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84301031e0 msgr2=0x7f8430198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f842f7fe640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84301031e0 0x7f8430198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f842f7fe640 1 -- 172.21.15.94:0/3633760084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8430103b00 msgr2=0x7f8430198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f842f7fe640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8430103b00 0x7f8430198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f842f7fe640 1 -- 172.21.15.94:0/3633760084 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843019d720 con 0x7f843010c520 2024-04-21T08:42:46.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f842f7fe640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f843010c520 0x7f843019cff0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f842800eaa0 tx=0x7f842800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f8434891640 1 -- 172.21.15.94:0/3633760084 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842800cdb0 con 0x7f843010c520 2024-04-21T08:42:46.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f8435893640 1 -- 172.21.15.94:0/3633760084 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f843019da10 con 0x7f843010c520 2024-04-21T08:42:46.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.878+0000 7f8434891640 1 -- 172.21.15.94:0/3633760084 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8428005020 con 0x7f843010c520 2024-04-21T08:42:46.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.879+0000 7f8434891640 1 -- 172.21.15.94:0/3633760084 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84280116d0 con 0x7f843010c520 2024-04-21T08:42:46.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.879+0000 7f8435893640 1 -- 172.21.15.94:0/3633760084 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84301a52a0 con 0x7f843010c520 2024-04-21T08:42:46.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f8434891640 1 -- 172.21.15.94:0/3633760084 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f84280051d0 con 0x7f843010c520 2024-04-21T08:42:46.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f8438565740 1 Processor -- start 2024-04-21T08:42:46.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f8438565740 1 -- start start 2024-04-21T08:42:46.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f8438565740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebc38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f8438565740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebcc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f8438565740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 0x55ed5ebcee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f8438565740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ed5eb281f0 con 0x55ed5ebc3e80 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f8438565740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ed5eac54b0 con 0x55ed5ebcca00 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f8438565740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ed5eb241b0 con 0x55ed5ebc34d0 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f8434891640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84180a3d20 0x7f84180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f842effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebcc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f842effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebcc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35694/0 (socket says 172.21.15.94:35694) 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f842effd640 1 -- 172.21.15.94:0/647019215 learned_addr learned my addr 172.21.15.94:0/647019215 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.881+0000 7f842f7fe640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84180a3d20 0x7f84180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f842e7fc640 1 --2- 172.21.15.94:0/647019215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 0x55ed5ebcee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f8434891640 1 -- 172.21.15.94:0/3633760084 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f84280d0d40 con 0x7f843010c520 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f842e7fc640 1 --2- 172.21.15.94:0/647019215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebc38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f842effd640 1 -- 172.21.15.94:0/647019215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc34d0 msgr2=0x55ed5ebc38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f842effd640 1 --2- 172.21.15.94:0/647019215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebc38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f842effd640 1 -- 172.21.15.94:0/647019215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 msgr2=0x55ed5ebcee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f842effd640 1 --2- 172.21.15.94:0/647019215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 0x55ed5ebcee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f842effd640 1 -- 172.21.15.94:0/647019215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed5eab71b0 con 0x55ed5ebc3e80 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f842e7fc640 1 --2- 172.21.15.94:0/647019215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebc38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f842effd640 1 --2- 172.21.15.94:0/647019215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebcc430 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8424036040 tx=0x7f8424005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45de478cf110e3e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f842f7fe640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84180a3d20 0x7f84180a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f84280ce6b0 tx=0x7f84280bd040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f8415ffb640 1 -- 172.21.15.94:0/647019215 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424042020 con 0x55ed5ebc3e80 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f8415ffb640 1 -- 172.21.15.94:0/647019215 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f842402cdc0 con 0x55ed5ebc3e80 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.882+0000 7f8415ffb640 1 -- 172.21.15.94:0/647019215 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424003af0 con 0x55ed5ebc3e80 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 -- 172.21.15.94:0/647019215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 msgr2=0x55ed5ebcc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 --2- 172.21.15.94:0/647019215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebcc430 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8424036040 tx=0x7f8424005d50 comp rx=0 tx=0).stop 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 -- 172.21.15.94:0/647019215 shutdown_connections 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 --2- 172.21.15.94:0/647019215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 0x55ed5ebcee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 --2- 172.21.15.94:0/647019215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebcc430 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 --2- 172.21.15.94:0/647019215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebc38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 -- 172.21.15.94:0/647019215 >> 172.21.15.94:0/647019215 conn(0x55ed5ebbdba0 msgr2=0x55ed5ebbdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 -- 172.21.15.94:0/647019215 shutdown_connections 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 -- 172.21.15.94:0/647019215 wait complete. 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 -- start start 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 Processor -- start 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.883+0000 7f8438565740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebcc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f8438565740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebc9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f8438565740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebcca00 0x55ed5ebc9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f8438565740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ed5ebcf910 con 0x55ed5ebc3e80 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f8438565740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ed5eae1500 con 0x55ed5ebc34d0 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f8438565740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ed5ebcfa90 con 0x55ed5ebcca00 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f8438565740 10 client.0 _ll_register_callbacks cb 0x55ed5eaafb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebc9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebc9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35698/0 (socket says 172.21.15.94:35698) 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842effd640 1 -- 172.21.15.94:0/138092171 learned_addr learned my addr 172.21.15.94:0/138092171 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842e7fc640 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebcc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842effd640 1 -- 172.21.15.94:0/138092171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebcca00 msgr2=0x55ed5ebc9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842effd640 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebcca00 0x55ed5ebc9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842effd640 1 -- 172.21.15.94:0/138092171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebc34d0 msgr2=0x55ed5ebcc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842effd640 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebcc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842effd640 1 -- 172.21.15.94:0/138092171 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed5e83a3f0 con 0x55ed5ebc3e80 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842e7fc640 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebcc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f842effd640 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebc9660 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8424006010 tx=0x7f84240040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ddd6f0e71d7ec92 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f840effd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f840effd640 1 -- 172.21.15.94:0/138092171 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424042030 con 0x55ed5ebc3e80 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f840effd640 1 -- 172.21.15.94:0/138092171 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f8424003a70 con 0x55ed5ebc3e80 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f8438565740 1 -- 172.21.15.94:0/138092171 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f842402cdc0 con 0x55ed5ebc3e80 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f840effd640 1 -- 172.21.15.94:0/138092171 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8424003c70 con 0x55ed5ebc3e80 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f8438565740 10 client.16599 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:46.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.884+0000 7f8438565740 1 -- 172.21.15.94:0/138092171 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ed5e9047d0 con 0x55ed5ebc3e80 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.885+0000 7f840effd640 1 -- 172.21.15.94:0/138092171 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f842402cdc0 con 0x55ed5ebc3e80 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.885+0000 7f840effd640 1 -- 172.21.15.94:0/138092171 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f8424047420 con 0x55ed5ebc3e80 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.885+0000 7f8438565740 10 client.16599 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.885+0000 7f8438565740 10 client.16599 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.885+0000 7f8438565740 20 client.16599 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.885+0000 7f8438565740 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55ed5ebdc970 0x55ed5ebded60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.885+0000 7f8438565740 4 client.16599 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.885+0000 7f8438565740 1 -- 172.21.15.94:0/138092171 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed5eac7930 con 0x55ed5ebdc970 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.885+0000 7f842e7fc640 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55ed5ebdc970 0x55ed5ebded60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.886+0000 7f840effd640 10 client.16599 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.886+0000 7f842e7fc640 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55ed5ebdc970 0x55ed5ebded60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b2faeac7a2aeb8f2 server_cookie=7075ce2fca91e57a in_seq=0 out_seq=0 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.886+0000 7f840effd640 1 -- 172.21.15.94:0/138092171 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ed5eac7930 con 0x55ed5ebdc970 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.886+0000 7f840effd640 10 client.16599 handle_command_reply: tid=0 2024-04-21T08:42:46.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.886+0000 7f8438565740 1 client.16599 shutdown 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 1 -- 172.21.15.94:0/138092171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 msgr2=0x55ed5ebc9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebc9660 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8424006010 tx=0x7f84240040c0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 1 -- 172.21.15.94:0/138092171 shutdown_connections 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebcca00 0x55ed5ebc9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55ed5ebdc970 0x55ed5ebded60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebc9660 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f840effd640 0 client.16599 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 1 --2- 172.21.15.94:0/138092171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebcc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 1 -- 172.21.15.94:0/138092171 >> 172.21.15.94:0/138092171 conn(0x55ed5ebbdba0 msgr2=0x55ed5ebbf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 1 -- 172.21.15.94:0/138092171 shutdown_connections 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 1 -- 172.21.15.94:0/138092171 wait complete. 2024-04-21T08:42:46.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.887+0000 7f8438565740 20 client.16599 trim_cache size 0 max 16384 2024-04-21T08:42:46.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f8438565740 1 Processor -- start 2024-04-21T08:42:46.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f8438565740 1 -- start start 2024-04-21T08:42:46.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f8438565740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f8438565740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebbca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f8438565740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 0x55ed5ebcbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f8438565740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f84301b1d00 con 0x55ed5ebc34d0 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f8438565740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f84301af5d0 con 0x55ed5ebcca00 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f8438565740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f84301ad420 con 0x55ed5ebc3e80 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f842e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f842e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35700/0 (socket says 172.21.15.94:35700) 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f842e7fc640 1 -- 172.21.15.94:0/3876236771 learned_addr learned my addr 172.21.15.94:0/3876236771 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f842e7fc640 1 -- 172.21.15.94:0/3876236771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc3e80 msgr2=0x55ed5ebbca80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f842e7fc640 1 --2- 172.21.15.94:0/3876236771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebbca80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f842e7fc640 1 -- 172.21.15.94:0/3876236771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 msgr2=0x55ed5ebcbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f842e7fc640 1 --2- 172.21.15.94:0/3876236771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 0x55ed5ebcbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.900+0000 7f842e7fc640 1 -- 172.21.15.94:0/3876236771 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed5ebc27c0 con 0x55ed5ebc34d0 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f842e7fc640 1 --2- 172.21.15.94:0/3876236771 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd87d0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8420004700 tx=0x7f842000bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ee330182f73e6e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f84157fa640 1 -- 172.21.15.94:0/3876236771 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8420005220 con 0x55ed5ebc34d0 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f84157fa640 1 -- 172.21.15.94:0/3876236771 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8420022030 con 0x55ed5ebc34d0 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f84157fa640 1 -- 172.21.15.94:0/3876236771 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8420003850 con 0x55ed5ebc34d0 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f8438565740 1 -- 172.21.15.94:0/3876236771 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 msgr2=0x55ed5ebd87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f8438565740 1 --2- 172.21.15.94:0/3876236771 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd87d0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8420004700 tx=0x7f842000bee0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f8438565740 1 -- 172.21.15.94:0/3876236771 shutdown_connections 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f8438565740 1 --2- 172.21.15.94:0/3876236771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebbca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f8438565740 1 --2- 172.21.15.94:0/3876236771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 0x55ed5ebcbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f8438565740 1 --2- 172.21.15.94:0/3876236771 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd87d0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f8438565740 1 -- 172.21.15.94:0/3876236771 >> 172.21.15.94:0/3876236771 conn(0x55ed5ebbdba0 msgr2=0x55ed5ebbc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f8438565740 1 -- 172.21.15.94:0/3876236771 shutdown_connections 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.901+0000 7f8438565740 1 -- 172.21.15.94:0/3876236771 wait complete. 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f8438565740 1 -- start start 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f8438565740 1 Processor -- start 2024-04-21T08:42:46.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f8438565740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f8438565740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebd0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f8438565740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 0x55ed5ebd08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f8438565740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f84301ada90 con 0x55ed5ebc34d0 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f8438565740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f84301aef00 con 0x55ed5ebcca00 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f8438565740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f84301ae160 con 0x55ed5ebc3e80 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f8438565740 10 client.0 _ll_register_callbacks cb 0x7f842402c460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f842e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f842e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35716/0 (socket says 172.21.15.94:35716) 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f842e7fc640 1 -- 172.21.15.94:0/452791884 learned_addr learned my addr 172.21.15.94:0/452791884 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f842e7fc640 1 -- 172.21.15.94:0/452791884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc3e80 msgr2=0x55ed5ebd0380 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f842e7fc640 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebd0380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f842e7fc640 1 -- 172.21.15.94:0/452791884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 msgr2=0x55ed5ebd08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f842e7fc640 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 0x55ed5ebd08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f842e7fc640 1 -- 172.21.15.94:0/452791884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed5ebd10c0 con 0x55ed5ebc34d0 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f842e7fc640 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd5010 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f842000b3f0 tx=0x7f8420005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae0c00ea74149b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.902+0000 7f840effd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f840effd640 1 -- 172.21.15.94:0/452791884 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8420005120 con 0x55ed5ebc34d0 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f840effd640 1 -- 172.21.15.94:0/452791884 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f84200052c0 con 0x55ed5ebc34d0 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f8438565740 1 -- 172.21.15.94:0/452791884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8420022030 con 0x55ed5ebc34d0 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f840effd640 1 -- 172.21.15.94:0/452791884 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f842003b050 con 0x55ed5ebc34d0 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f8438565740 10 client.16605 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f8438565740 1 -- 172.21.15.94:0/452791884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ed5ec42bf0 con 0x55ed5ebc34d0 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f840effd640 1 -- 172.21.15.94:0/452791884 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8420022030 con 0x55ed5ebc34d0 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f840effd640 1 -- 172.21.15.94:0/452791884 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f8420027960 con 0x55ed5ebc34d0 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f8438565740 10 client.16605 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f8438565740 10 client.16605 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f8438565740 20 client.16605 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f8438565740 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55ed5ec43230 0x55ed5ec45650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f8438565740 4 client.16605 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.903+0000 7f8438565740 1 -- 172.21.15.94:0/452791884 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ed5eb35c10 con 0x55ed5ec43230 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.904+0000 7f842effd640 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55ed5ec43230 0x55ed5ec45650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.904+0000 7f840effd640 10 client.16605 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.904+0000 7f842effd640 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55ed5ec43230 0x55ed5ec45650 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=992dd1823976bfe4 server_cookie=a578f1b9b8aa830b in_seq=0 out_seq=0 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.904+0000 7f840effd640 1 -- 172.21.15.94:0/452791884 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ed5eb35c10 con 0x55ed5ec43230 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.904+0000 7f840effd640 10 client.16605 handle_command_reply: tid=0 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.904+0000 7f8438565740 1 client.16605 shutdown 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 1 -- 172.21.15.94:0/452791884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 msgr2=0x55ed5ebd5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd5010 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f842000b3f0 tx=0x7f8420005770 comp rx=0 tx=0).stop 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 1 -- 172.21.15.94:0/452791884 shutdown_connections 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed5ebc3e80 0x55ed5ebd0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55ed5ec43230 0x55ed5ec45650 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ed5ebcca00 0x55ed5ebd08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 1 --2- 172.21.15.94:0/452791884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ed5ebc34d0 0x55ed5ebd5010 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f840effd640 0 client.16605 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:42:46.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 1 -- 172.21.15.94:0/452791884 >> 172.21.15.94:0/452791884 conn(0x55ed5ebbdba0 msgr2=0x55ed5ebbcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 1 -- 172.21.15.94:0/452791884 shutdown_connections 2024-04-21T08:42:46.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 1 -- 172.21.15.94:0/452791884 wait complete. 2024-04-21T08:42:46.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8438565740 20 client.16605 trim_cache size 0 max 16384 2024-04-21T08:42:46.910 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:42:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 -- 172.21.15.94:0/3633760084 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84180a3d20 msgr2=0x7f84180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84180a3d20 0x7f84180a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f84280ce6b0 tx=0x7f84280bd040 comp rx=0 tx=0).stop 2024-04-21T08:42:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 -- 172.21.15.94:0/3633760084 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f843010c520 msgr2=0x7f843019cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f843010c520 0x7f843019cff0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f842800eaa0 tx=0x7f842800ef60 comp rx=0 tx=0).stop 2024-04-21T08:42:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 -- 172.21.15.94:0/3633760084 shutdown_connections 2024-04-21T08:42:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f84180a3d20 0x7f84180a61e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f843010c520 0x7f843019cff0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8430103b00 0x7f8430198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 --2- 172.21.15.94:0/3633760084 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84301031e0 0x7f8430198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 -- 172.21.15.94:0/3633760084 >> 172.21.15.94:0/3633760084 conn(0x7f84300771d0 msgr2=0x7f84300fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 -- 172.21.15.94:0/3633760084 shutdown_connections 2024-04-21T08:42:46.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:46.905+0000 7f8435893640 1 -- 172.21.15.94:0/3633760084 wait complete. 2024-04-21T08:42:46.915 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:42:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 -- 172.21.15.94:0/672070167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bc8106f30 msgr2=0x7f9bc8107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 --2- 172.21.15.94:0/672070167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bc8106f30 0x7f9bc8107310 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009930 tx=0x7f9bbc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 -- 172.21.15.94:0/672070167 shutdown_connections 2024-04-21T08:42:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 --2- 172.21.15.94:0/672070167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc8100450 0x7f9bc81008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 --2- 172.21.15.94:0/672070167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc8100e10 0x7f9bc810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 --2- 172.21.15.94:0/672070167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bc8106f30 0x7f9bc8107310 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 -- 172.21.15.94:0/672070167 >> 172.21.15.94:0/672070167 conn(0x7f9bc80773e0 msgr2=0x7f9bc80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 -- 172.21.15.94:0/672070167 shutdown_connections 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 -- 172.21.15.94:0/672070167 wait complete. 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 Processor -- start 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.017+0000 7f9bce759640 1 -- start start 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bce759640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc8100450 0x7f9bc8101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bce759640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bc8100e10 0x7f9bc8101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bce759640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc8106f30 0x7f9bc81020b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bce759640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc8112290 con 0x7f9bc8100e10 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bce759640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc8112110 con 0x7f9bc8106f30 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bce759640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc8112410 con 0x7f9bc8100450 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bc77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bc8100e10 0x7f9bc8101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bc77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bc8100e10 0x7f9bc8101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35738/0 (socket says 172.21.15.94:35738) 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bc77fe640 1 -- 172.21.15.94:0/3210982579 learned_addr learned my addr 172.21.15.94:0/3210982579 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bc7fff640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc8100450 0x7f9bc8101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc8106f30 0x7f9bc81020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bc77fe640 1 -- 172.21.15.94:0/3210982579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc8100450 msgr2=0x7f9bc8101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bc77fe640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc8100450 0x7f9bc8101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bc77fe640 1 -- 172.21.15.94:0/3210982579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc8106f30 msgr2=0x7f9bc81020b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bc77fe640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc8106f30 0x7f9bc81020b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bc77fe640 1 -- 172.21.15.94:0/3210982579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc81a6d30 con 0x7f9bc8100e10 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc8106f30 0x7f9bc81020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.018+0000 7f9bc77fe640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bc8100e10 0x7f9bc8101b70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800d770 tx=0x7f9bb800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.019+0000 7f9bc57fa640 1 -- 172.21.15.94:0/3210982579 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8017070 con 0x7f9bc8100e10 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.019+0000 7f9bc57fa640 1 -- 172.21.15.94:0/3210982579 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bb80054d0 con 0x7f9bc8100e10 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.019+0000 7f9bce759640 1 -- 172.21.15.94:0/3210982579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc81a6fc0 con 0x7f9bc8100e10 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.019+0000 7f9bc57fa640 1 -- 172.21.15.94:0/3210982579 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8004c30 con 0x7f9bc8100e10 2024-04-21T08:42:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.019+0000 7f9bce759640 1 -- 172.21.15.94:0/3210982579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bc81a7490 con 0x7f9bc8100e10 2024-04-21T08:42:47.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bc57fa640 1 -- 172.21.15.94:0/3210982579 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9bb8004e10 con 0x7f9bc8100e10 2024-04-21T08:42:47.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bd142c740 1 Processor -- start 2024-04-21T08:42:47.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bd142c740 1 -- start start 2024-04-21T08:42:47.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bc57fa640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bb00a3d20 0x7f9bb00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c5884d0 0x55ce5c5888b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c591430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c591a00 0x55ce5c593e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bd142c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ce5c4ed1f0 con 0x55ce5c5884d0 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bd142c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ce5c48a4b0 con 0x55ce5c591a00 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bd142c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ce5c4e91b0 con 0x55ce5c588e80 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bb00a3d20 0x7f9bb00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bc57fa640 1 -- 172.21.15.94:0/3210982579 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9bb80d0730 con 0x7f9bc8100e10 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bc7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c5884d0 0x55ce5c5888b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bc7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c5884d0 0x55ce5c5888b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35740/0 (socket says 172.21.15.94:35740) 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.021+0000 7f9bc7fff640 1 -- 172.21.15.94:0/441172055 learned_addr learned my addr 172.21.15.94:0/441172055 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bc77fe640 1 --2- 172.21.15.94:0/441172055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c591430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/441172055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c591a00 0x55ce5c593e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bc7fff640 1 -- 172.21.15.94:0/441172055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 msgr2=0x55ce5c591430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bc7fff640 1 --2- 172.21.15.94:0/441172055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c591430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bc7fff640 1 -- 172.21.15.94:0/441172055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c591a00 msgr2=0x55ce5c593e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bc7fff640 1 --2- 172.21.15.94:0/441172055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c591a00 0x55ce5c593e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bc7fff640 1 -- 172.21.15.94:0/441172055 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce5c47c1b0 con 0x55ce5c5884d0 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bb00a3d20 0x7f9bb00a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0005fd0 tx=0x7f9bc0013b00 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bc7fff640 1 --2- 172.21.15.94:0/441172055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c5884d0 0x55ce5c5888b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc000c00 tx=0x7f9bbc005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca3d1fa3bd16522 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bae7fc640 1 -- 172.21.15.94:0/441172055 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc044020 con 0x55ce5c5884d0 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bae7fc640 1 -- 172.21.15.94:0/441172055 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bbc02cc90 con 0x55ce5c5884d0 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.022+0000 7f9bae7fc640 1 -- 172.21.15.94:0/441172055 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc0046c0 con 0x55ce5c5884d0 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 -- 172.21.15.94:0/441172055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c5884d0 msgr2=0x55ce5c5888b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 --2- 172.21.15.94:0/441172055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c5884d0 0x55ce5c5888b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc000c00 tx=0x7f9bbc005dd0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 -- 172.21.15.94:0/441172055 shutdown_connections 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 --2- 172.21.15.94:0/441172055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c591430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 --2- 172.21.15.94:0/441172055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c591a00 0x55ce5c593e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 --2- 172.21.15.94:0/441172055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c5884d0 0x55ce5c5888b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 -- 172.21.15.94:0/441172055 >> 172.21.15.94:0/441172055 conn(0x55ce5c582ba0 msgr2=0x55ce5c582f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 -- 172.21.15.94:0/441172055 shutdown_connections 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 -- 172.21.15.94:0/441172055 wait complete. 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 -- start start 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.023+0000 7f9bd142c740 1 Processor -- start 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c5884d0 0x55ce5c591310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c58e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c58ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bd142c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ce5c594910 con 0x55ce5c591a00 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bd142c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ce5c4a6500 con 0x55ce5c5884d0 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bd142c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ce5c594a90 con 0x55ce5c588e80 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bd142c740 10 client.0 _ll_register_callbacks cb 0x55ce5c474b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:47.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bcccf6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c58ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bcccf6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c58ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35748/0 (socket says 172.21.15.94:35748) 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bcccf6640 1 -- 172.21.15.94:0/2064315953 learned_addr learned my addr 172.21.15.94:0/2064315953 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bc7fff640 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c5884d0 0x55ce5c591310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bcccf6640 1 -- 172.21.15.94:0/2064315953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 msgr2=0x55ce5c58e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c58e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bcccf6640 1 -- 172.21.15.94:0/2064315953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c5884d0 msgr2=0x55ce5c591310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c5884d0 0x55ce5c591310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bcccf6640 1 -- 172.21.15.94:0/2064315953 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce5c1ff3f0 con 0x55ce5c591a00 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bc7fff640 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c5884d0 0x55ce5c591310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c58ec40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0011960 tx=0x7f9bc0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2611bcfc14d63f31 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9ba77fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2064315953 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0024030 con 0x55ce5c591a00 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2064315953 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9bc000ea60 con 0x55ce5c591a00 2024-04-21T08:42:47.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bd142c740 1 -- 172.21.15.94:0/2064315953 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bbc02cc90 con 0x55ce5c591a00 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2064315953 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9bc0031050 con 0x55ce5c591a00 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bd142c740 10 client.16617 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.024+0000 7f9bd142c740 1 -- 172.21.15.94:0/2064315953 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ce5c2c97d0 con 0x55ce5c591a00 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2064315953 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bbc02cc90 con 0x55ce5c591a00 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2064315953 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9bc0020cb0 con 0x55ce5c591a00 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9bd142c740 10 client.16617 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9bd142c740 10 client.16617 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9bd142c740 20 client.16617 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ce5c5a1b50 0x55ce5c5a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9bd142c740 4 client.16617 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9bd142c740 1 -- 172.21.15.94:0/2064315953 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce5c48c930 con 0x55ce5c5a1b50 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9bc7fff640 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ce5c5a1b50 0x55ce5c5a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9bc7fff640 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ce5c5a1b50 0x55ce5c5a3fc0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8679ad7722f5311 server_cookie=4847ed57333f70e3 in_seq=0 out_seq=0 2024-04-21T08:42:47.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.025+0000 7f9ba77fe640 10 client.16617 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2064315953 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce5c48c930 con 0x55ce5c5a1b50 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9ba77fe640 10 client.16617 handle_command_reply: tid=0 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9bd142c740 1 client.16617 shutdown 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9bd142c740 1 -- 172.21.15.94:0/2064315953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 msgr2=0x55ce5c58ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c58ec40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0011960 tx=0x7f9bc0004ab0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9bd142c740 1 -- 172.21.15.94:0/2064315953 shutdown_connections 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ce5c5a1b50 0x55ce5c5a3fc0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c58e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c58ec40 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2064315953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c5884d0 0x55ce5c591310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9ba77fe640 0 client.16617 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9bd142c740 1 -- 172.21.15.94:0/2064315953 >> 172.21.15.94:0/2064315953 conn(0x55ce5c582ba0 msgr2=0x55ce5c5848f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.026+0000 7f9bd142c740 1 -- 172.21.15.94:0/2064315953 shutdown_connections 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.027+0000 7f9bd142c740 1 -- 172.21.15.94:0/2064315953 wait complete. 2024-04-21T08:42:47.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.027+0000 7f9bd142c740 20 client.16617 trim_cache size 0 max 16384 2024-04-21T08:42:47.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bd142c740 1 Processor -- start 2024-04-21T08:42:47.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bd142c740 1 -- start start 2024-04-21T08:42:47.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c5884d0 0x55ce5c59d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c581a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c590dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bd142c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc81b3f40 con 0x55ce5c591a00 2024-04-21T08:42:47.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bd142c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc81b1810 con 0x55ce5c5884d0 2024-04-21T08:42:47.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bd142c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc81af660 con 0x55ce5c588e80 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bcccf6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c590dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bcccf6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c590dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35764/0 (socket says 172.21.15.94:35764) 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bcccf6640 1 -- 172.21.15.94:0/2626069390 learned_addr learned my addr 172.21.15.94:0/2626069390 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bcccf6640 1 -- 172.21.15.94:0/2626069390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 msgr2=0x55ce5c581a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bc77fe640 1 --2- 172.21.15.94:0/2626069390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c581a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/2626069390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c581a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bcccf6640 1 -- 172.21.15.94:0/2626069390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c5884d0 msgr2=0x55ce5c59d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/2626069390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c5884d0 0x55ce5c59d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bcccf6640 1 -- 172.21.15.94:0/2626069390 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce5c58a270 con 0x55ce5c591a00 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bc77fe640 1 --2- 172.21.15.94:0/2626069390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c581a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/2626069390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c590dd0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000a010 tx=0x7f9bc00093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b6771b2bb16890a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9badffb640 1 -- 172.21.15.94:0/2626069390 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0024070 con 0x55ce5c591a00 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9badffb640 1 -- 172.21.15.94:0/2626069390 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bc002b780 con 0x55ce5c591a00 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.040+0000 7f9badffb640 1 -- 172.21.15.94:0/2626069390 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0003c60 con 0x55ce5c591a00 2024-04-21T08:42:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.041+0000 7f9bd142c740 1 -- 172.21.15.94:0/2626069390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 msgr2=0x55ce5c590dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.041+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2626069390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c590dd0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000a010 tx=0x7f9bc00093a0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.041+0000 7f9bd142c740 1 -- 172.21.15.94:0/2626069390 shutdown_connections 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.041+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2626069390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c588e80 0x55ce5c581a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.041+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2626069390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c591a00 0x55ce5c590dd0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.041+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2626069390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c5884d0 0x55ce5c59d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.041+0000 7f9bd142c740 1 -- 172.21.15.94:0/2626069390 >> 172.21.15.94:0/2626069390 conn(0x55ce5c582ba0 msgr2=0x55ce5c581370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.041+0000 7f9bd142c740 1 -- 172.21.15.94:0/2626069390 shutdown_connections 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.041+0000 7f9bd142c740 1 -- 172.21.15.94:0/2626069390 wait complete. 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.041+0000 7f9bd142c740 1 -- start start 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bd142c740 1 Processor -- start 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c5884d0 0x55ce5c59a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c588e80 0x55ce5c5953f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bd142c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c591a00 0x55ce5c595930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bd142c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc81afcd0 con 0x55ce5c588e80 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bd142c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc81b1140 con 0x55ce5c591a00 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bd142c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc81b03a0 con 0x55ce5c5884d0 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bd142c740 10 client.0 _ll_register_callbacks cb 0x7f9bc0011750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bc77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c588e80 0x55ce5c5953f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bc77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c588e80 0x55ce5c5953f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35766/0 (socket says 172.21.15.94:35766) 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bc77fe640 1 -- 172.21.15.94:0/2584248612 learned_addr learned my addr 172.21.15.94:0/2584248612 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bcccf6640 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c591a00 0x55ce5c595930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bc77fe640 1 -- 172.21.15.94:0/2584248612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c5884d0 msgr2=0x55ce5c59a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bc77fe640 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c5884d0 0x55ce5c59a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bc77fe640 1 -- 172.21.15.94:0/2584248612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c591a00 msgr2=0x55ce5c595930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bc77fe640 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c591a00 0x55ce5c595930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bc77fe640 1 -- 172.21.15.94:0/2584248612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce5c596160 con 0x55ce5c588e80 2024-04-21T08:42:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bc77fe640 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c588e80 0x55ce5c5953f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9bb809bff0 tx=0x7f9bb8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202da732a9c1d067 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9ba77fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2584248612 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8096e60 con 0x55ce5c588e80 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2584248612 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f9bb80cdd20 con 0x55ce5c588e80 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.042+0000 7f9bd142c740 1 -- 172.21.15.94:0/2584248612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bc002b780 con 0x55ce5c588e80 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2584248612 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9bb80a06a0 con 0x55ce5c588e80 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9bd142c740 10 client.16623 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9bd142c740 1 -- 172.21.15.94:0/2584248612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ce5c608060 con 0x55ce5c588e80 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2584248612 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bc002b780 con 0x55ce5c588e80 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2584248612 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9bb804a840 con 0x55ce5c588e80 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9bd142c740 10 client.16623 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9bd142c740 10 client.16623 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9bd142c740 20 client.16623 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ce5c6087b0 0x55ce5c60abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9bd142c740 4 client.16623 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9bd142c740 1 -- 172.21.15.94:0/2584248612 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce5c4fac10 con 0x55ce5c6087b0 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.043+0000 7f9bc7fff640 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ce5c6087b0 0x55ce5c60abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9ba77fe640 10 client.16623 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bc7fff640 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ce5c6087b0 0x55ce5c60abf0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=729d24af8b34270a server_cookie=59aca6e1c54f45e9 in_seq=0 out_seq=0 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9ba77fe640 1 -- 172.21.15.94:0/2584248612 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce5c4fac10 con 0x55ce5c6087b0 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9ba77fe640 10 client.16623 handle_command_reply: tid=0 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 client.16623 shutdown 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 -- 172.21.15.94:0/2584248612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c588e80 msgr2=0x55ce5c5953f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c588e80 0x55ce5c5953f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9bb809bff0 tx=0x7f9bb8005740 comp rx=0 tx=0).stop 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 -- 172.21.15.94:0/2584248612 shutdown_connections 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ce5c6087b0 0x55ce5c60abf0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce5c591a00 0x55ce5c595930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ce5c588e80 0x55ce5c5953f0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 --2- 172.21.15.94:0/2584248612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ce5c5884d0 0x55ce5c59a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 -- 172.21.15.94:0/2584248612 >> 172.21.15.94:0/2584248612 conn(0x55ce5c582ba0 msgr2=0x55ce5c589af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:47.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9ba77fe640 0 client.16623 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:47.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 -- 172.21.15.94:0/2584248612 shutdown_connections 2024-04-21T08:42:47.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 1 -- 172.21.15.94:0/2584248612 wait complete. 2024-04-21T08:42:47.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.044+0000 7f9bd142c740 20 client.16623 trim_cache size 0 max 16384 2024-04-21T08:42:47.050 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:42:47.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 -- 172.21.15.94:0/3210982579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bb00a3d20 msgr2=0x7f9bb00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:47.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bb00a3d20 0x7f9bb00a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0005fd0 tx=0x7f9bc0013b00 comp rx=0 tx=0).stop 2024-04-21T08:42:47.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 -- 172.21.15.94:0/3210982579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bc8100e10 msgr2=0x7f9bc8101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:47.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bc8100e10 0x7f9bc8101b70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800d770 tx=0x7f9bb800dc30 comp rx=0 tx=0).stop 2024-04-21T08:42:47.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 -- 172.21.15.94:0/3210982579 shutdown_connections 2024-04-21T08:42:47.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9bb00a3d20 0x7f9bb00a61e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc8106f30 0x7f9bc81020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bc8100e10 0x7f9bc8101b70 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 --2- 172.21.15.94:0/3210982579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc8100450 0x7f9bc8101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:47.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 -- 172.21.15.94:0/3210982579 >> 172.21.15.94:0/3210982579 conn(0x7f9bc80773e0 msgr2=0x7f9bc81051a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:47.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 -- 172.21.15.94:0/3210982579 shutdown_connections 2024-04-21T08:42:47.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:47.045+0000 7f9bce759640 1 -- 172.21.15.94:0/3210982579 wait complete. 2024-04-21T08:42:47.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:47 smithi094 ceph-mon[27988]: pgmap v87: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:42:47.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:47 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1425294778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:47 smithi111 ceph-mon[31493]: pgmap v87: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:42:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:47 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1425294778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:47.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:47 smithi173 ceph-mon[31428]: pgmap v87: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:42:47.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:47 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1425294778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:48.055 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:42:48.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:48.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:42:48.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.174+0000 7f60caae4640 1 -- 172.21.15.94:0/415188452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60c4102b40 msgr2=0x7f60c4106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.174+0000 7f60caae4640 1 --2- 172.21.15.94:0/415188452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60c4102b40 0x7f60c4106b80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009900 tx=0x7f60b002bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:48.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 -- 172.21.15.94:0/415188452 shutdown_connections 2024-04-21T08:42:48.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 --2- 172.21.15.94:0/415188452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4107260 0x7f60c410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 --2- 172.21.15.94:0/415188452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60c4102b40 0x7f60c4106b80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 --2- 172.21.15.94:0/415188452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c4102190 0x7f60c4102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 -- 172.21.15.94:0/415188452 >> 172.21.15.94:0/415188452 conn(0x7f60c40793a0 msgr2=0x7f60c40797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 -- 172.21.15.94:0/415188452 shutdown_connections 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 -- 172.21.15.94:0/415188452 wait complete. 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 Processor -- start 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 -- start start 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c4102190 0x7f60c4074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60c4107260 0x7f60c4074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4074cd0 0x7f60c41a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f60c41121a0 con 0x7f60c4107260 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f60c4112020 con 0x7f60c4102190 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60caae4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f60c4112320 con 0x7f60c4074cd0 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.175+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60c4107260 0x7f60c4074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60c4107260 0x7f60c4074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35786/0 (socket says 172.21.15.94:35786) 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c3fff640 1 -- 172.21.15.94:0/1002107314 learned_addr learned my addr 172.21.15.94:0/1002107314 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c8880640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c4102190 0x7f60c4074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c9081640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4074cd0 0x7f60c41a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c3fff640 1 -- 172.21.15.94:0/1002107314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4074cd0 msgr2=0x7f60c41a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c3fff640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4074cd0 0x7f60c41a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c3fff640 1 -- 172.21.15.94:0/1002107314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c4102190 msgr2=0x7f60c4074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c3fff640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c4102190 0x7f60c4074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c3fff640 1 -- 172.21.15.94:0/1002107314 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c41a6fd0 con 0x7f60c4107260 2024-04-21T08:42:48.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c8880640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c4102190 0x7f60c4074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:48.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c3fff640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60c4107260 0x7f60c4074790 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f60b0036040 tx=0x7f60b00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c1ffb640 1 -- 172.21.15.94:0/1002107314 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b0043020 con 0x7f60c4107260 2024-04-21T08:42:48.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c1ffb640 1 -- 172.21.15.94:0/1002107314 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b002cb70 con 0x7f60c4107260 2024-04-21T08:42:48.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60caae4640 1 -- 172.21.15.94:0/1002107314 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c41a7260 con 0x7f60c4107260 2024-04-21T08:42:48.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.176+0000 7f60c1ffb640 1 -- 172.21.15.94:0/1002107314 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b0003bf0 con 0x7f60c4107260 2024-04-21T08:42:48.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.177+0000 7f60caae4640 1 -- 172.21.15.94:0/1002107314 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c41a77b0 con 0x7f60c4107260 2024-04-21T08:42:48.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60c1ffb640 1 -- 172.21.15.94:0/1002107314 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f60b0046020 con 0x7f60c4107260 2024-04-21T08:42:48.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60cd7b4740 1 Processor -- start 2024-04-21T08:42:48.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60cd7b4740 1 -- start start 2024-04-21T08:42:48.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 0x564324dcecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dcf290 0x564324dd7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324dda260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564324d32f00 con 0x564324dd7e10 2024-04-21T08:42:48.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564324cd0590 con 0x564324dcf290 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564324d2eec0 con 0x564324dce8e0 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60c1ffb640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f60b40a3d50 0x7f60b40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324dda260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60c8880640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dcf290 0x564324dd7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60c9081640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 0x564324dcecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324dda260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35796/0 (socket says 172.21.15.94:35796) 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60c8880640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dcf290 0x564324dd7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:38378/0 (socket says 172.21.15.94:38378) 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.179+0000 7f60c3fff640 1 -- 172.21.15.94:0/4194385433 learned_addr learned my addr 172.21.15.94:0/4194385433 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60c9081640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f60b40a3d50 0x7f60b40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60c1ffb640 1 -- 172.21.15.94:0/1002107314 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f60b00f2d00 con 0x7f60c4107260 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60c3fff640 1 -- 172.21.15.94:0/4194385433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 msgr2=0x564324dcecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60c3fff640 1 --2- 172.21.15.94:0/4194385433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 0x564324dcecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60c3fff640 1 -- 172.21.15.94:0/4194385433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dcf290 msgr2=0x564324dd7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60c3fff640 1 --2- 172.21.15.94:0/4194385433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dcf290 0x564324dd7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60c3fff640 1 -- 172.21.15.94:0/4194385433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564324b0d7e0 con 0x564324dd7e10 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60c8880640 1 --2- 172.21.15.94:0/4194385433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dcf290 0x564324dd7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60c3fff640 1 --2- 172.21.15.94:0/4194385433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324dda260 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f60b00f01e0 tx=0x7f60b002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b977d8f7f0b17ef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60c9081640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f60b40a3d50 0x7f60b40a6210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009fd0 tx=0x7f60bc009500 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60aeffd640 1 -- 172.21.15.94:0/4194385433 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00be510 con 0x564324dd7e10 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60aeffd640 1 -- 172.21.15.94:0/4194385433 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b0089070 con 0x564324dd7e10 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.180+0000 7f60aeffd640 1 -- 172.21.15.94:0/4194385433 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00c3f10 con 0x564324dd7e10 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 -- 172.21.15.94:0/4194385433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 msgr2=0x564324dda260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/4194385433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324dda260 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f60b00f01e0 tx=0x7f60b002bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 -- 172.21.15.94:0/4194385433 shutdown_connections 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/4194385433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324dda260 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/4194385433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dcf290 0x564324dd7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/4194385433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 0x564324dcecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 -- 172.21.15.94:0/4194385433 >> 172.21.15.94:0/4194385433 conn(0x564324dc8fb0 msgr2=0x564324dc9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 -- 172.21.15.94:0/4194385433 shutdown_connections 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 -- 172.21.15.94:0/4194385433 wait complete. 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 -- start start 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.181+0000 7f60cd7b4740 1 Processor -- start 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 0x564324dd4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324dd5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dd7e10 0x564324dd57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564324ddab40 con 0x564324dcf290 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564324cec580 con 0x564324dd7e10 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564324ddacc0 con 0x564324dce8e0 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60cd7b4740 10 client.0 _ll_register_callbacks cb 0x564324cf2650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324dd5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324dd5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35800/0 (socket says 172.21.15.94:35800) 2024-04-21T08:42:48.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c3fff640 1 -- 172.21.15.94:0/1114795907 learned_addr learned my addr 172.21.15.94:0/1114795907 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c9081640 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dd7e10 0x564324dd57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c8880640 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 0x564324dd4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c3fff640 1 -- 172.21.15.94:0/1114795907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 msgr2=0x564324dd4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c3fff640 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 0x564324dd4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c3fff640 1 -- 172.21.15.94:0/1114795907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dd7e10 msgr2=0x564324dd57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c3fff640 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dd7e10 0x564324dd57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c3fff640 1 -- 172.21.15.94:0/1114795907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564324b6df70 con 0x564324dcf290 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c9081640 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dd7e10 0x564324dd57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c8880640 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 0x564324dd4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60c3fff640 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324dd5290 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f60b00f0670 tx=0x7f60b00f0120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b156362e62e350 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60a3fff640 1 -- 172.21.15.94:0/1114795907 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00be330 con 0x564324dcf290 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60a3fff640 1 -- 172.21.15.94:0/1114795907 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f60b00bb040 con 0x564324dcf290 2024-04-21T08:42:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.182+0000 7f60cd7b4740 1 -- 172.21.15.94:0/1114795907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60b0089070 con 0x564324dcf290 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60a3fff640 1 -- 172.21.15.94:0/1114795907 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f60b00be510 con 0x564324dcf290 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60cd7b4740 10 client.16635 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60cd7b4740 1 -- 172.21.15.94:0/1114795907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564324a05170 con 0x564324dcf290 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60a3fff640 1 -- 172.21.15.94:0/1114795907 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b0089070 con 0x564324dcf290 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60a3fff640 1 -- 172.21.15.94:0/1114795907 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f60b00bec30 con 0x564324dcf290 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60cd7b4740 10 client.16635 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60cd7b4740 10 client.16635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60cd7b4740 20 client.16635 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564324de7ec0 0x564324dea2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60cd7b4740 4 client.16635 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60cd7b4740 1 -- 172.21.15.94:0/1114795907 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564324d40920 con 0x564324de7ec0 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.183+0000 7f60c8880640 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564324de7ec0 0x564324dea2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.184+0000 7f60a3fff640 10 client.16635 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:48.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.184+0000 7f60c8880640 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564324de7ec0 0x564324dea2b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2ee8e674bd6e526 server_cookie=5438d25b6c12fdf0 in_seq=0 out_seq=0 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.184+0000 7f60a3fff640 1 -- 172.21.15.94:0/1114795907 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564324d40920 con 0x564324de7ec0 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.184+0000 7f60a3fff640 10 client.16635 handle_command_reply: tid=0 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 client.16635 shutdown 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 -- 172.21.15.94:0/1114795907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 msgr2=0x564324dd5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324dd5290 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f60b00f0670 tx=0x7f60b00f0120 comp rx=0 tx=0).stop 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 -- 172.21.15.94:0/1114795907 shutdown_connections 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564324de7ec0 0x564324dea2b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dd7e10 0x564324dd57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324dd5290 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60a3fff640 0 client.16635 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/1114795907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dce8e0 0x564324dd4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 -- 172.21.15.94:0/1114795907 >> 172.21.15.94:0/1114795907 conn(0x564324dc8fb0 msgr2=0x564324dcad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 -- 172.21.15.94:0/1114795907 shutdown_connections 2024-04-21T08:42:48.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 1 -- 172.21.15.94:0/1114795907 wait complete. 2024-04-21T08:42:48.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.185+0000 7f60cd7b4740 20 client.16635 trim_cache size 0 max 16384 2024-04-21T08:42:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60cd7b4740 1 Processor -- start 2024-04-21T08:42:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60cd7b4740 1 -- start start 2024-04-21T08:42:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dce8e0 0x564324dd7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324de3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dd7e10 0x564324defa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f60c41b3da0 con 0x564324dcf290 2024-04-21T08:42:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f60c41b1670 con 0x564324dce8e0 2024-04-21T08:42:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f60c41af4c0 con 0x564324dd7e10 2024-04-21T08:42:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324de3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324de3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35812/0 (socket says 172.21.15.94:35812) 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.203+0000 7f60c3fff640 1 -- 172.21.15.94:0/2102877533 learned_addr learned my addr 172.21.15.94:0/2102877533 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60c9081640 1 --2- 172.21.15.94:0/2102877533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dd7e10 0x564324defa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60c3fff640 1 -- 172.21.15.94:0/2102877533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dd7e10 msgr2=0x564324defa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60c3fff640 1 --2- 172.21.15.94:0/2102877533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dd7e10 0x564324defa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60c3fff640 1 -- 172.21.15.94:0/2102877533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dce8e0 msgr2=0x564324dd7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60c3fff640 1 --2- 172.21.15.94:0/2102877533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dce8e0 0x564324dd7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60c3fff640 1 -- 172.21.15.94:0/2102877533 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564324dd04c0 con 0x564324dcf290 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60c3fff640 1 --2- 172.21.15.94:0/2102877533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324de3a00 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f60b00bae30 tx=0x7f60b00bcee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ddb3c40f13666b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60ae7fc640 1 -- 172.21.15.94:0/2102877533 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00be330 con 0x564324dcf290 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60ae7fc640 1 -- 172.21.15.94:0/2102877533 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b0089040 con 0x564324dcf290 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60ae7fc640 1 -- 172.21.15.94:0/2102877533 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00bac70 con 0x564324dcf290 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60cd7b4740 1 -- 172.21.15.94:0/2102877533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 msgr2=0x564324de3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.204+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/2102877533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324de3a00 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f60b00bae30 tx=0x7f60b00bcee0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 -- 172.21.15.94:0/2102877533 shutdown_connections 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/2102877533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dd7e10 0x564324defa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/2102877533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dcf290 0x564324de3a00 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/2102877533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dce8e0 0x564324dd7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 -- 172.21.15.94:0/2102877533 >> 172.21.15.94:0/2102877533 conn(0x564324dc8fb0 msgr2=0x564324dd55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 -- 172.21.15.94:0/2102877533 shutdown_connections 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 -- 172.21.15.94:0/2102877533 wait complete. 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 -- start start 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 Processor -- start 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dce8e0 0x564324ddc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dcf290 0x564324ddc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324e4d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f60c41afb30 con 0x564324dd7e10 2024-04-21T08:42:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f60c41b0fa0 con 0x564324dce8e0 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f60c41b0200 con 0x564324dcf290 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60cd7b4740 10 client.0 _ll_register_callbacks cb 0x7f60b0005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.205+0000 7f60c9081640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324e4d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60c9081640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324e4d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35820/0 (socket says 172.21.15.94:35820) 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60c9081640 1 -- 172.21.15.94:0/602770974 learned_addr learned my addr 172.21.15.94:0/602770974 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60c8880640 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dce8e0 0x564324ddc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60c3fff640 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dcf290 0x564324ddc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60c9081640 1 -- 172.21.15.94:0/602770974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dcf290 msgr2=0x564324ddc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60c9081640 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dcf290 0x564324ddc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60c9081640 1 -- 172.21.15.94:0/602770974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dce8e0 msgr2=0x564324ddc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60c9081640 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dce8e0 0x564324ddc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60c9081640 1 -- 172.21.15.94:0/602770974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564324e4d7d0 con 0x564324dd7e10 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60c9081640 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324e4d290 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0098e0 tx=0x7f60bc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229f57208ba577b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60a3fff640 1 -- 172.21.15.94:0/602770974 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00de80 con 0x564324dd7e10 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60a3fff640 1 -- 172.21.15.94:0/602770974 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f60bc01b070 con 0x564324dd7e10 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60cd7b4740 1 -- 172.21.15.94:0/602770974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60b0089040 con 0x564324dd7e10 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60a3fff640 1 -- 172.21.15.94:0/602770974 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f60bc003b80 con 0x564324dd7e10 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60cd7b4740 10 client.16641 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60cd7b4740 1 -- 172.21.15.94:0/602770974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564324e4eb10 con 0x564324dd7e10 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.206+0000 7f60a3fff640 1 -- 172.21.15.94:0/602770974 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b0089040 con 0x564324dd7e10 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.207+0000 7f60a3fff640 1 -- 172.21.15.94:0/602770974 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f60bc0043d0 con 0x564324dd7e10 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.207+0000 7f60cd7b4740 10 client.16641 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.207+0000 7f60cd7b4740 10 client.16641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.207+0000 7f60cd7b4740 20 client.16641 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.207+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564324de7ec0 0x564324e51320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.207+0000 7f60cd7b4740 4 client.16641 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:42:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.207+0000 7f60cd7b4740 1 -- 172.21.15.94:0/602770974 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564324d3a790 con 0x564324de7ec0 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.207+0000 7f60c8880640 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564324de7ec0 0x564324e51320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.207+0000 7f60a3fff640 10 client.16641 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.207+0000 7f60c8880640 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564324de7ec0 0x564324e51320 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f59932506323228 server_cookie=81fb1bfb91ea1a12 in_seq=0 out_seq=0 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60a3fff640 1 -- 172.21.15.94:0/602770974 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564324d3a790 con 0x564324de7ec0 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60a3fff640 10 client.16641 handle_command_reply: tid=0 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60cd7b4740 1 client.16641 shutdown 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60cd7b4740 1 -- 172.21.15.94:0/602770974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 msgr2=0x564324e4d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324e4d290 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0098e0 tx=0x7f60bc004ab0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60cd7b4740 1 -- 172.21.15.94:0/602770974 shutdown_connections 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564324de7ec0 0x564324e51320 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60a3fff640 0 client.16641 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564324dcf290 0x564324ddc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564324dd7e10 0x564324e4d290 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.208+0000 7f60cd7b4740 1 --2- 172.21.15.94:0/602770974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564324dce8e0 0x564324ddc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60cd7b4740 1 -- 172.21.15.94:0/602770974 >> 172.21.15.94:0/602770974 conn(0x564324dc8fb0 msgr2=0x564324dc7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60cd7b4740 1 -- 172.21.15.94:0/602770974 shutdown_connections 2024-04-21T08:42:48.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60cd7b4740 1 -- 172.21.15.94:0/602770974 wait complete. 2024-04-21T08:42:48.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60cd7b4740 20 client.16641 trim_cache size 0 max 16384 2024-04-21T08:42:48.214 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:42:48.214 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:42:48.214 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "e4926b60-9b8c-4c8c-9880-0b237d8bdbf7", 2024-04-21T08:42:48.214 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:42:48.215 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:42:48.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 -- 172.21.15.94:0/1002107314 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f60b40a3d50 msgr2=0x7f60b40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f60b40a3d50 0x7f60b40a6210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009fd0 tx=0x7f60bc009500 comp rx=0 tx=0).stop 2024-04-21T08:42:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 -- 172.21.15.94:0/1002107314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60c4107260 msgr2=0x7f60c4074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60c4107260 0x7f60c4074790 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f60b0036040 tx=0x7f60b00045e0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 -- 172.21.15.94:0/1002107314 shutdown_connections 2024-04-21T08:42:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f60b40a3d50 0x7f60b40a6210 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60c4074cd0 0x7f60c41a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60c4107260 0x7f60c4074790 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 --2- 172.21.15.94:0/1002107314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c4102190 0x7f60c4074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 -- 172.21.15.94:0/1002107314 >> 172.21.15.94:0/1002107314 conn(0x7f60c40793a0 msgr2=0x7f60c40ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.209+0000 7f60caae4640 1 -- 172.21.15.94:0/1002107314 shutdown_connections 2024-04-21T08:42:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.210+0000 7f60caae4640 1 -- 172.21.15.94:0/1002107314 wait complete. 2024-04-21T08:42:48.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:42:48.220 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:42:48.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.321+0000 7fdf030a8640 1 -- 172.21.15.94:0/599524930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdefc108000 msgr2=0x7fdefc1083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.321+0000 7fdf030a8640 1 --2- 172.21.15.94:0/599524930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdefc108000 0x7fdefc1083e0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009900 tx=0x7fdef002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.321+0000 7fdf030a8640 1 -- 172.21.15.94:0/599524930 shutdown_connections 2024-04-21T08:42:48.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.321+0000 7fdf030a8640 1 --2- 172.21.15.94:0/599524930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdefc108920 0x7fdefc06a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.321+0000 7fdf030a8640 1 --2- 172.21.15.94:0/599524930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdefc06ace0 0x7fdefc10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.321+0000 7fdf030a8640 1 --2- 172.21.15.94:0/599524930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdefc108000 0x7fdefc1083e0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.321+0000 7fdf030a8640 1 -- 172.21.15.94:0/599524930 >> 172.21.15.94:0/599524930 conn(0x7fdefc076c50 msgr2=0x7fdefc077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.321+0000 7fdf030a8640 1 -- 172.21.15.94:0/599524930 shutdown_connections 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.321+0000 7fdf030a8640 1 -- 172.21.15.94:0/599524930 wait complete. 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.321+0000 7fdf030a8640 1 Processor -- start 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdf030a8640 1 -- start start 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdf030a8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdefc06ace0 0x7fdefc198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdf030a8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdefc108000 0x7fdefc198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdf030a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdefc108920 0x7fdefc19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdf030a8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc112200 con 0x7fdefc108000 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdf030a8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc112080 con 0x7fdefc06ace0 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdf030a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc112380 con 0x7fdefc108920 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdefc108000 0x7fdefc198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdefc108000 0x7fdefc198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35848/0 (socket says 172.21.15.94:35848) 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdefbfff640 1 -- 172.21.15.94:0/96161865 learned_addr learned my addr 172.21.15.94:0/96161865 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdf01645640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdefc108920 0x7fdefc19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdf00e44640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdefc06ace0 0x7fdefc198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdefbfff640 1 -- 172.21.15.94:0/96161865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdefc108920 msgr2=0x7fdefc19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdefbfff640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdefc108920 0x7fdefc19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdefbfff640 1 -- 172.21.15.94:0/96161865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdefc06ace0 msgr2=0x7fdefc198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdefbfff640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdefc06ace0 0x7fdefc198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdefbfff640 1 -- 172.21.15.94:0/96161865 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdefc19d750 con 0x7fdefc108000 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdf00e44640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdefc06ace0 0x7fdefc198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdefbfff640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdefc108000 0x7fdefc198ca0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fdee800d770 tx=0x7fdee800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.322+0000 7fdef9ffb640 1 -- 172.21.15.94:0/96161865 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee8017070 con 0x7fdefc108000 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.323+0000 7fdf030a8640 1 -- 172.21.15.94:0/96161865 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdefc19da40 con 0x7fdefc108000 2024-04-21T08:42:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.323+0000 7fdef9ffb640 1 -- 172.21.15.94:0/96161865 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdee80054d0 con 0x7fdefc108000 2024-04-21T08:42:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.323+0000 7fdef9ffb640 1 -- 172.21.15.94:0/96161865 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee8004c30 con 0x7fdefc108000 2024-04-21T08:42:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.323+0000 7fdf030a8640 1 -- 172.21.15.94:0/96161865 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdefc1a52d0 con 0x7fdefc108000 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdef9ffb640 1 -- 172.21.15.94:0/96161865 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fdee8004e10 con 0x7fdefc108000 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf05d79740 1 Processor -- start 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf05d79740 1 -- start start 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5580 0x5626a26f5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdef9ffb640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdeec0a3d20 0x7fdeec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5ea0 0x5626a26fe540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26fea80 0x5626a2700e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf05d79740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5626a265a1e0 con 0x5626a26f5580 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf05d79740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5626a25f7390 con 0x5626a26f5ea0 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf05d79740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5626a26561a0 con 0x5626a26fea80 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf00e44640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5580 0x5626a26f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf01645640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdeec0a3d20 0x7fdeec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf00e44640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5580 0x5626a26f5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35856/0 (socket says 172.21.15.94:35856) 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdf00e44640 1 -- 172.21.15.94:0/3699469327 learned_addr learned my addr 172.21.15.94:0/3699469327 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.325+0000 7fdefbfff640 1 --2- 172.21.15.94:0/3699469327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5ea0 0x5626a26fe540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdef9ffb640 1 -- 172.21.15.94:0/96161865 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fdee80d06b0 con 0x7fdefc108000 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdf00e44640 1 -- 172.21.15.94:0/3699469327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26fea80 msgr2=0x5626a2700e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdf00e44640 1 --2- 172.21.15.94:0/3699469327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26fea80 0x5626a2700e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdf00e44640 1 -- 172.21.15.94:0/3699469327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5ea0 msgr2=0x5626a26fe540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdf00e44640 1 --2- 172.21.15.94:0/3699469327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5ea0 0x5626a26fe540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdf00e44640 1 -- 172.21.15.94:0/3699469327 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626a24367d0 con 0x5626a26f5580 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdefbfff640 1 --2- 172.21.15.94:0/3699469327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5ea0 0x5626a26fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdf01645640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdeec0a3d20 0x7fdeec0a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009850 tx=0x7fdef40095a0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdf00e44640 1 --2- 172.21.15.94:0/3699469327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5580 0x5626a26f5960 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fdef0036040 tx=0x7fdef002c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee1e3ccc4de840f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdee6ffd640 1 -- 172.21.15.94:0/3699469327 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0041020 con 0x5626a26f5580 2024-04-21T08:42:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdee6ffd640 1 -- 172.21.15.94:0/3699469327 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdef0005040 con 0x5626a26f5580 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.326+0000 7fdee6ffd640 1 -- 172.21.15.94:0/3699469327 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef002cbb0 con 0x5626a26f5580 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 -- 172.21.15.94:0/3699469327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5580 msgr2=0x5626a26f5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 --2- 172.21.15.94:0/3699469327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5580 0x5626a26f5960 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fdef0036040 tx=0x7fdef002c720 comp rx=0 tx=0).stop 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 -- 172.21.15.94:0/3699469327 shutdown_connections 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 --2- 172.21.15.94:0/3699469327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26fea80 0x5626a2700e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 --2- 172.21.15.94:0/3699469327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5ea0 0x5626a26fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 --2- 172.21.15.94:0/3699469327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5580 0x5626a26f5960 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 -- 172.21.15.94:0/3699469327 >> 172.21.15.94:0/3699469327 conn(0x5626a26efcb0 msgr2=0x5626a26f0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 -- 172.21.15.94:0/3699469327 shutdown_connections 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 -- 172.21.15.94:0/3699469327 wait complete. 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 -- start start 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 Processor -- start 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5580 0x5626a26fba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5ea0 0x5626a26fbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26fea80 0x5626a26fc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5626a2701810 con 0x5626a26f5ea0 2024-04-21T08:42:48.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.327+0000 7fdf05d79740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5626a26132f0 con 0x5626a26fea80 2024-04-21T08:42:48.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdf05d79740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5626a2701990 con 0x5626a26f5580 2024-04-21T08:42:48.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdf05d79740 10 client.0 _ll_register_callbacks cb 0x5626a2620580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:48.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5ea0 0x5626a26fbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5ea0 0x5626a26fbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35860/0 (socket says 172.21.15.94:35860) 2024-04-21T08:42:48.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdefbfff640 1 -- 172.21.15.94:0/2515374268 learned_addr learned my addr 172.21.15.94:0/2515374268 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:48.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdf01645640 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26fea80 0x5626a26fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdefbfff640 1 -- 172.21.15.94:0/2515374268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5580 msgr2=0x5626a26fba90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdefbfff640 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5580 0x5626a26fba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdefbfff640 1 -- 172.21.15.94:0/2515374268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26fea80 msgr2=0x5626a26fc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdefbfff640 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26fea80 0x5626a26fc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdefbfff640 1 -- 172.21.15.94:0/2515374268 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626a236c3f0 con 0x5626a26f5ea0 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdefbfff640 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5ea0 0x5626a26fbfd0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdee8098c40 tx=0x7fdee80209a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1559f065b204070a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdedbfff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdedbfff640 1 -- 172.21.15.94:0/2515374268 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80cf070 con 0x5626a26f5ea0 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdedbfff640 1 -- 172.21.15.94:0/2515374268 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fdee80a0120 con 0x5626a26f5ea0 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdf05d79740 1 -- 172.21.15.94:0/2515374268 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdef0005040 con 0x5626a26f5ea0 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdedbfff640 1 -- 172.21.15.94:0/2515374268 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdee8005680 con 0x5626a26f5ea0 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdf05d79740 10 client.16653 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdf05d79740 1 -- 172.21.15.94:0/2515374268 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5626a232c0f0 con 0x5626a26f5ea0 2024-04-21T08:42:48.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.328+0000 7fdedbfff640 1 -- 172.21.15.94:0/2515374268 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdef0005040 con 0x5626a26f5ea0 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.329+0000 7fdedbfff640 1 -- 172.21.15.94:0/2515374268 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdee80cdca0 con 0x5626a26f5ea0 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.329+0000 7fdf05d79740 10 client.16653 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.329+0000 7fdf05d79740 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.329+0000 7fdf05d79740 20 client.16653 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.329+0000 7fdf05d79740 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626a270eb10 0x5626a2710f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.329+0000 7fdf05d79740 4 client.16653 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.329+0000 7fdf05d79740 1 -- 172.21.15.94:0/2515374268 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626a2667c00 con 0x5626a270eb10 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.329+0000 7fdf00e44640 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626a270eb10 0x5626a2710f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.329+0000 7fdedbfff640 10 client.16653 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.329+0000 7fdf00e44640 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626a270eb10 0x5626a2710f90 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c7c5e40e02ee0a9 server_cookie=ea29e8a7a22eb93f in_seq=0 out_seq=0 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdedbfff640 1 -- 172.21.15.94:0/2515374268 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5626a2667c00 con 0x5626a270eb10 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdedbfff640 10 client.16653 handle_command_reply: tid=0 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 client.16653 shutdown 2024-04-21T08:42:48.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 -- 172.21.15.94:0/2515374268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5ea0 msgr2=0x5626a26fbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5ea0 0x5626a26fbfd0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdee8098c40 tx=0x7fdee80209a0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 -- 172.21.15.94:0/2515374268 shutdown_connections 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626a270eb10 0x5626a2710f90 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26fea80 0x5626a26fc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26f5ea0 0x5626a26fbfd0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdedbfff640 0 client.16653 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 --2- 172.21.15.94:0/2515374268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5580 0x5626a26fba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 -- 172.21.15.94:0/2515374268 >> 172.21.15.94:0/2515374268 conn(0x5626a26efcb0 msgr2=0x5626a26f1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 -- 172.21.15.94:0/2515374268 shutdown_connections 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 1 -- 172.21.15.94:0/2515374268 wait complete. 2024-04-21T08:42:48.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.330+0000 7fdf05d79740 20 client.16653 trim_cache size 0 max 16384 2024-04-21T08:42:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.347+0000 7fdf05d79740 1 Processor -- start 2024-04-21T08:42:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.347+0000 7fdf05d79740 1 -- start start 2024-04-21T08:42:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5580 0x5626a26fe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 0x5626a270a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a26fabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf05d79740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1b1c70 con 0x5626a26fea80 2024-04-21T08:42:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf05d79740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1af540 con 0x5626a26f5580 2024-04-21T08:42:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf05d79740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1ad390 con 0x5626a26f5ea0 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf01645640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a26fabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf01645640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a26fabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35864/0 (socket says 172.21.15.94:35864) 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf01645640 1 -- 172.21.15.94:0/4035456843 learned_addr learned my addr 172.21.15.94:0/4035456843 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdefbfff640 1 --2- 172.21.15.94:0/4035456843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 0x5626a270a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf01645640 1 -- 172.21.15.94:0/4035456843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 msgr2=0x5626a270a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf01645640 1 --2- 172.21.15.94:0/4035456843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 0x5626a270a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf00e44640 1 --2- 172.21.15.94:0/4035456843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5580 0x5626a26fe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf01645640 1 -- 172.21.15.94:0/4035456843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5580 msgr2=0x5626a26fe1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf01645640 1 --2- 172.21.15.94:0/4035456843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5580 0x5626a26fe1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf01645640 1 -- 172.21.15.94:0/4035456843 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626a26f5cd0 con 0x5626a26fea80 2024-04-21T08:42:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdefbfff640 1 --2- 172.21.15.94:0/4035456843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 0x5626a270a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdf01645640 1 --2- 172.21.15.94:0/4035456843 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a26fabf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdef401a720 tx=0x7fdef4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96185201d502686 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdee67fc640 1 -- 172.21.15.94:0/4035456843 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef4028070 con 0x5626a26fea80 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdee67fc640 1 -- 172.21.15.94:0/4035456843 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdef4018b80 con 0x5626a26fea80 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.348+0000 7fdee67fc640 1 -- 172.21.15.94:0/4035456843 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef4023c00 con 0x5626a26fea80 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 -- 172.21.15.94:0/4035456843 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 msgr2=0x5626a26fabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 --2- 172.21.15.94:0/4035456843 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a26fabf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdef401a720 tx=0x7fdef4017000 comp rx=0 tx=0).stop 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 -- 172.21.15.94:0/4035456843 shutdown_connections 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 --2- 172.21.15.94:0/4035456843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 0x5626a270a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 --2- 172.21.15.94:0/4035456843 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a26fabf0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 --2- 172.21.15.94:0/4035456843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5580 0x5626a26fe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 -- 172.21.15.94:0/4035456843 >> 172.21.15.94:0/4035456843 conn(0x5626a26efcb0 msgr2=0x5626a26ee4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 -- 172.21.15.94:0/4035456843 shutdown_connections 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 -- 172.21.15.94:0/4035456843 wait complete. 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 -- start start 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.349+0000 7fdf05d79740 1 Processor -- start 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5580 0x5626a26f8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 0x5626a26f9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf05d79740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a27740d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf05d79740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1ada00 con 0x5626a26fea80 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf05d79740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1aee70 con 0x5626a26f5580 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf05d79740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc1ae0d0 con 0x5626a26f5ea0 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf05d79740 10 client.0 _ll_register_callbacks cb 0x7fdee8097e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf01645640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a27740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf01645640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a27740d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35878/0 (socket says 172.21.15.94:35878) 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf01645640 1 -- 172.21.15.94:0/3752745025 learned_addr learned my addr 172.21.15.94:0/3752745025 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf00e44640 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5580 0x5626a26f8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdefbfff640 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 0x5626a26f9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf01645640 1 -- 172.21.15.94:0/3752745025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 msgr2=0x5626a26f9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf01645640 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 0x5626a26f9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf01645640 1 -- 172.21.15.94:0/3752745025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5580 msgr2=0x5626a26f8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf01645640 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5580 0x5626a26f8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf01645640 1 -- 172.21.15.94:0/3752745025 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626a2774660 con 0x5626a26fea80 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdefbfff640 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 0x5626a26f9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf01645640 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a27740d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdef40186d0 tx=0x7fdef4018700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e1096046db2891 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdedbfff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:42:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdedbfff640 1 -- 172.21.15.94:0/3752745025 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef4031060 con 0x5626a26fea80 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdedbfff640 1 -- 172.21.15.94:0/3752745025 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fdef4023880 con 0x5626a26fea80 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.350+0000 7fdf05d79740 1 -- 172.21.15.94:0/3752745025 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdef4018b80 con 0x5626a26fea80 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdedbfff640 1 -- 172.21.15.94:0/3752745025 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdef4028070 con 0x5626a26fea80 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdf05d79740 10 client.16659 fetch_fsmap learned FSMap version 24 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdf05d79740 1 -- 172.21.15.94:0/3752745025 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5626a27758e0 con 0x5626a26fea80 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdedbfff640 1 -- 172.21.15.94:0/3752745025 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdef4018b80 con 0x5626a26fea80 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdedbfff640 1 -- 172.21.15.94:0/3752745025 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdef4022070 con 0x5626a26fea80 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdf05d79740 10 client.16659 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdf05d79740 10 client.16659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdf05d79740 20 client.16659 populate_metadata read hostname 'smithi094' 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdf05d79740 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626a2775fb0 0x5626a27783d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdf05d79740 4 client.16659 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdf05d79740 1 -- 172.21.15.94:0/3752745025 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5626a2661a70 con 0x5626a2775fb0 2024-04-21T08:42:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.351+0000 7fdf00e44640 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626a2775fb0 0x5626a27783d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdedbfff640 10 client.16659 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf00e44640 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626a2775fb0 0x5626a27783d0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb188e7726032226 server_cookie=8a74e943592b5194 in_seq=0 out_seq=0 2024-04-21T08:42:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdedbfff640 1 -- 172.21.15.94:0/3752745025 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5626a2661a70 con 0x5626a2775fb0 2024-04-21T08:42:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdedbfff640 10 client.16659 handle_command_reply: tid=0 2024-04-21T08:42:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 client.16659 shutdown 2024-04-21T08:42:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 -- 172.21.15.94:0/3752745025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 msgr2=0x5626a27740d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a27740d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdef40186d0 tx=0x7fdef4018700 comp rx=0 tx=0).stop 2024-04-21T08:42:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 -- 172.21.15.94:0/3752745025 shutdown_connections 2024-04-21T08:42:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626a2775fb0 0x5626a27783d0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626a26f5ea0 0x5626a26f9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626a26fea80 0x5626a27740d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdedbfff640 0 client.16659 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:42:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 --2- 172.21.15.94:0/3752745025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626a26f5580 0x5626a26f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 -- 172.21.15.94:0/3752745025 >> 172.21.15.94:0/3752745025 conn(0x5626a26efcb0 msgr2=0x5626a26eed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 -- 172.21.15.94:0/3752745025 shutdown_connections 2024-04-21T08:42:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 1 -- 172.21.15.94:0/3752745025 wait complete. 2024-04-21T08:42:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.352+0000 7fdf05d79740 20 client.16659 trim_cache size 0 max 16384 2024-04-21T08:42:48.359 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:42:48.359 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:42:48.359 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:42:48.360 INFO:teuthology.orchestra.run.smithi094.stdout: "e4926b60-9b8c-4c8c-9880-0b237d8bdbf7": { 2024-04-21T08:42:48.360 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:42:48.360 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "e4926b60-9b8c-4c8c-9880-0b237d8bdbf7", 2024-04-21T08:42:48.360 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:42:48.360 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:42:48.360 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:42:48.360 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:42:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 -- 172.21.15.94:0/96161865 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdeec0a3d20 msgr2=0x7fdeec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdeec0a3d20 0x7fdeec0a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009850 tx=0x7fdef40095a0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 -- 172.21.15.94:0/96161865 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdefc108000 msgr2=0x7fdefc198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdefc108000 0x7fdefc198ca0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fdee800d770 tx=0x7fdee800dc30 comp rx=0 tx=0).stop 2024-04-21T08:42:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 -- 172.21.15.94:0/96161865 shutdown_connections 2024-04-21T08:42:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdeec0a3d20 0x7fdeec0a61e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdefc108920 0x7fdefc19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdefc108000 0x7fdefc198ca0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 --2- 172.21.15.94:0/96161865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdefc06ace0 0x7fdefc198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 -- 172.21.15.94:0/96161865 >> 172.21.15.94:0/96161865 conn(0x7fdefc076c50 msgr2=0x7fdefc0fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 -- 172.21.15.94:0/96161865 shutdown_connections 2024-04-21T08:42:48.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:42:48.353+0000 7fdf030a8640 1 -- 172.21.15.94:0/96161865 wait complete. 2024-04-21T08:42:48.363 INFO:tasks.cephfs.filesystem:scrub status for tag:e4926b60-9b8c-4c8c-9880-0b237d8bdbf7 - {'path': '/', 'tag': 'e4926b60-9b8c-4c8c-9880-0b237d8bdbf7', 'options': 'recursive,force'} 2024-04-21T08:42:49.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:49 smithi111 ceph-mon[31493]: pgmap v88: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:42:49.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:49 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:42:49.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:49 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:49.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:49 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:42:49.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:49 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:49.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:49 smithi094 ceph-mon[27988]: pgmap v88: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:42:49.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:49 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:42:49.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:49 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:49.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:49 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:42:49.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:49 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:49.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:49 smithi173 ceph-mon[31428]: pgmap v88: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:42:49.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:49 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:42:49.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:49 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:49.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:49 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:42:49.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:49 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:42:51.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:51 smithi111 ceph-mon[31493]: pgmap v89: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:42:51.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:51 smithi094 ceph-mon[27988]: pgmap v89: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:42:51.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:51 smithi173 ceph-mon[31428]: pgmap v89: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:42:52.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:52 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:42:52.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:52 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:42:52.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:52 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:42:53.411 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:53 smithi094 ceph-mon[27988]: pgmap v90: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:42:53.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:53 smithi111 ceph-mon[31493]: pgmap v90: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:42:53.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:53 smithi173 ceph-mon[31428]: pgmap v90: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:42:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:55 smithi111 ceph-mon[31493]: pgmap v91: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:42:55.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:55 smithi094 ceph-mon[27988]: pgmap v91: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:42:55.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:55 smithi173 ceph-mon[31428]: pgmap v91: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:42:57.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:57 smithi111 ceph-mon[31493]: pgmap v92: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:42:57.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:57 smithi094 ceph-mon[27988]: pgmap v92: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:42:57.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:57 smithi173 ceph-mon[31428]: pgmap v92: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:42:59.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:42:59 smithi111 ceph-mon[31493]: pgmap v93: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:42:59.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:42:59 smithi094 ceph-mon[27988]: pgmap v93: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:42:59.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:42:59 smithi173 ceph-mon[31428]: pgmap v93: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:43:01.439 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:01 smithi111 ceph-mon[31493]: pgmap v94: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:43:01.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:01 smithi094 ceph-mon[27988]: pgmap v94: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:43:01.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:01 smithi173 ceph-mon[31428]: pgmap v94: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:43:03.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:03 smithi111 ceph-mon[31493]: pgmap v95: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:03.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:03 smithi094 ceph-mon[27988]: pgmap v95: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:03.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:03 smithi173 ceph-mon[31428]: pgmap v95: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:05.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:05 smithi111 ceph-mon[31493]: pgmap v96: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-21T08:43:05.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:05 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:05 smithi094 ceph-mon[27988]: pgmap v96: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-21T08:43:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:05 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:05.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:05 smithi173 ceph-mon[31428]: pgmap v96: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-21T08:43:05.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:05 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:06.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:06 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:06.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:06 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:06.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:06 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:43:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:06 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:06 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:06.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:06 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:43:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:06 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:06 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:06.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:06 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:43:07.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:07 smithi111 ceph-mon[31493]: pgmap v97: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:43:07.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:07 smithi094 ceph-mon[27988]: pgmap v97: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:43:07.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:07 smithi173 ceph-mon[31428]: pgmap v97: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:43:09.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:09 smithi111 ceph-mon[31493]: pgmap v98: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:43:09.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:09 smithi094 ceph-mon[27988]: pgmap v98: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:43:09.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:09 smithi173 ceph-mon[31428]: pgmap v98: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:43:11.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:11 smithi111 ceph-mon[31493]: pgmap v99: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:43:11.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:11 smithi094 ceph-mon[27988]: pgmap v99: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:43:11.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:11 smithi173 ceph-mon[31428]: pgmap v99: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:43:13.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:13 smithi111 ceph-mon[31493]: pgmap v100: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:43:13.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:13 smithi094 ceph-mon[27988]: pgmap v100: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:43:13.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:13 smithi173 ceph-mon[31428]: pgmap v100: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:43:15.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:15 smithi111 ceph-mon[31493]: pgmap v101: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:43:15.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:15 smithi094 ceph-mon[27988]: pgmap v101: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:43:15.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:15 smithi173 ceph-mon[31428]: pgmap v101: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:43:17.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:17 smithi111 ceph-mon[31493]: pgmap v102: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:43:17.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:17 smithi094 ceph-mon[27988]: pgmap v102: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:43:17.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:17 smithi173 ceph-mon[31428]: pgmap v102: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:43:18.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:43:18.364 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:43:18.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.481+0000 7f5915eeb640 1 -- 172.21.15.94:0/1018904495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59101031b0 msgr2=0x7f5910103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.481+0000 7f5915eeb640 1 --2- 172.21.15.94:0/1018904495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59101031b0 0x7f5910103590 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5900009900 tx=0x7f590002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.481+0000 7f5915eeb640 1 -- 172.21.15.94:0/1018904495 shutdown_connections 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.481+0000 7f5915eeb640 1 --2- 172.21.15.94:0/1018904495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f591010c580 0x7f591010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.481+0000 7f5915eeb640 1 --2- 172.21.15.94:0/1018904495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5910103ad0 0x7f591010bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.481+0000 7f5915eeb640 1 --2- 172.21.15.94:0/1018904495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59101031b0 0x7f5910103590 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.481+0000 7f5915eeb640 1 -- 172.21.15.94:0/1018904495 >> 172.21.15.94:0/1018904495 conn(0x7f59100771d0 msgr2=0x7f59100775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.481+0000 7f5915eeb640 1 -- 172.21.15.94:0/1018904495 shutdown_connections 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.481+0000 7f5915eeb640 1 -- 172.21.15.94:0/1018904495 wait complete. 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f5915eeb640 1 Processor -- start 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f5915eeb640 1 -- start start 2024-04-21T08:43:18.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f5915eeb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59101031b0 0x7f5910198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f5915eeb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5910103ad0 0x7f5910198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f5915eeb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f591010c580 0x7f591019cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f5915eeb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f59101121d0 con 0x7f59101031b0 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f5915eeb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5910112050 con 0x7f591010c580 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f5915eeb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5910112350 con 0x7f5910103ad0 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f590f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59101031b0 0x7f5910198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f590f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59101031b0 0x7f5910198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39236/0 (socket says 172.21.15.94:39236) 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.482+0000 7f590f7fe640 1 -- 172.21.15.94:0/971555999 learned_addr learned my addr 172.21.15.94:0/971555999 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590ffff640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f591010c580 0x7f591019cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590effd640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5910103ad0 0x7f5910198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590f7fe640 1 -- 172.21.15.94:0/971555999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5910103ad0 msgr2=0x7f5910198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590f7fe640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5910103ad0 0x7f5910198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590f7fe640 1 -- 172.21.15.94:0/971555999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f591010c580 msgr2=0x7f591019cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590f7fe640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f591010c580 0x7f591019cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590f7fe640 1 -- 172.21.15.94:0/971555999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591019d720 con 0x7f59101031b0 2024-04-21T08:43:18.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590effd640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5910103ad0 0x7f5910198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:18.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590ffff640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f591010c580 0x7f591019cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:18.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590f7fe640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59101031b0 0x7f5910198730 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f5900005940 tx=0x7f590002d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590cff9640 1 -- 172.21.15.94:0/971555999 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900043020 con 0x7f59101031b0 2024-04-21T08:43:18.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590cff9640 1 -- 172.21.15.94:0/971555999 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f590002dc90 con 0x7f59101031b0 2024-04-21T08:43:18.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f5915eeb640 1 -- 172.21.15.94:0/971555999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591019d9b0 con 0x7f59101031b0 2024-04-21T08:43:18.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f590cff9640 1 -- 172.21.15.94:0/971555999 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900003ab0 con 0x7f59101031b0 2024-04-21T08:43:18.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.483+0000 7f5915eeb640 1 -- 172.21.15.94:0/971555999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59101a52a0 con 0x7f59101031b0 2024-04-21T08:43:18.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f590cff9640 1 -- 172.21.15.94:0/971555999 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5900046020 con 0x7f59101031b0 2024-04-21T08:43:18.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f5918bb9740 1 Processor -- start 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f5918bb9740 1 -- start start 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed67580 0x55653ed67960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 0x55653ed70540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed70a80 0x55653ed72e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f5918bb9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55653eccc1e0 con 0x55653ed70a80 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f5918bb9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55653ec69390 con 0x55653ed67ea0 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f5918bb9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55653ecc81a0 con 0x55653ed67580 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f590cff9640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58f80a3d20 0x7f58f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f590effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed70a80 0x55653ed72e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f590effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed70a80 0x55653ed72e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39252/0 (socket says 172.21.15.94:39252) 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f590effd640 1 -- 172.21.15.94:0/138048250 learned_addr learned my addr 172.21.15.94:0/138048250 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f590ffff640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58f80a3d20 0x7f58f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f590f7fe640 1 --2- 172.21.15.94:0/138048250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 0x55653ed70540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.486+0000 7f590ffff640 1 --2- 172.21.15.94:0/138048250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed67580 0x55653ed67960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f590effd640 1 -- 172.21.15.94:0/138048250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed67580 msgr2=0x55653ed67960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f590effd640 1 --2- 172.21.15.94:0/138048250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed67580 0x55653ed67960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f590effd640 1 -- 172.21.15.94:0/138048250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 msgr2=0x55653ed70540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f590effd640 1 --2- 172.21.15.94:0/138048250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 0x55653ed70540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f590effd640 1 -- 172.21.15.94:0/138048250 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55653eaa87d0 con 0x55653ed70a80 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f590ffff640 1 --2- 172.21.15.94:0/138048250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed67580 0x55653ed67960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f590cff9640 1 -- 172.21.15.94:0/971555999 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f59000f2cd0 con 0x7f59101031b0 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f590effd640 1 --2- 172.21.15.94:0/138048250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed70a80 0x55653ed72e70 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f590400cc50 tx=0x7f590400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bb5dcd8b2b03d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f590ffff640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58f80a3d20 0x7f58f80a61e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5908005c40 tx=0x7f5908009530 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f58f5ffb640 1 -- 172.21.15.94:0/138048250 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904014070 con 0x55653ed70a80 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f58f5ffb640 1 -- 172.21.15.94:0/138048250 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5904019b10 con 0x55653ed70a80 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.487+0000 7f58f5ffb640 1 -- 172.21.15.94:0/138048250 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904010ba0 con 0x55653ed70a80 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 -- 172.21.15.94:0/138048250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed70a80 msgr2=0x55653ed72e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 --2- 172.21.15.94:0/138048250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed70a80 0x55653ed72e70 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f590400cc50 tx=0x7f590400e720 comp rx=0 tx=0).stop 2024-04-21T08:43:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 -- 172.21.15.94:0/138048250 shutdown_connections 2024-04-21T08:43:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 --2- 172.21.15.94:0/138048250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed70a80 0x55653ed72e70 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 --2- 172.21.15.94:0/138048250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 0x55653ed70540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 --2- 172.21.15.94:0/138048250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed67580 0x55653ed67960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 -- 172.21.15.94:0/138048250 >> 172.21.15.94:0/138048250 conn(0x55653ed61cb0 msgr2=0x55653ed62090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 -- 172.21.15.94:0/138048250 shutdown_connections 2024-04-21T08:43:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 -- 172.21.15.94:0/138048250 wait complete. 2024-04-21T08:43:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 -- start start 2024-04-21T08:43:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.488+0000 7f5918bb9740 1 Processor -- start 2024-04-21T08:43:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed6da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 0x55653ed6dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 0x55653ed6e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f5918bb9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55653ed73870 con 0x55653ed67580 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f5918bb9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55653ec852f0 con 0x55653ed67ea0 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f5918bb9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55653ed739f0 con 0x55653ed70a80 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f5918bb9740 10 client.0 _ll_register_callbacks cb 0x55653ec92580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f590f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed6da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f590f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed6da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39268/0 (socket says 172.21.15.94:39268) 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f590f7fe640 1 -- 172.21.15.94:0/4002173530 learned_addr learned my addr 172.21.15.94:0/4002173530 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f590ffff640 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 0x55653ed6e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f590f7fe640 1 -- 172.21.15.94:0/4002173530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 msgr2=0x55653ed6e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f590f7fe640 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 0x55653ed6e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f590f7fe640 1 -- 172.21.15.94:0/4002173530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 msgr2=0x55653ed6dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f590f7fe640 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 0x55653ed6dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f590f7fe640 1 -- 172.21.15.94:0/4002173530 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55653e9de3f0 con 0x55653ed67580 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f590f7fe640 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed6da60 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f59000f0640 tx=0x7f59000be420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef782ab13698b1ef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f58eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f58eeffd640 1 -- 172.21.15.94:0/4002173530 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59000beac0 con 0x55653ed67580 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f58eeffd640 1 -- 172.21.15.94:0/4002173530 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f59000bb130 con 0x55653ed67580 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.489+0000 7f5918bb9740 1 -- 172.21.15.94:0/4002173530 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5904019b10 con 0x55653ed67580 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f58eeffd640 1 -- 172.21.15.94:0/4002173530 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f59000c2e70 con 0x55653ed67580 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f5918bb9740 10 client.16671 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f5918bb9740 1 -- 172.21.15.94:0/4002173530 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55653e99e0f0 con 0x55653ed67580 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f58eeffd640 1 -- 172.21.15.94:0/4002173530 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5904019b10 con 0x55653ed67580 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f58eeffd640 1 -- 172.21.15.94:0/4002173530 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f59000c72a0 con 0x55653ed67580 2024-04-21T08:43:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f5918bb9740 10 client.16671 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f5918bb9740 10 client.16671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f5918bb9740 20 client.16671 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f5918bb9740 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55653ed80b00 0x55653ed82fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f5918bb9740 4 client.16671 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f5918bb9740 1 -- 172.21.15.94:0/4002173530 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55653ecd9c00 con 0x55653ed80b00 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.490+0000 7f590effd640 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55653ed80b00 0x55653ed82fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.491+0000 7f58eeffd640 10 client.16671 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.491+0000 7f590effd640 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55653ed80b00 0x55653ed82fa0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca3b6e745eb6eb1b server_cookie=7434ce3f1ccf9aa7 in_seq=0 out_seq=0 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f58eeffd640 1 -- 172.21.15.94:0/4002173530 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55653ecd9c00 con 0x55653ed80b00 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f58eeffd640 10 client.16671 handle_command_reply: tid=0 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 client.16671 shutdown 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 -- 172.21.15.94:0/4002173530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 msgr2=0x55653ed6da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed6da60 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f59000f0640 tx=0x7f59000be420 comp rx=0 tx=0).stop 2024-04-21T08:43:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 -- 172.21.15.94:0/4002173530 shutdown_connections 2024-04-21T08:43:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55653ed80b00 0x55653ed82fa0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 0x55653ed6e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 0x55653ed6dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f58eeffd640 0 client.16671 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 --2- 172.21.15.94:0/4002173530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed6da60 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 -- 172.21.15.94:0/4002173530 >> 172.21.15.94:0/4002173530 conn(0x55653ed61cb0 msgr2=0x55653ed63a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 -- 172.21.15.94:0/4002173530 shutdown_connections 2024-04-21T08:43:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 1 -- 172.21.15.94:0/4002173530 wait complete. 2024-04-21T08:43:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.492+0000 7f5918bb9740 20 client.16671 trim_cache size 0 max 16384 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.509+0000 7f5918bb9740 1 Processor -- start 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.509+0000 7f5918bb9740 1 -- start start 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed70190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 0x55653ed6c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 0x55653ed7c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f5918bb9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f59101afa50 con 0x55653ed67580 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f5918bb9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f59101ad320 con 0x55653ed67ea0 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f5918bb9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59101ab170 con 0x55653ed70a80 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f590f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed70190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f590f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed70190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39278/0 (socket says 172.21.15.94:39278) 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f590f7fe640 1 -- 172.21.15.94:0/2924378265 learned_addr learned my addr 172.21.15.94:0/2924378265 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f590f7fe640 1 -- 172.21.15.94:0/2924378265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 msgr2=0x55653ed7c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f590f7fe640 1 --2- 172.21.15.94:0/2924378265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 0x55653ed7c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f590f7fe640 1 -- 172.21.15.94:0/2924378265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 msgr2=0x55653ed6c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f590f7fe640 1 --2- 172.21.15.94:0/2924378265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 0x55653ed6c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f590f7fe640 1 -- 172.21.15.94:0/2924378265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55653ed68e00 con 0x55653ed67580 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f590f7fe640 1 --2- 172.21.15.94:0/2924378265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed70190 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f5900006010 tx=0x7f59000be700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f6e3d665f80a71 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f58f57fa640 1 -- 172.21.15.94:0/2924378265 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59000bad60 con 0x55653ed67580 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f58f57fa640 1 -- 172.21.15.94:0/2924378265 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f590002de40 con 0x55653ed67580 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.510+0000 7f58f57fa640 1 -- 172.21.15.94:0/2924378265 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900005340 con 0x55653ed67580 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 -- 172.21.15.94:0/2924378265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 msgr2=0x55653ed70190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 --2- 172.21.15.94:0/2924378265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed70190 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f5900006010 tx=0x7f59000be700 comp rx=0 tx=0).stop 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 -- 172.21.15.94:0/2924378265 shutdown_connections 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 --2- 172.21.15.94:0/2924378265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 0x55653ed7c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 --2- 172.21.15.94:0/2924378265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67ea0 0x55653ed6c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 --2- 172.21.15.94:0/2924378265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67580 0x55653ed70190 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 -- 172.21.15.94:0/2924378265 >> 172.21.15.94:0/2924378265 conn(0x55653ed61cb0 msgr2=0x55653ed60d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 -- 172.21.15.94:0/2924378265 shutdown_connections 2024-04-21T08:43:18.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 -- 172.21.15.94:0/2924378265 wait complete. 2024-04-21T08:43:18.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 -- start start 2024-04-21T08:43:18.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.511+0000 7f5918bb9740 1 Processor -- start 2024-04-21T08:43:18.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67580 0x55653ed78e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67ea0 0x55653ed75c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f5918bb9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 0x55653ed76180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f5918bb9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f59101ab7e0 con 0x55653ed67ea0 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f5918bb9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f59101acc50 con 0x55653ed67580 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f5918bb9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59101abeb0 con 0x55653ed70a80 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f5918bb9740 10 client.0 _ll_register_callbacks cb 0x7f5900009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f590effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67ea0 0x55653ed75c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f590effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67ea0 0x55653ed75c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39280/0 (socket says 172.21.15.94:39280) 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f590effd640 1 -- 172.21.15.94:0/1790612068 learned_addr learned my addr 172.21.15.94:0/1790612068 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f590f7fe640 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67580 0x55653ed78e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f590effd640 1 -- 172.21.15.94:0/1790612068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 msgr2=0x55653ed76180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f590effd640 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 0x55653ed76180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f590effd640 1 -- 172.21.15.94:0/1790612068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67580 msgr2=0x55653ed78e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f590effd640 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67580 0x55653ed78e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f590effd640 1 -- 172.21.15.94:0/1790612068 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55653ed766c0 con 0x55653ed67ea0 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f590effd640 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67ea0 0x55653ed75c40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f590400ec20 tx=0x7f590400c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a29a882a14ef44 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f58eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f58eeffd640 1 -- 172.21.15.94:0/1790612068 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904027070 con 0x55653ed67ea0 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f58eeffd640 1 -- 172.21.15.94:0/1790612068 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5904004220 con 0x55653ed67ea0 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f5918bb9740 1 -- 172.21.15.94:0/1790612068 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55653ecd3a70 con 0x55653ed67ea0 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.512+0000 7f58eeffd640 1 -- 172.21.15.94:0/1790612068 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5904030050 con 0x55653ed67ea0 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f5918bb9740 10 client.16677 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f5918bb9740 1 -- 172.21.15.94:0/1790612068 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55653ede7e80 con 0x55653ed67ea0 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f58eeffd640 1 -- 172.21.15.94:0/1790612068 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55653ecd3a70 con 0x55653ed67ea0 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f58eeffd640 1 -- 172.21.15.94:0/1790612068 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f590402a020 con 0x55653ed67ea0 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f5918bb9740 10 client.16677 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f5918bb9740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:18.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f5918bb9740 20 client.16677 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f5918bb9740 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55653ede83d0 0x55653edea7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f5918bb9740 4 client.16677 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f5918bb9740 1 -- 172.21.15.94:0/1790612068 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55653ec4f590 con 0x55653ede83d0 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f590f7fe640 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55653ede83d0 0x55653edea7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f58eeffd640 10 client.16677 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.513+0000 7f590f7fe640 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55653ede83d0 0x55653edea7f0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c502542a23c12cb6 server_cookie=d3f7b29b86029d5d in_seq=0 out_seq=0 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f58eeffd640 1 -- 172.21.15.94:0/1790612068 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55653ec4f590 con 0x55653ede83d0 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f58eeffd640 10 client.16677 handle_command_reply: tid=0 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 client.16677 shutdown 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 -- 172.21.15.94:0/1790612068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67ea0 msgr2=0x55653ed75c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67ea0 0x55653ed75c40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f590400ec20 tx=0x7f590400c4f0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 -- 172.21.15.94:0/1790612068 shutdown_connections 2024-04-21T08:43:18.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55653ede83d0 0x55653edea7f0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55653ed70a80 0x55653ed76180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55653ed67ea0 0x55653ed75c40 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f58eeffd640 0 client.16677 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 --2- 172.21.15.94:0/1790612068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55653ed67580 0x55653ed78e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 -- 172.21.15.94:0/1790612068 >> 172.21.15.94:0/1790612068 conn(0x55653ed61cb0 msgr2=0x55653ed604e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 -- 172.21.15.94:0/1790612068 shutdown_connections 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 1 -- 172.21.15.94:0/1790612068 wait complete. 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5918bb9740 20 client.16677 trim_cache size 0 max 16384 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:43:18.520 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:43:18.521 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:43:18.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5915eeb640 1 -- 172.21.15.94:0/971555999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58f80a3d20 msgr2=0x7f58f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5915eeb640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58f80a3d20 0x7f58f80a61e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5908005c40 tx=0x7f5908009530 comp rx=0 tx=0).stop 2024-04-21T08:43:18.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5915eeb640 1 -- 172.21.15.94:0/971555999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59101031b0 msgr2=0x7f5910198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.514+0000 7f5915eeb640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59101031b0 0x7f5910198730 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f5900005940 tx=0x7f590002d890 comp rx=0 tx=0).stop 2024-04-21T08:43:18.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.515+0000 7f5915eeb640 1 -- 172.21.15.94:0/971555999 shutdown_connections 2024-04-21T08:43:18.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.515+0000 7f5915eeb640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58f80a3d20 0x7f58f80a61e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.515+0000 7f5915eeb640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5910103ad0 0x7f5910198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.515+0000 7f5915eeb640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f591010c580 0x7f591019cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.515+0000 7f5915eeb640 1 --2- 172.21.15.94:0/971555999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59101031b0 0x7f5910198730 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.515+0000 7f5915eeb640 1 -- 172.21.15.94:0/971555999 >> 172.21.15.94:0/971555999 conn(0x7f59100771d0 msgr2=0x7f59100fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.515+0000 7f5915eeb640 1 -- 172.21.15.94:0/971555999 shutdown_connections 2024-04-21T08:43:18.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.515+0000 7f5915eeb640 1 -- 172.21.15.94:0/971555999 wait complete. 2024-04-21T08:43:18.526 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:43:18.526 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:43:18.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.625+0000 7fa9adbf4640 1 -- 172.21.15.94:0/3576426850 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9a8106f50 msgr2=0x7fa9a8107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.625+0000 7fa9adbf4640 1 --2- 172.21.15.94:0/3576426850 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9a8106f50 0x7fa9a8107330 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa998009cd0 tx=0x7fa99802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.625+0000 7fa9adbf4640 1 -- 172.21.15.94:0/3576426850 shutdown_connections 2024-04-21T08:43:18.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.625+0000 7fa9adbf4640 1 --2- 172.21.15.94:0/3576426850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9a81004c0 0x7fa9a8100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.625+0000 7fa9adbf4640 1 --2- 172.21.15.94:0/3576426850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a8100e80 0x7fa9a810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.625+0000 7fa9adbf4640 1 --2- 172.21.15.94:0/3576426850 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9a8106f50 0x7fa9a8107330 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.625+0000 7fa9adbf4640 1 -- 172.21.15.94:0/3576426850 >> 172.21.15.94:0/3576426850 conn(0x7fa9a8077450 msgr2=0x7fa9a80fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.625+0000 7fa9adbf4640 1 -- 172.21.15.94:0/3576426850 shutdown_connections 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.625+0000 7fa9adbf4640 1 -- 172.21.15.94:0/3576426850 wait complete. 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.625+0000 7fa9adbf4640 1 Processor -- start 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9adbf4640 1 -- start start 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9adbf4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9a81004c0 0x7fa9a81016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9adbf4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9a8100e80 0x7fa9a8101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9adbf4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a8106f50 0x7fa9a8102120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9adbf4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a8112260 con 0x7fa9a81004c0 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9adbf4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a81120e0 con 0x7fa9a8106f50 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9adbf4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a81123e0 con 0x7fa9a8100e80 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9a81004c0 0x7fa9a81016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9a81004c0 0x7fa9a81016a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39292/0 (socket says 172.21.15.94:39292) 2024-04-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a77fe640 1 -- 172.21.15.94:0/2746194546 learned_addr learned my addr 172.21.15.94:0/2746194546 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a7fff640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a8106f50 0x7fa9a8102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a77fe640 1 -- 172.21.15.94:0/2746194546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9a8100e80 msgr2=0x7fa9a8101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a77fe640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9a8100e80 0x7fa9a8101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a77fe640 1 -- 172.21.15.94:0/2746194546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a8106f50 msgr2=0x7fa9a8102120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a77fe640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a8106f50 0x7fa9a8102120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a77fe640 1 -- 172.21.15.94:0/2746194546 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a81a6c60 con 0x7fa9a81004c0 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a7fff640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a8106f50 0x7fa9a8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a77fe640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9a81004c0 0x7fa9a81016a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa99802c700 tx=0x7fa998035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a4ff9640 1 -- 172.21.15.94:0/2746194546 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998039040 con 0x7fa9a81004c0 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9adbf4640 1 -- 172.21.15.94:0/2746194546 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a81a6ef0 con 0x7fa9a81004c0 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a4ff9640 1 -- 172.21.15.94:0/2746194546 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa998032960 con 0x7fa9a81004c0 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9a4ff9640 1 -- 172.21.15.94:0/2746194546 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9980042c0 con 0x7fa9a81004c0 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.626+0000 7fa9adbf4640 1 -- 172.21.15.94:0/2746194546 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a81a7360 con 0x7fa9a81004c0 2024-04-21T08:43:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.627+0000 7fa9adbf4640 1 -- 172.21.15.94:0/2746194546 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a8069230 con 0x7fa9a81004c0 2024-04-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.628+0000 7fa9a4ff9640 1 -- 172.21.15.94:0/2746194546 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa998046030 con 0x7fa9a81004c0 2024-04-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.629+0000 7fa9a4ff9640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa9940a3d20 0x7fa9940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.629+0000 7fa9a6ffd640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa9940a3d20 0x7fa9940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.629+0000 7fa9a4ff9640 1 -- 172.21.15.94:0/2746194546 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fa9980f2da0 con 0x7fa9a81004c0 2024-04-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.629+0000 7fa9a6ffd640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa9940a3d20 0x7fa9940a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8103130 tx=0x7fa990005c70 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.633+0000 7fa9a4ff9640 1 -- 172.21.15.94:0/2746194546 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa99804d020 con 0x7fa9a81004c0 2024-04-21T08:43:18.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.848+0000 7fa9adbf4640 1 -- 172.21.15.94:0/2746194546 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9a810a480 con 0x7fa9a81004c0 2024-04-21T08:43:18.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.849+0000 7fa9a4ff9640 1 -- 172.21.15.94:0/2746194546 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7fa9980bf200 con 0x7fa9a81004c0 2024-04-21T08:43:18.849 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:43:18.849 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:43:18.849 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:43:18.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 -- 172.21.15.94:0/2746194546 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa9940a3d20 msgr2=0x7fa9940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa9940a3d20 0x7fa9940a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8103130 tx=0x7fa990005c70 comp rx=0 tx=0).stop 2024-04-21T08:43:18.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 -- 172.21.15.94:0/2746194546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9a81004c0 msgr2=0x7fa9a81016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9a81004c0 0x7fa9a81016a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa99802c700 tx=0x7fa998035ce0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 -- 172.21.15.94:0/2746194546 shutdown_connections 2024-04-21T08:43:18.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa9940a3d20 0x7fa9940a61e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9a8100e80 0x7fa9a8101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a8106f50 0x7fa9a8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 --2- 172.21.15.94:0/2746194546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9a81004c0 0x7fa9a81016a0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 -- 172.21.15.94:0/2746194546 >> 172.21.15.94:0/2746194546 conn(0x7fa9a8077450 msgr2=0x7fa9a8104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 -- 172.21.15.94:0/2746194546 shutdown_connections 2024-04-21T08:43:18.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.851+0000 7fa9adbf4640 1 -- 172.21.15.94:0/2746194546 wait complete. 2024-04-21T08:43:18.862 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:43:18.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 -- 172.21.15.94:0/1715214460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f918410e810 msgr2=0x7f9184110c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 --2- 172.21.15.94:0/1715214460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f918410e810 0x7f9184110c00 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f917c009ca0 tx=0x7f917c02c180 comp rx=0 tx=0).stop 2024-04-21T08:43:18.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 -- 172.21.15.94:0/1715214460 shutdown_connections 2024-04-21T08:43:18.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 --2- 172.21.15.94:0/1715214460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f918410e810 0x7f9184110c00 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 --2- 172.21.15.94:0/1715214460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f918406aa90 0x7f918406af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 --2- 172.21.15.94:0/1715214460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9184102f00 0x7f918406a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 -- 172.21.15.94:0/1715214460 >> 172.21.15.94:0/1715214460 conn(0x7f9184076c50 msgr2=0x7f9184077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 -- 172.21.15.94:0/1715214460 shutdown_connections 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 -- 172.21.15.94:0/1715214460 wait complete. 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 Processor -- start 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.964+0000 7f918c550640 1 -- start start 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918c550640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f918406aa90 0x7f91841a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918c550640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184102f00 0x7f91841a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918c550640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f918410e810 0x7f918419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918c550640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9184114440 con 0x7f918406aa90 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918c550640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91841142c0 con 0x7f9184102f00 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918c550640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f91841145c0 con 0x7f918410e810 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918a2ec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f918406aa90 0x7f91841a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918a2ec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f918406aa90 0x7f91841a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39308/0 (socket says 172.21.15.94:39308) 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918a2ec640 1 -- 172.21.15.94:0/2369545105 learned_addr learned my addr 172.21.15.94:0/2369545105 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918aaed640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f918410e810 0x7f918419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f9189aeb640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184102f00 0x7f91841a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918a2ec640 1 -- 172.21.15.94:0/2369545105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f918410e810 msgr2=0x7f918419a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918a2ec640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f918410e810 0x7f918419a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918a2ec640 1 -- 172.21.15.94:0/2369545105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184102f00 msgr2=0x7f91841a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918a2ec640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184102f00 0x7f91841a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918a2ec640 1 -- 172.21.15.94:0/2369545105 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918419af70 con 0x7f918406aa90 2024-04-21T08:43:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918a2ec640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f918406aa90 0x7f91841a0630 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f917800c970 tx=0x7f917800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f91777fe640 1 -- 172.21.15.94:0/2369545105 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9178007bb0 con 0x7f918406aa90 2024-04-21T08:43:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f91777fe640 1 -- 172.21.15.94:0/2369545105 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9178007d50 con 0x7f918406aa90 2024-04-21T08:43:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f91777fe640 1 -- 172.21.15.94:0/2369545105 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917800f450 con 0x7f918406aa90 2024-04-21T08:43:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918c550640 1 -- 172.21.15.94:0/2369545105 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f918419b260 con 0x7f918406aa90 2024-04-21T08:43:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.965+0000 7f918c550640 1 -- 172.21.15.94:0/2369545105 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91841a7480 con 0x7f918406aa90 2024-04-21T08:43:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.967+0000 7f91777fe640 1 -- 172.21.15.94:0/2369545105 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9178004f20 con 0x7f918406aa90 2024-04-21T08:43:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.968+0000 7f918f219740 1 Processor -- start 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.968+0000 7f918f219740 1 -- start start 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.968+0000 7f91777fe640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f91700a3d20 0x7f91700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.968+0000 7f918f219740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5585800124d0 0x5585800128b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.968+0000 7f918f219740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 0x55858001b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.968+0000 7f918f219740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55858001ba00 0x55858001de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.968+0000 7f918f219740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55857ff771f0 con 0x5585800124d0 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.968+0000 7f918f219740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55857ff144b0 con 0x55858001ba00 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.968+0000 7f918f219740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55857ff731b0 con 0x558580012e80 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.968+0000 7f918a2ec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5585800124d0 0x5585800128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918aaed640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f91700a3d20 0x7f91700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918a2ec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5585800124d0 0x5585800128b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39310/0 (socket says 172.21.15.94:39310) 2024-04-21T08:43:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918a2ec640 1 -- 172.21.15.94:0/871895346 learned_addr learned my addr 172.21.15.94:0/871895346 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f91777fe640 1 -- 172.21.15.94:0/2369545105 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f91780d0a10 con 0x7f918406aa90 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f9189aeb640 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 0x55858001b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918aaed640 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55858001ba00 0x55858001de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918a2ec640 1 -- 172.21.15.94:0/871895346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 msgr2=0x55858001b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918a2ec640 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 0x55858001b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918a2ec640 1 -- 172.21.15.94:0/871895346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55858001ba00 msgr2=0x55858001de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918a2ec640 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55858001ba00 0x55858001de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918a2ec640 1 -- 172.21.15.94:0/871895346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55857ff061b0 con 0x5585800124d0 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f9189aeb640 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 0x55858001b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918aaed640 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55858001ba00 0x55858001de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918a2ec640 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5585800124d0 0x5585800128b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f917800c940 tx=0x7f9178091a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a096cf2117987474 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f918aaed640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f91700a3d20 0x7f91700a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f917c006010 tx=0x7f917c02e040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f9174bfe640 1 -- 172.21.15.94:0/871895346 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780055f0 con 0x5585800124d0 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f9174bfe640 1 -- 172.21.15.94:0/871895346 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f917809ccd0 con 0x5585800124d0 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.969+0000 7f9174bfe640 1 -- 172.21.15.94:0/871895346 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917809a8d0 con 0x5585800124d0 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 -- 172.21.15.94:0/871895346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5585800124d0 msgr2=0x5585800128b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5585800124d0 0x5585800128b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f917800c940 tx=0x7f9178091a60 comp rx=0 tx=0).stop 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 -- 172.21.15.94:0/871895346 shutdown_connections 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 0x55858001b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55858001ba00 0x55858001de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 --2- 172.21.15.94:0/871895346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5585800124d0 0x5585800128b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 -- 172.21.15.94:0/871895346 >> 172.21.15.94:0/871895346 conn(0x55858000cba0 msgr2=0x55858000cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 -- 172.21.15.94:0/871895346 shutdown_connections 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 -- 172.21.15.94:0/871895346 wait complete. 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 -- start start 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.970+0000 7f918f219740 1 Processor -- start 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918f219740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5585800124d0 0x55858001b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918f219740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 0x5585800186f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918f219740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x558580018d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918f219740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55858001e910 con 0x55858001ba00 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918f219740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55857ff30500 con 0x5585800124d0 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918f219740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55858001ea90 con 0x558580012e80 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918f219740 10 client.0 _ll_register_callbacks cb 0x55857fefeb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918aaed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x558580018d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918aaed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x558580018d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39314/0 (socket says 172.21.15.94:39314) 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918aaed640 1 -- 172.21.15.94:0/694645109 learned_addr learned my addr 172.21.15.94:0/694645109 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f9189aeb640 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 0x5585800186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918a2ec640 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5585800124d0 0x55858001b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918aaed640 1 -- 172.21.15.94:0/694645109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 msgr2=0x5585800186f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918aaed640 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 0x5585800186f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918aaed640 1 -- 172.21.15.94:0/694645109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5585800124d0 msgr2=0x55858001b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918aaed640 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5585800124d0 0x55858001b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918aaed640 1 -- 172.21.15.94:0/694645109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55857fc893f0 con 0x55858001ba00 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918a2ec640 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5585800124d0 0x55858001b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:18.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f9189aeb640 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 0x5585800186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918aaed640 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x558580018d00 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f917c036000 tx=0x7f917c02c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3413d6593634b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f916dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f916dffb640 1 -- 172.21.15.94:0/694645109 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c04f070 con 0x55858001ba00 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f916dffb640 1 -- 172.21.15.94:0/694645109 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f917c032560 con 0x55858001ba00 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918f219740 1 -- 172.21.15.94:0/694645109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f917809ccd0 con 0x55858001ba00 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f916dffb640 1 -- 172.21.15.94:0/694645109 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f917c0327a0 con 0x55858001ba00 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918f219740 10 client.16695 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.971+0000 7f918f219740 1 -- 172.21.15.94:0/694645109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55857fd537d0 con 0x55858001ba00 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f916dffb640 1 -- 172.21.15.94:0/694645109 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f917809ccd0 con 0x55858001ba00 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f916dffb640 1 -- 172.21.15.94:0/694645109 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f917c04a070 con 0x55858001ba00 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f918f219740 10 client.16695 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f918f219740 10 client.16695 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f918f219740 20 client.16695 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f918f219740 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55858002bb00 0x55858002df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f918f219740 4 client.16695 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f918f219740 1 -- 172.21.15.94:0/694645109 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55857ff16930 con 0x55858002bb00 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f918a2ec640 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55858002bb00 0x55858002df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f918a2ec640 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55858002bb00 0x55858002df30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dfccfe2997546c32 server_cookie=b87063d1d3941b15 in_seq=0 out_seq=0 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.972+0000 7f916dffb640 10 client.16695 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f916dffb640 1 -- 172.21.15.94:0/694645109 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55857ff16930 con 0x55858002bb00 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f916dffb640 10 client.16695 handle_command_reply: tid=0 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 client.16695 shutdown 2024-04-21T08:43:18.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 -- 172.21.15.94:0/694645109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 msgr2=0x558580018d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x558580018d00 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f917c036000 tx=0x7f917c02c180 comp rx=0 tx=0).stop 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 -- 172.21.15.94:0/694645109 shutdown_connections 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55858002bb00 0x55858002df30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558580012e80 0x5585800186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x558580018d00 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 --2- 172.21.15.94:0/694645109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5585800124d0 0x55858001b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f916dffb640 0 client.16695 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 -- 172.21.15.94:0/694645109 >> 172.21.15.94:0/694645109 conn(0x55858000cba0 msgr2=0x55858000e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 -- 172.21.15.94:0/694645109 shutdown_connections 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.973+0000 7f918f219740 1 -- 172.21.15.94:0/694645109 wait complete. 2024-04-21T08:43:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.974+0000 7f918f219740 20 client.16695 trim_cache size 0 max 16384 2024-04-21T08:43:18.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918f219740 1 Processor -- start 2024-04-21T08:43:18.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918f219740 1 -- start start 2024-04-21T08:43:18.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918f219740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5585800124d0 0x5585800277d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918f219740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558580012e80 0x55858000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918f219740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x55858001ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918f219740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f91841b3f30 con 0x55858001ba00 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918f219740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91841b1800 con 0x558580012e80 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918f219740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f91841af650 con 0x5585800124d0 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918aaed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x55858001ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918aaed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x55858001ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39322/0 (socket says 172.21.15.94:39322) 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918aaed640 1 -- 172.21.15.94:0/1811113739 learned_addr learned my addr 172.21.15.94:0/1811113739 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918aaed640 1 -- 172.21.15.94:0/1811113739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5585800124d0 msgr2=0x5585800277d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918aaed640 1 --2- 172.21.15.94:0/1811113739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5585800124d0 0x5585800277d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918aaed640 1 -- 172.21.15.94:0/1811113739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558580012e80 msgr2=0x55858000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918aaed640 1 --2- 172.21.15.94:0/1811113739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558580012e80 0x55858000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918aaed640 1 -- 172.21.15.94:0/1811113739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585800120b0 con 0x55858001ba00 2024-04-21T08:43:18.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f918aaed640 1 --2- 172.21.15.94:0/1811113739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x55858001ae20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f917c02c6c0 tx=0x7f917c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb41dcfed5232bfb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f916ffff640 1 -- 172.21.15.94:0/1811113739 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c064070 con 0x55858001ba00 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.987+0000 7f916ffff640 1 -- 172.21.15.94:0/1811113739 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f917c045070 con 0x55858001ba00 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f916ffff640 1 -- 172.21.15.94:0/1811113739 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c04f040 con 0x55858001ba00 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 -- 172.21.15.94:0/1811113739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 msgr2=0x55858001ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 --2- 172.21.15.94:0/1811113739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x55858001ae20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f917c02c6c0 tx=0x7f917c0095c0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 -- 172.21.15.94:0/1811113739 shutdown_connections 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 --2- 172.21.15.94:0/1811113739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x55858001ae20 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 --2- 172.21.15.94:0/1811113739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558580012e80 0x55858000ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 --2- 172.21.15.94:0/1811113739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5585800124d0 0x5585800277d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 -- 172.21.15.94:0/1811113739 >> 172.21.15.94:0/1811113739 conn(0x55858000cba0 msgr2=0x55858000b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 -- 172.21.15.94:0/1811113739 shutdown_connections 2024-04-21T08:43:18.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 -- 172.21.15.94:0/1811113739 wait complete. 2024-04-21T08:43:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 -- start start 2024-04-21T08:43:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.988+0000 7f918f219740 1 Processor -- start 2024-04-21T08:43:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918f219740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5585800124d0 0x558580024250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918f219740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558580012e80 0x5585800151f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918f219740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x5585800158e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918f219740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f91841afcc0 con 0x55858001ba00 2024-04-21T08:43:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918f219740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91841b1130 con 0x558580012e80 2024-04-21T08:43:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918f219740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f91841b0390 con 0x5585800124d0 2024-04-21T08:43:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918f219740 10 client.0 _ll_register_callbacks cb 0x7f917c009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918aaed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x5585800158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918aaed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x5585800158e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39326/0 (socket says 172.21.15.94:39326) 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918aaed640 1 -- 172.21.15.94:0/2793274958 learned_addr learned my addr 172.21.15.94:0/2793274958 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918aaed640 1 -- 172.21.15.94:0/2793274958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5585800124d0 msgr2=0x558580024250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918aaed640 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5585800124d0 0x558580024250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918aaed640 1 -- 172.21.15.94:0/2793274958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558580012e80 msgr2=0x5585800151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918aaed640 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558580012e80 0x5585800151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918aaed640 1 -- 172.21.15.94:0/2793274958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558580091070 con 0x55858001ba00 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918aaed640 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x5585800158e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f917c02e010 tx=0x7f917c032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8436b40ab5f3bdb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f916dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f916dffb640 1 -- 172.21.15.94:0/2793274958 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c064070 con 0x55858001ba00 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f916dffb640 1 -- 172.21.15.94:0/2793274958 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f917c034640 con 0x55858001ba00 2024-04-21T08:43:18.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918f219740 1 -- 172.21.15.94:0/2793274958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f917c045070 con 0x55858001ba00 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f916dffb640 1 -- 172.21.15.94:0/2793274958 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f917c071050 con 0x55858001ba00 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918f219740 10 client.16701 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.989+0000 7f918f219740 1 -- 172.21.15.94:0/2793274958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558580092210 con 0x55858001ba00 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f916dffb640 1 -- 172.21.15.94:0/2793274958 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f917c045070 con 0x55858001ba00 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f916dffb640 1 -- 172.21.15.94:0/2793274958 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f917c04a040 con 0x55858001ba00 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f918f219740 10 client.16701 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f918f219740 10 client.16701 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f918f219740 20 client.16701 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f918f219740 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x558580092870 0x558580094cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f918f219740 4 client.16701 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f918f219740 1 -- 172.21.15.94:0/2793274958 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55857ff84c10 con 0x558580092870 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f918a2ec640 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x558580092870 0x558580094cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f918a2ec640 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x558580092870 0x558580094cb0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec3eedbe7ddcf908 server_cookie=3af5d05cd4af985 in_seq=0 out_seq=0 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.990+0000 7f916dffb640 10 client.16701 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f916dffb640 1 -- 172.21.15.94:0/2793274958 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55857ff84c10 con 0x558580092870 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f916dffb640 10 client.16701 handle_command_reply: tid=0 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 client.16701 shutdown 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 -- 172.21.15.94:0/2793274958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 msgr2=0x5585800158e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x5585800158e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f917c02e010 tx=0x7f917c032e90 comp rx=0 tx=0).stop 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 -- 172.21.15.94:0/2793274958 shutdown_connections 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x558580092870 0x558580094cb0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55858001ba00 0x5585800158e0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558580012e80 0x5585800151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 --2- 172.21.15.94:0/2793274958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5585800124d0 0x558580024250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f916dffb640 0 client.16701 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:43:18.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 -- 172.21.15.94:0/2793274958 >> 172.21.15.94:0/2793274958 conn(0x55858000cba0 msgr2=0x55858000bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 -- 172.21.15.94:0/2793274958 shutdown_connections 2024-04-21T08:43:18.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 1 -- 172.21.15.94:0/2793274958 wait complete. 2024-04-21T08:43:18.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918f219740 20 client.16701 trim_cache size 0 max 16384 2024-04-21T08:43:18.999 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:43:19.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 -- 172.21.15.94:0/2369545105 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f91700a3d20 msgr2=0x7f91700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f91700a3d20 0x7f91700a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f917c006010 tx=0x7f917c02e040 comp rx=0 tx=0).stop 2024-04-21T08:43:19.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 -- 172.21.15.94:0/2369545105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f918406aa90 msgr2=0x7f91841a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f918406aa90 0x7f91841a0630 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f917800c970 tx=0x7f917800ce30 comp rx=0 tx=0).stop 2024-04-21T08:43:19.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 -- 172.21.15.94:0/2369545105 shutdown_connections 2024-04-21T08:43:19.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f91700a3d20 0x7f91700a61e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f918410e810 0x7f918419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184102f00 0x7f91841a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 --2- 172.21.15.94:0/2369545105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f918406aa90 0x7f91841a0630 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 -- 172.21.15.94:0/2369545105 >> 172.21.15.94:0/2369545105 conn(0x7f9184076c50 msgr2=0x7f918410f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 -- 172.21.15.94:0/2369545105 shutdown_connections 2024-04-21T08:43:19.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:18.991+0000 7f918c550640 1 -- 172.21.15.94:0/2369545105 wait complete. 2024-04-21T08:43:19.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:43:19.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.103+0000 7ff2329b6640 1 -- 172.21.15.94:0/667789588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff22c1073c0 msgr2=0x7ff22c10eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.103+0000 7ff2329b6640 1 --2- 172.21.15.94:0/667789588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff22c1073c0 0x7ff22c10eeb0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff224009ca0 tx=0x7ff22402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.103+0000 7ff2329b6640 1 -- 172.21.15.94:0/667789588 shutdown_connections 2024-04-21T08:43:19.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.103+0000 7ff2329b6640 1 --2- 172.21.15.94:0/667789588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff22c1073c0 0x7ff22c10eeb0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.103+0000 7ff2329b6640 1 --2- 172.21.15.94:0/667789588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff22c100cb0 0x7ff22c106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.103+0000 7ff2329b6640 1 --2- 172.21.15.94:0/667789588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff22c100300 0x7ff22c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.103+0000 7ff2329b6640 1 -- 172.21.15.94:0/667789588 >> 172.21.15.94:0/667789588 conn(0x7ff22c076fb0 msgr2=0x7ff22c0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.103+0000 7ff2329b6640 1 -- 172.21.15.94:0/667789588 shutdown_connections 2024-04-21T08:43:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.103+0000 7ff2329b6640 1 -- 172.21.15.94:0/667789588 wait complete. 2024-04-21T08:43:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.103+0000 7ff2329b6640 1 Processor -- start 2024-04-21T08:43:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff2329b6640 1 -- start start 2024-04-21T08:43:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff2329b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff22c100300 0x7ff22c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff2329b6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff22c1073c0 0x7ff22c19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff2329b6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff22c1985a0 0x7ff22c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff2329b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1122c0 con 0x7ff22c100300 2024-04-21T08:43:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff2329b6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c112140 con 0x7ff22c1985a0 2024-04-21T08:43:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff2329b6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c112440 con 0x7ff22c1073c0 2024-04-21T08:43:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff22bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff22c100300 0x7ff22c19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff22bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff22c100300 0x7ff22c19e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39350/0 (socket says 172.21.15.94:39350) 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff22bfff640 1 -- 172.21.15.94:0/3733091007 learned_addr learned my addr 172.21.15.94:0/3733091007 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff22c1073c0 0x7ff22c19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff230f53640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff22c1985a0 0x7ff22c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff22bfff640 1 -- 172.21.15.94:0/3733091007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff22c1073c0 msgr2=0x7ff22c19e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff22bfff640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff22c1073c0 0x7ff22c19e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff22bfff640 1 -- 172.21.15.94:0/3733091007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff22c1985a0 msgr2=0x7ff22c198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff22bfff640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff22c1985a0 0x7ff22c198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff22bfff640 1 -- 172.21.15.94:0/3733091007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22c1990c0 con 0x7ff22c100300 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff230f53640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff22c1985a0 0x7ff22c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.104+0000 7ff22bfff640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff22c100300 0x7ff22c19e3f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00d770 tx=0x7ff21c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.105+0000 7ff2297fa640 1 -- 172.21.15.94:0/3733091007 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c017070 con 0x7ff22c100300 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.105+0000 7ff2329b6640 1 -- 172.21.15.94:0/3733091007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff22c1993b0 con 0x7ff22c100300 2024-04-21T08:43:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.105+0000 7ff2297fa640 1 -- 172.21.15.94:0/3733091007 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff21c0054d0 con 0x7ff22c100300 2024-04-21T08:43:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.105+0000 7ff2297fa640 1 -- 172.21.15.94:0/3733091007 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c004c30 con 0x7ff22c100300 2024-04-21T08:43:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.105+0000 7ff2329b6640 1 -- 172.21.15.94:0/3733091007 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff22c1a5600 con 0x7ff22c100300 2024-04-21T08:43:19.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.107+0000 7ff2355aa740 1 Processor -- start 2024-04-21T08:43:19.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.107+0000 7ff2297fa640 1 -- 172.21.15.94:0/3733091007 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff21c004e10 con 0x7ff22c100300 2024-04-21T08:43:19.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff2355aa740 1 -- start start 2024-04-21T08:43:19.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 0x563b963538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b96353e80 0x563b9635c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b9635ca00 0x563b9635ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff2355aa740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563b962b81f0 con 0x563b9635ca00 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff2355aa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563b962554b0 con 0x563b963534d0 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff2355aa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563b962b41b0 con 0x563b96353e80 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b9635ca00 0x563b9635ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b9635ca00 0x563b9635ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39354/0 (socket says 172.21.15.94:39354) 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff22b7fe640 1 -- 172.21.15.94:0/504548935 learned_addr learned my addr 172.21.15.94:0/504548935 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff230f53640 1 --2- 172.21.15.94:0/504548935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 0x563b963538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff22bfff640 1 --2- 172.21.15.94:0/504548935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b96353e80 0x563b9635c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff22b7fe640 1 -- 172.21.15.94:0/504548935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b96353e80 msgr2=0x563b9635c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff2297fa640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2140a3d20 0x7ff2140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/504548935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b96353e80 0x563b9635c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff22b7fe640 1 -- 172.21.15.94:0/504548935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 msgr2=0x563b963538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/504548935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 0x563b963538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff22b7fe640 1 -- 172.21.15.94:0/504548935 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b962471b0 con 0x563b9635ca00 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff230f53640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2140a3d20 0x7ff2140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff2297fa640 1 -- 172.21.15.94:0/3733091007 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff21c0d0760 con 0x7ff22c100300 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.108+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/504548935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b9635ca00 0x563b9635ee50 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff220005940 tx=0x7ff22000e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7505cdff4c190aa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2127fc640 1 -- 172.21.15.94:0/504548935 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220014070 con 0x563b9635ca00 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2127fc640 1 -- 172.21.15.94:0/504548935 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff220005020 con 0x563b9635ca00 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2127fc640 1 -- 172.21.15.94:0/504548935 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220004bc0 con 0x563b9635ca00 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2355aa740 1 -- 172.21.15.94:0/504548935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b9635ca00 msgr2=0x563b9635ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff230f53640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2140a3d20 0x7ff2140a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff224009430 tx=0x7ff2240095c0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2355aa740 1 --2- 172.21.15.94:0/504548935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b9635ca00 0x563b9635ee50 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff220005940 tx=0x7ff22000e350 comp rx=0 tx=0).stop 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2355aa740 1 -- 172.21.15.94:0/504548935 shutdown_connections 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2355aa740 1 --2- 172.21.15.94:0/504548935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b96353e80 0x563b9635c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2355aa740 1 --2- 172.21.15.94:0/504548935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b9635ca00 0x563b9635ee50 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2355aa740 1 --2- 172.21.15.94:0/504548935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 0x563b963538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2355aa740 1 -- 172.21.15.94:0/504548935 >> 172.21.15.94:0/504548935 conn(0x563b9634dba0 msgr2=0x563b9634df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2355aa740 1 -- 172.21.15.94:0/504548935 shutdown_connections 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.109+0000 7ff2355aa740 1 -- 172.21.15.94:0/504548935 wait complete. 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff2355aa740 1 -- start start 2024-04-21T08:43:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff2355aa740 1 Processor -- start 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 0x563b9635c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96359620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 0x563b96359b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff2355aa740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563b9635f970 con 0x563b96353e80 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff2355aa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563b96271500 con 0x563b963534d0 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff2355aa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563b9635faf0 con 0x563b9635ca00 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff2355aa740 10 client.0 _ll_register_callbacks cb 0x563b9623fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96359620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96359620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39362/0 (socket says 172.21.15.94:39362) 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff22b7fe640 1 -- 172.21.15.94:0/2228621292 learned_addr learned my addr 172.21.15.94:0/2228621292 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff230f53640 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 0x563b96359b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.110+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 0x563b9635c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff22b7fe640 1 -- 172.21.15.94:0/2228621292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 msgr2=0x563b96359b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 0x563b96359b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff22b7fe640 1 -- 172.21.15.94:0/2228621292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 msgr2=0x563b9635c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 0x563b9635c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff22b7fe640 1 -- 172.21.15.94:0/2228621292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b95fca3f0 con 0x563b96353e80 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff230f53640 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 0x563b96359b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 0x563b9635c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96359620 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ff22000eff0 tx=0x7ff220004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a22eea8ae3d4aa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff20b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2228621292 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220027060 con 0x563b96353e80 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2228621292 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff220004b50 con 0x563b96353e80 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff2355aa740 1 -- 172.21.15.94:0/2228621292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff220005020 con 0x563b96353e80 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2228621292 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff22002e050 con 0x563b96353e80 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff2355aa740 10 client.16713 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff2355aa740 1 -- 172.21.15.94:0/2228621292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563b960947d0 con 0x563b96353e80 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2228621292 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff220005020 con 0x563b96353e80 2024-04-21T08:43:19.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2228621292 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff220010430 con 0x563b96353e80 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff2355aa740 10 client.16713 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff2355aa740 10 client.16713 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff2355aa740 20 client.16713 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x563b9636cac0 0x563b9636eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff2355aa740 4 client.16713 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.111+0000 7ff2355aa740 1 -- 172.21.15.94:0/2228621292 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b96257930 con 0x563b9636cac0 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.112+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x563b9636cac0 0x563b9636eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.112+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x563b9636cac0 0x563b9636eeb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=176cecdb02b0e1fc server_cookie=e16ee346685e3d9b in_seq=0 out_seq=0 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.112+0000 7ff20b7fe640 10 client.16713 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2228621292 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b96257930 con 0x563b9636cac0 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff20b7fe640 10 client.16713 handle_command_reply: tid=0 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 client.16713 shutdown 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 -- 172.21.15.94:0/2228621292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 msgr2=0x563b96359620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96359620 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ff22000eff0 tx=0x7ff220004740 comp rx=0 tx=0).stop 2024-04-21T08:43:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 -- 172.21.15.94:0/2228621292 shutdown_connections 2024-04-21T08:43:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 0x563b96359b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x563b9636cac0 0x563b9636eeb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96359620 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff20b7fe640 0 client.16713 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:43:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2228621292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b963534d0 0x563b9635c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 -- 172.21.15.94:0/2228621292 >> 172.21.15.94:0/2228621292 conn(0x563b9634dba0 msgr2=0x563b9634f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 -- 172.21.15.94:0/2228621292 shutdown_connections 2024-04-21T08:43:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 1 -- 172.21.15.94:0/2228621292 wait complete. 2024-04-21T08:43:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.113+0000 7ff2355aa740 20 client.16713 trim_cache size 0 max 16384 2024-04-21T08:43:19.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.126+0000 7ff2355aa740 1 Processor -- start 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.126+0000 7ff2355aa740 1 -- start start 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.126+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b963534d0 0x563b96368830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.126+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b96353e80 0x563b96358780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.126+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 0x563b9634ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.126+0000 7ff2355aa740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1b1c80 con 0x563b963534d0 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.126+0000 7ff2355aa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1af550 con 0x563b96353e80 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.126+0000 7ff2355aa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1ad3a0 con 0x563b9635ca00 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff22bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b963534d0 0x563b96368830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff22bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b963534d0 0x563b96368830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39378/0 (socket says 172.21.15.94:39378) 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff22bfff640 1 -- 172.21.15.94:0/2320812823 learned_addr learned my addr 172.21.15.94:0/2320812823 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff230f53640 1 --2- 172.21.15.94:0/2320812823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 0x563b9634ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff22bfff640 1 -- 172.21.15.94:0/2320812823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 msgr2=0x563b9634ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2320812823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 0x563b9634ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff22bfff640 1 -- 172.21.15.94:0/2320812823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b96353e80 msgr2=0x563b96358780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2320812823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b96353e80 0x563b96358780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff22bfff640 1 -- 172.21.15.94:0/2320812823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b9634cfc0 con 0x563b963534d0 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff230f53640 1 --2- 172.21.15.94:0/2320812823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 0x563b9634ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2320812823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b963534d0 0x563b96368830 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00d740 tx=0x7ff21c0d6550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a79f4920f67c503 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff211ffb640 1 -- 172.21.15.94:0/2320812823 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c020040 con 0x563b963534d0 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff211ffb640 1 -- 172.21.15.94:0/2320812823 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff21c005680 con 0x563b963534d0 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.127+0000 7ff211ffb640 1 -- 172.21.15.94:0/2320812823 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c0a4f90 con 0x563b963534d0 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 -- 172.21.15.94:0/2320812823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b963534d0 msgr2=0x563b96368830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2320812823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b963534d0 0x563b96368830 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00d740 tx=0x7ff21c0d6550 comp rx=0 tx=0).stop 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 -- 172.21.15.94:0/2320812823 shutdown_connections 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2320812823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b9635ca00 0x563b9634ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2320812823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b96353e80 0x563b96358780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2320812823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b963534d0 0x563b96368830 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 -- 172.21.15.94:0/2320812823 >> 172.21.15.94:0/2320812823 conn(0x563b9634dba0 msgr2=0x563b9634c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 -- 172.21.15.94:0/2320812823 shutdown_connections 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 -- 172.21.15.94:0/2320812823 wait complete. 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 -- start start 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 Processor -- start 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b963534d0 0x563b96360320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96360860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b9635ca00 0x563b96364740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1ada10 con 0x563b96353e80 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1aee80 con 0x563b9635ca00 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1ae0e0 con 0x563b963534d0 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.128+0000 7ff2355aa740 10 client.0 _ll_register_callbacks cb 0x7ff22000c9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96360860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96360860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39386/0 (socket says 172.21.15.94:39386) 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff22b7fe640 1 -- 172.21.15.94:0/2003119691 learned_addr learned my addr 172.21.15.94:0/2003119691 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b963534d0 0x563b96360320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff22b7fe640 1 -- 172.21.15.94:0/2003119691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b963534d0 msgr2=0x563b96360320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b963534d0 0x563b96360320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff22b7fe640 1 -- 172.21.15.94:0/2003119691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b9635ca00 msgr2=0x563b96364740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b9635ca00 0x563b96364740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff22b7fe640 1 -- 172.21.15.94:0/2003119691 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b96364d80 con 0x563b96353e80 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96360860 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff22000efc0 tx=0x7ff220010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f671a563679f1d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff20b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2003119691 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220027070 con 0x563b96353e80 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2003119691 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff220010ba0 con 0x563b96353e80 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff2355aa740 1 -- 172.21.15.94:0/2003119691 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff21c005680 con 0x563b96353e80 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2003119691 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff22003b050 con 0x563b96353e80 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff2355aa740 10 client.16719 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff2355aa740 1 -- 172.21.15.94:0/2003119691 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563b963d2740 con 0x563b96353e80 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2003119691 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff21c005680 con 0x563b96353e80 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.129+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2003119691 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff2200035f0 con 0x563b96353e80 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff2355aa740 10 client.16719 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:19.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff2355aa740 10 client.16719 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff2355aa740 20 client.16719 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x563b963d2e30 0x563b963d5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff2355aa740 4 client.16719 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff2355aa740 1 -- 172.21.15.94:0/2003119691 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563b962c5c10 con 0x563b963d2e30 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x563b963d2e30 0x563b963d5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x563b963d2e30 0x563b963d5250 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa6db3b186ad98d2 server_cookie=cd048c67172cad8f in_seq=0 out_seq=0 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff20b7fe640 10 client.16719 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff20b7fe640 1 -- 172.21.15.94:0/2003119691 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563b962c5c10 con 0x563b963d2e30 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff20b7fe640 10 client.16719 handle_command_reply: tid=0 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.130+0000 7ff2355aa740 1 client.16719 shutdown 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 1 -- 172.21.15.94:0/2003119691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 msgr2=0x563b96360860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96360860 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff22000efc0 tx=0x7ff220010ec0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 1 -- 172.21.15.94:0/2003119691 shutdown_connections 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x563b963d2e30 0x563b963d5250 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b9635ca00 0x563b96364740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563b96353e80 0x563b96360860 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff20b7fe640 0 client.16719 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:43:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 1 --2- 172.21.15.94:0/2003119691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563b963534d0 0x563b96360320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 1 -- 172.21.15.94:0/2003119691 >> 172.21.15.94:0/2003119691 conn(0x563b9634dba0 msgr2=0x563b96354260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 1 -- 172.21.15.94:0/2003119691 shutdown_connections 2024-04-21T08:43:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 1 -- 172.21.15.94:0/2003119691 wait complete. 2024-04-21T08:43:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2355aa740 20 client.16719 trim_cache size 0 max 16384 2024-04-21T08:43:19.137 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:43:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 -- 172.21.15.94:0/3733091007 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2140a3d20 msgr2=0x7ff2140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2140a3d20 0x7ff2140a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff224009430 tx=0x7ff2240095c0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 -- 172.21.15.94:0/3733091007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff22c100300 msgr2=0x7ff22c19e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff22c100300 0x7ff22c19e3f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00d770 tx=0x7ff21c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:43:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 -- 172.21.15.94:0/3733091007 shutdown_connections 2024-04-21T08:43:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff2140a3d20 0x7ff2140a61e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff22c1073c0 0x7ff22c19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff22c1985a0 0x7ff22c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 --2- 172.21.15.94:0/3733091007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff22c100300 0x7ff22c19e3f0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 -- 172.21.15.94:0/3733091007 >> 172.21.15.94:0/3733091007 conn(0x7ff22c076fb0 msgr2=0x7ff22c105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 -- 172.21.15.94:0/3733091007 shutdown_connections 2024-04-21T08:43:19.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.131+0000 7ff2329b6640 1 -- 172.21.15.94:0/3733091007 wait complete. 2024-04-21T08:43:19.142 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:43:19.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.243+0000 7ff779ec1640 1 -- 172.21.15.94:0/261157158 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff774101090 msgr2=0x7ff774101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.243+0000 7ff779ec1640 1 --2- 172.21.15.94:0/261157158 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff774101090 0x7ff774101470 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff764009900 tx=0x7ff76402bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:19.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.243+0000 7ff779ec1640 1 -- 172.21.15.94:0/261157158 shutdown_connections 2024-04-21T08:43:19.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.243+0000 7ff779ec1640 1 --2- 172.21.15.94:0/261157158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff774106160 0x7ff77410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.243+0000 7ff779ec1640 1 --2- 172.21.15.94:0/261157158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774101a40 0x7ff774105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.243+0000 7ff779ec1640 1 --2- 172.21.15.94:0/261157158 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff774101090 0x7ff774101470 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.243+0000 7ff779ec1640 1 -- 172.21.15.94:0/261157158 >> 172.21.15.94:0/261157158 conn(0x7ff774077f50 msgr2=0x7ff774078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.243+0000 7ff779ec1640 1 -- 172.21.15.94:0/261157158 shutdown_connections 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.243+0000 7ff779ec1640 1 -- 172.21.15.94:0/261157158 wait complete. 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff779ec1640 1 Processor -- start 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff779ec1640 1 -- start start 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff779ec1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff774101090 0x7ff774102b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff779ec1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106160 0x7ff774103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff779ec1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7741035a0 0x7ff7741a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff779ec1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741110a0 con 0x7ff774101090 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff779ec1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff774110f20 con 0x7ff774106160 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff779ec1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff774111220 con 0x7ff7741035a0 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff7737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff774101090 0x7ff774102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff7737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff774101090 0x7ff774102b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39414/0 (socket says 172.21.15.94:39414) 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff7737fe640 1 -- 172.21.15.94:0/6267554 learned_addr learned my addr 172.21.15.94:0/6267554 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:19.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff773fff640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7741035a0 0x7ff7741a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff772ffd640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106160 0x7ff774103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff7737fe640 1 -- 172.21.15.94:0/6267554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7741035a0 msgr2=0x7ff7741a69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff7737fe640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7741035a0 0x7ff7741a69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff7737fe640 1 -- 172.21.15.94:0/6267554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106160 msgr2=0x7ff774103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff7737fe640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106160 0x7ff774103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.244+0000 7ff7737fe640 1 -- 172.21.15.94:0/6267554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7741a7040 con 0x7ff774101090 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.245+0000 7ff7737fe640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff774101090 0x7ff774102b20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff764005940 tx=0x7ff76402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.245+0000 7ff770ff9640 1 -- 172.21.15.94:0/6267554 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff764043020 con 0x7ff774101090 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.245+0000 7ff770ff9640 1 -- 172.21.15.94:0/6267554 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff76402cbe0 con 0x7ff774101090 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.245+0000 7ff770ff9640 1 -- 172.21.15.94:0/6267554 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff764003bf0 con 0x7ff774101090 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.245+0000 7ff779ec1640 1 -- 172.21.15.94:0/6267554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7741a72d0 con 0x7ff774101090 2024-04-21T08:43:19.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.245+0000 7ff779ec1640 1 -- 172.21.15.94:0/6267554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7741a7820 con 0x7ff774101090 2024-04-21T08:43:19.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.247+0000 7ff770ff9640 1 -- 172.21.15.94:0/6267554 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff764046020 con 0x7ff774101090 2024-04-21T08:43:19.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.247+0000 7ff77cb91740 1 Processor -- start 2024-04-21T08:43:19.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.247+0000 7ff77cb91740 1 -- start start 2024-04-21T08:43:19.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.247+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 0x55cc1e6718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.247+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e67a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.247+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 0x55cc1e67ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.247+0000 7ff77cb91740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55cc1e5d61f0 con 0x55cc1e671e80 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.247+0000 7ff77cb91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cc1e5734b0 con 0x55cc1e6714d0 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.247+0000 7ff77cb91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cc1e5d21b0 con 0x55cc1e67aa00 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.247+0000 7ff770ff9640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff75c0a3d20 0x7ff75c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff7737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e67a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff773fff640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff75c0a3d20 0x7ff75c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff7737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e67a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39424/0 (socket says 172.21.15.94:39424) 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff7737fe640 1 -- 172.21.15.94:0/2684799989 learned_addr learned my addr 172.21.15.94:0/2684799989 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff772ffd640 1 --2- 172.21.15.94:0/2684799989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 0x55cc1e67ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff770ff9640 1 -- 172.21.15.94:0/6267554 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff7640f2ca0 con 0x7ff774101090 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff773fff640 1 --2- 172.21.15.94:0/2684799989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 0x55cc1e6718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff7737fe640 1 -- 172.21.15.94:0/2684799989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 msgr2=0x55cc1e67ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff7737fe640 1 --2- 172.21.15.94:0/2684799989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 0x55cc1e67ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff7737fe640 1 -- 172.21.15.94:0/2684799989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 msgr2=0x55cc1e6718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff7737fe640 1 --2- 172.21.15.94:0/2684799989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 0x55cc1e6718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff7737fe640 1 -- 172.21.15.94:0/2684799989 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc1e5651b0 con 0x55cc1e671e80 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff773fff640 1 --2- 172.21.15.94:0/2684799989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 0x55cc1e6718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff773fff640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff75c0a3d20 0x7ff75c0a61e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff76c005fd0 tx=0x7ff76c0127c0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff7737fe640 1 --2- 172.21.15.94:0/2684799989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e67a430 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff7640be670 tx=0x7ff7640903b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973541db761ac298 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff759ffb640 1 -- 172.21.15.94:0/2684799989 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7640be2d0 con 0x55cc1e671e80 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff759ffb640 1 -- 172.21.15.94:0/2684799989 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7640bf070 con 0x55cc1e671e80 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.248+0000 7ff759ffb640 1 -- 172.21.15.94:0/2684799989 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7640054d0 con 0x55cc1e671e80 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 -- 172.21.15.94:0/2684799989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 msgr2=0x55cc1e67a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 --2- 172.21.15.94:0/2684799989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e67a430 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff7640be670 tx=0x7ff7640903b0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 -- 172.21.15.94:0/2684799989 shutdown_connections 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 --2- 172.21.15.94:0/2684799989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 0x55cc1e67ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 --2- 172.21.15.94:0/2684799989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e67a430 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 --2- 172.21.15.94:0/2684799989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 0x55cc1e6718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 -- 172.21.15.94:0/2684799989 >> 172.21.15.94:0/2684799989 conn(0x55cc1e66bba0 msgr2=0x55cc1e66bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 -- 172.21.15.94:0/2684799989 shutdown_connections 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 -- 172.21.15.94:0/2684799989 wait complete. 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 -- start start 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.249+0000 7ff77cb91740 1 Processor -- start 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 0x55cc1e67a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e6776b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 0x55cc1e677c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff77cb91740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55cc1e67d730 con 0x55cc1e671e80 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff77cb91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cc1e58f500 con 0x55cc1e6714d0 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff77cb91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cc1e67d8b0 con 0x55cc1e67aa00 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff77cb91740 10 client.0 _ll_register_callbacks cb 0x55cc1e55db80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:19.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff772ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e6776b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff772ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e6776b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39434/0 (socket says 172.21.15.94:39434) 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff772ffd640 1 -- 172.21.15.94:0/823531708 learned_addr learned my addr 172.21.15.94:0/823531708 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff773fff640 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 0x55cc1e677c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff7737fe640 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 0x55cc1e67a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff772ffd640 1 -- 172.21.15.94:0/823531708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 msgr2=0x55cc1e677c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff772ffd640 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 0x55cc1e677c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff772ffd640 1 -- 172.21.15.94:0/823531708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 msgr2=0x55cc1e67a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff772ffd640 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 0x55cc1e67a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff772ffd640 1 -- 172.21.15.94:0/823531708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc1e2e83f0 con 0x55cc1e671e80 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff773fff640 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 0x55cc1e677c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff7737fe640 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 0x55cc1e67a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff772ffd640 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e6776b0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff768010b70 tx=0x7ff76800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a584211a753606a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff752ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff752ffd640 1 -- 172.21.15.94:0/823531708 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768004b20 con 0x55cc1e671e80 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff752ffd640 1 -- 172.21.15.94:0/823531708 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff768004cc0 con 0x55cc1e671e80 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff77cb91740 1 -- 172.21.15.94:0/823531708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7640bf070 con 0x55cc1e671e80 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff752ffd640 1 -- 172.21.15.94:0/823531708 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff768003970 con 0x55cc1e671e80 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff77cb91740 10 client.16731 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.250+0000 7ff77cb91740 1 -- 172.21.15.94:0/823531708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cc1e3b27d0 con 0x55cc1e671e80 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.251+0000 7ff752ffd640 1 -- 172.21.15.94:0/823531708 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7640bf070 con 0x55cc1e671e80 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.251+0000 7ff752ffd640 1 -- 172.21.15.94:0/823531708 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff76800e5c0 con 0x55cc1e671e80 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.251+0000 7ff77cb91740 10 client.16731 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.251+0000 7ff77cb91740 10 client.16731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.251+0000 7ff77cb91740 20 client.16731 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.251+0000 7ff77cb91740 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55cc1e68a9c0 0x55cc1e68ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.251+0000 7ff77cb91740 4 client.16731 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.251+0000 7ff77cb91740 1 -- 172.21.15.94:0/823531708 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc1e575930 con 0x55cc1e68a9c0 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.251+0000 7ff7737fe640 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55cc1e68a9c0 0x55cc1e68ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff752ffd640 10 client.16731 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff7737fe640 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55cc1e68a9c0 0x55cc1e68ce20 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c35fe2618a92ad94 server_cookie=99dfb05ef1ec01d7 in_seq=0 out_seq=0 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff752ffd640 1 -- 172.21.15.94:0/823531708 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc1e575930 con 0x55cc1e68a9c0 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff752ffd640 10 client.16731 handle_command_reply: tid=0 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff77cb91740 1 client.16731 shutdown 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff77cb91740 1 -- 172.21.15.94:0/823531708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 msgr2=0x55cc1e6776b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff77cb91740 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e6776b0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff768010b70 tx=0x7ff76800f7f0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff77cb91740 1 -- 172.21.15.94:0/823531708 shutdown_connections 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff77cb91740 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55cc1e68a9c0 0x55cc1e68ce20 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff77cb91740 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e67aa00 0x55cc1e677c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff752ffd640 0 client.16731 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff77cb91740 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e6776b0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff77cb91740 1 --2- 172.21.15.94:0/823531708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e6714d0 0x55cc1e67a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff77cb91740 1 -- 172.21.15.94:0/823531708 >> 172.21.15.94:0/823531708 conn(0x55cc1e66bba0 msgr2=0x55cc1e66d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.252+0000 7ff77cb91740 1 -- 172.21.15.94:0/823531708 shutdown_connections 2024-04-21T08:43:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.253+0000 7ff77cb91740 1 -- 172.21.15.94:0/823531708 wait complete. 2024-04-21T08:43:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.253+0000 7ff77cb91740 20 client.16731 trim_cache size 0 max 16384 2024-04-21T08:43:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff77cb91740 1 Processor -- start 2024-04-21T08:43:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff77cb91740 1 -- start start 2024-04-21T08:43:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e6714d0 0x55cc1e686430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e679de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 0x55cc1e67a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff77cb91740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741b3ef0 con 0x55cc1e671e80 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff77cb91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741b17c0 con 0x55cc1e67aa00 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff77cb91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741af610 con 0x55cc1e6714d0 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff772ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e679de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff772ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e679de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39442/0 (socket says 172.21.15.94:39442) 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff772ffd640 1 -- 172.21.15.94:0/3781113379 learned_addr learned my addr 172.21.15.94:0/3781113379 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff773fff640 1 --2- 172.21.15.94:0/3781113379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 0x55cc1e67a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff772ffd640 1 -- 172.21.15.94:0/3781113379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e6714d0 msgr2=0x55cc1e686430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff772ffd640 1 --2- 172.21.15.94:0/3781113379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e6714d0 0x55cc1e686430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff772ffd640 1 -- 172.21.15.94:0/3781113379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 msgr2=0x55cc1e67a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff772ffd640 1 --2- 172.21.15.94:0/3781113379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 0x55cc1e67a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff772ffd640 1 -- 172.21.15.94:0/3781113379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc1e677fc0 con 0x55cc1e671e80 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff773fff640 1 --2- 172.21.15.94:0/3781113379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 0x55cc1e67a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.266+0000 7ff772ffd640 1 --2- 172.21.15.94:0/3781113379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e679de0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff76800ea30 tx=0x7ff76800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ce12eadf438a47 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff7597fa640 1 -- 172.21.15.94:0/3781113379 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76800e5c0 con 0x55cc1e671e80 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff7597fa640 1 -- 172.21.15.94:0/3781113379 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff76802d070 con 0x55cc1e671e80 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff7597fa640 1 -- 172.21.15.94:0/3781113379 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7680054d0 con 0x55cc1e671e80 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff77cb91740 1 -- 172.21.15.94:0/3781113379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 msgr2=0x55cc1e679de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff77cb91740 1 --2- 172.21.15.94:0/3781113379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e679de0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff76800ea30 tx=0x7ff76800ee30 comp rx=0 tx=0).stop 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff77cb91740 1 -- 172.21.15.94:0/3781113379 shutdown_connections 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff77cb91740 1 --2- 172.21.15.94:0/3781113379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 0x55cc1e67a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff77cb91740 1 --2- 172.21.15.94:0/3781113379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e671e80 0x55cc1e679de0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff77cb91740 1 --2- 172.21.15.94:0/3781113379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e6714d0 0x55cc1e686430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff77cb91740 1 -- 172.21.15.94:0/3781113379 >> 172.21.15.94:0/3781113379 conn(0x55cc1e66bba0 msgr2=0x55cc1e66ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff77cb91740 1 -- 172.21.15.94:0/3781113379 shutdown_connections 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff77cb91740 1 -- 172.21.15.94:0/3781113379 wait complete. 2024-04-21T08:43:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.267+0000 7ff77cb91740 1 -- start start 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff77cb91740 1 Processor -- start 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e6714d0 0x55cc1e682940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e671e80 0x55cc1e682e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff77cb91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 0x55cc1e67f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff77cb91740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741afc80 con 0x55cc1e6714d0 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff77cb91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741b10f0 con 0x55cc1e67aa00 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff77cb91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741b0350 con 0x55cc1e671e80 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff77cb91740 10 client.0 _ll_register_callbacks cb 0x7ff768010780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff7737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e6714d0 0x55cc1e682940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff7737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e6714d0 0x55cc1e682940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39448/0 (socket says 172.21.15.94:39448) 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff7737fe640 1 -- 172.21.15.94:0/3711774773 learned_addr learned my addr 172.21.15.94:0/3711774773 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff773fff640 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 0x55cc1e67f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff772ffd640 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e671e80 0x55cc1e682e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff7737fe640 1 -- 172.21.15.94:0/3711774773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e671e80 msgr2=0x55cc1e682e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff7737fe640 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e671e80 0x55cc1e682e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff7737fe640 1 -- 172.21.15.94:0/3711774773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 msgr2=0x55cc1e67f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff7737fe640 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 0x55cc1e67f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff7737fe640 1 -- 172.21.15.94:0/3711774773 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc1e6801a0 con 0x55cc1e6714d0 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff7737fe640 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e6714d0 0x55cc1e682940 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ff7640f0610 tx=0x7ff76402c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16ac116e9ad58d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.268+0000 7ff752ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff752ffd640 1 -- 172.21.15.94:0/3711774773 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7640d0060 con 0x55cc1e6714d0 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff752ffd640 1 -- 172.21.15.94:0/3711774773 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff7640054d0 con 0x55cc1e6714d0 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff77cb91740 1 -- 172.21.15.94:0/3711774773 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff76802d070 con 0x55cc1e6714d0 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff752ffd640 1 -- 172.21.15.94:0/3711774773 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff7640be4b0 con 0x55cc1e6714d0 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff77cb91740 10 client.16737 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff77cb91740 1 -- 172.21.15.94:0/3711774773 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cc1e6f1a60 con 0x55cc1e6714d0 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff752ffd640 1 -- 172.21.15.94:0/3711774773 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff76802d070 con 0x55cc1e6714d0 2024-04-21T08:43:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff752ffd640 1 -- 172.21.15.94:0/3711774773 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff7640bafe0 con 0x55cc1e6714d0 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff77cb91740 10 client.16737 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff77cb91740 10 client.16737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff77cb91740 20 client.16737 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff77cb91740 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55cc1e6f1fe0 0x55cc1e6f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff77cb91740 4 client.16737 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff77cb91740 1 -- 172.21.15.94:0/3711774773 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cc1e5e3c10 con 0x55cc1e6f1fe0 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.269+0000 7ff772ffd640 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55cc1e6f1fe0 0x55cc1e6f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff772ffd640 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55cc1e6f1fe0 0x55cc1e6f43d0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d152b8a55669778b server_cookie=983157553df5e3a in_seq=0 out_seq=0 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff752ffd640 10 client.16737 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff752ffd640 1 -- 172.21.15.94:0/3711774773 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cc1e5e3c10 con 0x55cc1e6f1fe0 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff752ffd640 10 client.16737 handle_command_reply: tid=0 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 client.16737 shutdown 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 -- 172.21.15.94:0/3711774773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e6714d0 msgr2=0x55cc1e682940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e6714d0 0x55cc1e682940 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ff7640f0610 tx=0x7ff76402c950 comp rx=0 tx=0).stop 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 -- 172.21.15.94:0/3711774773 shutdown_connections 2024-04-21T08:43:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55cc1e6f1fe0 0x55cc1e6f43d0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc1e671e80 0x55cc1e682e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc1e67aa00 0x55cc1e67f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff752ffd640 0 client.16737 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 --2- 172.21.15.94:0/3711774773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cc1e6714d0 0x55cc1e682940 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 -- 172.21.15.94:0/3711774773 >> 172.21.15.94:0/3711774773 conn(0x55cc1e66bba0 msgr2=0x55cc1e66cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 -- 172.21.15.94:0/3711774773 shutdown_connections 2024-04-21T08:43:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 1 -- 172.21.15.94:0/3711774773 wait complete. 2024-04-21T08:43:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.270+0000 7ff77cb91740 20 client.16737 trim_cache size 0 max 16384 2024-04-21T08:43:19.277 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:43:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 -- 172.21.15.94:0/6267554 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff75c0a3d20 msgr2=0x7ff75c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff75c0a3d20 0x7ff75c0a61e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff76c005fd0 tx=0x7ff76c0127c0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 -- 172.21.15.94:0/6267554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff774101090 msgr2=0x7ff774102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff774101090 0x7ff774102b20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff764005940 tx=0x7ff76402c7e0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 -- 172.21.15.94:0/6267554 shutdown_connections 2024-04-21T08:43:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff75c0a3d20 0x7ff75c0a61e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7741035a0 0x7ff7741a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106160 0x7ff774103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 --2- 172.21.15.94:0/6267554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff774101090 0x7ff774102b20 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 -- 172.21.15.94:0/6267554 >> 172.21.15.94:0/6267554 conn(0x7ff774077f50 msgr2=0x7ff774069500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 -- 172.21.15.94:0/6267554 shutdown_connections 2024-04-21T08:43:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:19.271+0000 7ff779ec1640 1 -- 172.21.15.94:0/6267554 wait complete. 2024-04-21T08:43:19.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:19 smithi094 ceph-mon[27988]: pgmap v103: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:43:19.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:19 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2746194546' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:19.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:19 smithi173 ceph-mon[31428]: pgmap v103: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:43:19.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:19 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2746194546' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:19.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:19 smithi111 ceph-mon[31493]: pgmap v103: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:43:19.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:19 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2746194546' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:20.282 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:43:20.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:20.282 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:43:20.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.389+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3085767167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41c8102b40 msgr2=0x7f41c8106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.389+0000 7f41ce2c1640 1 --2- 172.21.15.94:0/3085767167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41c8102b40 0x7f41c8106b80 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009900 tx=0x7f41bc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.389+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3085767167 shutdown_connections 2024-04-21T08:43:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.389+0000 7f41ce2c1640 1 --2- 172.21.15.94:0/3085767167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c8107260 0x7f41c810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.389+0000 7f41ce2c1640 1 --2- 172.21.15.94:0/3085767167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41c8102b40 0x7f41c8106b80 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.389+0000 7f41ce2c1640 1 --2- 172.21.15.94:0/3085767167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41c8102190 0x7f41c8102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.389+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3085767167 >> 172.21.15.94:0/3085767167 conn(0x7f41c80793a0 msgr2=0x7f41c80797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.389+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3085767167 shutdown_connections 2024-04-21T08:43:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.389+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3085767167 wait complete. 2024-04-21T08:43:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41ce2c1640 1 Processor -- start 2024-04-21T08:43:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41ce2c1640 1 -- start start 2024-04-21T08:43:20.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41ce2c1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c8102190 0x7f41c8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41ce2c1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41c8107260 0x7f41c8198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41ce2c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41c819cfe0 0x7f41c819d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41ce2c1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f41c81121a0 con 0x7f41c819cfe0 2024-04-21T08:43:20.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41ce2c1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8112020 con 0x7f41c8102190 2024-04-21T08:43:20.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41ce2c1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8112320 con 0x7f41c8107260 2024-04-21T08:43:20.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41cc85e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41c819cfe0 0x7f41c819d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41cc85e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41c819cfe0 0x7f41c819d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39452/0 (socket says 172.21.15.94:39452) 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41cc85e640 1 -- 172.21.15.94:0/3430879860 learned_addr learned my addr 172.21.15.94:0/3430879860 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41c7fff640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c8102190 0x7f41c8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.390+0000 7f41c77fe640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41c8107260 0x7f41c8198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41cc85e640 1 -- 172.21.15.94:0/3430879860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41c8107260 msgr2=0x7f41c8198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41cc85e640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41c8107260 0x7f41c8198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41cc85e640 1 -- 172.21.15.94:0/3430879860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c8102190 msgr2=0x7f41c8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41cc85e640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c8102190 0x7f41c8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41cc85e640 1 -- 172.21.15.94:0/3430879860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c819dad0 con 0x7f41c819cfe0 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41cc85e640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41c819cfe0 0x7f41c819d490 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f41c000cce0 tx=0x7f41c000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41c57fa640 1 -- 172.21.15.94:0/3430879860 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c000f450 con 0x7f41c819cfe0 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3430879860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c819ddc0 con 0x7f41c819cfe0 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41c57fa640 1 -- 172.21.15.94:0/3430879860 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41c000faa0 con 0x7f41c819cfe0 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41c57fa640 1 -- 172.21.15.94:0/3430879860 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c0018b70 con 0x7f41c819cfe0 2024-04-21T08:43:20.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.391+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3430879860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c81a5530 con 0x7f41c819cfe0 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.393+0000 7f41c57fa640 1 -- 172.21.15.94:0/3430879860 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f41c0018d10 con 0x7f41c819cfe0 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.393+0000 7f41d0f8c740 1 Processor -- start 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.393+0000 7f41d0f8c740 1 -- start start 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.393+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a69dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a6a6e10 0x56369a6a9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56369a601f00 con 0x56369a69e290 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56369a59f590 con 0x56369a6a6e10 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56369a5fdec0 con 0x56369a69d8e0 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c57fa640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f41b00a3d20 0x7f41b00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39460/0 (socket says 172.21.15.94:39460) 2024-04-21T08:43:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c7fff640 1 -- 172.21.15.94:0/3793127488 learned_addr learned my addr 172.21.15.94:0/3793127488 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41cc85e640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f41b00a3d20 0x7f41b00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c57fa640 1 -- 172.21.15.94:0/3430879860 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f41c00d0f20 con 0x7f41c819cfe0 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c77fe640 1 --2- 172.21.15.94:0/3793127488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a69dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c7fff640 1 -- 172.21.15.94:0/3793127488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 msgr2=0x56369a69dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c7fff640 1 --2- 172.21.15.94:0/3793127488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a69dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c7fff640 1 -- 172.21.15.94:0/3793127488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a6a6e10 msgr2=0x56369a6a9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c7fff640 1 --2- 172.21.15.94:0/3793127488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a6a6e10 0x56369a6a9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c7fff640 1 -- 172.21.15.94:0/3793127488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56369a3dc7e0 con 0x56369a69e290 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c77fe640 1 --2- 172.21.15.94:0/3793127488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a69dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41cc85e640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f41b00a3d20 0x7f41b00a61e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f41c00ce340 tx=0x7f41c0098d80 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41c7fff640 1 --2- 172.21.15.94:0/3793127488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a6840 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f41b800b760 tx=0x7f41b800bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccef22ed9c3d3a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41ae7fc640 1 -- 172.21.15.94:0/3793127488 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8003dd0 con 0x56369a69e290 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41ae7fc640 1 -- 172.21.15.94:0/3793127488 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41b8015070 con 0x56369a69e290 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.394+0000 7f41ae7fc640 1 -- 172.21.15.94:0/3793127488 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8004d90 con 0x56369a69e290 2024-04-21T08:43:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.395+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3793127488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 msgr2=0x56369a6a6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.395+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/3793127488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a6840 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f41b800b760 tx=0x7f41b800bc20 comp rx=0 tx=0).stop 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.395+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3793127488 shutdown_connections 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.395+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/3793127488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a6a6e10 0x56369a6a9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.395+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/3793127488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a6840 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.395+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/3793127488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a69dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.395+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3793127488 >> 172.21.15.94:0/3793127488 conn(0x56369a697fb0 msgr2=0x56369a698390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.395+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3793127488 shutdown_connections 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.395+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3793127488 wait complete. 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.395+0000 7f41d0f8c740 1 -- start start 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41d0f8c740 1 Processor -- start 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a6a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a69e290 0x56369a6a4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56369a6a9d80 con 0x56369a6a6e10 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56369a5bb580 con 0x56369a69e290 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56369a6a9f00 con 0x56369a69d8e0 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41d0f8c740 10 client.0 _ll_register_callbacks cb 0x56369a5c1650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39464/0 (socket says 172.21.15.94:39464) 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c77fe640 1 -- 172.21.15.94:0/2450329975 learned_addr learned my addr 172.21.15.94:0/2450329975 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c77fe640 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a6a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c7fff640 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a69e290 0x56369a6a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c77fe640 1 -- 172.21.15.94:0/2450329975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 msgr2=0x56369a6a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c77fe640 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a6a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c77fe640 1 -- 172.21.15.94:0/2450329975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a69e290 msgr2=0x56369a6a4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c77fe640 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a69e290 0x56369a6a4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c77fe640 1 -- 172.21.15.94:0/2450329975 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56369a43cf70 con 0x56369a6a6e10 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41c77fe640 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6a48c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0096c0 tx=0x7f41bc005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b97234990d3d27 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.396+0000 7f41a77fe640 1 -- 172.21.15.94:0/2450329975 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc02cc90 con 0x56369a6a6e10 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2450329975 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41b8015070 con 0x56369a6a6e10 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41a77fe640 1 -- 172.21.15.94:0/2450329975 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f41bc005250 con 0x56369a6a6e10 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41a77fe640 1 -- 172.21.15.94:0/2450329975 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f41bc053050 con 0x56369a6a6e10 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41d0f8c740 10 client.16749 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2450329975 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56369a2d4170 con 0x56369a6a6e10 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41a77fe640 1 -- 172.21.15.94:0/2450329975 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41b8015070 con 0x56369a6a6e10 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41a77fe640 1 -- 172.21.15.94:0/2450329975 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f41bc0035f0 con 0x56369a6a6e10 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41d0f8c740 10 client.16749 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41d0f8c740 10 client.16749 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41d0f8c740 20 client.16749 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56369a6b72e0 0x56369a6b96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41d0f8c740 4 client.16749 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2450329975 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56369a60f920 con 0x56369a6b72e0 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.397+0000 7f41c7fff640 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56369a6b72e0 0x56369a6b96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.398+0000 7f41a77fe640 10 client.16749 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.398+0000 7f41c7fff640 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56369a6b72e0 0x56369a6b96f0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc9d19c8899e6fd4 server_cookie=d0b698ebc50c5ecc in_seq=0 out_seq=0 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.398+0000 7f41a77fe640 1 -- 172.21.15.94:0/2450329975 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56369a60f920 con 0x56369a6b72e0 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.398+0000 7f41a77fe640 10 client.16749 handle_command_reply: tid=0 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.398+0000 7f41d0f8c740 1 client.16749 shutdown 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2450329975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 msgr2=0x56369a6a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6a48c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0096c0 tx=0x7f41bc005b50 comp rx=0 tx=0).stop 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2450329975 shutdown_connections 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56369a6b72e0 0x56369a6b96f0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6a48c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a69e290 0x56369a6a4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41a77fe640 0 client.16749 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:20.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/2450329975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a6a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2450329975 >> 172.21.15.94:0/2450329975 conn(0x56369a697fb0 msgr2=0x56369a699d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2450329975 shutdown_connections 2024-04-21T08:43:20.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2450329975 wait complete. 2024-04-21T08:43:20.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.399+0000 7f41d0f8c740 20 client.16749 trim_cache size 0 max 16384 2024-04-21T08:43:20.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41d0f8c740 1 Processor -- start 2024-04-21T08:43:20.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41d0f8c740 1 -- start start 2024-04-21T08:43:20.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a69d8e0 0x56369a6a6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69e290 0x56369a6af560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f41c81b1c20 con 0x56369a6a6e10 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f41c81af4f0 con 0x56369a69d8e0 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f41c81ad340 con 0x56369a69e290 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41c77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41c77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39476/0 (socket says 172.21.15.94:39476) 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.416+0000 7f41c77fe640 1 -- 172.21.15.94:0/2623369087 learned_addr learned my addr 172.21.15.94:0/2623369087 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41c77fe640 1 --2- 172.21.15.94:0/2623369087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a69d8e0 0x56369a6a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41c7fff640 1 --2- 172.21.15.94:0/2623369087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69e290 0x56369a6af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41c77fe640 1 -- 172.21.15.94:0/2623369087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69e290 msgr2=0x56369a6af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41c77fe640 1 --2- 172.21.15.94:0/2623369087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69e290 0x56369a6af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41c77fe640 1 -- 172.21.15.94:0/2623369087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a69d8e0 msgr2=0x56369a6a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41c77fe640 1 --2- 172.21.15.94:0/2623369087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a69d8e0 0x56369a6a6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41c77fe640 1 -- 172.21.15.94:0/2623369087 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56369a69e030 con 0x56369a6a6e10 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41c77fe640 1 --2- 172.21.15.94:0/2623369087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6b2c70 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f41bc004650 tx=0x7f41bc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49bfb05968a8189e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41adffb640 1 -- 172.21.15.94:0/2623369087 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc000e60 con 0x56369a6a6e10 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41adffb640 1 -- 172.21.15.94:0/2623369087 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41bc0040b0 con 0x56369a6a6e10 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.417+0000 7f41adffb640 1 -- 172.21.15.94:0/2623369087 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc004920 con 0x56369a6a6e10 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2623369087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 msgr2=0x56369a6b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/2623369087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6b2c70 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f41bc004650 tx=0x7f41bc005fd0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2623369087 shutdown_connections 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/2623369087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69e290 0x56369a6af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/2623369087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a6a6e10 0x56369a6b2c70 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/2623369087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a69d8e0 0x56369a6a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2623369087 >> 172.21.15.94:0/2623369087 conn(0x56369a697fb0 msgr2=0x56369a697010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2623369087 shutdown_connections 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 -- 172.21.15.94:0/2623369087 wait complete. 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 -- start start 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 Processor -- start 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a6a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a6a6e10 0x56369a6a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f41c81ad9b0 con 0x56369a69e290 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f41c81aee20 con 0x56369a6a6e10 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f41c81ae080 con 0x56369a69d8e0 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.418+0000 7f41d0f8c740 10 client.0 _ll_register_callbacks cb 0x7f41bc02c6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41c7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41c7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39484/0 (socket says 172.21.15.94:39484) 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41c7fff640 1 -- 172.21.15.94:0/3619569130 learned_addr learned my addr 172.21.15.94:0/3619569130 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41c77fe640 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a6a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41c7fff640 1 -- 172.21.15.94:0/3619569130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 msgr2=0x56369a6a00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41c7fff640 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a6a00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41c7fff640 1 -- 172.21.15.94:0/3619569130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a6a6e10 msgr2=0x56369a6a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41c7fff640 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a6a6e10 0x56369a6a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41c7fff640 1 -- 172.21.15.94:0/3619569130 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56369a6a1350 con 0x56369a69e290 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41c7fff640 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a0610 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f41b80044d0 tx=0x7f41b8005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28301c023c685cc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41a77fe640 1 -- 172.21.15.94:0/3619569130 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b80272f0 con 0x56369a69e290 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41a77fe640 1 -- 172.21.15.94:0/3619569130 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f41b8027490 con 0x56369a69e290 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3619569130 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41bc0040b0 con 0x56369a69e290 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41a77fe640 1 -- 172.21.15.94:0/3619569130 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f41b8035050 con 0x56369a69e290 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41d0f8c740 10 client.16755 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3619569130 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56369a721e20 con 0x56369a69e290 2024-04-21T08:43:20.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41a77fe640 1 -- 172.21.15.94:0/3619569130 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41bc0040b0 con 0x56369a69e290 2024-04-21T08:43:20.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.419+0000 7f41a77fe640 1 -- 172.21.15.94:0/3619569130 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f41b8011980 con 0x56369a69e290 2024-04-21T08:43:20.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.420+0000 7f41d0f8c740 10 client.16755 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:20.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.420+0000 7f41d0f8c740 10 client.16755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:20.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.420+0000 7f41d0f8c740 20 client.16755 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:20.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.420+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56369a722370 0x56369a724790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.420+0000 7f41d0f8c740 4 client.16755 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:43:20.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.420+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3619569130 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56369a609790 con 0x56369a722370 2024-04-21T08:43:20.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.420+0000 7f41c77fe640 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56369a722370 0x56369a724790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.420+0000 7f41a77fe640 10 client.16755 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:20.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.420+0000 7f41c77fe640 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56369a722370 0x56369a724790 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=271c0f782a078c1a server_cookie=ab8501dfa46aa63b in_seq=0 out_seq=0 2024-04-21T08:43:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.421+0000 7f41a77fe640 1 -- 172.21.15.94:0/3619569130 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56369a609790 con 0x56369a722370 2024-04-21T08:43:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.421+0000 7f41a77fe640 10 client.16755 handle_command_reply: tid=0 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.421+0000 7f41d0f8c740 1 client.16755 shutdown 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3619569130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 msgr2=0x56369a6a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a0610 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f41b80044d0 tx=0x7f41b8005fc0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3619569130 shutdown_connections 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56369a722370 0x56369a724790 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56369a6a6e10 0x56369a6a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56369a69e290 0x56369a6a0610 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 1 --2- 172.21.15.94:0/3619569130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56369a69d8e0 0x56369a6a00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3619569130 >> 172.21.15.94:0/3619569130 conn(0x56369a697fb0 msgr2=0x56369a6a3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41a77fe640 0 client.16755 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3619569130 shutdown_connections 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 1 -- 172.21.15.94:0/3619569130 wait complete. 2024-04-21T08:43:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41d0f8c740 20 client.16755 trim_cache size 0 max 16384 2024-04-21T08:43:20.428 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:43:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:43:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "fb89b90f-a7ea-405e-9f73-f69ad1c796cf", 2024-04-21T08:43:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:43:20.428 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:43:20.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3430879860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f41b00a3d20 msgr2=0x7f41b00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41ce2c1640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f41b00a3d20 0x7f41b00a61e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f41c00ce340 tx=0x7f41c0098d80 comp rx=0 tx=0).stop 2024-04-21T08:43:20.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3430879860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41c819cfe0 msgr2=0x7f41c819d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41ce2c1640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41c819cfe0 0x7f41c819d490 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f41c000cce0 tx=0x7f41c000a620 comp rx=0 tx=0).stop 2024-04-21T08:43:20.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3430879860 shutdown_connections 2024-04-21T08:43:20.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41ce2c1640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f41b00a3d20 0x7f41b00a61e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41ce2c1640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41c8107260 0x7f41c8198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41ce2c1640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41c819cfe0 0x7f41c819d490 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41ce2c1640 1 --2- 172.21.15.94:0/3430879860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c8102190 0x7f41c8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.422+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3430879860 >> 172.21.15.94:0/3430879860 conn(0x7f41c80793a0 msgr2=0x7f41c80fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.423+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3430879860 shutdown_connections 2024-04-21T08:43:20.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.423+0000 7f41ce2c1640 1 -- 172.21.15.94:0/3430879860 wait complete. 2024-04-21T08:43:20.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:43:20.434 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:43:20.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.534+0000 7f6e73ee4640 1 -- 172.21.15.94:0/778957201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e6c1021a0 msgr2=0x7f6e6c102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.534+0000 7f6e73ee4640 1 --2- 172.21.15.94:0/778957201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e6c1021a0 0x7f6e6c102580 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009900 tx=0x7f6e5c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:20.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.534+0000 7f6e73ee4640 1 -- 172.21.15.94:0/778957201 shutdown_connections 2024-04-21T08:43:20.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.534+0000 7f6e73ee4640 1 --2- 172.21.15.94:0/778957201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e6c107270 0x7f6e6c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.534+0000 7f6e73ee4640 1 --2- 172.21.15.94:0/778957201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e6c102b50 0x7f6e6c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.534+0000 7f6e73ee4640 1 --2- 172.21.15.94:0/778957201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e6c1021a0 0x7f6e6c102580 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.534+0000 7f6e73ee4640 1 -- 172.21.15.94:0/778957201 >> 172.21.15.94:0/778957201 conn(0x7f6e6c0793e0 msgr2=0x7f6e6c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.534+0000 7f6e73ee4640 1 -- 172.21.15.94:0/778957201 shutdown_connections 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.534+0000 7f6e73ee4640 1 -- 172.21.15.94:0/778957201 wait complete. 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e73ee4640 1 Processor -- start 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e73ee4640 1 -- start start 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e73ee4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e6c1021a0 0x7f6e6c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e73ee4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e6c107270 0x7f6e6c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e73ee4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e6c19cf40 0x7f6e6c19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e73ee4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c112200 con 0x7f6e6c107270 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e73ee4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c112080 con 0x7f6e6c19cf40 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e73ee4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c112380 con 0x7f6e6c1021a0 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e7147f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e6c107270 0x7f6e6c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e7147f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e6c107270 0x7f6e6c198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39498/0 (socket says 172.21.15.94:39498) 2024-04-21T08:43:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e7147f640 1 -- 172.21.15.94:0/556781845 learned_addr learned my addr 172.21.15.94:0/556781845 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e72481640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e6c19cf40 0x7f6e6c19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e71c80640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e6c1021a0 0x7f6e6c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e7147f640 1 -- 172.21.15.94:0/556781845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e6c1021a0 msgr2=0x7f6e6c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e7147f640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e6c1021a0 0x7f6e6c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e7147f640 1 -- 172.21.15.94:0/556781845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e6c19cf40 msgr2=0x7f6e6c19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e7147f640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e6c19cf40 0x7f6e6c19d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e7147f640 1 -- 172.21.15.94:0/556781845 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e6c19da30 con 0x7f6e6c107270 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.535+0000 7f6e7147f640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e6c107270 0x7f6e6c198bc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800c970 tx=0x7f6e6800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.536+0000 7f6e62ffd640 1 -- 172.21.15.94:0/556781845 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68014070 con 0x7f6e6c107270 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.536+0000 7f6e62ffd640 1 -- 172.21.15.94:0/556781845 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e68004440 con 0x7f6e6c107270 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.536+0000 7f6e73ee4640 1 -- 172.21.15.94:0/556781845 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e6c19dd20 con 0x7f6e6c107270 2024-04-21T08:43:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.536+0000 7f6e62ffd640 1 -- 172.21.15.94:0/556781845 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68003e50 con 0x7f6e6c107270 2024-04-21T08:43:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.536+0000 7f6e73ee4640 1 -- 172.21.15.94:0/556781845 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e6c1a5510 con 0x7f6e6c107270 2024-04-21T08:43:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.538+0000 7f6e62ffd640 1 -- 172.21.15.94:0/556781845 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f6e68017070 con 0x7f6e6c107270 2024-04-21T08:43:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.538+0000 7f6e76bb1740 1 Processor -- start 2024-04-21T08:43:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.538+0000 7f6e76bb1740 1 -- start start 2024-04-21T08:43:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.538+0000 7f6e62ffd640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6e580a3ec0 0x7f6e580a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.538+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315ddd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315e6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315e9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e72481640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6e580a3ec0 0x7f6e580a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55a53153e1e0 con 0x55a5315e6e90 2024-04-21T08:43:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a5314db390 con 0x55a5315dd990 2024-04-21T08:43:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a53153a1a0 con 0x55a5315de2b0 2024-04-21T08:43:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e62ffd640 1 -- 172.21.15.94:0/556781845 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f6e680d0770 con 0x7f6e6c107270 2024-04-21T08:43:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e72481640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315e9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e72481640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315e9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39514/0 (socket says 172.21.15.94:39514) 2024-04-21T08:43:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e71c80640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315ddd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e72481640 1 -- 172.21.15.94:0/346688017 learned_addr learned my addr 172.21.15.94:0/346688017 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e7147f640 1 --2- 172.21.15.94:0/346688017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315e6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e72481640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6e580a3ec0 0x7f6e580a6380 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e64014820 tx=0x7f6e64014c20 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e71c80640 1 -- 172.21.15.94:0/346688017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 msgr2=0x55a5315e6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e71c80640 1 --2- 172.21.15.94:0/346688017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315e6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e71c80640 1 -- 172.21.15.94:0/346688017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 msgr2=0x55a5315e9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e71c80640 1 --2- 172.21.15.94:0/346688017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315e9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e71c80640 1 -- 172.21.15.94:0/346688017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a53131a7d0 con 0x55a5315dd990 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e72481640 1 --2- 172.21.15.94:0/346688017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315e9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e71c80640 1 --2- 172.21.15.94:0/346688017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315ddd70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c02c170 tx=0x7f6e5c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9090692daf930f7b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e57fff640 1 -- 172.21.15.94:0/346688017 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c044020 con 0x55a5315dd990 2024-04-21T08:43:20.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e57fff640 1 -- 172.21.15.94:0/346688017 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e5c02cc90 con 0x55a5315dd990 2024-04-21T08:43:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.539+0000 7f6e57fff640 1 -- 172.21.15.94:0/346688017 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c0046c0 con 0x55a5315dd990 2024-04-21T08:43:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.540+0000 7f6e76bb1740 1 -- 172.21.15.94:0/346688017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 msgr2=0x55a5315ddd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.540+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/346688017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315ddd70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c02c170 tx=0x7f6e5c02c650 comp rx=0 tx=0).stop 2024-04-21T08:43:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.540+0000 7f6e76bb1740 1 -- 172.21.15.94:0/346688017 shutdown_connections 2024-04-21T08:43:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.540+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/346688017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315e6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.540+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/346688017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315e9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.540+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/346688017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315ddd70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.540+0000 7f6e76bb1740 1 -- 172.21.15.94:0/346688017 >> 172.21.15.94:0/346688017 conn(0x55a5315d80c0 msgr2=0x55a5315d84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.540+0000 7f6e76bb1740 1 -- 172.21.15.94:0/346688017 shutdown_connections 2024-04-21T08:43:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.540+0000 7f6e76bb1740 1 -- 172.21.15.94:0/346688017 wait complete. 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e76bb1740 1 -- start start 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e76bb1740 1 Processor -- start 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315dd990 0x55a5315e3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315de2b0 0x55a5315e43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315e6e90 0x55a5315e4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55a5315e9c20 con 0x55a5315de2b0 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a5314f72f0 con 0x55a5315e6e90 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a5315e9da0 con 0x55a5315dd990 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e76bb1740 10 client.0 _ll_register_callbacks cb 0x55a531504580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e7147f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315de2b0 0x55a5315e43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e7147f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315de2b0 0x55a5315e43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39518/0 (socket says 172.21.15.94:39518) 2024-04-21T08:43:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e7147f640 1 -- 172.21.15.94:0/2350090645 learned_addr learned my addr 172.21.15.94:0/2350090645 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e72481640 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315e6e90 0x55a5315e4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e71c80640 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315dd990 0x55a5315e3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e7147f640 1 -- 172.21.15.94:0/2350090645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315dd990 msgr2=0x55a5315e3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e7147f640 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315dd990 0x55a5315e3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e7147f640 1 -- 172.21.15.94:0/2350090645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315e6e90 msgr2=0x55a5315e4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e7147f640 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315e6e90 0x55a5315e4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e7147f640 1 -- 172.21.15.94:0/2350090645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5312503f0 con 0x55a5315de2b0 2024-04-21T08:43:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.541+0000 7f6e71c80640 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315dd990 0x55a5315e3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e7147f640 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315de2b0 0x55a5315e43d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f6e680cdb40 tx=0x7f6e6809cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e5bcbb1bfd893f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e557fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e557fa640 1 -- 172.21.15.94:0/2350090645 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e680a4fa0 con 0x55a5315de2b0 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e557fa640 1 -- 172.21.15.94:0/2350090645 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f6e680d0320 con 0x55a5315de2b0 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2350090645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e5c02cc90 con 0x55a5315de2b0 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e557fa640 1 -- 172.21.15.94:0/2350090645 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6e680d0520 con 0x55a5315de2b0 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e76bb1740 10 client.16767 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2350090645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a5312100f0 con 0x55a5315de2b0 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e557fa640 1 -- 172.21.15.94:0/2350090645 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e5c02cc90 con 0x55a5315de2b0 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e557fa640 1 -- 172.21.15.94:0/2350090645 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6e680a0310 con 0x55a5315de2b0 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e76bb1740 10 client.16767 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e76bb1740 10 client.16767 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e76bb1740 20 client.16767 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a5315f6df0 0x55a5315f9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e76bb1740 4 client.16767 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.542+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2350090645 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a53154bc00 con 0x55a5315f6df0 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.543+0000 7f6e71c80640 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a5315f6df0 0x55a5315f9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.543+0000 7f6e71c80640 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a5315f6df0 0x55a5315f9240 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9058d717afc192e6 server_cookie=53d15dd6dd8b81b9 in_seq=0 out_seq=0 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.543+0000 7f6e557fa640 10 client.16767 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e557fa640 1 -- 172.21.15.94:0/2350090645 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a53154bc00 con 0x55a5315f6df0 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e557fa640 10 client.16767 handle_command_reply: tid=0 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 client.16767 shutdown 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2350090645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315de2b0 msgr2=0x55a5315e43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315de2b0 0x55a5315e43d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f6e680cdb40 tx=0x7f6e6809cb60 comp rx=0 tx=0).stop 2024-04-21T08:43:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2350090645 shutdown_connections 2024-04-21T08:43:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a5315f6df0 0x55a5315f9240 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315e6e90 0x55a5315e4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e557fa640 0 client.16767 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315de2b0 0x55a5315e43d0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2350090645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315dd990 0x55a5315e3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2350090645 >> 172.21.15.94:0/2350090645 conn(0x55a5315d80c0 msgr2=0x55a5315d9e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2350090645 shutdown_connections 2024-04-21T08:43:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2350090645 wait complete. 2024-04-21T08:43:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.544+0000 7f6e76bb1740 20 client.16767 trim_cache size 0 max 16384 2024-04-21T08:43:20.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e76bb1740 1 Processor -- start 2024-04-21T08:43:20.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e76bb1740 1 -- start start 2024-04-21T08:43:20.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c1afaa0 con 0x55a5315e6e90 2024-04-21T08:43:20.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c1ad370 con 0x55a5315dd990 2024-04-21T08:43:20.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c1ab1c0 con 0x55a5315de2b0 2024-04-21T08:43:20.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e72481640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e72481640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39526/0 (socket says 172.21.15.94:39526) 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.561+0000 7f6e72481640 1 -- 172.21.15.94:0/3106775835 learned_addr learned my addr 172.21.15.94:0/3106775835 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.562+0000 7f6e7147f640 1 --2- 172.21.15.94:0/3106775835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.562+0000 7f6e72481640 1 -- 172.21.15.94:0/3106775835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 msgr2=0x55a5315d2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.562+0000 7f6e72481640 1 --2- 172.21.15.94:0/3106775835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315d2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.562+0000 7f6e72481640 1 -- 172.21.15.94:0/3106775835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 msgr2=0x55a5315f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.562+0000 7f6e72481640 1 --2- 172.21.15.94:0/3106775835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.562+0000 7f6e72481640 1 -- 172.21.15.94:0/3106775835 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5315e23e0 con 0x55a5315e6e90 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.562+0000 7f6e72481640 1 --2- 172.21.15.94:0/3106775835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315feb10 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f6e64012c70 tx=0x7f6e6401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5edd394c20461d85 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.562+0000 7f6e577fe640 1 -- 172.21.15.94:0/3106775835 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64003970 con 0x55a5315e6e90 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.562+0000 7f6e577fe640 1 -- 172.21.15.94:0/3106775835 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e64004020 con 0x55a5315e6e90 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.562+0000 7f6e577fe640 1 -- 172.21.15.94:0/3106775835 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e640206a0 con 0x55a5315e6e90 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 -- 172.21.15.94:0/3106775835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 msgr2=0x55a5315feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/3106775835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315feb10 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f6e64012c70 tx=0x7f6e6401d000 comp rx=0 tx=0).stop 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 -- 172.21.15.94:0/3106775835 shutdown_connections 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/3106775835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315d2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/3106775835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315feb10 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/3106775835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 -- 172.21.15.94:0/3106775835 >> 172.21.15.94:0/3106775835 conn(0x55a5315d80c0 msgr2=0x55a5315d24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 -- 172.21.15.94:0/3106775835 shutdown_connections 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 -- 172.21.15.94:0/3106775835 wait complete. 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 -- start start 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 Processor -- start 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315eaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315eafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315eb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c1ab830 con 0x55a5315e6e90 2024-04-21T08:43:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c1acca0 con 0x55a5315dd990 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c1abf00 con 0x55a5315de2b0 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.563+0000 7f6e76bb1740 10 client.0 _ll_register_callbacks cb 0x7f6e6809bd00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e72481640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315eb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e72481640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315eb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39540/0 (socket says 172.21.15.94:39540) 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e72481640 1 -- 172.21.15.94:0/2656147726 learned_addr learned my addr 172.21.15.94:0/2656147726 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e7147f640 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315eafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e71c80640 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315eaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e72481640 1 -- 172.21.15.94:0/2656147726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 msgr2=0x55a5315eafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e72481640 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315eafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e72481640 1 -- 172.21.15.94:0/2656147726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 msgr2=0x55a5315eaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e72481640 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315eaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e72481640 1 -- 172.21.15.94:0/2656147726 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a53165c3d0 con 0x55a5315e6e90 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e7147f640 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315eafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e71c80640 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315eaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e72481640 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315eb4e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009a90 tx=0x7f6e64005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423e175eebe9675c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e557fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e557fa640 1 -- 172.21.15.94:0/2656147726 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6403c060 con 0x55a5315e6e90 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e557fa640 1 -- 172.21.15.94:0/2656147726 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f6e64003970 con 0x55a5315e6e90 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2656147726 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e64004020 con 0x55a5315e6e90 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e557fa640 1 -- 172.21.15.94:0/2656147726 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6e64043050 con 0x55a5315e6e90 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e76bb1740 10 client.16773 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2656147726 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a53165d6e0 con 0x55a5315e6e90 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.564+0000 7f6e557fa640 1 -- 172.21.15.94:0/2656147726 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e64004020 con 0x55a5315e6e90 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.565+0000 7f6e557fa640 1 -- 172.21.15.94:0/2656147726 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6e64034030 con 0x55a5315e6e90 2024-04-21T08:43:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.565+0000 7f6e76bb1740 10 client.16773 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.565+0000 7f6e76bb1740 10 client.16773 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.565+0000 7f6e76bb1740 20 client.16773 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.565+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a53165ddd0 0x55a5316601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.565+0000 7f6e76bb1740 4 client.16773 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.565+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2656147726 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a531545a70 con 0x55a53165ddd0 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.565+0000 7f6e71c80640 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a53165ddd0 0x55a5316601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.565+0000 7f6e71c80640 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a53165ddd0 0x55a5316601f0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d333d24a91b01e5 server_cookie=acfaa5d2c35164f8 in_seq=0 out_seq=0 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.565+0000 7f6e557fa640 10 client.16773 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e557fa640 1 -- 172.21.15.94:0/2656147726 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a531545a70 con 0x55a53165ddd0 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e557fa640 10 client.16773 handle_command_reply: tid=0 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 client.16773 shutdown 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2656147726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 msgr2=0x55a5315eb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315eb4e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009a90 tx=0x7f6e64005c80 comp rx=0 tx=0).stop 2024-04-21T08:43:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2656147726 shutdown_connections 2024-04-21T08:43:20.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55a53165ddd0 0x55a5316601f0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5315de2b0 0x55a5315eafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a5315e6e90 0x55a5315eb4e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e557fa640 0 client.16773 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:20.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 --2- 172.21.15.94:0/2656147726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5315dd990 0x55a5315eaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2656147726 >> 172.21.15.94:0/2656147726 conn(0x55a5315d80c0 msgr2=0x55a5315d2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2656147726 shutdown_connections 2024-04-21T08:43:20.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 1 -- 172.21.15.94:0/2656147726 wait complete. 2024-04-21T08:43:20.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e76bb1740 20 client.16773 trim_cache size 0 max 16384 2024-04-21T08:43:20.573 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:43:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:43:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:43:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "fb89b90f-a7ea-405e-9f73-f69ad1c796cf": { 2024-04-21T08:43:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:43:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "fb89b90f-a7ea-405e-9f73-f69ad1c796cf", 2024-04-21T08:43:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:43:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:43:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:43:20.574 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:43:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 -- 172.21.15.94:0/556781845 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6e580a3ec0 msgr2=0x7f6e580a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6e580a3ec0 0x7f6e580a6380 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e64014820 tx=0x7f6e64014c20 comp rx=0 tx=0).stop 2024-04-21T08:43:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 -- 172.21.15.94:0/556781845 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e6c107270 msgr2=0x7f6e6c198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e6c107270 0x7f6e6c198bc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800c970 tx=0x7f6e6800ce30 comp rx=0 tx=0).stop 2024-04-21T08:43:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 -- 172.21.15.94:0/556781845 shutdown_connections 2024-04-21T08:43:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6e580a3ec0 0x7f6e580a6380 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e6c19cf40 0x7f6e6c19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e6c107270 0x7f6e6c198bc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 --2- 172.21.15.94:0/556781845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e6c1021a0 0x7f6e6c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 -- 172.21.15.94:0/556781845 >> 172.21.15.94:0/556781845 conn(0x7f6e6c0793e0 msgr2=0x7f6e6c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 -- 172.21.15.94:0/556781845 shutdown_connections 2024-04-21T08:43:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:20.566+0000 7f6e73ee4640 1 -- 172.21.15.94:0/556781845 wait complete. 2024-04-21T08:43:20.577 INFO:tasks.cephfs.filesystem:scrub status for tag:fb89b90f-a7ea-405e-9f73-f69ad1c796cf - {'path': '/', 'tag': 'fb89b90f-a7ea-405e-9f73-f69ad1c796cf', 'options': 'recursive,force'} 2024-04-21T08:43:21.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:21 smithi094 ceph-mon[27988]: pgmap v104: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:43:21.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:21 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:43:21.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:21 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:21.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:21 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:43:21.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:21 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:21.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:21 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:43:21.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:21 smithi173 ceph-mon[31428]: pgmap v104: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:43:21.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:21 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:43:21.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:21 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:21.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:21 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:43:21.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:21 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:21.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:21 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:43:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:21 smithi111 ceph-mon[31493]: pgmap v104: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:43:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:21 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:43:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:21 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:21.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:21 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:43:21.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:21 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:21.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:21 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:43:23.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:23 smithi094 ceph-mon[27988]: pgmap v105: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:23.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:23 smithi173 ceph-mon[31428]: pgmap v105: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:23.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:23 smithi111 ceph-mon[31493]: pgmap v105: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:25.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:25 smithi094 ceph-mon[27988]: pgmap v106: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:43:25.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:25 smithi173 ceph-mon[31428]: pgmap v106: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:43:25.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:25 smithi111 ceph-mon[31493]: pgmap v106: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:43:27.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:27 smithi094 ceph-mon[27988]: pgmap v107: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:43:27.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:27 smithi173 ceph-mon[31428]: pgmap v107: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:43:27.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:27 smithi111 ceph-mon[31493]: pgmap v107: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:43:29.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:29 smithi094 ceph-mon[27988]: pgmap v108: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:43:29.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:29 smithi173 ceph-mon[31428]: pgmap v108: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:43:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:29 smithi111 ceph-mon[31493]: pgmap v108: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:43:31.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:31 smithi094 ceph-mon[27988]: pgmap v109: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:43:31.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:31 smithi173 ceph-mon[31428]: pgmap v109: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:43:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:31 smithi111 ceph-mon[31493]: pgmap v109: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:43:33.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:33 smithi094 ceph-mon[27988]: pgmap v110: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:43:33.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:33 smithi173 ceph-mon[31428]: pgmap v110: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:43:33.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:33 smithi111 ceph-mon[31493]: pgmap v110: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:43:35.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:35 smithi094 ceph-mon[27988]: pgmap v111: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:43:35.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:35 smithi173 ceph-mon[31428]: pgmap v111: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:43:35.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:35 smithi111 ceph-mon[31493]: pgmap v111: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:43:37.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:37 smithi094 ceph-mon[27988]: pgmap v112: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:43:37.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:37 smithi173 ceph-mon[31428]: pgmap v112: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:43:37.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:37 smithi111 ceph-mon[31493]: pgmap v112: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:43:39.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:39 smithi094 ceph-mon[27988]: pgmap v113: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:39.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:39 smithi173 ceph-mon[31428]: pgmap v113: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:39.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:39 smithi111 ceph-mon[31493]: pgmap v113: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:41.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:41 smithi111 ceph-mon[31493]: pgmap v114: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:43:41.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:41 smithi094 ceph-mon[27988]: pgmap v114: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:43:41.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:41 smithi173 ceph-mon[31428]: pgmap v114: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:43:43.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:43 smithi094 ceph-mon[27988]: pgmap v115: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:43.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:43 smithi173 ceph-mon[31428]: pgmap v115: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:43.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:43 smithi111 ceph-mon[31493]: pgmap v115: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:45.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:45 smithi094 ceph-mon[27988]: pgmap v116: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:43:45.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:45 smithi173 ceph-mon[31428]: pgmap v116: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:43:45.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:45 smithi111 ceph-mon[31493]: pgmap v116: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:43:47.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:47 smithi094 ceph-mon[27988]: pgmap v117: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:43:47.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:47 smithi173 ceph-mon[31428]: pgmap v117: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:43:47.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:47 smithi111 ceph-mon[31493]: pgmap v117: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:43:49.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:49 smithi094 ceph-mon[27988]: pgmap v118: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:49.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:49 smithi173 ceph-mon[31428]: pgmap v118: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:49.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:49 smithi111 ceph-mon[31493]: pgmap v118: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:50.578 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:43:50.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:43:50.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.695+0000 7f3878ef2640 1 -- 172.21.15.94:0/1912963388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3874106150 msgr2=0x7f387410dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.695+0000 7f3878ef2640 1 --2- 172.21.15.94:0/1912963388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3874106150 0x7f387410dd20 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f386c009900 tx=0x7f386c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:50.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.695+0000 7f3878ef2640 1 -- 172.21.15.94:0/1912963388 shutdown_connections 2024-04-21T08:43:50.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.695+0000 7f3878ef2640 1 --2- 172.21.15.94:0/1912963388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3874101a30 0x7f3874105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.695+0000 7f3878ef2640 1 --2- 172.21.15.94:0/1912963388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3874106150 0x7f387410dd20 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.695+0000 7f3878ef2640 1 --2- 172.21.15.94:0/1912963388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3874101080 0x7f3874101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.695+0000 7f3878ef2640 1 -- 172.21.15.94:0/1912963388 >> 172.21.15.94:0/1912963388 conn(0x7f3874078000 msgr2=0x7f3874078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:50.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.695+0000 7f3878ef2640 1 -- 172.21.15.94:0/1912963388 shutdown_connections 2024-04-21T08:43:50.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.695+0000 7f3878ef2640 1 -- 172.21.15.94:0/1912963388 wait complete. 2024-04-21T08:43:50.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3878ef2640 1 Processor -- start 2024-04-21T08:43:50.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3878ef2640 1 -- start start 2024-04-21T08:43:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3878ef2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3874101080 0x7f3874199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3878ef2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3874106150 0x7f3874199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3878ef2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f387419a030 0x7f38741a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3878ef2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f38741110c0 con 0x7f387419a030 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3878ef2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3874110f40 con 0x7f3874101080 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3878ef2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3874111240 con 0x7f3874106150 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3872d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f387419a030 0x7f38741a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3872d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f387419a030 0x7f38741a45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33618/0 (socket says 172.21.15.94:33618) 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3872d9d640 1 -- 172.21.15.94:0/1247664684 learned_addr learned my addr 172.21.15.94:0/1247664684 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f387259c640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3874101080 0x7f3874199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3871d9b640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3874106150 0x7f3874199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3872d9d640 1 -- 172.21.15.94:0/1247664684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3874106150 msgr2=0x7f3874199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3872d9d640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3874106150 0x7f3874199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3872d9d640 1 -- 172.21.15.94:0/1247664684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3874101080 msgr2=0x7f3874199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3872d9d640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3874101080 0x7f3874199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.696+0000 7f3872d9d640 1 -- 172.21.15.94:0/1247664684 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38741a4db0 con 0x7f387419a030 2024-04-21T08:43:50.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.697+0000 7f3872d9d640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f387419a030 0x7f38741a45e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f386c009230 tx=0x7f386c02c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.697+0000 7f38637fe640 1 -- 172.21.15.94:0/1247664684 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c043020 con 0x7f387419a030 2024-04-21T08:43:50.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.697+0000 7f38637fe640 1 -- 172.21.15.94:0/1247664684 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f386c02caf0 con 0x7f387419a030 2024-04-21T08:43:50.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.697+0000 7f3878ef2640 1 -- 172.21.15.94:0/1247664684 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38741a5040 con 0x7f387419a030 2024-04-21T08:43:50.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.697+0000 7f38637fe640 1 -- 172.21.15.94:0/1247664684 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c003990 con 0x7f387419a030 2024-04-21T08:43:50.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.697+0000 7f3878ef2640 1 -- 172.21.15.94:0/1247664684 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38741a55e0 con 0x7f387419a030 2024-04-21T08:43:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.699+0000 7f38637fe640 1 -- 172.21.15.94:0/1247664684 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f386c046020 con 0x7f387419a030 2024-04-21T08:43:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.699+0000 7f387bbc3740 1 Processor -- start 2024-04-21T08:43:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f387bbc3740 1 -- start start 2024-04-21T08:43:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f38637fe640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f385c0a3da0 0x7f385c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686d580 0x5627b686d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6876540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b6878e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f387bbc3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5627b67d21e0 con 0x5627b6876a80 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f387bbc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5627b676f390 con 0x5627b686dea0 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f387bbc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5627b67ce1a0 con 0x5627b686d580 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f3871d9b640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f385c0a3da0 0x7f385c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f3872d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b6878e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f3872d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b6878e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33630/0 (socket says 172.21.15.94:33630) 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f3872d9d640 1 -- 172.21.15.94:0/830524525 learned_addr learned my addr 172.21.15.94:0/830524525 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f38637fe640 1 -- 172.21.15.94:0/1247664684 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f386c0f2c60 con 0x7f387419a030 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f3871d9b640 1 --2- 172.21.15.94:0/830524525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6876540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.700+0000 7f387259c640 1 --2- 172.21.15.94:0/830524525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686d580 0x5627b686d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.701+0000 7f3872d9d640 1 -- 172.21.15.94:0/830524525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686d580 msgr2=0x5627b686d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.701+0000 7f3872d9d640 1 --2- 172.21.15.94:0/830524525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686d580 0x5627b686d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.701+0000 7f3872d9d640 1 -- 172.21.15.94:0/830524525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 msgr2=0x5627b6876540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.701+0000 7f3872d9d640 1 --2- 172.21.15.94:0/830524525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6876540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.701+0000 7f3872d9d640 1 -- 172.21.15.94:0/830524525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627b65ae7d0 con 0x5627b6876a80 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.701+0000 7f3871d9b640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f385c0a3da0 0x7f385c0a6260 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3874069c30 tx=0x7f38680137d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.701+0000 7f3872d9d640 1 --2- 172.21.15.94:0/830524525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b6878e70 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f386c0f0050 tx=0x7f386c03ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff018d2c81b46c12 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.701+0000 7f3860ff9640 1 -- 172.21.15.94:0/830524525 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c0be290 con 0x5627b6876a80 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.701+0000 7f3860ff9640 1 -- 172.21.15.94:0/830524525 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f386c0bf030 con 0x5627b6876a80 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.701+0000 7f3860ff9640 1 -- 172.21.15.94:0/830524525 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c0f0250 con 0x5627b6876a80 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 -- 172.21.15.94:0/830524525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 msgr2=0x5627b6878e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 --2- 172.21.15.94:0/830524525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b6878e70 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f386c0f0050 tx=0x7f386c03ac10 comp rx=0 tx=0).stop 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 -- 172.21.15.94:0/830524525 shutdown_connections 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 --2- 172.21.15.94:0/830524525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b6878e70 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 --2- 172.21.15.94:0/830524525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6876540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 --2- 172.21.15.94:0/830524525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686d580 0x5627b686d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 -- 172.21.15.94:0/830524525 >> 172.21.15.94:0/830524525 conn(0x5627b6867cb0 msgr2=0x5627b6868090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 -- 172.21.15.94:0/830524525 shutdown_connections 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 -- 172.21.15.94:0/830524525 wait complete. 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 -- start start 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 Processor -- start 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b686d580 0x5627b6873a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6873fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b6876a80 0x5627b6874500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5627b6879630 con 0x5627b686d580 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5627b678b2f0 con 0x5627b686dea0 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5627b68797b0 con 0x5627b6876a80 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387bbc3740 10 client.0 _ll_register_callbacks cb 0x5627b6798580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b686d580 0x5627b6873a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b686d580 0x5627b6873a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33646/0 (socket says 172.21.15.94:33646) 2024-04-21T08:43:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.702+0000 7f387259c640 1 -- 172.21.15.94:0/4030222852 learned_addr learned my addr 172.21.15.94:0/4030222852 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f3872d9d640 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b6876a80 0x5627b6874500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f3871d9b640 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6873fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f387259c640 1 -- 172.21.15.94:0/4030222852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b6876a80 msgr2=0x5627b6874500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f387259c640 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b6876a80 0x5627b6874500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f387259c640 1 -- 172.21.15.94:0/4030222852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 msgr2=0x5627b6873fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f387259c640 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6873fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f387259c640 1 -- 172.21.15.94:0/4030222852 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627b64e43f0 con 0x5627b686d580 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f387259c640 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b686d580 0x5627b6873a80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3864010b70 tx=0x7f386400f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6568e993ffdbf10c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f3859ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f3859ffb640 1 -- 172.21.15.94:0/4030222852 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386401e060 con 0x5627b686d580 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f3859ffb640 1 -- 172.21.15.94:0/4030222852 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f386400e5c0 con 0x5627b686d580 2024-04-21T08:43:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f387bbc3740 1 -- 172.21.15.94:0/4030222852 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f386c0bf030 con 0x5627b686d580 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f3859ffb640 1 -- 172.21.15.94:0/4030222852 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3864025050 con 0x5627b686d580 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f387bbc3740 10 client.16785 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f387bbc3740 1 -- 172.21.15.94:0/4030222852 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5627b64a40f0 con 0x5627b686d580 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f3859ffb640 1 -- 172.21.15.94:0/4030222852 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f386c0bf030 con 0x5627b686d580 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.703+0000 7f3859ffb640 1 -- 172.21.15.94:0/4030222852 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3864019910 con 0x5627b686d580 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.704+0000 7f387bbc3740 10 client.16785 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.704+0000 7f387bbc3740 10 client.16785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.704+0000 7f387bbc3740 20 client.16785 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.704+0000 7f387bbc3740 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5627b6886810 0x5627b6888c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.704+0000 7f387bbc3740 4 client.16785 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.704+0000 7f387bbc3740 1 -- 172.21.15.94:0/4030222852 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627b67dfc00 con 0x5627b6886810 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.704+0000 7f3871d9b640 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5627b6886810 0x5627b6888c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.704+0000 7f3859ffb640 10 client.16785 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.704+0000 7f3871d9b640 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5627b6886810 0x5627b6888c80 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=195ae239d6096e6d server_cookie=6ec6522f8166ddf in_seq=0 out_seq=0 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f3859ffb640 1 -- 172.21.15.94:0/4030222852 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627b67dfc00 con 0x5627b6886810 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f3859ffb640 10 client.16785 handle_command_reply: tid=0 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 client.16785 shutdown 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 -- 172.21.15.94:0/4030222852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b686d580 msgr2=0x5627b6873a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b686d580 0x5627b6873a80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3864010b70 tx=0x7f386400f7f0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 -- 172.21.15.94:0/4030222852 shutdown_connections 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5627b6886810 0x5627b6888c80 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b6876a80 0x5627b6874500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6873fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f3859ffb640 0 client.16785 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 --2- 172.21.15.94:0/4030222852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b686d580 0x5627b6873a80 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 -- 172.21.15.94:0/4030222852 >> 172.21.15.94:0/4030222852 conn(0x5627b6867cb0 msgr2=0x5627b6869a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 -- 172.21.15.94:0/4030222852 shutdown_connections 2024-04-21T08:43:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 1 -- 172.21.15.94:0/4030222852 wait complete. 2024-04-21T08:43:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.705+0000 7f387bbc3740 20 client.16785 trim_cache size 0 max 16384 2024-04-21T08:43:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f387bbc3740 1 Processor -- start 2024-04-21T08:43:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f387bbc3740 1 -- start start 2024-04-21T08:43:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686d580 0x5627b68761b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6882460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b687ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f387bbc3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f38741b1d30 con 0x5627b6876a80 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f387bbc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f38741af600 con 0x5627b686dea0 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f387bbc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f38741ad450 con 0x5627b686d580 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f3872d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b687ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f3872d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b687ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33662/0 (socket says 172.21.15.94:33662) 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f3872d9d640 1 -- 172.21.15.94:0/2869961041 learned_addr learned my addr 172.21.15.94:0/2869961041 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f3871d9b640 1 --2- 172.21.15.94:0/2869961041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6882460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f3872d9d640 1 -- 172.21.15.94:0/2869961041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686d580 msgr2=0x5627b68761b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f3872d9d640 1 --2- 172.21.15.94:0/2869961041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686d580 0x5627b68761b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f3872d9d640 1 -- 172.21.15.94:0/2869961041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 msgr2=0x5627b6882460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f3872d9d640 1 --2- 172.21.15.94:0/2869961041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6882460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.723+0000 7f3872d9d640 1 -- 172.21.15.94:0/2869961041 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627b6879630 con 0x5627b6876a80 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f3872d9d640 1 --2- 172.21.15.94:0/2869961041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b687ed50 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f386c0f05d0 tx=0x7f386c0f0080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0e73dded78748d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f385bfff640 1 -- 172.21.15.94:0/2869961041 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c0be290 con 0x5627b6876a80 2024-04-21T08:43:50.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f385bfff640 1 -- 172.21.15.94:0/2869961041 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f386c0bf030 con 0x5627b6876a80 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f385bfff640 1 -- 172.21.15.94:0/2869961041 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c0be470 con 0x5627b6876a80 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f387bbc3740 1 -- 172.21.15.94:0/2869961041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 msgr2=0x5627b687ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f387bbc3740 1 --2- 172.21.15.94:0/2869961041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b687ed50 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f386c0f05d0 tx=0x7f386c0f0080 comp rx=0 tx=0).stop 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f387bbc3740 1 -- 172.21.15.94:0/2869961041 shutdown_connections 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f387bbc3740 1 --2- 172.21.15.94:0/2869961041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b687ed50 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f387bbc3740 1 --2- 172.21.15.94:0/2869961041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686dea0 0x5627b6882460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f387bbc3740 1 --2- 172.21.15.94:0/2869961041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686d580 0x5627b68761b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f387bbc3740 1 -- 172.21.15.94:0/2869961041 >> 172.21.15.94:0/2869961041 conn(0x5627b6867cb0 msgr2=0x5627b6866d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f387bbc3740 1 -- 172.21.15.94:0/2869961041 shutdown_connections 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.724+0000 7f387bbc3740 1 -- 172.21.15.94:0/2869961041 wait complete. 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f387bbc3740 1 -- start start 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f387bbc3740 1 Processor -- start 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686d580 0x5627b68e4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686dea0 0x5627b68e54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f387bbc3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b68e9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f387bbc3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f38741adac0 con 0x5627b6876a80 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f387bbc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f38741aef30 con 0x5627b686d580 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f387bbc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f38741ae190 con 0x5627b686dea0 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f387bbc3740 10 client.0 _ll_register_callbacks cb 0x7f386400aa50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3872d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b68e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3872d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b68e9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33668/0 (socket says 172.21.15.94:33668) 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3872d9d640 1 -- 172.21.15.94:0/3732150776 learned_addr learned my addr 172.21.15.94:0/3732150776 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f387259c640 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686d580 0x5627b68e4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3871d9b640 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686dea0 0x5627b68e54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3872d9d640 1 -- 172.21.15.94:0/3732150776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686dea0 msgr2=0x5627b68e54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3872d9d640 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686dea0 0x5627b68e54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3872d9d640 1 -- 172.21.15.94:0/3732150776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686d580 msgr2=0x5627b68e4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:50.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3872d9d640 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686d580 0x5627b68e4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3872d9d640 1 -- 172.21.15.94:0/3732150776 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627b68e97c0 con 0x5627b6876a80 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3872d9d640 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b68e9150 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f386c0df040 tx=0x7f386c0f04a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce18d5e048cc61b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.725+0000 7f3859ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f3859ffb640 1 -- 172.21.15.94:0/3732150776 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c0be470 con 0x5627b6876a80 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f3859ffb640 1 -- 172.21.15.94:0/3732150776 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f386c0beb90 con 0x5627b6876a80 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f387bbc3740 1 -- 172.21.15.94:0/3732150776 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f386c0bf030 con 0x5627b6876a80 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f3859ffb640 1 -- 172.21.15.94:0/3732150776 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f386c0be290 con 0x5627b6876a80 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f387bbc3740 10 client.16791 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f387bbc3740 1 -- 172.21.15.94:0/3732150776 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5627b68e97c0 con 0x5627b6876a80 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f3859ffb640 1 -- 172.21.15.94:0/3732150776 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f386c0bf030 con 0x5627b6876a80 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f3859ffb640 1 -- 172.21.15.94:0/3732150776 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f386c04abe0 con 0x5627b6876a80 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f387bbc3740 10 client.16791 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f387bbc3740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f387bbc3740 20 client.16791 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f387bbc3740 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5627b68f1f90 0x5627b68f4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f387bbc3740 4 client.16791 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f387bbc3740 1 -- 172.21.15.94:0/3732150776 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5627b67d9a70 con 0x5627b68f1f90 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.726+0000 7f387259c640 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5627b68f1f90 0x5627b68f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387259c640 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5627b68f1f90 0x5627b68f4380 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ce8173d9efdde6a server_cookie=53f150413fd41423 in_seq=0 out_seq=0 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f3859ffb640 10 client.16791 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f3859ffb640 1 -- 172.21.15.94:0/3732150776 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5627b67d9a70 con 0x5627b68f1f90 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f3859ffb640 10 client.16791 handle_command_reply: tid=0 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 client.16791 shutdown 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 -- 172.21.15.94:0/3732150776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 msgr2=0x5627b68e9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b68e9150 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f386c0df040 tx=0x7f386c0f04a0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 -- 172.21.15.94:0/3732150776 shutdown_connections 2024-04-21T08:43:50.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5627b68f1f90 0x5627b68f4380 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627b686dea0 0x5627b68e54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5627b6876a80 0x5627b68e9150 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 --2- 172.21.15.94:0/3732150776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5627b686d580 0x5627b68e4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 -- 172.21.15.94:0/3732150776 >> 172.21.15.94:0/3732150776 conn(0x5627b6867cb0 msgr2=0x5627b6873a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:50.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f3859ffb640 0 client.16791 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:50.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 -- 172.21.15.94:0/3732150776 shutdown_connections 2024-04-21T08:43:50.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 1 -- 172.21.15.94:0/3732150776 wait complete. 2024-04-21T08:43:50.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.727+0000 7f387bbc3740 20 client.16791 trim_cache size 0 max 16384 2024-04-21T08:43:50.733 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:43:50.733 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:43:50.734 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:43:50.734 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:43:50.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 -- 172.21.15.94:0/1247664684 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f385c0a3da0 msgr2=0x7f385c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f385c0a3da0 0x7f385c0a6260 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3874069c30 tx=0x7f38680137d0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 -- 172.21.15.94:0/1247664684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f387419a030 msgr2=0x7f38741a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f387419a030 0x7f38741a45e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f386c009230 tx=0x7f386c02c710 comp rx=0 tx=0).stop 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 -- 172.21.15.94:0/1247664684 shutdown_connections 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f385c0a3da0 0x7f385c0a6260 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3874106150 0x7f3874199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f387419a030 0x7f38741a45e0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 --2- 172.21.15.94:0/1247664684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3874101080 0x7f3874199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 -- 172.21.15.94:0/1247664684 >> 172.21.15.94:0/1247664684 conn(0x7f3874078000 msgr2=0x7f38740feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 -- 172.21.15.94:0/1247664684 shutdown_connections 2024-04-21T08:43:50.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.728+0000 7f3878ef2640 1 -- 172.21.15.94:0/1247664684 wait complete. 2024-04-21T08:43:50.737 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:43:50.738 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:43:50.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.836+0000 7fe87e7ad640 1 -- 172.21.15.94:0/2772617559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe878107340 msgr2=0x7fe87810eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.836+0000 7fe87e7ad640 1 --2- 172.21.15.94:0/2772617559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe878107340 0x7fe87810eec0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fe870009ca0 tx=0x7fe87002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.836+0000 7fe87e7ad640 1 -- 172.21.15.94:0/2772617559 shutdown_connections 2024-04-21T08:43:50.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.836+0000 7fe87e7ad640 1 --2- 172.21.15.94:0/2772617559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe878107340 0x7fe87810eec0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.836+0000 7fe87e7ad640 1 --2- 172.21.15.94:0/2772617559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe87806b260 0x7fe878106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.836+0000 7fe87e7ad640 1 --2- 172.21.15.94:0/2772617559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe87806a8b0 0x7fe87806ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.836+0000 7fe87e7ad640 1 -- 172.21.15.94:0/2772617559 >> 172.21.15.94:0/2772617559 conn(0x7fe878077e30 msgr2=0x7fe878078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.836+0000 7fe87e7ad640 1 -- 172.21.15.94:0/2772617559 shutdown_connections 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.836+0000 7fe87e7ad640 1 -- 172.21.15.94:0/2772617559 wait complete. 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.837+0000 7fe87e7ad640 1 Processor -- start 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.837+0000 7fe87e7ad640 1 -- start start 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.837+0000 7fe87e7ad640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe87806a8b0 0x7fe87819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.837+0000 7fe87e7ad640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe878107340 0x7fe87819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.837+0000 7fe87e7ad640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8781985e0 0x7fe878198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.837+0000 7fe87e7ad640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe878112260 con 0x7fe8781985e0 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.837+0000 7fe87e7ad640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe8781120e0 con 0x7fe878107340 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.837+0000 7fe87e7ad640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe8781123e0 con 0x7fe87806a8b0 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.837+0000 7fe87cd4a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8781985e0 0x7fe878198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe87cd4a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8781985e0 0x7fe878198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33680/0 (socket says 172.21.15.94:33680) 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe87cd4a640 1 -- 172.21.15.94:0/1556894696 learned_addr learned my addr 172.21.15.94:0/1556894696 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe877fff640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe87806a8b0 0x7fe87819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe8777fe640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe878107340 0x7fe87819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe87cd4a640 1 -- 172.21.15.94:0/1556894696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe87806a8b0 msgr2=0x7fe87819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe87cd4a640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe87806a8b0 0x7fe87819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe87cd4a640 1 -- 172.21.15.94:0/1556894696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe878107340 msgr2=0x7fe87819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe87cd4a640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe878107340 0x7fe87819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe87cd4a640 1 -- 172.21.15.94:0/1556894696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe878199100 con 0x7fe8781985e0 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe8777fe640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe878107340 0x7fe87819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe87cd4a640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8781985e0 0x7fe878198a90 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fe870005e30 tx=0x7fe870034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe8757fa640 1 -- 172.21.15.94:0/1556894696 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870039040 con 0x7fe8781985e0 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe8757fa640 1 -- 172.21.15.94:0/1556894696 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe870044070 con 0x7fe8781985e0 2024-04-21T08:43:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe87e7ad640 1 -- 172.21.15.94:0/1556894696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe878199390 con 0x7fe8781985e0 2024-04-21T08:43:50.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe8757fa640 1 -- 172.21.15.94:0/1556894696 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870003e60 con 0x7fe8781985e0 2024-04-21T08:43:50.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.838+0000 7fe87e7ad640 1 -- 172.21.15.94:0/1556894696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8781a55e0 con 0x7fe8781985e0 2024-04-21T08:43:50.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.839+0000 7fe87e7ad640 1 -- 172.21.15.94:0/1556894696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe878069230 con 0x7fe8781985e0 2024-04-21T08:43:50.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.840+0000 7fe8757fa640 1 -- 172.21.15.94:0/1556894696 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fe8700324c0 con 0x7fe8781985e0 2024-04-21T08:43:50.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.841+0000 7fe8757fa640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe8600a3da0 0x7fe8600a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.841+0000 7fe877fff640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe8600a3da0 0x7fe8600a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.841+0000 7fe8757fa640 1 -- 172.21.15.94:0/1556894696 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe8700f3790 con 0x7fe8781985e0 2024-04-21T08:43:50.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.841+0000 7fe877fff640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe8600a3da0 0x7fe8600a6260 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe868005fd0 tx=0x7fe86800b390 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:50.844+0000 7fe8757fa640 1 -- 172.21.15.94:0/1556894696 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe8700bfb60 con 0x7fe8781985e0 2024-04-21T08:43:51.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.059+0000 7fe87e7ad640 1 -- 172.21.15.94:0/1556894696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe87810a460 con 0x7fe8781985e0 2024-04-21T08:43:51.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.060+0000 7fe8757fa640 1 -- 172.21.15.94:0/1556894696 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7fe8700bf460 con 0x7fe8781985e0 2024-04-21T08:43:51.059 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:43:51.059 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:43:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:43:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 -- 172.21.15.94:0/1556894696 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe8600a3da0 msgr2=0x7fe8600a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe8600a3da0 0x7fe8600a6260 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe868005fd0 tx=0x7fe86800b390 comp rx=0 tx=0).stop 2024-04-21T08:43:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 -- 172.21.15.94:0/1556894696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8781985e0 msgr2=0x7fe878198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8781985e0 0x7fe878198a90 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fe870005e30 tx=0x7fe870034760 comp rx=0 tx=0).stop 2024-04-21T08:43:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 -- 172.21.15.94:0/1556894696 shutdown_connections 2024-04-21T08:43:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe8600a3da0 0x7fe8600a6260 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8781985e0 0x7fe878198a90 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe878107340 0x7fe87819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 --2- 172.21.15.94:0/1556894696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe87806a8b0 0x7fe87819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 -- 172.21.15.94:0/1556894696 >> 172.21.15.94:0/1556894696 conn(0x7fe878077e30 msgr2=0x7fe8780ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 -- 172.21.15.94:0/1556894696 shutdown_connections 2024-04-21T08:43:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.062+0000 7fe87e7ad640 1 -- 172.21.15.94:0/1556894696 wait complete. 2024-04-21T08:43:51.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:43:51.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.173+0000 7f4cca25c640 1 -- 172.21.15.94:0/3753833601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cc4102b40 msgr2=0x7f4cc4106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.173+0000 7f4cca25c640 1 --2- 172.21.15.94:0/3753833601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cc4102b40 0x7f4cc4106b80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009900 tx=0x7f4cb802bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.173+0000 7f4cca25c640 1 -- 172.21.15.94:0/3753833601 shutdown_connections 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.173+0000 7f4cca25c640 1 --2- 172.21.15.94:0/3753833601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc4107260 0x7f4cc410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.173+0000 7f4cca25c640 1 --2- 172.21.15.94:0/3753833601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cc4102b40 0x7f4cc4106b80 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.173+0000 7f4cca25c640 1 --2- 172.21.15.94:0/3753833601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cc4102190 0x7f4cc4102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.173+0000 7f4cca25c640 1 -- 172.21.15.94:0/3753833601 >> 172.21.15.94:0/3753833601 conn(0x7f4cc40793a0 msgr2=0x7f4cc40797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.173+0000 7f4cca25c640 1 -- 172.21.15.94:0/3753833601 shutdown_connections 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.173+0000 7f4cca25c640 1 -- 172.21.15.94:0/3753833601 wait complete. 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cca25c640 1 Processor -- start 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cca25c640 1 -- start start 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cca25c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cc4102190 0x7f4cc4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cca25c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cc4107260 0x7f4cc4198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cca25c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc419cff0 0x7f4cc419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cca25c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc41121d0 con 0x7f4cc4102190 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cca25c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc4112050 con 0x7f4cc419cff0 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cca25c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc4112350 con 0x7f4cc4107260 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cc37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cc4102190 0x7f4cc4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cc37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cc4102190 0x7f4cc4198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33688/0 (socket says 172.21.15.94:33688) 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cc37fe640 1 -- 172.21.15.94:0/3136317513 learned_addr learned my addr 172.21.15.94:0/3136317513 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cc3fff640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc419cff0 0x7f4cc419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cc37fe640 1 -- 172.21.15.94:0/3136317513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cc4107260 msgr2=0x7f4cc4198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cc37fe640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cc4107260 0x7f4cc4198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cc37fe640 1 -- 172.21.15.94:0/3136317513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc419cff0 msgr2=0x7f4cc419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cc37fe640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc419cff0 0x7f4cc419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.174+0000 7f4cc37fe640 1 -- 172.21.15.94:0/3136317513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc419dae0 con 0x7f4cc4102190 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.175+0000 7f4cc37fe640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cc4102190 0x7f4cc4198730 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400c920 tx=0x7f4cb400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.175+0000 7f4cc0ff9640 1 -- 172.21.15.94:0/3136317513 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4007c70 con 0x7f4cc4102190 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.175+0000 7f4cc0ff9640 1 -- 172.21.15.94:0/3136317513 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cb4007e10 con 0x7f4cc4102190 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.175+0000 7f4cca25c640 1 -- 172.21.15.94:0/3136317513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc419ddd0 con 0x7f4cc4102190 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.175+0000 7f4cc0ff9640 1 -- 172.21.15.94:0/3136317513 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4018820 con 0x7f4cc4102190 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.175+0000 7f4cca25c640 1 -- 172.21.15.94:0/3136317513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc41a55c0 con 0x7f4cc4102190 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4cc0ff9640 1 -- 172.21.15.94:0/3136317513 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4cb400f450 con 0x7f4cc4102190 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4ccce51740 1 Processor -- start 2024-04-21T08:43:51.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4ccce51740 1 -- start start 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48250 0x5581d5e48630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48c00 0x5581d5e55840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e58220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4cc0ff9640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4cac0a3d20 0x7f4cac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4ccce51740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5581d5dad1f0 con 0x5581d5e55e10 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4ccce51740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5581d5d4a4b0 con 0x5581d5e48250 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4ccce51740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5581d5da91b0 con 0x5581d5e48c00 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4cc2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e58220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4cc2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e58220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33696/0 (socket says 172.21.15.94:33696) 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4cc2ffd640 1 -- 172.21.15.94:0/2332467115 learned_addr learned my addr 172.21.15.94:0/2332467115 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.177+0000 7f4cc3fff640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4cac0a3d20 0x7f4cac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4cc37fe640 1 --2- 172.21.15.94:0/2332467115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48c00 0x5581d5e55840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4cc3fff640 1 --2- 172.21.15.94:0/2332467115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48250 0x5581d5e48630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4cc0ff9640 1 -- 172.21.15.94:0/3136317513 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4cb40d0db0 con 0x7f4cc4102190 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4cc2ffd640 1 -- 172.21.15.94:0/2332467115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48c00 msgr2=0x5581d5e55840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4cc2ffd640 1 --2- 172.21.15.94:0/2332467115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48c00 0x5581d5e55840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4cc2ffd640 1 -- 172.21.15.94:0/2332467115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48250 msgr2=0x5581d5e48630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4cc2ffd640 1 --2- 172.21.15.94:0/2332467115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48250 0x5581d5e48630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4cc2ffd640 1 -- 172.21.15.94:0/2332467115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581d5d3c1b0 con 0x5581d5e55e10 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4cc2ffd640 1 --2- 172.21.15.94:0/2332467115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e58220 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f4cb802c170 tx=0x7f4cb8005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767f345dd892fef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4cc3fff640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4cac0a3d20 0x7f4cac0a61e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc005cc0 tx=0x7f4cbc00a340 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4ca9ffb640 1 -- 172.21.15.94:0/2332467115 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8042020 con 0x5581d5e55e10 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4ca9ffb640 1 -- 172.21.15.94:0/2332467115 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cb802cc90 con 0x5581d5e55e10 2024-04-21T08:43:51.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.178+0000 7f4ca9ffb640 1 -- 172.21.15.94:0/2332467115 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8003ca0 con 0x5581d5e55e10 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 -- 172.21.15.94:0/2332467115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 msgr2=0x5581d5e58220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 --2- 172.21.15.94:0/2332467115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e58220 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f4cb802c170 tx=0x7f4cb8005e00 comp rx=0 tx=0).stop 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 -- 172.21.15.94:0/2332467115 shutdown_connections 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 --2- 172.21.15.94:0/2332467115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48c00 0x5581d5e55840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 --2- 172.21.15.94:0/2332467115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e58220 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 --2- 172.21.15.94:0/2332467115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48250 0x5581d5e48630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 -- 172.21.15.94:0/2332467115 >> 172.21.15.94:0/2332467115 conn(0x5581d5e42ba0 msgr2=0x5581d5e42f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 -- 172.21.15.94:0/2332467115 shutdown_connections 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 -- 172.21.15.94:0/2332467115 wait complete. 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 -- start start 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.179+0000 7f4ccce51740 1 Processor -- start 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48250 0x5581d5e556e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48c00 0x5581d5e52a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e55e10 0x5581d5e52fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ccce51740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5581d5e58ce0 con 0x5581d5e48c00 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ccce51740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5581d5d66500 con 0x5581d5e55e10 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ccce51740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5581d5e58e60 con 0x5581d5e48250 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ccce51740 10 client.0 _ll_register_callbacks cb 0x5581d5d34b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48c00 0x5581d5e52a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48c00 0x5581d5e52a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33698/0 (socket says 172.21.15.94:33698) 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc2ffd640 1 -- 172.21.15.94:0/1186303109 learned_addr learned my addr 172.21.15.94:0/1186303109 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc37fe640 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48250 0x5581d5e556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc3fff640 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e55e10 0x5581d5e52fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc2ffd640 1 -- 172.21.15.94:0/1186303109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48250 msgr2=0x5581d5e556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc2ffd640 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48250 0x5581d5e556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc2ffd640 1 -- 172.21.15.94:0/1186303109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e55e10 msgr2=0x5581d5e52fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc2ffd640 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e55e10 0x5581d5e52fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc2ffd640 1 -- 172.21.15.94:0/1186303109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581d5abf3f0 con 0x5581d5e48c00 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc3fff640 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e55e10 0x5581d5e52fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4cc2ffd640 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48c00 0x5581d5e52a70 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8006010 tx=0x7f4cb802cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d15c411ebb9958 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ca2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/1186303109 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8042030 con 0x5581d5e48c00 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/1186303109 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4cb8004040 con 0x5581d5e48c00 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ccce51740 1 -- 172.21.15.94:0/1186303109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cb802cc90 con 0x5581d5e48c00 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.180+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/1186303109 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4cb8059050 con 0x5581d5e48c00 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4ccce51740 10 client.16809 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4ccce51740 1 -- 172.21.15.94:0/1186303109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5581d5b897d0 con 0x5581d5e48c00 2024-04-21T08:43:51.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/1186303109 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cb802cc90 con 0x5581d5e48c00 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/1186303109 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4cb8048430 con 0x5581d5e48c00 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4ccce51740 10 client.16809 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4ccce51740 10 client.16809 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4ccce51740 20 client.16809 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4ccce51740 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5581d5e65e10 0x5581d5e68200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4ccce51740 4 client.16809 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4ccce51740 1 -- 172.21.15.94:0/1186303109 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581d5d4c930 con 0x5581d5e65e10 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.181+0000 7f4cc37fe640 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5581d5e65e10 0x5581d5e68200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ca2ffd640 10 client.16809 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4cc37fe640 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5581d5e65e10 0x5581d5e68200 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=838af7c622f843b5 server_cookie=7438b1881b3da8c5 in_seq=0 out_seq=0 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/1186303109 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581d5d4c930 con 0x5581d5e65e10 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ca2ffd640 10 client.16809 handle_command_reply: tid=0 2024-04-21T08:43:51.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ccce51740 1 client.16809 shutdown 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ccce51740 1 -- 172.21.15.94:0/1186303109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48c00 msgr2=0x5581d5e52a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ccce51740 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48c00 0x5581d5e52a70 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8006010 tx=0x7f4cb802cb60 comp rx=0 tx=0).stop 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ccce51740 1 -- 172.21.15.94:0/1186303109 shutdown_connections 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ccce51740 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5581d5e65e10 0x5581d5e68200 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ccce51740 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e55e10 0x5581d5e52fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ccce51740 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48c00 0x5581d5e52a70 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ca2ffd640 0 client.16809 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ccce51740 1 --2- 172.21.15.94:0/1186303109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48250 0x5581d5e556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.182+0000 7f4ccce51740 1 -- 172.21.15.94:0/1186303109 >> 172.21.15.94:0/1186303109 conn(0x5581d5e42ba0 msgr2=0x5581d5e448f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.183+0000 7f4ccce51740 1 -- 172.21.15.94:0/1186303109 shutdown_connections 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.183+0000 7f4ccce51740 1 -- 172.21.15.94:0/1186303109 wait complete. 2024-04-21T08:43:51.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.183+0000 7f4ccce51740 20 client.16809 trim_cache size 0 max 16384 2024-04-21T08:43:51.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.195+0000 7f4ccce51740 1 Processor -- start 2024-04-21T08:43:51.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.195+0000 7f4ccce51740 1 -- start start 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48250 0x5581d5e44880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48c00 0x5581d5e41a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4ccce51740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc41afa70 con 0x5581d5e55e10 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4ccce51740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc41ad340 con 0x5581d5e48c00 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4ccce51740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc41ab190 con 0x5581d5e48250 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4cc3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4cc3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33708/0 (socket says 172.21.15.94:33708) 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4cc3fff640 1 -- 172.21.15.94:0/828292990 learned_addr learned my addr 172.21.15.94:0/828292990 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4cc3fff640 1 -- 172.21.15.94:0/828292990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48250 msgr2=0x5581d5e44880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4cc3fff640 1 --2- 172.21.15.94:0/828292990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48250 0x5581d5e44880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4cc3fff640 1 -- 172.21.15.94:0/828292990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48c00 msgr2=0x5581d5e41a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4cc3fff640 1 --2- 172.21.15.94:0/828292990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48c00 0x5581d5e41a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4cc3fff640 1 -- 172.21.15.94:0/828292990 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581d5e47560 con 0x5581d5e55e10 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4cc3fff640 1 --2- 172.21.15.94:0/828292990 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e551a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc013120 tx=0x7f4cbc005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e14a7c2a3e5567 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4ca97fa640 1 -- 172.21.15.94:0/828292990 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc02a070 con 0x5581d5e55e10 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4ca97fa640 1 -- 172.21.15.94:0/828292990 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cbc011d90 con 0x5581d5e55e10 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.196+0000 7f4ca97fa640 1 -- 172.21.15.94:0/828292990 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc025740 con 0x5581d5e55e10 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.197+0000 7f4ccce51740 1 -- 172.21.15.94:0/828292990 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 msgr2=0x5581d5e551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.197+0000 7f4ccce51740 1 --2- 172.21.15.94:0/828292990 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e551a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc013120 tx=0x7f4cbc005bd0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.197+0000 7f4ccce51740 1 -- 172.21.15.94:0/828292990 shutdown_connections 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.197+0000 7f4ccce51740 1 --2- 172.21.15.94:0/828292990 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e55e10 0x5581d5e551a0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.197+0000 7f4ccce51740 1 --2- 172.21.15.94:0/828292990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48c00 0x5581d5e41a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.197+0000 7f4ccce51740 1 --2- 172.21.15.94:0/828292990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e48250 0x5581d5e44880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.197+0000 7f4ccce51740 1 -- 172.21.15.94:0/828292990 >> 172.21.15.94:0/828292990 conn(0x5581d5e42ba0 msgr2=0x5581d5e41370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.197+0000 7f4ccce51740 1 -- 172.21.15.94:0/828292990 shutdown_connections 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.197+0000 7f4ccce51740 1 -- 172.21.15.94:0/828292990 wait complete. 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.197+0000 7f4ccce51740 1 -- start start 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ccce51740 1 Processor -- start 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48250 0x5581d5e68ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48c00 0x5581d5e69530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ccce51740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e55e10 0x5581d5ecaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ccce51740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc41ab800 con 0x5581d5e48250 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ccce51740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc41acc70 con 0x5581d5e48c00 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ccce51740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc41abed0 con 0x5581d5e55e10 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ccce51740 10 client.0 _ll_register_callbacks cb 0x7f4cb802c7e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4cc37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48250 0x5581d5e68ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4cc37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48250 0x5581d5e68ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33722/0 (socket says 172.21.15.94:33722) 2024-04-21T08:43:51.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4cc37fe640 1 -- 172.21.15.94:0/2248856697 learned_addr learned my addr 172.21.15.94:0/2248856697 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4cc37fe640 1 -- 172.21.15.94:0/2248856697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e55e10 msgr2=0x5581d5ecaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4cc37fe640 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e55e10 0x5581d5ecaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4cc37fe640 1 -- 172.21.15.94:0/2248856697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48c00 msgr2=0x5581d5e69530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4cc37fe640 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48c00 0x5581d5e69530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4cc37fe640 1 -- 172.21.15.94:0/2248856697 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581d5e69e30 con 0x5581d5e48250 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4cc37fe640 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48250 0x5581d5e68ff0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4099340 tx=0x7f4cb4014400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91199df58d3ea9d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ca2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/2248856697 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb409da60 con 0x5581d5e48250 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/2248856697 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4cb400fa90 con 0x5581d5e48250 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.198+0000 7f4ccce51740 1 -- 172.21.15.94:0/2248856697 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cbc011d90 con 0x5581d5e48250 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/2248856697 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4cb409c970 con 0x5581d5e48250 2024-04-21T08:43:51.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ccce51740 10 client.16815 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ccce51740 1 -- 172.21.15.94:0/2248856697 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5581d5ecc6c0 con 0x5581d5e48250 2024-04-21T08:43:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/2248856697 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cbc011d90 con 0x5581d5e48250 2024-04-21T08:43:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/2248856697 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4cb400f740 con 0x5581d5e48250 2024-04-21T08:43:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ccce51740 10 client.16815 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ccce51740 10 client.16815 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:43:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ccce51740 20 client.16815 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ccce51740 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5581d5eccda0 0x5581d5ecf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ccce51740 4 client.16815 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:43:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ccce51740 1 -- 172.21.15.94:0/2248856697 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5581d5dbac10 con 0x5581d5eccda0 2024-04-21T08:43:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4cc2ffd640 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5581d5eccda0 0x5581d5ecf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4ca2ffd640 10 client.16815 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.199+0000 7f4cc2ffd640 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5581d5eccda0 0x5581d5ecf220 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9dc56eb922bd5b80 server_cookie=8d2a5789790b0697 in_seq=0 out_seq=0 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ca2ffd640 1 -- 172.21.15.94:0/2248856697 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5581d5dbac10 con 0x5581d5eccda0 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ca2ffd640 10 client.16815 handle_command_reply: tid=0 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 client.16815 shutdown 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 -- 172.21.15.94:0/2248856697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48250 msgr2=0x5581d5e68ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48250 0x5581d5e68ff0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4099340 tx=0x7f4cb4014400 comp rx=0 tx=0).stop 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 -- 172.21.15.94:0/2248856697 shutdown_connections 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5581d5eccda0 0x5581d5ecf220 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d5e55e10 0x5581d5ecaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d5e48c00 0x5581d5e69530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ca2ffd640 0 client.16815 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:43:51.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 --2- 172.21.15.94:0/2248856697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d5e48250 0x5581d5e68ff0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 -- 172.21.15.94:0/2248856697 >> 172.21.15.94:0/2248856697 conn(0x5581d5e42ba0 msgr2=0x5581d5e610f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 -- 172.21.15.94:0/2248856697 shutdown_connections 2024-04-21T08:43:51.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 1 -- 172.21.15.94:0/2248856697 wait complete. 2024-04-21T08:43:51.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4ccce51740 20 client.16815 trim_cache size 0 max 16384 2024-04-21T08:43:51.207 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4cca25c640 1 -- 172.21.15.94:0/3136317513 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4cac0a3d20 msgr2=0x7f4cac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4cca25c640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4cac0a3d20 0x7f4cac0a61e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc005cc0 tx=0x7f4cbc00a340 comp rx=0 tx=0).stop 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4cca25c640 1 -- 172.21.15.94:0/3136317513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cc4102190 msgr2=0x7f4cc4198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.200+0000 7f4cca25c640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cc4102190 0x7f4cc4198730 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400c920 tx=0x7f4cb400cde0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.201+0000 7f4cca25c640 1 -- 172.21.15.94:0/3136317513 shutdown_connections 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.201+0000 7f4cca25c640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4cac0a3d20 0x7f4cac0a61e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.201+0000 7f4cca25c640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cc4107260 0x7f4cc4198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.201+0000 7f4cca25c640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc419cff0 0x7f4cc419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.201+0000 7f4cca25c640 1 --2- 172.21.15.94:0/3136317513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cc4102190 0x7f4cc4198730 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.201+0000 7f4cca25c640 1 -- 172.21.15.94:0/3136317513 >> 172.21.15.94:0/3136317513 conn(0x7f4cc40793a0 msgr2=0x7f4cc40fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.201+0000 7f4cca25c640 1 -- 172.21.15.94:0/3136317513 shutdown_connections 2024-04-21T08:43:51.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.201+0000 7f4cca25c640 1 -- 172.21.15.94:0/3136317513 wait complete. 2024-04-21T08:43:51.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:43:51.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.312+0000 7fae09eb2640 1 -- 172.21.15.94:0/4209219142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae04106160 msgr2=0x7fae0410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.312+0000 7fae09eb2640 1 --2- 172.21.15.94:0/4209219142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae04106160 0x7fae0410dd30 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009ca0 tx=0x7fadfc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.312+0000 7fae09eb2640 1 -- 172.21.15.94:0/4209219142 shutdown_connections 2024-04-21T08:43:51.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.312+0000 7fae09eb2640 1 --2- 172.21.15.94:0/4209219142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae04106160 0x7fae0410dd30 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.312+0000 7fae09eb2640 1 --2- 172.21.15.94:0/4209219142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae04101a40 0x7fae04105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.312+0000 7fae09eb2640 1 --2- 172.21.15.94:0/4209219142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae04101090 0x7fae04101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.312+0000 7fae09eb2640 1 -- 172.21.15.94:0/4209219142 >> 172.21.15.94:0/4209219142 conn(0x7fae04077f50 msgr2=0x7fae04078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.312+0000 7fae09eb2640 1 -- 172.21.15.94:0/4209219142 shutdown_connections 2024-04-21T08:43:51.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.312+0000 7fae09eb2640 1 -- 172.21.15.94:0/4209219142 wait complete. 2024-04-21T08:43:51.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae09eb2640 1 Processor -- start 2024-04-21T08:43:51.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae09eb2640 1 -- start start 2024-04-21T08:43:51.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae09eb2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae04101090 0x7fae04198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae09eb2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae04106160 0x7fae04198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae09eb2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae0419cf30 0x7fae0419d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae09eb2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fae041110d0 con 0x7fae04106160 2024-04-21T08:43:51.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae09eb2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fae04110f50 con 0x7fae0419cf30 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae09eb2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fae04111250 con 0x7fae04101090 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae02ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae04106160 0x7fae04198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae02ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae04106160 0x7fae04198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33748/0 (socket says 172.21.15.94:33748) 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.313+0000 7fae02ffd640 1 -- 172.21.15.94:0/3535469947 learned_addr learned my addr 172.21.15.94:0/3535469947 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae037fe640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae04101090 0x7fae04198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae03fff640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae0419cf30 0x7fae0419d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae02ffd640 1 -- 172.21.15.94:0/3535469947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae04101090 msgr2=0x7fae04198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae02ffd640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae04101090 0x7fae04198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae02ffd640 1 -- 172.21.15.94:0/3535469947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae0419cf30 msgr2=0x7fae0419d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae02ffd640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae0419cf30 0x7fae0419d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae02ffd640 1 -- 172.21.15.94:0/3535469947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0419da20 con 0x7fae04106160 2024-04-21T08:43:51.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae03fff640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae0419cf30 0x7fae0419d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:51.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae02ffd640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae04106160 0x7fae04198bb0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fadf800c970 tx=0x7fadf800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae00ff9640 1 -- 172.21.15.94:0/3535469947 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf8014070 con 0x7fae04106160 2024-04-21T08:43:51.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae00ff9640 1 -- 172.21.15.94:0/3535469947 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadf8004440 con 0x7fae04106160 2024-04-21T08:43:51.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae09eb2640 1 -- 172.21.15.94:0/3535469947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae0419dd10 con 0x7fae04106160 2024-04-21T08:43:51.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae00ff9640 1 -- 172.21.15.94:0/3535469947 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf8003e50 con 0x7fae04106160 2024-04-21T08:43:51.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.314+0000 7fae09eb2640 1 -- 172.21.15.94:0/3535469947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae041a5500 con 0x7fae04106160 2024-04-21T08:43:51.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.316+0000 7fae00ff9640 1 -- 172.21.15.94:0/3535469947 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fadf8017070 con 0x7fae04106160 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.316+0000 7fae0cb76740 1 Processor -- start 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae0cb76740 1 -- start start 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995958b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562499595e80 0x56249959e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249959ea00 0x5624995a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae0cb76740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5624994fa1f0 con 0x5624995954d0 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae0cb76740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5624994974b0 con 0x562499595e80 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae0cb76740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5624994f61b0 con 0x56249959ea00 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae03fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae00ff9640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadec0a3ec0 0x7fadec0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae03fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33750/0 (socket says 172.21.15.94:33750) 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae03fff640 1 -- 172.21.15.94:0/1097735946 learned_addr learned my addr 172.21.15.94:0/1097735946 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae02ffd640 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249959ea00 0x5624995a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae03fff640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadec0a3ec0 0x7fadec0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae03fff640 1 -- 172.21.15.94:0/1097735946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249959ea00 msgr2=0x5624995a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae00ff9640 1 -- 172.21.15.94:0/3535469947 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fadf80d0780 con 0x7fae04106160 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae03fff640 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249959ea00 0x5624995a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae037fe640 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562499595e80 0x56249959e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae03fff640 1 -- 172.21.15.94:0/1097735946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562499595e80 msgr2=0x56249959e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae03fff640 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562499595e80 0x56249959e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae03fff640 1 -- 172.21.15.94:0/1097735946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624994891b0 con 0x5624995954d0 2024-04-21T08:43:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae02ffd640 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249959ea00 0x5624995a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae037fe640 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562499595e80 0x56249959e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.317+0000 7fae03fff640 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995958b0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009430 tx=0x7fadfc032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104f674458cc39af server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fade9ffb640 1 -- 172.21.15.94:0/1097735946 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0327a0 con 0x5624995954d0 2024-04-21T08:43:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fade9ffb640 1 -- 172.21.15.94:0/1097735946 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadfc032940 con 0x5624995954d0 2024-04-21T08:43:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fade9ffb640 1 -- 172.21.15.94:0/1097735946 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc034890 con 0x5624995954d0 2024-04-21T08:43:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fae03fff640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadec0a3ec0 0x7fadec0a6380 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fadfc04c000 tx=0x7fadfc048000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fae0cb76740 1 -- 172.21.15.94:0/1097735946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 msgr2=0x5624995958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995958b0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009430 tx=0x7fadfc032f30 comp rx=0 tx=0).stop 2024-04-21T08:43:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fae0cb76740 1 -- 172.21.15.94:0/1097735946 shutdown_connections 2024-04-21T08:43:51.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249959ea00 0x5624995a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562499595e80 0x56249959e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1097735946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995958b0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fae0cb76740 1 -- 172.21.15.94:0/1097735946 >> 172.21.15.94:0/1097735946 conn(0x56249958fba0 msgr2=0x56249958ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fae0cb76740 1 -- 172.21.15.94:0/1097735946 shutdown_connections 2024-04-21T08:43:51.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.318+0000 7fae0cb76740 1 -- 172.21.15.94:0/1097735946 wait complete. 2024-04-21T08:43:51.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae0cb76740 1 -- start start 2024-04-21T08:43:51.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae0cb76740 1 Processor -- start 2024-04-21T08:43:51.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5624995954d0 0x56249959e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562499595e80 0x56249959b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56249959ea00 0x56249959bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae0cb76740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5624995a1970 con 0x56249959ea00 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae0cb76740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5624994b3500 con 0x5624995954d0 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae0cb76740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5624995a1af0 con 0x562499595e80 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae0cb76740 10 client.0 _ll_register_callbacks cb 0x562499481b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae03fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56249959ea00 0x56249959bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae03fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56249959ea00 0x56249959bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33760/0 (socket says 172.21.15.94:33760) 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae03fff640 1 -- 172.21.15.94:0/1350268284 learned_addr learned my addr 172.21.15.94:0/1350268284 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae037fe640 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5624995954d0 0x56249959e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae03fff640 1 -- 172.21.15.94:0/1350268284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562499595e80 msgr2=0x56249959b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae02ffd640 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562499595e80 0x56249959b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae03fff640 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562499595e80 0x56249959b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae03fff640 1 -- 172.21.15.94:0/1350268284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5624995954d0 msgr2=0x56249959e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae03fff640 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5624995954d0 0x56249959e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae03fff640 1 -- 172.21.15.94:0/1350268284 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56249920c3f0 con 0x56249959ea00 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.319+0000 7fae037fe640 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5624995954d0 0x56249959e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fae03fff640 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56249959ea00 0x56249959bcf0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fadfc04f040 tx=0x7fadfc0324c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3356c2663cc5fc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fade2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fade2ffd640 1 -- 172.21.15.94:0/1350268284 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc039040 con 0x56249959ea00 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fade2ffd640 1 -- 172.21.15.94:0/1350268284 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fadfc034b30 con 0x56249959ea00 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fae0cb76740 1 -- 172.21.15.94:0/1350268284 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadfc032940 con 0x56249959ea00 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fade2ffd640 1 -- 172.21.15.94:0/1350268284 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fadfc060050 con 0x56249959ea00 2024-04-21T08:43:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fae0cb76740 10 client.16827 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fae0cb76740 1 -- 172.21.15.94:0/1350268284 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5624992d67d0 con 0x56249959ea00 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fade2ffd640 1 -- 172.21.15.94:0/1350268284 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadfc032940 con 0x56249959ea00 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fade2ffd640 1 -- 172.21.15.94:0/1350268284 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fadfc0434c0 con 0x56249959ea00 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fae0cb76740 10 client.16827 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fae0cb76740 10 client.16827 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fae0cb76740 20 client.16827 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5624995aea50 0x5624995b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fae0cb76740 4 client.16827 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.320+0000 7fae0cb76740 1 -- 172.21.15.94:0/1350268284 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562499499930 con 0x5624995aea50 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.321+0000 7fae037fe640 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5624995aea50 0x5624995b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.321+0000 7fade2ffd640 10 client.16827 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:43:51.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.321+0000 7fae037fe640 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5624995aea50 0x5624995b0e40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=18e17b00931f2ad1 server_cookie=13513238222b81a7 in_seq=0 out_seq=0 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fade2ffd640 1 -- 172.21.15.94:0/1350268284 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562499499930 con 0x5624995aea50 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fade2ffd640 10 client.16827 handle_command_reply: tid=0 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 client.16827 shutdown 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 -- 172.21.15.94:0/1350268284 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56249959ea00 msgr2=0x56249959bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56249959ea00 0x56249959bcf0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fadfc04f040 tx=0x7fadfc0324c0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 -- 172.21.15.94:0/1350268284 shutdown_connections 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562499595e80 0x56249959b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5624995aea50 0x5624995b0e40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56249959ea00 0x56249959bcf0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1350268284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5624995954d0 0x56249959e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fade2ffd640 0 client.16827 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:43:51.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 -- 172.21.15.94:0/1350268284 >> 172.21.15.94:0/1350268284 conn(0x56249958fba0 msgr2=0x5624995918f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 -- 172.21.15.94:0/1350268284 shutdown_connections 2024-04-21T08:43:51.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 1 -- 172.21.15.94:0/1350268284 wait complete. 2024-04-21T08:43:51.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.322+0000 7fae0cb76740 20 client.16827 trim_cache size 0 max 16384 2024-04-21T08:43:51.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.335+0000 7fae0cb76740 1 Processor -- start 2024-04-21T08:43:51.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.335+0000 7fae0cb76740 1 -- start start 2024-04-21T08:43:51.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562499595e80 0x56249959a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 0x5624995aa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae0cb76740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fae041b1c10 con 0x5624995954d0 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae0cb76740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fae041af4e0 con 0x56249959ea00 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae0cb76740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fae041ad330 con 0x562499595e80 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995b3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33762/0 (socket says 172.21.15.94:33762) 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae037fe640 1 -- 172.21.15.94:0/2631744246 learned_addr learned my addr 172.21.15.94:0/2631744246 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae03fff640 1 --2- 172.21.15.94:0/2631744246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 0x5624995aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae037fe640 1 -- 172.21.15.94:0/2631744246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562499595e80 msgr2=0x56249959a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae037fe640 1 --2- 172.21.15.94:0/2631744246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562499595e80 0x56249959a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae037fe640 1 -- 172.21.15.94:0/2631744246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 msgr2=0x5624995aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae037fe640 1 --2- 172.21.15.94:0/2631744246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 0x5624995aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae037fe640 1 -- 172.21.15.94:0/2631744246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624995904e0 con 0x5624995954d0 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fae037fe640 1 --2- 172.21.15.94:0/2631744246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995b3e70 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x562499594fa0 tx=0x7fadf400b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f65f9ac1a3d78f49 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fade97fa640 1 -- 172.21.15.94:0/2631744246 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf4013cb0 con 0x5624995954d0 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fade97fa640 1 -- 172.21.15.94:0/2631744246 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadf401e030 con 0x5624995954d0 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.336+0000 7fade97fa640 1 -- 172.21.15.94:0/2631744246 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf4011970 con 0x5624995954d0 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 -- 172.21.15.94:0/2631744246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 msgr2=0x5624995b3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 --2- 172.21.15.94:0/2631744246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995b3e70 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x562499594fa0 tx=0x7fadf400b3a0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 -- 172.21.15.94:0/2631744246 shutdown_connections 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 --2- 172.21.15.94:0/2631744246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562499595e80 0x56249959a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 --2- 172.21.15.94:0/2631744246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 0x5624995aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 --2- 172.21.15.94:0/2631744246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5624995954d0 0x5624995b3e70 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 -- 172.21.15.94:0/2631744246 >> 172.21.15.94:0/2631744246 conn(0x56249958fba0 msgr2=0x5624995a9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 -- 172.21.15.94:0/2631744246 shutdown_connections 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 -- 172.21.15.94:0/2631744246 wait complete. 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 -- start start 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.337+0000 7fae0cb76740 1 Processor -- start 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624995954d0 0x5624996103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562499595e80 0x562499610910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae0cb76740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 0x562499598650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae0cb76740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fae041ad9a0 con 0x562499595e80 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae0cb76740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fae041aee10 con 0x56249959ea00 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae0cb76740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fae041ae070 con 0x5624995954d0 2024-04-21T08:43:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae0cb76740 10 client.0 _ll_register_callbacks cb 0x7fadfc004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae02ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562499595e80 0x562499610910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae02ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562499595e80 0x562499610910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33770/0 (socket says 172.21.15.94:33770) 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae02ffd640 1 -- 172.21.15.94:0/1287792156 learned_addr learned my addr 172.21.15.94:0/1287792156 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae03fff640 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 0x562499598650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae037fe640 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624995954d0 0x5624996103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae02ffd640 1 -- 172.21.15.94:0/1287792156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624995954d0 msgr2=0x5624996103d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae02ffd640 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624995954d0 0x5624996103d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae02ffd640 1 -- 172.21.15.94:0/1287792156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 msgr2=0x562499598650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae02ffd640 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 0x562499598650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae02ffd640 1 -- 172.21.15.94:0/1287792156 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562499598e50 con 0x562499595e80 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae03fff640 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 0x562499598650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:51.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae02ffd640 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562499595e80 0x562499610910 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fadf809c2d0 tx=0x7fadf8066de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b04c6f6415f264 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fade2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fade2ffd640 1 -- 172.21.15.94:0/1287792156 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf8066e50 con 0x562499595e80 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae0cb76740 1 -- 172.21.15.94:0/1287792156 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadf401e030 con 0x562499595e80 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fade2ffd640 1 -- 172.21.15.94:0/1287792156 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fadf80d0380 con 0x562499595e80 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fade2ffd640 1 -- 172.21.15.94:0/1287792156 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fadf809ca40 con 0x562499595e80 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae0cb76740 10 client.16833 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.338+0000 7fae0cb76740 1 -- 172.21.15.94:0/1287792156 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562499618d20 con 0x562499595e80 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fade2ffd640 1 -- 172.21.15.94:0/1287792156 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadf401e030 con 0x562499595e80 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fade2ffd640 1 -- 172.21.15.94:0/1287792156 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fadf809c5a0 con 0x562499595e80 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fae0cb76740 10 client.16833 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:51.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fae0cb76740 10 client.16833 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fae0cb76740 20 client.16833 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5624996194a0 0x56249961b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fae0cb76740 4 client.16833 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fae0cb76740 1 -- 172.21.15.94:0/1287792156 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562499507c10 con 0x5624996194a0 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fae037fe640 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5624996194a0 0x56249961b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fade2ffd640 10 client.16833 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.339+0000 7fae037fe640 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5624996194a0 0x56249961b8c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=11722426737e7a2d server_cookie=928da5b5d8c06a7b in_seq=0 out_seq=0 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fade2ffd640 1 -- 172.21.15.94:0/1287792156 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562499507c10 con 0x5624996194a0 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fade2ffd640 10 client.16833 handle_command_reply: tid=0 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 client.16833 shutdown 2024-04-21T08:43:51.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 -- 172.21.15.94:0/1287792156 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562499595e80 msgr2=0x562499610910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562499595e80 0x562499610910 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fadf809c2d0 tx=0x7fadf8066de0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 -- 172.21.15.94:0/1287792156 shutdown_connections 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5624996194a0 0x56249961b8c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56249959ea00 0x562499598650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562499595e80 0x562499610910 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fade2ffd640 0 client.16833 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 --2- 172.21.15.94:0/1287792156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624995954d0 0x5624996103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 -- 172.21.15.94:0/1287792156 >> 172.21.15.94:0/1287792156 conn(0x56249958fba0 msgr2=0x562499590f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 -- 172.21.15.94:0/1287792156 shutdown_connections 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 1 -- 172.21.15.94:0/1287792156 wait complete. 2024-04-21T08:43:51.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae0cb76740 20 client.16833 trim_cache size 0 max 16384 2024-04-21T08:43:51.348 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:43:51.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae09eb2640 1 -- 172.21.15.94:0/3535469947 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadec0a3ec0 msgr2=0x7fadec0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae09eb2640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadec0a3ec0 0x7fadec0a6380 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fadfc04c000 tx=0x7fadfc048000 comp rx=0 tx=0).stop 2024-04-21T08:43:51.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae09eb2640 1 -- 172.21.15.94:0/3535469947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae04106160 msgr2=0x7fae04198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae09eb2640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae04106160 0x7fae04198bb0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fadf800c970 tx=0x7fadf800ce30 comp rx=0 tx=0).stop 2024-04-21T08:43:51.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae09eb2640 1 -- 172.21.15.94:0/3535469947 shutdown_connections 2024-04-21T08:43:51.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae09eb2640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadec0a3ec0 0x7fadec0a6380 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae09eb2640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae0419cf30 0x7fae0419d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae09eb2640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae04106160 0x7fae04198bb0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.340+0000 7fae09eb2640 1 --2- 172.21.15.94:0/3535469947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae04101090 0x7fae04198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.341+0000 7fae09eb2640 1 -- 172.21.15.94:0/3535469947 >> 172.21.15.94:0/3535469947 conn(0x7fae04077f50 msgr2=0x7fae040693c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.341+0000 7fae09eb2640 1 -- 172.21.15.94:0/3535469947 shutdown_connections 2024-04-21T08:43:51.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.341+0000 7fae09eb2640 1 -- 172.21.15.94:0/3535469947 wait complete. 2024-04-21T08:43:51.351 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:43:51.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 -- 172.21.15.94:0/1732116090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc28101130 msgr2=0x7efc28101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 --2- 172.21.15.94:0/1732116090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc28101130 0x7efc28101590 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7efc24009900 tx=0x7efc2402bc80 comp rx=0 tx=0).stop 2024-04-21T08:43:51.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 -- 172.21.15.94:0/1732116090 shutdown_connections 2024-04-21T08:43:51.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 --2- 172.21.15.94:0/1732116090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc28101ad0 0x7efc2810dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 --2- 172.21.15.94:0/1732116090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc28101130 0x7efc28101590 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 --2- 172.21.15.94:0/1732116090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc28105ea0 0x7efc28106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 -- 172.21.15.94:0/1732116090 >> 172.21.15.94:0/1732116090 conn(0x7efc28077e30 msgr2=0x7efc28078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 -- 172.21.15.94:0/1732116090 shutdown_connections 2024-04-21T08:43:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 -- 172.21.15.94:0/1732116090 wait complete. 2024-04-21T08:43:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 Processor -- start 2024-04-21T08:43:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.453+0000 7efc2fd2b640 1 -- start start 2024-04-21T08:43:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2fd2b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc28101130 0x7efc28199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2fd2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc28101ad0 0x7efc28199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2fd2b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc28105ea0 0x7efc281a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2fd2b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efc281111f0 con 0x7efc28101ad0 2024-04-21T08:43:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2fd2b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc28111070 con 0x7efc28101130 2024-04-21T08:43:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2fd2b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc28111370 con 0x7efc28105ea0 2024-04-21T08:43:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2d2c6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc28101ad0 0x7efc28199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2d2c6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc28101ad0 0x7efc28199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33792/0 (socket says 172.21.15.94:33792) 2024-04-21T08:43:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2d2c6640 1 -- 172.21.15.94:0/4143722898 learned_addr learned my addr 172.21.15.94:0/4143722898 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2dac7640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc28101130 0x7efc28199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc28105ea0 0x7efc281a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2d2c6640 1 -- 172.21.15.94:0/4143722898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc28105ea0 msgr2=0x7efc281a4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2d2c6640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc28105ea0 0x7efc281a4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2d2c6640 1 -- 172.21.15.94:0/4143722898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc28101130 msgr2=0x7efc28199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2d2c6640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc28101130 0x7efc28199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2d2c6640 1 -- 172.21.15.94:0/4143722898 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc281a4c50 con 0x7efc28101ad0 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc28105ea0 0x7efc281a4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2dac7640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc28101130 0x7efc28199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2d2c6640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc28101ad0 0x7efc28199bd0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7efc24006010 tx=0x7efc24005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc1effd640 1 -- 172.21.15.94:0/4143722898 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24043020 con 0x7efc28101ad0 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2fd2b640 1 -- 172.21.15.94:0/4143722898 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc281a4e20 con 0x7efc28101ad0 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc1effd640 1 -- 172.21.15.94:0/4143722898 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc24004350 con 0x7efc28101ad0 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc1effd640 1 -- 172.21.15.94:0/4143722898 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24003a10 con 0x7efc28101ad0 2024-04-21T08:43:51.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.454+0000 7efc2fd2b640 1 -- 172.21.15.94:0/4143722898 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc281a5290 con 0x7efc28101ad0 2024-04-21T08:43:51.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.456+0000 7efc1effd640 1 -- 172.21.15.94:0/4143722898 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7efc24046020 con 0x7efc28101ad0 2024-04-21T08:43:51.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc329f6740 1 Processor -- start 2024-04-21T08:43:51.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc329f6740 1 -- start start 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc329f6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f49088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc329f6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f4911430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc329f6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4911a00 0x5632f4913e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc329f6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5632f486d1f0 con 0x5632f4908e80 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc329f6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5632f480a4b0 con 0x5632f49084d0 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc329f6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5632f48691b0 con 0x5632f4911a00 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc1effd640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc140a3d20 0x7efc140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc2dac7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f4911430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc2e2c8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f49088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc2dac7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f4911430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33802/0 (socket says 172.21.15.94:33802) 2024-04-21T08:43:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.457+0000 7efc2dac7640 1 -- 172.21.15.94:0/770597823 learned_addr learned my addr 172.21.15.94:0/770597823 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc2d2c6640 1 --2- 172.21.15.94:0/770597823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4911a00 0x5632f4913e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc140a3d20 0x7efc140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc1effd640 1 -- 172.21.15.94:0/4143722898 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7efc240f2740 con 0x7efc28101ad0 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc2e2c8640 1 -- 172.21.15.94:0/770597823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4911a00 msgr2=0x5632f4913e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/770597823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4911a00 0x5632f4913e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc2e2c8640 1 -- 172.21.15.94:0/770597823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 msgr2=0x5632f4911430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/770597823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f4911430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc2e2c8640 1 -- 172.21.15.94:0/770597823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632f47fc1b0 con 0x5632f49084d0 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc2dac7640 1 --2- 172.21.15.94:0/770597823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f4911430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/770597823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f49088b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efc20014e70 tx=0x7efc200117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b830dbf83ed62ea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc140a3d20 0x7efc140a61e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efc20005c40 tx=0x7efc2001c000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc13fff640 1 -- 172.21.15.94:0/770597823 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc20022070 con 0x5632f49084d0 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc13fff640 1 -- 172.21.15.94:0/770597823 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc20003e20 con 0x5632f49084d0 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.458+0000 7efc13fff640 1 -- 172.21.15.94:0/770597823 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc20004030 con 0x5632f49084d0 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 -- 172.21.15.94:0/770597823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 msgr2=0x5632f49088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 --2- 172.21.15.94:0/770597823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f49088b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efc20014e70 tx=0x7efc200117b0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 -- 172.21.15.94:0/770597823 shutdown_connections 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 --2- 172.21.15.94:0/770597823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4911a00 0x5632f4913e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 --2- 172.21.15.94:0/770597823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f4911430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 --2- 172.21.15.94:0/770597823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f49088b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 -- 172.21.15.94:0/770597823 >> 172.21.15.94:0/770597823 conn(0x5632f4902ba0 msgr2=0x5632f4902f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 -- 172.21.15.94:0/770597823 shutdown_connections 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 -- 172.21.15.94:0/770597823 wait complete. 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 -- start start 2024-04-21T08:43:51.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.459+0000 7efc329f6740 1 Processor -- start 2024-04-21T08:43:51.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc329f6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f4911370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc329f6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 0x5632f490e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc329f6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4911a00 0x5632f490ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc329f6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5632f4914910 con 0x5632f4911a00 2024-04-21T08:43:51.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc329f6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5632f4826500 con 0x5632f49084d0 2024-04-21T08:43:51.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc329f6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5632f4914a90 con 0x5632f4908e80 2024-04-21T08:43:51.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc329f6740 10 client.0 _ll_register_callbacks cb 0x5632f47f4b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:51.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2e2c8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4911a00 0x5632f490ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2e2c8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4911a00 0x5632f490ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33808/0 (socket says 172.21.15.94:33808) 2024-04-21T08:43:51.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2e2c8640 1 -- 172.21.15.94:0/3811038227 learned_addr learned my addr 172.21.15.94:0/3811038227 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2dac7640 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f4911370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2d2c6640 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 0x5632f490e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2e2c8640 1 -- 172.21.15.94:0/3811038227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 msgr2=0x5632f490e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 0x5632f490e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2e2c8640 1 -- 172.21.15.94:0/3811038227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 msgr2=0x5632f4911370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f4911370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2e2c8640 1 -- 172.21.15.94:0/3811038227 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632f457f3f0 con 0x5632f4911a00 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2d2c6640 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 0x5632f490e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4911a00 0x5632f490ed10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7efc20009540 tx=0x7efc2000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b9cd4689c42287 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc117fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:51.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.460+0000 7efc117fa640 1 -- 172.21.15.94:0/3811038227 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc20022070 con 0x5632f4911a00 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc117fa640 1 -- 172.21.15.94:0/3811038227 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7efc200050c0 con 0x5632f4911a00 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc329f6740 1 -- 172.21.15.94:0/3811038227 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc20003e20 con 0x5632f4911a00 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc117fa640 1 -- 172.21.15.94:0/3811038227 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efc20038050 con 0x5632f4911a00 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc329f6740 10 client.16845 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc329f6740 1 -- 172.21.15.94:0/3811038227 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5632f46497d0 con 0x5632f4911a00 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc117fa640 1 -- 172.21.15.94:0/3811038227 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc20003e20 con 0x5632f4911a00 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc117fa640 1 -- 172.21.15.94:0/3811038227 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efc2001d070 con 0x5632f4911a00 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc329f6740 10 client.16845 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc329f6740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc329f6740 20 client.16845 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc329f6740 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5632f4921bf0 0x5632f4923fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc329f6740 4 client.16845 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc329f6740 1 -- 172.21.15.94:0/3811038227 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632f480c930 con 0x5632f4921bf0 2024-04-21T08:43:51.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.461+0000 7efc2dac7640 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5632f4921bf0 0x5632f4923fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.462+0000 7efc117fa640 10 client.16845 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.462+0000 7efc2dac7640 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5632f4921bf0 0x5632f4923fe0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bf944890424115 server_cookie=792853c22886823f in_seq=0 out_seq=0 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.462+0000 7efc117fa640 1 -- 172.21.15.94:0/3811038227 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632f480c930 con 0x5632f4921bf0 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.462+0000 7efc117fa640 10 client.16845 handle_command_reply: tid=0 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.462+0000 7efc329f6740 1 client.16845 shutdown 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.462+0000 7efc329f6740 1 -- 172.21.15.94:0/3811038227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4911a00 msgr2=0x5632f490ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.462+0000 7efc329f6740 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4911a00 0x5632f490ed10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7efc20009540 tx=0x7efc2000cf60 comp rx=0 tx=0).stop 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.463+0000 7efc329f6740 1 -- 172.21.15.94:0/3811038227 shutdown_connections 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.463+0000 7efc329f6740 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5632f4921bf0 0x5632f4923fe0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.463+0000 7efc329f6740 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 0x5632f490e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.463+0000 7efc329f6740 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4911a00 0x5632f490ed10 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.463+0000 7efc117fa640 0 client.16845 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:51.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.463+0000 7efc329f6740 1 --2- 172.21.15.94:0/3811038227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f4911370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.463+0000 7efc329f6740 1 -- 172.21.15.94:0/3811038227 >> 172.21.15.94:0/3811038227 conn(0x5632f4902ba0 msgr2=0x5632f49048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.463+0000 7efc329f6740 1 -- 172.21.15.94:0/3811038227 shutdown_connections 2024-04-21T08:43:51.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.463+0000 7efc329f6740 1 -- 172.21.15.94:0/3811038227 wait complete. 2024-04-21T08:43:51.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.463+0000 7efc329f6740 20 client.16845 trim_cache size 0 max 16384 2024-04-21T08:43:51.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc329f6740 1 Processor -- start 2024-04-21T08:43:51.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc329f6740 1 -- start start 2024-04-21T08:43:51.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc329f6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f49084d0 0x5632f491d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc329f6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 0x5632f4910e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc329f6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f4911a00 0x5632f4911370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc329f6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efc281b1d10 con 0x5632f49084d0 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc329f6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc281af5e0 con 0x5632f4911a00 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc329f6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc281ad430 con 0x5632f4908e80 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc2dac7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f49084d0 0x5632f491d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc2dac7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f49084d0 0x5632f491d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33824/0 (socket says 172.21.15.94:33824) 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc2dac7640 1 -- 172.21.15.94:0/2406654695 learned_addr learned my addr 172.21.15.94:0/2406654695 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc2d2c6640 1 --2- 172.21.15.94:0/2406654695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 0x5632f4910e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc2dac7640 1 -- 172.21.15.94:0/2406654695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 msgr2=0x5632f4910e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc2dac7640 1 --2- 172.21.15.94:0/2406654695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 0x5632f4910e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc2dac7640 1 -- 172.21.15.94:0/2406654695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f4911a00 msgr2=0x5632f4911370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc2dac7640 1 --2- 172.21.15.94:0/2406654695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f4911a00 0x5632f4911370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.476+0000 7efc2dac7640 1 -- 172.21.15.94:0/2406654695 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632f4924350 con 0x5632f49084d0 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc2dac7640 1 --2- 172.21.15.94:0/2406654695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f49084d0 0x5632f491d7d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x5632f4903c10 tx=0x7efc18005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f192635f3d2e6209 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc137fe640 1 -- 172.21.15.94:0/2406654695 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc18004250 con 0x5632f49084d0 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc137fe640 1 -- 172.21.15.94:0/2406654695 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc18023070 con 0x5632f49084d0 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc137fe640 1 -- 172.21.15.94:0/2406654695 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc18003bc0 con 0x5632f49084d0 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc329f6740 1 -- 172.21.15.94:0/2406654695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f49084d0 msgr2=0x5632f491d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc329f6740 1 --2- 172.21.15.94:0/2406654695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f49084d0 0x5632f491d7d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x5632f4903c10 tx=0x7efc18005900 comp rx=0 tx=0).stop 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc329f6740 1 -- 172.21.15.94:0/2406654695 shutdown_connections 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc329f6740 1 --2- 172.21.15.94:0/2406654695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4908e80 0x5632f4910e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc329f6740 1 --2- 172.21.15.94:0/2406654695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f4911a00 0x5632f4911370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc329f6740 1 --2- 172.21.15.94:0/2406654695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f49084d0 0x5632f491d7d0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc329f6740 1 -- 172.21.15.94:0/2406654695 >> 172.21.15.94:0/2406654695 conn(0x5632f4902ba0 msgr2=0x5632f4901c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc329f6740 1 -- 172.21.15.94:0/2406654695 shutdown_connections 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.477+0000 7efc329f6740 1 -- 172.21.15.94:0/2406654695 wait complete. 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc329f6740 1 -- start start 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc329f6740 1 Processor -- start 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc329f6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f491a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc329f6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f490b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc329f6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4911a00 0x5632f490b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc329f6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efc281adaa0 con 0x5632f4908e80 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc329f6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc281aef10 con 0x5632f49084d0 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc329f6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc281ae170 con 0x5632f4911a00 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc329f6740 10 client.0 _ll_register_callbacks cb 0x7efc20014c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc2d2c6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f490b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc2d2c6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f490b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33838/0 (socket says 172.21.15.94:33838) 2024-04-21T08:43:51.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc2d2c6640 1 -- 172.21.15.94:0/843911540 learned_addr learned my addr 172.21.15.94:0/843911540 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc2e2c8640 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4911a00 0x5632f490b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.478+0000 7efc2dac7640 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f491a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc2d2c6640 1 -- 172.21.15.94:0/843911540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4911a00 msgr2=0x5632f490b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc2d2c6640 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4911a00 0x5632f490b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc2d2c6640 1 -- 172.21.15.94:0/843911540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 msgr2=0x5632f491a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc2d2c6640 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f491a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc2d2c6640 1 -- 172.21.15.94:0/843911540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632f490bf10 con 0x5632f4908e80 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc2d2c6640 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f490b180 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7efc2402c710 tx=0x7efc24047f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8f669c0cf62f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc117fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc117fa640 1 -- 172.21.15.94:0/843911540 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc240be410 con 0x5632f4908e80 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc117fa640 1 -- 172.21.15.94:0/843911540 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7efc240be670 con 0x5632f4908e80 2024-04-21T08:43:51.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc329f6740 1 -- 172.21.15.94:0/843911540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc18023070 con 0x5632f4908e80 2024-04-21T08:43:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc117fa640 1 -- 172.21.15.94:0/843911540 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efc240c27c0 con 0x5632f4908e80 2024-04-21T08:43:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc329f6740 10 client.16851 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc329f6740 1 -- 172.21.15.94:0/843911540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5632f4987f40 con 0x5632f4908e80 2024-04-21T08:43:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc117fa640 1 -- 172.21.15.94:0/843911540 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc18023070 con 0x5632f4908e80 2024-04-21T08:43:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc117fa640 1 -- 172.21.15.94:0/843911540 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efc240c6bf0 con 0x5632f4908e80 2024-04-21T08:43:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc329f6740 10 client.16851 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc329f6740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc329f6740 20 client.16851 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc329f6740 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5632f4988660 0x5632f498aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc329f6740 4 client.16851 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.479+0000 7efc329f6740 1 -- 172.21.15.94:0/843911540 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632f487ac10 con 0x5632f4988660 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.480+0000 7efc2dac7640 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5632f4988660 0x5632f498aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.480+0000 7efc117fa640 10 client.16851 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.480+0000 7efc2dac7640 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5632f4988660 0x5632f498aa80 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83998ca61d5dec84 server_cookie=68d678118d715d57 in_seq=0 out_seq=0 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.480+0000 7efc117fa640 1 -- 172.21.15.94:0/843911540 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632f487ac10 con 0x5632f4988660 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.480+0000 7efc117fa640 10 client.16851 handle_command_reply: tid=0 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.480+0000 7efc329f6740 1 client.16851 shutdown 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.480+0000 7efc329f6740 1 -- 172.21.15.94:0/843911540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 msgr2=0x5632f490b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.480+0000 7efc329f6740 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f490b180 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7efc2402c710 tx=0x7efc24047f90 comp rx=0 tx=0).stop 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc329f6740 1 -- 172.21.15.94:0/843911540 shutdown_connections 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc329f6740 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5632f4988660 0x5632f498aa80 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc329f6740 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5632f4911a00 0x5632f490b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc329f6740 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5632f4908e80 0x5632f490b180 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc117fa640 0 client.16851 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:51.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc329f6740 1 --2- 172.21.15.94:0/843911540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5632f49084d0 0x5632f491a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc329f6740 1 -- 172.21.15.94:0/843911540 >> 172.21.15.94:0/843911540 conn(0x5632f4902ba0 msgr2=0x5632f4903f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc329f6740 1 -- 172.21.15.94:0/843911540 shutdown_connections 2024-04-21T08:43:51.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc329f6740 1 -- 172.21.15.94:0/843911540 wait complete. 2024-04-21T08:43:51.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc329f6740 20 client.16851 trim_cache size 0 max 16384 2024-04-21T08:43:51.488 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:43:51.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 -- 172.21.15.94:0/4143722898 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc140a3d20 msgr2=0x7efc140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc140a3d20 0x7efc140a61e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efc20005c40 tx=0x7efc2001c000 comp rx=0 tx=0).stop 2024-04-21T08:43:51.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 -- 172.21.15.94:0/4143722898 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc28101ad0 msgr2=0x7efc28199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc28101ad0 0x7efc28199bd0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7efc24006010 tx=0x7efc24005d90 comp rx=0 tx=0).stop 2024-04-21T08:43:51.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 -- 172.21.15.94:0/4143722898 shutdown_connections 2024-04-21T08:43:51.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7efc140a3d20 0x7efc140a61e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc28105ea0 0x7efc281a4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc28101ad0 0x7efc28199bd0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 --2- 172.21.15.94:0/4143722898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc28101130 0x7efc28199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 -- 172.21.15.94:0/4143722898 >> 172.21.15.94:0/4143722898 conn(0x7efc28077e30 msgr2=0x7efc280fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 -- 172.21.15.94:0/4143722898 shutdown_connections 2024-04-21T08:43:51.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:51.481+0000 7efc2fd2b640 1 -- 172.21.15.94:0/4143722898 wait complete. 2024-04-21T08:43:51.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:51 smithi094 ceph-mon[27988]: pgmap v119: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:43:51.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:51 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1556894696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:51.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:51 smithi173 ceph-mon[31428]: pgmap v119: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:43:51.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:51 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1556894696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:51.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:51 smithi111 ceph-mon[31493]: pgmap v119: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:43:51.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:51 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1556894696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:52.493 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:43:52.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:52.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:43:52.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.614+0000 7fc468d85640 1 -- 172.21.15.94:0/2138893309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc464107270 msgr2=0x7fc46410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.614+0000 7fc468d85640 1 --2- 172.21.15.94:0/2138893309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc464107270 0x7fc46410ee40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009ca0 tx=0x7fc45c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.615+0000 7fc468d85640 1 -- 172.21.15.94:0/2138893309 shutdown_connections 2024-04-21T08:43:52.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.615+0000 7fc468d85640 1 --2- 172.21.15.94:0/2138893309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc464107270 0x7fc46410ee40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.615+0000 7fc468d85640 1 --2- 172.21.15.94:0/2138893309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46406b1f0 0x7fc464106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.615+0000 7fc468d85640 1 --2- 172.21.15.94:0/2138893309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc46406a840 0x7fc46406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.615+0000 7fc468d85640 1 -- 172.21.15.94:0/2138893309 >> 172.21.15.94:0/2138893309 conn(0x7fc464077eb0 msgr2=0x7fc4640782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.615+0000 7fc468d85640 1 -- 172.21.15.94:0/2138893309 shutdown_connections 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.615+0000 7fc468d85640 1 -- 172.21.15.94:0/2138893309 wait complete. 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.615+0000 7fc468d85640 1 Processor -- start 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.615+0000 7fc468d85640 1 -- start start 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc468d85640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46406a840 0x7fc4641986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc468d85640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc464107270 0x7fc464198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc468d85640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc46419cf70 0x7fc46419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc468d85640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc4641121d0 con 0x7fc464107270 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc468d85640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc464112050 con 0x7fc46406a840 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc468d85640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc464112350 con 0x7fc46419cf70 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc45bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc464107270 0x7fc464198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc45bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc464107270 0x7fc464198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33848/0 (socket says 172.21.15.94:33848) 2024-04-21T08:43:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc45bfff640 1 -- 172.21.15.94:0/1333436598 learned_addr learned my addr 172.21.15.94:0/1333436598 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc462d9d640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc46419cf70 0x7fc46419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc46259c640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46406a840 0x7fc4641986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc45bfff640 1 -- 172.21.15.94:0/1333436598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc46419cf70 msgr2=0x7fc46419d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc45bfff640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc46419cf70 0x7fc46419d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc45bfff640 1 -- 172.21.15.94:0/1333436598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46406a840 msgr2=0x7fc4641986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc45bfff640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46406a840 0x7fc4641986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc45bfff640 1 -- 172.21.15.94:0/1333436598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46419da60 con 0x7fc464107270 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc462d9d640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc46419cf70 0x7fc46419d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc46259c640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46406a840 0x7fc4641986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc45bfff640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc464107270 0x7fc464198bf0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c970 tx=0x7fc44c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.616+0000 7fc45b7fe640 1 -- 172.21.15.94:0/1333436598 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c007bb0 con 0x7fc464107270 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.617+0000 7fc468d85640 1 -- 172.21.15.94:0/1333436598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc46419dd50 con 0x7fc464107270 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.617+0000 7fc45b7fe640 1 -- 172.21.15.94:0/1333436598 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc44c007d50 con 0x7fc464107270 2024-04-21T08:43:52.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.617+0000 7fc45b7fe640 1 -- 172.21.15.94:0/1333436598 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c00f450 con 0x7fc464107270 2024-04-21T08:43:52.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.617+0000 7fc468d85640 1 -- 172.21.15.94:0/1333436598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4641a5590 con 0x7fc464107270 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.619+0000 7fc45b7fe640 1 -- 172.21.15.94:0/1333436598 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fc44c004ec0 con 0x7fc464107270 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.619+0000 7fc46b973740 1 Processor -- start 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.619+0000 7fc46b973740 1 -- start start 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc46b973740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc46b973740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 0x556111ce9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc46b973740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 0x556111cec260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc46b973740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556111c44f00 con 0x556111ce08e0 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc46b973740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556111be2590 con 0x556111ce1290 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc46b973740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556111c40ec0 con 0x556111ce9e10 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc462d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc462d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33860/0 (socket says 172.21.15.94:33860) 2024-04-21T08:43:52.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc462d9d640 1 -- 172.21.15.94:0/1726177234 learned_addr learned my addr 172.21.15.94:0/1726177234 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc45b7fe640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4500a3d20 0x7fc4500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc46259c640 1 --2- 172.21.15.94:0/1726177234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 0x556111ce9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc45bfff640 1 --2- 172.21.15.94:0/1726177234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 0x556111cec260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc462d9d640 1 -- 172.21.15.94:0/1726177234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 msgr2=0x556111cec260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc462d9d640 1 --2- 172.21.15.94:0/1726177234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 0x556111cec260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc462d9d640 1 -- 172.21.15.94:0/1726177234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 msgr2=0x556111ce9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc462d9d640 1 --2- 172.21.15.94:0/1726177234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 0x556111ce9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc462d9d640 1 -- 172.21.15.94:0/1726177234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556111a1f7e0 con 0x556111ce08e0 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc45b7fe640 1 -- 172.21.15.94:0/1333436598 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc44c0d0930 con 0x7fc464107270 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc462d9d640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4500a3d20 0x7fc4500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.620+0000 7fc462d9d640 1 --2- 172.21.15.94:0/1726177234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce0cc0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009430 tx=0x7fc45c0098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca73bc8dbef4498e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc458ff9640 1 -- 172.21.15.94:0/1726177234 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c039050 con 0x556111ce08e0 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc458ff9640 1 -- 172.21.15.94:0/1726177234 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc45c046030 con 0x556111ce08e0 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc458ff9640 1 -- 172.21.15.94:0/1726177234 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c034440 con 0x556111ce08e0 2024-04-21T08:43:52.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc46b973740 1 -- 172.21.15.94:0/1726177234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 msgr2=0x556111ce0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc462d9d640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4500a3d20 0x7fc4500a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc45c04a000 tx=0x7fc45c04b000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc46b973740 1 --2- 172.21.15.94:0/1726177234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce0cc0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009430 tx=0x7fc45c0098a0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc46b973740 1 -- 172.21.15.94:0/1726177234 shutdown_connections 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc46b973740 1 --2- 172.21.15.94:0/1726177234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 0x556111cec260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc46b973740 1 --2- 172.21.15.94:0/1726177234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 0x556111ce9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc46b973740 1 --2- 172.21.15.94:0/1726177234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce0cc0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc46b973740 1 -- 172.21.15.94:0/1726177234 >> 172.21.15.94:0/1726177234 conn(0x556111cdafb0 msgr2=0x556111cdb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc46b973740 1 -- 172.21.15.94:0/1726177234 shutdown_connections 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.621+0000 7fc46b973740 1 -- 172.21.15.94:0/1726177234 wait complete. 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc46b973740 1 -- start start 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc46b973740 1 Processor -- start 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc46b973740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce08e0 0x556111ce6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc46b973740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111ce7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc46b973740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce9e10 0x556111ce78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc46b973740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556111cecd80 con 0x556111ce1290 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc46b973740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556111bfe580 con 0x556111ce9e10 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc46b973740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556111cecf00 con 0x556111ce08e0 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc46b973740 10 client.0 _ll_register_callbacks cb 0x556111c04650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc45bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111ce7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc45bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111ce7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33862/0 (socket says 172.21.15.94:33862) 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc45bfff640 1 -- 172.21.15.94:0/3248259980 learned_addr learned my addr 172.21.15.94:0/3248259980 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc462d9d640 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce9e10 0x556111ce78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc46259c640 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce08e0 0x556111ce6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc45bfff640 1 -- 172.21.15.94:0/3248259980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce08e0 msgr2=0x556111ce6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.622+0000 7fc45bfff640 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce08e0 0x556111ce6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc45bfff640 1 -- 172.21.15.94:0/3248259980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce9e10 msgr2=0x556111ce78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc45bfff640 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce9e10 0x556111ce78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc45bfff640 1 -- 172.21.15.94:0/3248259980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556111a7ff70 con 0x556111ce1290 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc45bfff640 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111ce7360 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc44c098950 tx=0x7fc44c09be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc1115025ae029a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc449ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc449ffb640 1 -- 172.21.15.94:0/3248259980 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c01d070 con 0x556111ce1290 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc449ffb640 1 -- 172.21.15.94:0/3248259980 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc44c09bb40 con 0x556111ce1290 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc46b973740 1 -- 172.21.15.94:0/3248259980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc45c046030 con 0x556111ce1290 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc449ffb640 1 -- 172.21.15.94:0/3248259980 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc44c09cbf0 con 0x556111ce1290 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc46b973740 10 client.16863 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc46b973740 1 -- 172.21.15.94:0/3248259980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556111917170 con 0x556111ce1290 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc449ffb640 1 -- 172.21.15.94:0/3248259980 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc45c046030 con 0x556111ce1290 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc449ffb640 1 -- 172.21.15.94:0/3248259980 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc44c09b3b0 con 0x556111ce1290 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc46b973740 10 client.16863 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc46b973740 10 client.16863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc46b973740 20 client.16863 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc46b973740 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556111cfa3f0 0x556111cfc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc46b973740 4 client.16863 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.623+0000 7fc46b973740 1 -- 172.21.15.94:0/3248259980 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556111c52920 con 0x556111cfa3f0 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.624+0000 7fc46259c640 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556111cfa3f0 0x556111cfc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.624+0000 7fc46259c640 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556111cfa3f0 0x556111cfc7e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c465adfd4d701c13 server_cookie=906dee5580bf04e8 in_seq=0 out_seq=0 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.624+0000 7fc449ffb640 10 client.16863 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc449ffb640 1 -- 172.21.15.94:0/3248259980 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556111c52920 con 0x556111cfa3f0 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc449ffb640 10 client.16863 handle_command_reply: tid=0 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 client.16863 shutdown 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 -- 172.21.15.94:0/3248259980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 msgr2=0x556111ce7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111ce7360 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc44c098950 tx=0x7fc44c09be50 comp rx=0 tx=0).stop 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 -- 172.21.15.94:0/3248259980 shutdown_connections 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556111cfa3f0 0x556111cfc7e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce9e10 0x556111ce78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111ce7360 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 --2- 172.21.15.94:0/3248259980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce08e0 0x556111ce6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc449ffb640 0 client.16863 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:52.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 -- 172.21.15.94:0/3248259980 >> 172.21.15.94:0/3248259980 conn(0x556111cdafb0 msgr2=0x556111cdcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 -- 172.21.15.94:0/3248259980 shutdown_connections 2024-04-21T08:43:52.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 1 -- 172.21.15.94:0/3248259980 wait complete. 2024-04-21T08:43:52.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.625+0000 7fc46b973740 20 client.16863 trim_cache size 0 max 16384 2024-04-21T08:43:52.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46b973740 1 Processor -- start 2024-04-21T08:43:52.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46b973740 1 -- start start 2024-04-21T08:43:52.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46b973740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46b973740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 0x556111cf5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46b973740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 0x556111d02090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46b973740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc4641b1c20 con 0x556111ce08e0 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46b973740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc4641af4f0 con 0x556111ce1290 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46b973740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc4641ad340 con 0x556111ce9e10 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33866/0 (socket says 172.21.15.94:33866) 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46259c640 1 -- 172.21.15.94:0/1013386662 learned_addr learned my addr 172.21.15.94:0/1013386662 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46259c640 1 -- 172.21.15.94:0/1013386662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 msgr2=0x556111d02090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46259c640 1 --2- 172.21.15.94:0/1013386662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 0x556111d02090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc45bfff640 1 --2- 172.21.15.94:0/1013386662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 0x556111cf5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46259c640 1 -- 172.21.15.94:0/1013386662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 msgr2=0x556111cf5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46259c640 1 --2- 172.21.15.94:0/1013386662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 0x556111cf5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46259c640 1 -- 172.21.15.94:0/1013386662 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556111ce5fb0 con 0x556111ce08e0 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc45bfff640 1 --2- 172.21.15.94:0/1013386662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 0x556111cf5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:52.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.643+0000 7fc46259c640 1 --2- 172.21.15.94:0/1013386662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce9550 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x556111cfd030 tx=0x7fc454005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcaa680d3c84d1b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc44bfff640 1 -- 172.21.15.94:0/1013386662 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45401f020 con 0x556111ce08e0 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc44bfff640 1 -- 172.21.15.94:0/1013386662 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4540044d0 con 0x556111ce08e0 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc44bfff640 1 -- 172.21.15.94:0/1013386662 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454012650 con 0x556111ce08e0 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc46b973740 1 -- 172.21.15.94:0/1013386662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 msgr2=0x556111ce9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc46b973740 1 --2- 172.21.15.94:0/1013386662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce9550 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x556111cfd030 tx=0x7fc454005f90 comp rx=0 tx=0).stop 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc46b973740 1 -- 172.21.15.94:0/1013386662 shutdown_connections 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc46b973740 1 --2- 172.21.15.94:0/1013386662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 0x556111d02090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc46b973740 1 --2- 172.21.15.94:0/1013386662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce1290 0x556111cf5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc46b973740 1 --2- 172.21.15.94:0/1013386662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce08e0 0x556111ce9550 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc46b973740 1 -- 172.21.15.94:0/1013386662 >> 172.21.15.94:0/1013386662 conn(0x556111cdafb0 msgr2=0x556111cdcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc46b973740 1 -- 172.21.15.94:0/1013386662 shutdown_connections 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.644+0000 7fc46b973740 1 -- 172.21.15.94:0/1013386662 wait complete. 2024-04-21T08:43:52.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc46b973740 1 -- start start 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc46b973740 1 Processor -- start 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc46b973740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce08e0 0x556111cf2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc46b973740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111cef150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc46b973740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 0x556111cef890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc46b973740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc4641ad9b0 con 0x556111ce1290 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc46b973740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc4641aee20 con 0x556111ce08e0 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc46b973740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc4641ae080 con 0x556111ce9e10 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc46b973740 10 client.0 _ll_register_callbacks cb 0x7fc44c09c2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc45bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111cef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc45bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111cef150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33868/0 (socket says 172.21.15.94:33868) 2024-04-21T08:43:52.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc45bfff640 1 -- 172.21.15.94:0/3586694398 learned_addr learned my addr 172.21.15.94:0/3586694398 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc45bfff640 1 -- 172.21.15.94:0/3586694398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 msgr2=0x556111cef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc45bfff640 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 0x556111cef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc45bfff640 1 -- 172.21.15.94:0/3586694398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce08e0 msgr2=0x556111cf2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc45bfff640 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce08e0 0x556111cf2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc45bfff640 1 -- 172.21.15.94:0/3586694398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556111d60910 con 0x556111ce1290 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc45bfff640 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111cef150 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc44c09b7d0 tx=0x7fc44c007730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7392ec329757dce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc449ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.645+0000 7fc449ffb640 1 -- 172.21.15.94:0/3586694398 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c01d070 con 0x556111ce1290 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc449ffb640 1 -- 172.21.15.94:0/3586694398 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc44c098240 con 0x556111ce1290 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc46b973740 1 -- 172.21.15.94:0/3586694398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4540044d0 con 0x556111ce1290 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc449ffb640 1 -- 172.21.15.94:0/3586694398 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc44c0a5160 con 0x556111ce1290 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc46b973740 10 client.16869 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:52.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc46b973740 1 -- 172.21.15.94:0/3586694398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556111d61970 con 0x556111ce1290 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc449ffb640 1 -- 172.21.15.94:0/3586694398 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4540044d0 con 0x556111ce1290 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc449ffb640 1 -- 172.21.15.94:0/3586694398 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc44c0bb2b0 con 0x556111ce1290 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc46b973740 10 client.16869 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc46b973740 10 client.16869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc46b973740 20 client.16869 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc46b973740 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556111d61f20 0x556111d64380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc46b973740 4 client.16869 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc46b973740 1 -- 172.21.15.94:0/3586694398 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556111c4c790 con 0x556111d61f20 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.646+0000 7fc46259c640 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556111d61f20 0x556111d64380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.647+0000 7fc449ffb640 10 client.16869 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:52.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.647+0000 7fc46259c640 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556111d61f20 0x556111d64380 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ede5afb54d5b293 server_cookie=3813886634729df3 in_seq=0 out_seq=0 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc449ffb640 1 -- 172.21.15.94:0/3586694398 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556111c4c790 con 0x556111d61f20 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc449ffb640 10 client.16869 handle_command_reply: tid=0 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 client.16869 shutdown 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 -- 172.21.15.94:0/3586694398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 msgr2=0x556111cef150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111cef150 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc44c09b7d0 tx=0x7fc44c007730 comp rx=0 tx=0).stop 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 -- 172.21.15.94:0/3586694398 shutdown_connections 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556111d61f20 0x556111d64380 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556111ce9e10 0x556111cef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556111ce1290 0x556111cef150 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 --2- 172.21.15.94:0/3586694398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556111ce08e0 0x556111cf2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc449ffb640 0 client.16869 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:52.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 -- 172.21.15.94:0/3586694398 >> 172.21.15.94:0/3586694398 conn(0x556111cdafb0 msgr2=0x556111ce6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 -- 172.21.15.94:0/3586694398 shutdown_connections 2024-04-21T08:43:52.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 1 -- 172.21.15.94:0/3586694398 wait complete. 2024-04-21T08:43:52.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc46b973740 20 client.16869 trim_cache size 0 max 16384 2024-04-21T08:43:52.655 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:43:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:43:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "2ffeaf72-afdd-42c8-ac81-54c9effd7ac0", 2024-04-21T08:43:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:43:52.655 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:43:52.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc468d85640 1 -- 172.21.15.94:0/1333436598 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4500a3d20 msgr2=0x7fc4500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.648+0000 7fc468d85640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4500a3d20 0x7fc4500a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc45c04a000 tx=0x7fc45c04b000 comp rx=0 tx=0).stop 2024-04-21T08:43:52.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.649+0000 7fc468d85640 1 -- 172.21.15.94:0/1333436598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc464107270 msgr2=0x7fc464198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.649+0000 7fc468d85640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc464107270 0x7fc464198bf0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c970 tx=0x7fc44c00ce30 comp rx=0 tx=0).stop 2024-04-21T08:43:52.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.649+0000 7fc468d85640 1 -- 172.21.15.94:0/1333436598 shutdown_connections 2024-04-21T08:43:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.649+0000 7fc468d85640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4500a3d20 0x7fc4500a61e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.649+0000 7fc468d85640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc46419cf70 0x7fc46419d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.649+0000 7fc468d85640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc464107270 0x7fc464198bf0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.649+0000 7fc468d85640 1 --2- 172.21.15.94:0/1333436598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46406a840 0x7fc4641986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.649+0000 7fc468d85640 1 -- 172.21.15.94:0/1333436598 >> 172.21.15.94:0/1333436598 conn(0x7fc464077eb0 msgr2=0x7fc4640fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.649+0000 7fc468d85640 1 -- 172.21.15.94:0/1333436598 shutdown_connections 2024-04-21T08:43:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.649+0000 7fc468d85640 1 -- 172.21.15.94:0/1333436598 wait complete. 2024-04-21T08:43:52.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:43:52.659 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.761+0000 7ffbeb752640 1 -- 172.21.15.94:0/2044358030 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe4107270 msgr2=0x7ffbe410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.761+0000 7ffbeb752640 1 --2- 172.21.15.94:0/2044358030 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe4107270 0x7ffbe410ee40 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009ca0 tx=0x7ffbdc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.761+0000 7ffbeb752640 1 -- 172.21.15.94:0/2044358030 shutdown_connections 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.761+0000 7ffbeb752640 1 --2- 172.21.15.94:0/2044358030 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe4107270 0x7ffbe410ee40 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.761+0000 7ffbeb752640 1 --2- 172.21.15.94:0/2044358030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbe4102b50 0x7ffbe4106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.761+0000 7ffbeb752640 1 --2- 172.21.15.94:0/2044358030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbe41021a0 0x7ffbe4102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.761+0000 7ffbeb752640 1 -- 172.21.15.94:0/2044358030 >> 172.21.15.94:0/2044358030 conn(0x7ffbe40fda40 msgr2=0x7ffbe40ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.761+0000 7ffbeb752640 1 -- 172.21.15.94:0/2044358030 shutdown_connections 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.761+0000 7ffbeb752640 1 -- 172.21.15.94:0/2044358030 wait complete. 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbeb752640 1 Processor -- start 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbeb752640 1 -- start start 2024-04-21T08:43:52.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbeb752640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbe41021a0 0x7ffbe419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbeb752640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbe4107270 0x7ffbe419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbeb752640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe4198620 0x7ffbe4198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbeb752640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41121e0 con 0x7ffbe4198620 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbeb752640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4112060 con 0x7ffbe4107270 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbeb752640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4112360 con 0x7ffbe41021a0 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbe9cef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe4198620 0x7ffbe4198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbe9cef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe4198620 0x7ffbe4198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33896/0 (socket says 172.21.15.94:33896) 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbe9cef640 1 -- 172.21.15.94:0/1397831788 learned_addr learned my addr 172.21.15.94:0/1397831788 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbe41021a0 0x7ffbe419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.762+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbe4107270 0x7ffbe419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbe9cef640 1 -- 172.21.15.94:0/1397831788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbe41021a0 msgr2=0x7ffbe419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbe9cef640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbe41021a0 0x7ffbe419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbe9cef640 1 -- 172.21.15.94:0/1397831788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbe4107270 msgr2=0x7ffbe419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbe9cef640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbe4107270 0x7ffbe419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbe9cef640 1 -- 172.21.15.94:0/1397831788 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe4199140 con 0x7ffbe4198620 2024-04-21T08:43:52.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbe4107270 0x7ffbe419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:52.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbe41021a0 0x7ffbe419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:52.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbe9cef640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe4198620 0x7ffbe4198ad0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc005e30 tx=0x7ffbdc034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbda7fc640 1 -- 172.21.15.94:0/1397831788 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc039040 con 0x7ffbe4198620 2024-04-21T08:43:52.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbda7fc640 1 -- 172.21.15.94:0/1397831788 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbdc044070 con 0x7ffbe4198620 2024-04-21T08:43:52.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbeb752640 1 -- 172.21.15.94:0/1397831788 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe41993d0 con 0x7ffbe4198620 2024-04-21T08:43:52.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbda7fc640 1 -- 172.21.15.94:0/1397831788 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc003e60 con 0x7ffbe4198620 2024-04-21T08:43:52.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.763+0000 7ffbeb752640 1 -- 172.21.15.94:0/1397831788 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe41a5570 con 0x7ffbe4198620 2024-04-21T08:43:52.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.765+0000 7ffbda7fc640 1 -- 172.21.15.94:0/1397831788 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ffbdc0324c0 con 0x7ffbe4198620 2024-04-21T08:43:52.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.765+0000 7ffbee429740 1 Processor -- start 2024-04-21T08:43:52.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.765+0000 7ffbee429740 1 -- start start 2024-04-21T08:43:52.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbee429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 0x559090307960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbee429740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090310540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbee429740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x559090312e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbee429740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55909026c1e0 con 0x559090307ea0 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbee429740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559090209390 con 0x559090310a80 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbee429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5590902681a0 con 0x559090307580 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbda7fc640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbd00a3d20 0x7ffbd00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe94ee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090310540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe94ee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090310540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33904/0 (socket says 172.21.15.94:33904) 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe94ee640 1 -- 172.21.15.94:0/4093146679 learned_addr learned my addr 172.21.15.94:0/4093146679 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe9cef640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbd00a3d20 0x7ffbd00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/4093146679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 0x559090307960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/4093146679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x559090312e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe94ee640 1 -- 172.21.15.94:0/4093146679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 msgr2=0x559090307960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/4093146679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 0x559090307960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbda7fc640 1 -- 172.21.15.94:0/1397831788 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ffbdc0f2e60 con 0x7ffbe4198620 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe94ee640 1 -- 172.21.15.94:0/4093146679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 msgr2=0x559090312e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/4093146679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x559090312e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe94ee640 1 -- 172.21.15.94:0/4093146679 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590900487d0 con 0x559090307ea0 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/4093146679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 0x559090307960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/4093146679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090310540 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ffbe000bd40 tx=0x7ffbe000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7f59a213b09b51 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbe9cef640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbd00a3d20 0x7ffbd00a61e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc048000 tx=0x7ffbdc091620 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbcf7fe640 1 -- 172.21.15.94:0/4093146679 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0015070 con 0x559090307ea0 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbcf7fe640 1 -- 172.21.15.94:0/4093146679 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbe000be30 con 0x559090307ea0 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.766+0000 7ffbcf7fe640 1 -- 172.21.15.94:0/4093146679 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0011940 con 0x559090307ea0 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.767+0000 7ffbee429740 1 -- 172.21.15.94:0/4093146679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 msgr2=0x559090310540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.767+0000 7ffbee429740 1 --2- 172.21.15.94:0/4093146679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090310540 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ffbe000bd40 tx=0x7ffbe000f730 comp rx=0 tx=0).stop 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.767+0000 7ffbee429740 1 -- 172.21.15.94:0/4093146679 shutdown_connections 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.767+0000 7ffbee429740 1 --2- 172.21.15.94:0/4093146679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x559090312e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.767+0000 7ffbee429740 1 --2- 172.21.15.94:0/4093146679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090310540 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.767+0000 7ffbee429740 1 --2- 172.21.15.94:0/4093146679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 0x559090307960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.767+0000 7ffbee429740 1 -- 172.21.15.94:0/4093146679 >> 172.21.15.94:0/4093146679 conn(0x559090301cb0 msgr2=0x559090302090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.767+0000 7ffbee429740 1 -- 172.21.15.94:0/4093146679 shutdown_connections 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.767+0000 7ffbee429740 1 -- 172.21.15.94:0/4093146679 wait complete. 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.767+0000 7ffbee429740 1 -- start start 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbee429740 1 Processor -- start 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbee429740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307580 0x55909030db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbee429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307ea0 0x55909030e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbee429740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x55909030e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbee429740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x559090313870 con 0x559090307580 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbee429740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5590902252f0 con 0x559090310a80 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbee429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5590903139f0 con 0x559090307ea0 2024-04-21T08:43:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbee429740 10 client.0 _ll_register_callbacks cb 0x559090232580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbe8ced640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307580 0x55909030db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbe8ced640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307580 0x55909030db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33918/0 (socket says 172.21.15.94:33918) 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbe8ced640 1 -- 172.21.15.94:0/1955303799 learned_addr learned my addr 172.21.15.94:0/1955303799 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x55909030e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbe8ced640 1 -- 172.21.15.94:0/1955303799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307ea0 msgr2=0x55909030e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307ea0 0x55909030e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbe8ced640 1 -- 172.21.15.94:0/1955303799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 msgr2=0x55909030e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x55909030e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbe8ced640 1 -- 172.21.15.94:0/1955303799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55908ff7e3f0 con 0x559090307580 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307580 0x55909030db20 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7ffbe406b630 tx=0x7ffbd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a20efd228622cd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.768+0000 7ffbccff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbccff9640 1 -- 172.21.15.94:0/1955303799 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4014110 con 0x559090307580 2024-04-21T08:43:52.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbccff9640 1 -- 172.21.15.94:0/1955303799 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ffbd4004020 con 0x559090307580 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbee429740 1 -- 172.21.15.94:0/1955303799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbe000be30 con 0x559090307580 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbccff9640 1 -- 172.21.15.94:0/1955303799 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffbd4026050 con 0x559090307580 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbee429740 10 client.16881 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbee429740 1 -- 172.21.15.94:0/1955303799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55908ff3e0f0 con 0x559090307580 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbccff9640 1 -- 172.21.15.94:0/1955303799 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbe000be30 con 0x559090307580 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbccff9640 1 -- 172.21.15.94:0/1955303799 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ffbd40047a0 con 0x559090307580 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbee429740 10 client.16881 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbee429740 10 client.16881 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbee429740 20 client.16881 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbee429740 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559090320c20 0x559090323080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbee429740 4 client.16881 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbee429740 1 -- 172.21.15.94:0/1955303799 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559090279c00 con 0x559090320c20 2024-04-21T08:43:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.769+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559090320c20 0x559090323080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.770+0000 7ffbccff9640 10 client.16881 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:52.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.770+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559090320c20 0x559090323080 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48c3a6daa6f15be5 server_cookie=a81ce95fe3ca4f62 in_seq=0 out_seq=0 2024-04-21T08:43:52.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.770+0000 7ffbccff9640 1 -- 172.21.15.94:0/1955303799 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559090279c00 con 0x559090320c20 2024-04-21T08:43:52.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.770+0000 7ffbccff9640 10 client.16881 handle_command_reply: tid=0 2024-04-21T08:43:52.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.770+0000 7ffbee429740 1 client.16881 shutdown 2024-04-21T08:43:52.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 1 -- 172.21.15.94:0/1955303799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307580 msgr2=0x55909030db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307580 0x55909030db20 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7ffbe406b630 tx=0x7ffbd4004730 comp rx=0 tx=0).stop 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 1 -- 172.21.15.94:0/1955303799 shutdown_connections 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559090320c20 0x559090323080 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307ea0 0x55909030e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x55909030e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbccff9640 0 client.16881 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 1 --2- 172.21.15.94:0/1955303799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307580 0x55909030db20 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 1 -- 172.21.15.94:0/1955303799 >> 172.21.15.94:0/1955303799 conn(0x559090301cb0 msgr2=0x559090303a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 1 -- 172.21.15.94:0/1955303799 shutdown_connections 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 1 -- 172.21.15.94:0/1955303799 wait complete. 2024-04-21T08:43:52.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.771+0000 7ffbee429740 20 client.16881 trim_cache size 0 max 16384 2024-04-21T08:43:52.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbee429740 1 Processor -- start 2024-04-21T08:43:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbee429740 1 -- start start 2024-04-21T08:43:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbee429740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090307580 0x559090310250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbee429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307ea0 0x55909031c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbee429740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090310a80 0x55909030e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbee429740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41b1cc0 con 0x559090310a80 2024-04-21T08:43:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbee429740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41af590 con 0x559090307580 2024-04-21T08:43:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbee429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41ad3e0 con 0x559090307ea0 2024-04-21T08:43:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbe8ced640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090310a80 0x55909030e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbe8ced640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090310a80 0x55909030e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33926/0 (socket says 172.21.15.94:33926) 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbe8ced640 1 -- 172.21.15.94:0/2798688413 learned_addr learned my addr 172.21.15.94:0/2798688413 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/2798688413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090307580 0x559090310250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/2798688413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307ea0 0x55909031c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbe8ced640 1 -- 172.21.15.94:0/2798688413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307ea0 msgr2=0x55909031c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.788+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/2798688413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307ea0 0x55909031c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbe8ced640 1 -- 172.21.15.94:0/2798688413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090307580 msgr2=0x559090310250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/2798688413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090307580 0x559090310250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbe8ced640 1 -- 172.21.15.94:0/2798688413 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590903131e0 con 0x559090310a80 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/2798688413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090310a80 0x55909030e1f0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4000c00 tx=0x7ffbd4005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d17dc175e866742 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbceffd640 1 -- 172.21.15.94:0/2798688413 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd400bcf0 con 0x559090310a80 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbceffd640 1 -- 172.21.15.94:0/2798688413 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbd400be90 con 0x559090310a80 2024-04-21T08:43:52.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbceffd640 1 -- 172.21.15.94:0/2798688413 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4005060 con 0x559090310a80 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbee429740 1 -- 172.21.15.94:0/2798688413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090310a80 msgr2=0x55909030e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbee429740 1 --2- 172.21.15.94:0/2798688413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090310a80 0x55909030e1f0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4000c00 tx=0x7ffbd4005d30 comp rx=0 tx=0).stop 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbee429740 1 -- 172.21.15.94:0/2798688413 shutdown_connections 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbee429740 1 --2- 172.21.15.94:0/2798688413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307ea0 0x55909031c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbee429740 1 --2- 172.21.15.94:0/2798688413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090310a80 0x55909030e1f0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbee429740 1 --2- 172.21.15.94:0/2798688413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090307580 0x559090310250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.789+0000 7ffbee429740 1 -- 172.21.15.94:0/2798688413 >> 172.21.15.94:0/2798688413 conn(0x559090301cb0 msgr2=0x55909030dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 1 -- 172.21.15.94:0/2798688413 shutdown_connections 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 1 -- 172.21.15.94:0/2798688413 wait complete. 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 1 -- start start 2024-04-21T08:43:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 1 Processor -- start 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 0x5590903190c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090314430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x5590903149d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41ada50 con 0x559090307ea0 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41aeec0 con 0x559090310a80 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41ae120 con 0x559090307580 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbee429740 10 client.0 _ll_register_callbacks cb 0x7ffbd400b6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbe94ee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090314430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbe94ee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090314430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33930/0 (socket says 172.21.15.94:33930) 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbe94ee640 1 -- 172.21.15.94:0/667907125 learned_addr learned my addr 172.21.15.94:0/667907125 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.790+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 0x5590903190c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x5590903149d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbe94ee640 1 -- 172.21.15.94:0/667907125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 msgr2=0x5590903190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 0x5590903190c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbe94ee640 1 -- 172.21.15.94:0/667907125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 msgr2=0x5590903149d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x5590903149d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbe94ee640 1 -- 172.21.15.94:0/667907125 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55909031f720 con 0x559090307ea0 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 0x5590903190c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbe94ee640 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090314430 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0006010 tx=0x7ffbe00114e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa17715529ed6df server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbccff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbccff9640 1 -- 172.21.15.94:0/667907125 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0015040 con 0x559090307ea0 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbccff9640 1 -- 172.21.15.94:0/667907125 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ffbe000be30 con 0x559090307ea0 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbee429740 1 -- 172.21.15.94:0/667907125 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbd400be90 con 0x559090307ea0 2024-04-21T08:43:52.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbccff9640 1 -- 172.21.15.94:0/667907125 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffbe0033050 con 0x559090307ea0 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbee429740 10 client.16887 fetch_fsmap learned FSMap version 24 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbee429740 1 -- 172.21.15.94:0/667907125 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5590903870f0 con 0x559090307ea0 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbccff9640 1 -- 172.21.15.94:0/667907125 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbd400be90 con 0x559090307ea0 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbccff9640 1 -- 172.21.15.94:0/667907125 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ffbe0023070 con 0x559090307ea0 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbee429740 10 client.16887 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbee429740 10 client.16887 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbee429740 20 client.16887 populate_metadata read hostname 'smithi094' 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbee429740 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5590903875c0 0x5590903899e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbee429740 4 client.16887 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.791+0000 7ffbee429740 1 -- 172.21.15.94:0/667907125 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559090273a70 con 0x5590903875c0 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.792+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5590903875c0 0x5590903899e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.792+0000 7ffbccff9640 10 client.16887 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.792+0000 7ffbe8ced640 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5590903875c0 0x5590903899e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7b02f900147dd5b server_cookie=df39a8d408130064 in_seq=0 out_seq=0 2024-04-21T08:43:52.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.792+0000 7ffbccff9640 1 -- 172.21.15.94:0/667907125 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559090273a70 con 0x5590903875c0 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.792+0000 7ffbccff9640 10 client.16887 handle_command_reply: tid=0 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.792+0000 7ffbee429740 1 client.16887 shutdown 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.792+0000 7ffbee429740 1 -- 172.21.15.94:0/667907125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 msgr2=0x559090314430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.792+0000 7ffbee429740 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090314430 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0006010 tx=0x7ffbe00114e0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbee429740 1 -- 172.21.15.94:0/667907125 shutdown_connections 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbee429740 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5590903875c0 0x5590903899e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbee429740 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559090310a80 0x5590903149d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbee429740 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559090307ea0 0x559090314430 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbccff9640 0 client.16887 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbee429740 1 --2- 172.21.15.94:0/667907125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559090307580 0x5590903190c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbee429740 1 -- 172.21.15.94:0/667907125 >> 172.21.15.94:0/667907125 conn(0x559090301cb0 msgr2=0x559090303890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbee429740 1 -- 172.21.15.94:0/667907125 shutdown_connections 2024-04-21T08:43:52.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbee429740 1 -- 172.21.15.94:0/667907125 wait complete. 2024-04-21T08:43:52.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbee429740 20 client.16887 trim_cache size 0 max 16384 2024-04-21T08:43:52.801 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:43:52.801 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:43:52.801 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:43:52.801 INFO:teuthology.orchestra.run.smithi094.stdout: "2ffeaf72-afdd-42c8-ac81-54c9effd7ac0": { 2024-04-21T08:43:52.801 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:43:52.801 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "2ffeaf72-afdd-42c8-ac81-54c9effd7ac0", 2024-04-21T08:43:52.802 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:43:52.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:43:52.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:43:52.802 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:43:52.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 -- 172.21.15.94:0/1397831788 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbd00a3d20 msgr2=0x7ffbd00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbd00a3d20 0x7ffbd00a61e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc048000 tx=0x7ffbdc091620 comp rx=0 tx=0).stop 2024-04-21T08:43:52.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 -- 172.21.15.94:0/1397831788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe4198620 msgr2=0x7ffbe4198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe4198620 0x7ffbe4198ad0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc005e30 tx=0x7ffbdc034760 comp rx=0 tx=0).stop 2024-04-21T08:43:52.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 -- 172.21.15.94:0/1397831788 shutdown_connections 2024-04-21T08:43:52.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbd00a3d20 0x7ffbd00a61e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe4198620 0x7ffbe4198ad0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbe4107270 0x7ffbe419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 --2- 172.21.15.94:0/1397831788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbe41021a0 0x7ffbe419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 -- 172.21.15.94:0/1397831788 >> 172.21.15.94:0/1397831788 conn(0x7ffbe40fda40 msgr2=0x7ffbe40fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 -- 172.21.15.94:0/1397831788 shutdown_connections 2024-04-21T08:43:52.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:43:52.793+0000 7ffbeb752640 1 -- 172.21.15.94:0/1397831788 wait complete. 2024-04-21T08:43:52.805 INFO:tasks.cephfs.filesystem:scrub status for tag:2ffeaf72-afdd-42c8-ac81-54c9effd7ac0 - {'path': '/', 'tag': '2ffeaf72-afdd-42c8-ac81-54c9effd7ac0', 'options': 'recursive,force'} 2024-04-21T08:43:53.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:53 smithi094 ceph-mon[27988]: pgmap v120: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:53.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:53 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:43:53.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:53 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:53.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:53 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:43:53.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:53 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:53.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:53 smithi173 ceph-mon[31428]: pgmap v120: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:53.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:53 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:43:53.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:53 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:53.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:53 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:43:53.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:53 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:53 smithi111 ceph-mon[31493]: pgmap v120: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:43:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:53 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:43:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:53 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:53.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:53 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:43:53.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:53 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:43:55.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:55 smithi094 ceph-mon[27988]: pgmap v121: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:43:55.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:55 smithi173 ceph-mon[31428]: pgmap v121: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:43:55.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:55 smithi111 ceph-mon[31493]: pgmap v121: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:43:56.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:56 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:43:56.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:56 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:43:56.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:56 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:43:57.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:57 smithi094 ceph-mon[27988]: pgmap v122: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:43:57.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:57 smithi173 ceph-mon[31428]: pgmap v122: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:43:57.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:57 smithi111 ceph-mon[31493]: pgmap v122: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:43:59.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:43:59 smithi094 ceph-mon[27988]: pgmap v123: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:43:59.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:43:59 smithi173 ceph-mon[31428]: pgmap v123: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:43:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:43:59 smithi111 ceph-mon[31493]: pgmap v123: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:44:01.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:01 smithi094 ceph-mon[27988]: pgmap v124: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:44:01.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:01 smithi173 ceph-mon[31428]: pgmap v124: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:44:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:01 smithi111 ceph-mon[31493]: pgmap v124: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:44:03.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:03 smithi094 ceph-mon[27988]: pgmap v125: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:44:03.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:03 smithi173 ceph-mon[31428]: pgmap v125: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:44:03.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:03 smithi111 ceph-mon[31493]: pgmap v125: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:44:05.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:05 smithi094 ceph-mon[27988]: pgmap v126: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-21T08:44:05.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:05 smithi173 ceph-mon[31428]: pgmap v126: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-21T08:44:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:05 smithi111 ceph-mon[31493]: pgmap v126: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-21T08:44:06.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:06 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:06.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:06 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:06.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:06 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:06.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:06 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:44:07.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:06 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:07.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:06 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:07.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:06 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:07.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:06 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:44:07.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:06 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:07.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:06 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:07.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:06 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:07.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:06 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:44:07.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:07 smithi111 ceph-mon[31493]: pgmap v127: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:44:08.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:07 smithi094 ceph-mon[27988]: pgmap v127: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:44:08.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:07 smithi173 ceph-mon[31428]: pgmap v127: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:44:09.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:09 smithi111 ceph-mon[31493]: pgmap v128: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:44:10.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:09 smithi094 ceph-mon[27988]: pgmap v128: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:44:10.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:09 smithi173 ceph-mon[31428]: pgmap v128: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:44:11.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:11 smithi111 ceph-mon[31493]: pgmap v129: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:44:12.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:11 smithi094 ceph-mon[27988]: pgmap v129: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:44:12.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:11 smithi173 ceph-mon[31428]: pgmap v129: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:44:13.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:13 smithi111 ceph-mon[31493]: pgmap v130: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:44:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:13 smithi094 ceph-mon[27988]: pgmap v130: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:44:14.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:13 smithi173 ceph-mon[31428]: pgmap v130: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:44:15.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:15 smithi111 ceph-mon[31493]: pgmap v131: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:44:16.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:15 smithi094 ceph-mon[27988]: pgmap v131: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:44:16.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:15 smithi173 ceph-mon[31428]: pgmap v131: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:44:17.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:17 smithi173 ceph-mon[31428]: pgmap v132: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:44:17.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:17 smithi111 ceph-mon[31493]: pgmap v132: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:44:18.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:17 smithi094 ceph-mon[27988]: pgmap v132: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:44:19.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:19 smithi111 ceph-mon[31493]: pgmap v133: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:44:20.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:19 smithi094 ceph-mon[27988]: pgmap v133: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:44:20.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:19 smithi173 ceph-mon[31428]: pgmap v133: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:44:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:21 smithi111 ceph-mon[31493]: pgmap v134: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:44:22.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:21 smithi094 ceph-mon[27988]: pgmap v134: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:44:22.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:21 smithi173 ceph-mon[31428]: pgmap v134: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:44:22.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:44:22.807 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:44:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.919+0000 7fed4b752640 1 -- 172.21.15.94:0/1715479122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed4410c580 msgr2=0x7fed4410e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.919+0000 7fed4b752640 1 --2- 172.21.15.94:0/1715479122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed4410c580 0x7fed4410e9b0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009ca0 tx=0x7fed3c02c180 comp rx=0 tx=0).stop 2024-04-21T08:44:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.919+0000 7fed4b752640 1 -- 172.21.15.94:0/1715479122 shutdown_connections 2024-04-21T08:44:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.919+0000 7fed4b752640 1 --2- 172.21.15.94:0/1715479122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed44103ad0 0x7fed4410bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.919+0000 7fed4b752640 1 --2- 172.21.15.94:0/1715479122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed4410c580 0x7fed4410e9b0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.919+0000 7fed4b752640 1 --2- 172.21.15.94:0/1715479122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed441031b0 0x7fed44103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.919+0000 7fed4b752640 1 -- 172.21.15.94:0/1715479122 >> 172.21.15.94:0/1715479122 conn(0x7fed440771d0 msgr2=0x7fed440775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed4b752640 1 -- 172.21.15.94:0/1715479122 shutdown_connections 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed4b752640 1 -- 172.21.15.94:0/1715479122 wait complete. 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed4b752640 1 Processor -- start 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed4b752640 1 -- start start 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed4b752640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed441031b0 0x7fed441986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed4b752640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed44103ad0 0x7fed44198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed4b752640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4410c580 0x7fed4419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed4b752640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fed441121a0 con 0x7fed441031b0 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed4b752640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fed44112020 con 0x7fed44103ad0 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed4b752640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed44112320 con 0x7fed4410c580 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed494ee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed441031b0 0x7fed441986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed494ee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed441031b0 0x7fed441986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55296/0 (socket says 172.21.15.94:55296) 2024-04-21T08:44:22.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.920+0000 7fed494ee640 1 -- 172.21.15.94:0/255954729 learned_addr learned my addr 172.21.15.94:0/255954729 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed49cef640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4410c580 0x7fed4419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed48ced640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed44103ad0 0x7fed44198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed494ee640 1 -- 172.21.15.94:0/255954729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4410c580 msgr2=0x7fed4419cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed494ee640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4410c580 0x7fed4419cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed494ee640 1 -- 172.21.15.94:0/255954729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed44103ad0 msgr2=0x7fed44198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed494ee640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed44103ad0 0x7fed44198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed494ee640 1 -- 172.21.15.94:0/255954729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4419d6b0 con 0x7fed441031b0 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed49cef640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4410c580 0x7fed4419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed48ced640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed44103ad0 0x7fed44198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed494ee640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed441031b0 0x7fed441986c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fed4000d950 tx=0x7fed4000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed3a7fc640 1 -- 172.21.15.94:0/255954729 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed40017070 con 0x7fed441031b0 2024-04-21T08:44:22.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed4b752640 1 -- 172.21.15.94:0/255954729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4419d9a0 con 0x7fed441031b0 2024-04-21T08:44:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed3a7fc640 1 -- 172.21.15.94:0/255954729 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed40009c30 con 0x7fed441031b0 2024-04-21T08:44:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed3a7fc640 1 -- 172.21.15.94:0/255954729 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4000f620 con 0x7fed441031b0 2024-04-21T08:44:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.921+0000 7fed4b752640 1 -- 172.21.15.94:0/255954729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed441a5230 con 0x7fed441031b0 2024-04-21T08:44:22.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.923+0000 7fed3a7fc640 1 -- 172.21.15.94:0/255954729 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fed40010460 con 0x7fed441031b0 2024-04-21T08:44:22.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed4e429740 1 Processor -- start 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed4e429740 1 -- start start 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed3a7fc640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fed300a3d20 0x7fed300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed4e429740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c94960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed4e429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94ea0 0x556976c9d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed4e429740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976c9fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed4e429740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556976bf91e0 con 0x556976c94580 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed4e429740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556976b96390 con 0x556976c9da80 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed4e429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556976bf51a0 con 0x556976c94ea0 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed49cef640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fed300a3d20 0x7fed300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed3a7fc640 1 -- 172.21.15.94:0/255954729 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fed400d08f0 con 0x7fed441031b0 2024-04-21T08:44:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed494ee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c94960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed494ee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c94960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55310/0 (socket says 172.21.15.94:55310) 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed494ee640 1 -- 172.21.15.94:0/963006915 learned_addr learned my addr 172.21.15.94:0/963006915 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.924+0000 7fed48ced640 1 --2- 172.21.15.94:0/963006915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94ea0 0x556976c9d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.925+0000 7fed494ee640 1 -- 172.21.15.94:0/963006915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94ea0 msgr2=0x556976c9d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.925+0000 7fed494ee640 1 --2- 172.21.15.94:0/963006915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94ea0 0x556976c9d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.925+0000 7fed494ee640 1 -- 172.21.15.94:0/963006915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 msgr2=0x556976c9fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.925+0000 7fed494ee640 1 --2- 172.21.15.94:0/963006915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976c9fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.925+0000 7fed494ee640 1 -- 172.21.15.94:0/963006915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569769d57d0 con 0x556976c94580 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.925+0000 7fed49cef640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fed300a3d20 0x7fed300a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fed3c006010 tx=0x7fed3c009a00 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.925+0000 7fed494ee640 1 --2- 172.21.15.94:0/963006915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c94960 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fed40098e80 tx=0x7fed40098880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c09148f602aefd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.925+0000 7fed2f7fe640 1 -- 172.21.15.94:0/963006915 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4000ee20 con 0x556976c94580 2024-04-21T08:44:22.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.925+0000 7fed2f7fe640 1 -- 172.21.15.94:0/963006915 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed4009cbb0 con 0x556976c94580 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.925+0000 7fed2f7fe640 1 -- 172.21.15.94:0/963006915 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4001e9f0 con 0x556976c94580 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 -- 172.21.15.94:0/963006915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 msgr2=0x556976c94960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 --2- 172.21.15.94:0/963006915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c94960 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fed40098e80 tx=0x7fed40098880 comp rx=0 tx=0).stop 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 -- 172.21.15.94:0/963006915 shutdown_connections 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 --2- 172.21.15.94:0/963006915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94ea0 0x556976c9d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 --2- 172.21.15.94:0/963006915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976c9fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 --2- 172.21.15.94:0/963006915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c94960 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 -- 172.21.15.94:0/963006915 >> 172.21.15.94:0/963006915 conn(0x556976c8ecb0 msgr2=0x556976c8f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 -- 172.21.15.94:0/963006915 shutdown_connections 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 -- 172.21.15.94:0/963006915 wait complete. 2024-04-21T08:44:22.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 -- start start 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 Processor -- start 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94580 0x556976c9aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976c9b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976c9b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556976ca4c20 con 0x556976c94ea0 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556976bb22f0 con 0x556976c9da80 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556976ca4da0 con 0x556976c94580 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.926+0000 7fed4e429740 10 client.0 _ll_register_callbacks cb 0x556976bbf580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed48ced640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976c9b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed48ced640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976c9b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55318/0 (socket says 172.21.15.94:55318) 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed48ced640 1 -- 172.21.15.94:0/3235384622 learned_addr learned my addr 172.21.15.94:0/3235384622 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:22.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed494ee640 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94580 0x556976c9aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed48ced640 1 -- 172.21.15.94:0/3235384622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94580 msgr2=0x556976c9aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed48ced640 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94580 0x556976c9aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed48ced640 1 -- 172.21.15.94:0/3235384622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 msgr2=0x556976c9b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed48ced640 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976c9b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed48ced640 1 -- 172.21.15.94:0/3235384622 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55697690b3f0 con 0x556976c94ea0 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed48ced640 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976c9b000 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fed3400f6a0 tx=0x7fed3400fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc0760fd60d1344 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed2cff9640 1 -- 172.21.15.94:0/3235384622 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed34016070 con 0x556976c94ea0 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed2cff9640 1 -- 172.21.15.94:0/3235384622 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fed3400bb00 con 0x556976c94ea0 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed4e429740 1 -- 172.21.15.94:0/3235384622 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed4009cbb0 con 0x556976c94ea0 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed2cff9640 1 -- 172.21.15.94:0/3235384622 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fed34025050 con 0x556976c94ea0 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed4e429740 10 client.16899 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:22.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed4e429740 1 -- 172.21.15.94:0/3235384622 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5569768cb0f0 con 0x556976c94ea0 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.927+0000 7fed2cff9640 1 -- 172.21.15.94:0/3235384622 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed4009cbb0 con 0x556976c94ea0 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.928+0000 7fed2cff9640 1 -- 172.21.15.94:0/3235384622 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fed3401a400 con 0x556976c94ea0 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.928+0000 7fed4e429740 10 client.16899 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.928+0000 7fed4e429740 10 client.16899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.928+0000 7fed4e429740 20 client.16899 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.928+0000 7fed4e429740 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556976cb1fb0 0x556976cb4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.928+0000 7fed4e429740 4 client.16899 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.928+0000 7fed4e429740 1 -- 172.21.15.94:0/3235384622 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556976c06c00 con 0x556976cb1fb0 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.928+0000 7fed494ee640 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556976cb1fb0 0x556976cb4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.928+0000 7fed2cff9640 10 client.16899 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.928+0000 7fed494ee640 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556976cb1fb0 0x556976cb4410 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17fe394786cc7f6e server_cookie=617852de3f313a65 in_seq=0 out_seq=0 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed2cff9640 1 -- 172.21.15.94:0/3235384622 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556976c06c00 con 0x556976cb1fb0 2024-04-21T08:44:22.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed2cff9640 10 client.16899 handle_command_reply: tid=0 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed4e429740 1 client.16899 shutdown 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed4e429740 1 -- 172.21.15.94:0/3235384622 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 msgr2=0x556976c9b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed4e429740 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976c9b000 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fed3400f6a0 tx=0x7fed3400fb60 comp rx=0 tx=0).stop 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed4e429740 1 -- 172.21.15.94:0/3235384622 shutdown_connections 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed4e429740 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556976cb1fb0 0x556976cb4410 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed4e429740 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976c9b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed4e429740 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976c9b000 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed2cff9640 0 client.16899 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed4e429740 1 --2- 172.21.15.94:0/3235384622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94580 0x556976c9aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.929+0000 7fed4e429740 1 -- 172.21.15.94:0/3235384622 >> 172.21.15.94:0/3235384622 conn(0x556976c8ecb0 msgr2=0x556976c90a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.930+0000 7fed4e429740 1 -- 172.21.15.94:0/3235384622 shutdown_connections 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.930+0000 7fed4e429740 1 -- 172.21.15.94:0/3235384622 wait complete. 2024-04-21T08:44:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.930+0000 7fed4e429740 20 client.16899 trim_cache size 0 max 16384 2024-04-21T08:44:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.947+0000 7fed4e429740 1 Processor -- start 2024-04-21T08:44:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.947+0000 7fed4e429740 1 -- start start 2024-04-21T08:44:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.947+0000 7fed4e429740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c9d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.947+0000 7fed4e429740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c94ea0 0x556976cadae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.947+0000 7fed4e429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c9da80 0x556976c8dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.947+0000 7fed4e429740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fed441afab0 con 0x556976c94580 2024-04-21T08:44:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.947+0000 7fed4e429740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fed441ad380 con 0x556976c94ea0 2024-04-21T08:44:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.947+0000 7fed4e429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed441ab1d0 con 0x556976c9da80 2024-04-21T08:44:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.947+0000 7fed494ee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c9d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed494ee640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c9d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55326/0 (socket says 172.21.15.94:55326) 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed494ee640 1 -- 172.21.15.94:0/3096517958 learned_addr learned my addr 172.21.15.94:0/3096517958 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed49cef640 1 --2- 172.21.15.94:0/3096517958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c9da80 0x556976c8dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed48ced640 1 --2- 172.21.15.94:0/3096517958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c94ea0 0x556976cadae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed494ee640 1 -- 172.21.15.94:0/3096517958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c9da80 msgr2=0x556976c8dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed494ee640 1 --2- 172.21.15.94:0/3096517958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c9da80 0x556976c8dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed494ee640 1 -- 172.21.15.94:0/3096517958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c94ea0 msgr2=0x556976cadae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed494ee640 1 --2- 172.21.15.94:0/3096517958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c94ea0 0x556976cadae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed494ee640 1 -- 172.21.15.94:0/3096517958 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556976c94cd0 con 0x556976c94580 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed48ced640 1 --2- 172.21.15.94:0/3096517958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c94ea0 0x556976cadae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed494ee640 1 --2- 172.21.15.94:0/3096517958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c9d1f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fed400ce260 tx=0x7fed400987e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7854f5b1424a9d7c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed2effd640 1 -- 172.21.15.94:0/3096517958 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed40009de0 con 0x556976c94580 2024-04-21T08:44:22.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed2effd640 1 -- 172.21.15.94:0/3096517958 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed4009c4b0 con 0x556976c94580 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.948+0000 7fed2effd640 1 -- 172.21.15.94:0/3096517958 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4001ea30 con 0x556976c94580 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 -- 172.21.15.94:0/3096517958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 msgr2=0x556976c9d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 --2- 172.21.15.94:0/3096517958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c9d1f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fed400ce260 tx=0x7fed400987e0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 -- 172.21.15.94:0/3096517958 shutdown_connections 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 --2- 172.21.15.94:0/3096517958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c9da80 0x556976c8dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 --2- 172.21.15.94:0/3096517958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c94ea0 0x556976cadae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 --2- 172.21.15.94:0/3096517958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94580 0x556976c9d1f0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 -- 172.21.15.94:0/3096517958 >> 172.21.15.94:0/3096517958 conn(0x556976c8ecb0 msgr2=0x556976c8d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 -- 172.21.15.94:0/3096517958 shutdown_connections 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 -- 172.21.15.94:0/3096517958 wait complete. 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 -- start start 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 Processor -- start 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94580 0x556976ca5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976ca60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.949+0000 7fed4e429740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976ca65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed4e429740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fed441ab840 con 0x556976c94ea0 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed4e429740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fed441accb0 con 0x556976c9da80 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed4e429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed441abf10 con 0x556976c94580 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed4e429740 10 client.0 _ll_register_callbacks cb 0x7fed3400d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed48ced640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976ca60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed48ced640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976ca60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55328/0 (socket says 172.21.15.94:55328) 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed48ced640 1 -- 172.21.15.94:0/3777041179 learned_addr learned my addr 172.21.15.94:0/3777041179 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed494ee640 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94580 0x556976ca5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed49cef640 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976ca65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed48ced640 1 -- 172.21.15.94:0/3777041179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94580 msgr2=0x556976ca5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed48ced640 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94580 0x556976ca5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed48ced640 1 -- 172.21.15.94:0/3777041179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 msgr2=0x556976ca65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed48ced640 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976ca65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed48ced640 1 -- 172.21.15.94:0/3777041179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556976d176f0 con 0x556976c94ea0 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed49cef640 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976ca65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed48ced640 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976ca60a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fed3401aa10 tx=0x7fed3400de20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d21fa6e2f301f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed2cff9640 1 -- 172.21.15.94:0/3777041179 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3402f070 con 0x556976c94ea0 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed2cff9640 1 -- 172.21.15.94:0/3777041179 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fed3400b900 con 0x556976c94ea0 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed4e429740 1 -- 172.21.15.94:0/3777041179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed4009c4b0 con 0x556976c94ea0 2024-04-21T08:44:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed2cff9640 1 -- 172.21.15.94:0/3777041179 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fed34038050 con 0x556976c94ea0 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed4e429740 10 client.16905 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.950+0000 7fed4e429740 1 -- 172.21.15.94:0/3777041179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556976d189b0 con 0x556976c94ea0 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed2cff9640 1 -- 172.21.15.94:0/3777041179 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed4009c4b0 con 0x556976c94ea0 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed2cff9640 1 -- 172.21.15.94:0/3777041179 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fed34032020 con 0x556976c94ea0 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed4e429740 10 client.16905 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed4e429740 10 client.16905 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed4e429740 20 client.16905 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed4e429740 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556976d191b0 0x556976d1b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed4e429740 4 client.16905 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed4e429740 1 -- 172.21.15.94:0/3777041179 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556976c00a70 con 0x556976d191b0 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed494ee640 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556976d191b0 0x556976d1b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed494ee640 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556976d191b0 0x556976d1b5d0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee5850f296f74437 server_cookie=aeceb2154ce33b37 in_seq=0 out_seq=0 2024-04-21T08:44:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.951+0000 7fed2cff9640 10 client.16905 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed2cff9640 1 -- 172.21.15.94:0/3777041179 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556976c00a70 con 0x556976d191b0 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed2cff9640 10 client.16905 handle_command_reply: tid=0 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 client.16905 shutdown 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 -- 172.21.15.94:0/3777041179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 msgr2=0x556976ca60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976ca60a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fed3401aa10 tx=0x7fed3400de20 comp rx=0 tx=0).stop 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 -- 172.21.15.94:0/3777041179 shutdown_connections 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556976d191b0 0x556976d1b5d0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556976c9da80 0x556976ca65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556976c94ea0 0x556976ca60a0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 --2- 172.21.15.94:0/3777041179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556976c94580 0x556976ca5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed2cff9640 0 client.16905 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:22.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 -- 172.21.15.94:0/3777041179 >> 172.21.15.94:0/3777041179 conn(0x556976c8ecb0 msgr2=0x556976c8dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:22.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 -- 172.21.15.94:0/3777041179 shutdown_connections 2024-04-21T08:44:22.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 1 -- 172.21.15.94:0/3777041179 wait complete. 2024-04-21T08:44:22.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4e429740 20 client.16905 trim_cache size 0 max 16384 2024-04-21T08:44:22.960 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:44:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:44:22.960 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:44:22.960 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:44:22.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4b752640 1 -- 172.21.15.94:0/255954729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fed300a3d20 msgr2=0x7fed300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4b752640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fed300a3d20 0x7fed300a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fed3c006010 tx=0x7fed3c009a00 comp rx=0 tx=0).stop 2024-04-21T08:44:22.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4b752640 1 -- 172.21.15.94:0/255954729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed441031b0 msgr2=0x7fed441986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.952+0000 7fed4b752640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed441031b0 0x7fed441986c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fed4000d950 tx=0x7fed4000de10 comp rx=0 tx=0).stop 2024-04-21T08:44:22.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.953+0000 7fed4b752640 1 -- 172.21.15.94:0/255954729 shutdown_connections 2024-04-21T08:44:22.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.953+0000 7fed4b752640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fed300a3d20 0x7fed300a61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.953+0000 7fed4b752640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4410c580 0x7fed4419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.953+0000 7fed4b752640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed44103ad0 0x7fed44198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.953+0000 7fed4b752640 1 --2- 172.21.15.94:0/255954729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed441031b0 0x7fed441986c0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.953+0000 7fed4b752640 1 -- 172.21.15.94:0/255954729 >> 172.21.15.94:0/255954729 conn(0x7fed440771d0 msgr2=0x7fed440fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:22.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.953+0000 7fed4b752640 1 -- 172.21.15.94:0/255954729 shutdown_connections 2024-04-21T08:44:22.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:22.953+0000 7fed4b752640 1 -- 172.21.15.94:0/255954729 wait complete. 2024-04-21T08:44:22.963 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:44:22.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:23.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.062+0000 7f750ad96640 1 -- 172.21.15.94:0/1516662652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7504102b40 msgr2=0x7f7504106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.062+0000 7f750ad96640 1 --2- 172.21.15.94:0/1516662652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7504102b40 0x7f7504106b80 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009900 tx=0x7f74f802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:23.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.062+0000 7f750ad96640 1 -- 172.21.15.94:0/1516662652 shutdown_connections 2024-04-21T08:44:23.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.062+0000 7f750ad96640 1 --2- 172.21.15.94:0/1516662652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7504107260 0x7f750410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.062+0000 7f750ad96640 1 --2- 172.21.15.94:0/1516662652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7504102b40 0x7f7504106b80 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.062+0000 7f750ad96640 1 --2- 172.21.15.94:0/1516662652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102190 0x7f7504102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.062+0000 7f750ad96640 1 -- 172.21.15.94:0/1516662652 >> 172.21.15.94:0/1516662652 conn(0x7f75040793e0 msgr2=0x7f75040797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.062+0000 7f750ad96640 1 -- 172.21.15.94:0/1516662652 shutdown_connections 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.062+0000 7f750ad96640 1 -- 172.21.15.94:0/1516662652 wait complete. 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f750ad96640 1 Processor -- start 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f750ad96640 1 -- start start 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f750ad96640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102190 0x7f7504198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f750ad96640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7504107260 0x7f7504198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f750ad96640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f750419d000 0x7f750419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f750ad96640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7504112180 con 0x7f750419d000 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f750ad96640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7504112000 con 0x7f7504102190 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f750ad96640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7504112300 con 0x7f7504107260 2024-04-21T08:44:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f7509333640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f750419d000 0x7f750419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f7509333640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f750419d000 0x7f750419d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55344/0 (socket says 172.21.15.94:55344) 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f7509333640 1 -- 172.21.15.94:0/2201569446 learned_addr learned my addr 172.21.15.94:0/2201569446 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f7508b32640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102190 0x7f7504198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f7503fff640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7504107260 0x7f7504198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.063+0000 7f7509333640 1 -- 172.21.15.94:0/2201569446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7504107260 msgr2=0x7f7504198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f7509333640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7504107260 0x7f7504198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f7509333640 1 -- 172.21.15.94:0/2201569446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102190 msgr2=0x7f7504198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f7509333640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102190 0x7f7504198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f7509333640 1 -- 172.21.15.94:0/2201569446 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750419daf0 con 0x7f750419d000 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f7508b32640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102190 0x7f7504198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:23.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f7503fff640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7504107260 0x7f7504198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f7509333640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f750419d000 0x7f750419d4b0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009fd0 tx=0x7f74fc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f7501ffb640 1 -- 172.21.15.94:0/2201569446 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc017070 con 0x7f750419d000 2024-04-21T08:44:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f750ad96640 1 -- 172.21.15.94:0/2201569446 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750419dde0 con 0x7f750419d000 2024-04-21T08:44:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f7501ffb640 1 -- 172.21.15.94:0/2201569446 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74fc003e20 con 0x7f750419d000 2024-04-21T08:44:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f7501ffb640 1 -- 172.21.15.94:0/2201569446 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc020060 con 0x7f750419d000 2024-04-21T08:44:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.064+0000 7f750ad96640 1 -- 172.21.15.94:0/2201569446 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75041a55a0 con 0x7f750419d000 2024-04-21T08:44:23.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.065+0000 7f750ad96640 1 -- 172.21.15.94:0/2201569446 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7504069230 con 0x7f750419d000 2024-04-21T08:44:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.066+0000 7f7501ffb640 1 -- 172.21.15.94:0/2201569446 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f74fc003970 con 0x7f750419d000 2024-04-21T08:44:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.067+0000 7f7501ffb640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74ec0a3d20 0x7f74ec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.067+0000 7f7508b32640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74ec0a3d20 0x7f74ec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.067+0000 7f7501ffb640 1 -- 172.21.15.94:0/2201569446 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f74fc0d0ac0 con 0x7f750419d000 2024-04-21T08:44:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.067+0000 7f7508b32640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74ec0a3d20 0x7f74ec0a61e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f74f40058c0 tx=0x7f74f4005850 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.072+0000 7f7501ffb640 1 -- 172.21.15.94:0/2201569446 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74fc09cd80 con 0x7f750419d000 2024-04-21T08:44:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.287+0000 7f750ad96640 1 -- 172.21.15.94:0/2201569446 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f750410a4f0 con 0x7f750419d000 2024-04-21T08:44:23.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.287+0000 7f7501ffb640 1 -- 172.21.15.94:0/2201569446 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f74fc09c680 con 0x7f750419d000 2024-04-21T08:44:23.290 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:44:23.290 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:44:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.289+0000 7f750ad96640 1 -- 172.21.15.94:0/2201569446 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74ec0a3d20 msgr2=0x7f74ec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.289+0000 7f750ad96640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74ec0a3d20 0x7f74ec0a61e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f74f40058c0 tx=0x7f74f4005850 comp rx=0 tx=0).stop 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.289+0000 7f750ad96640 1 -- 172.21.15.94:0/2201569446 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f750419d000 msgr2=0x7f750419d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.289+0000 7f750ad96640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f750419d000 0x7f750419d4b0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009fd0 tx=0x7f74fc00b260 comp rx=0 tx=0).stop 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.289+0000 7f750ad96640 1 -- 172.21.15.94:0/2201569446 shutdown_connections 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.289+0000 7f750ad96640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f74ec0a3d20 0x7f74ec0a61e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.289+0000 7f750ad96640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7504107260 0x7f7504198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.289+0000 7f750ad96640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f750419d000 0x7f750419d4b0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.289+0000 7f750ad96640 1 --2- 172.21.15.94:0/2201569446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102190 0x7f7504198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.289+0000 7f750ad96640 1 -- 172.21.15.94:0/2201569446 >> 172.21.15.94:0/2201569446 conn(0x7f75040793e0 msgr2=0x7f75040ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.290+0000 7f750ad96640 1 -- 172.21.15.94:0/2201569446 shutdown_connections 2024-04-21T08:44:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.290+0000 7f750ad96640 1 -- 172.21.15.94:0/2201569446 wait complete. 2024-04-21T08:44:23.301 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 -- 172.21.15.94:0/1569512546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93fc106170 msgr2=0x7f93fc10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 --2- 172.21.15.94:0/1569512546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93fc106170 0x7f93fc10dd40 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009ca0 tx=0x7f93f402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 -- 172.21.15.94:0/1569512546 shutdown_connections 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 --2- 172.21.15.94:0/1569512546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93fc101a50 0x7f93fc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 --2- 172.21.15.94:0/1569512546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93fc106170 0x7f93fc10dd40 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 --2- 172.21.15.94:0/1569512546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93fc1010a0 0x7f93fc101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 -- 172.21.15.94:0/1569512546 >> 172.21.15.94:0/1569512546 conn(0x7f93fc077eb0 msgr2=0x7f93fc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 -- 172.21.15.94:0/1569512546 shutdown_connections 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 -- 172.21.15.94:0/1569512546 wait complete. 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 Processor -- start 2024-04-21T08:44:23.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 -- start start 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93fc1010a0 0x7f93fc199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93fc106170 0x7f93fc199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93fc19a060 0x7f93fc1a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc111150 con 0x7f93fc106170 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc110fd0 con 0x7f93fc19a060 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.402+0000 7f9401e25640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc1112d0 con 0x7f93fc1010a0 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93fc106170 0x7f93fc199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93fc106170 0x7f93fc199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55366/0 (socket says 172.21.15.94:55366) 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f3fff640 1 -- 172.21.15.94:0/973719963 learned_addr learned my addr 172.21.15.94:0/973719963 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93fbfff640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93fc19a060 0x7f93fc1a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f3fff640 1 -- 172.21.15.94:0/973719963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93fc1010a0 msgr2=0x7f93fc199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f3fff640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93fc1010a0 0x7f93fc199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f3fff640 1 -- 172.21.15.94:0/973719963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93fc19a060 msgr2=0x7f93fc1a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f3fff640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93fc19a060 0x7f93fc1a4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f3fff640 1 -- 172.21.15.94:0/973719963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93fc1a4e10 con 0x7f93fc106170 2024-04-21T08:44:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f3fff640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93fc106170 0x7f93fc199ad0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f93e400d740 tx=0x7f93e400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f97fa640 1 -- 172.21.15.94:0/973719963 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4017070 con 0x7f93fc106170 2024-04-21T08:44:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f9401e25640 1 -- 172.21.15.94:0/973719963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93fc1a5100 con 0x7f93fc106170 2024-04-21T08:44:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f97fa640 1 -- 172.21.15.94:0/973719963 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93e400de50 con 0x7f93fc106170 2024-04-21T08:44:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f93f97fa640 1 -- 172.21.15.94:0/973719963 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4004ce0 con 0x7f93fc106170 2024-04-21T08:44:23.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.403+0000 7f9401e25640 1 -- 172.21.15.94:0/973719963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93fc1a5600 con 0x7f93fc106170 2024-04-21T08:44:23.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.405+0000 7f93f97fa640 1 -- 172.21.15.94:0/973719963 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f93e4004e80 con 0x7f93fc106170 2024-04-21T08:44:23.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f9404a11740 1 Processor -- start 2024-04-21T08:44:23.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f9404a11740 1 -- start start 2024-04-21T08:44:23.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f9404a11740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ab790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f9404a11740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4abd60 0x563ccc4b4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f9404a11740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f9404a11740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563ccc4101f0 con 0x563ccc4ab3b0 2024-04-21T08:44:23.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f9404a11740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563ccc3ad4b0 con 0x563ccc4abd60 2024-04-21T08:44:23.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f9404a11740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563ccc40c1b0 con 0x563ccc4b4e60 2024-04-21T08:44:23.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ab790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55372/0 (socket says 172.21.15.94:55372) 2024-04-21T08:44:23.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fbfff640 1 -- 172.21.15.94:0/1745579773 learned_addr learned my addr 172.21.15.94:0/1745579773 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93f97fa640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93e80a3d20 0x7f93e80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fbfff640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93e80a3d20 0x7f93e80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fb7fe640 1 --2- 172.21.15.94:0/1745579773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4abd60 0x563ccc4b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93f97fa640 1 -- 172.21.15.94:0/973719963 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f93e40d0640 con 0x7f93fc106170 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93f3fff640 1 --2- 172.21.15.94:0/1745579773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fbfff640 1 -- 172.21.15.94:0/1745579773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 msgr2=0x563ccc4b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fbfff640 1 --2- 172.21.15.94:0/1745579773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fbfff640 1 -- 172.21.15.94:0/1745579773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4abd60 msgr2=0x563ccc4b4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fbfff640 1 --2- 172.21.15.94:0/1745579773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4abd60 0x563ccc4b4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fbfff640 1 -- 172.21.15.94:0/1745579773 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ccc39f1b0 con 0x563ccc4ab3b0 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93f3fff640 1 --2- 172.21.15.94:0/1745579773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.406+0000 7f93fbfff640 1 --2- 172.21.15.94:0/1745579773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ab790 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009430 tx=0x7f93f4032d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5811c74a8e201bf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f93f1ffb640 1 -- 172.21.15.94:0/1745579773 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f4039040 con 0x563ccc4ab3b0 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f93f1ffb640 1 -- 172.21.15.94:0/1745579773 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93f4003b70 con 0x563ccc4ab3b0 2024-04-21T08:44:23.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f93f1ffb640 1 -- 172.21.15.94:0/1745579773 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f4004860 con 0x563ccc4ab3b0 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f9404a11740 1 -- 172.21.15.94:0/1745579773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 msgr2=0x563ccc4ab790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f93fbfff640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93e80a3d20 0x7f93e80a61e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f93f404c000 tx=0x7f93f4048000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f9404a11740 1 --2- 172.21.15.94:0/1745579773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ab790 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009430 tx=0x7f93f4032d90 comp rx=0 tx=0).stop 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f9404a11740 1 -- 172.21.15.94:0/1745579773 shutdown_connections 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f9404a11740 1 --2- 172.21.15.94:0/1745579773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f9404a11740 1 --2- 172.21.15.94:0/1745579773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4abd60 0x563ccc4b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f9404a11740 1 --2- 172.21.15.94:0/1745579773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ab790 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f9404a11740 1 -- 172.21.15.94:0/1745579773 >> 172.21.15.94:0/1745579773 conn(0x563ccc4a5ba0 msgr2=0x563ccc4a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f9404a11740 1 -- 172.21.15.94:0/1745579773 shutdown_connections 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.407+0000 7f9404a11740 1 -- 172.21.15.94:0/1745579773 wait complete. 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f9404a11740 1 -- start start 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f9404a11740 1 Processor -- start 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f9404a11740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4bb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f9404a11740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4abd60 0x563ccc4b8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f9404a11740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f9404a11740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563ccc4bbda0 con 0x563ccc4ab3b0 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f9404a11740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563ccc3c9500 con 0x563ccc4abd60 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f9404a11740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563ccc4bbf20 con 0x563ccc4b4e60 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f9404a11740 10 client.0 _ll_register_callbacks cb 0x563ccc397b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f93fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4bb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f93fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4bb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55388/0 (socket says 172.21.15.94:55388) 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f93fb7fe640 1 -- 172.21.15.94:0/2708333435 learned_addr learned my addr 172.21.15.94:0/2708333435 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f93f3fff640 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4abd60 0x563ccc4b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f93fbfff640 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f93fb7fe640 1 -- 172.21.15.94:0/2708333435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 msgr2=0x563ccc4b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f93fb7fe640 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f93fb7fe640 1 -- 172.21.15.94:0/2708333435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4abd60 msgr2=0x563ccc4b8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f93fb7fe640 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4abd60 0x563ccc4b8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.408+0000 7f93fb7fe640 1 -- 172.21.15.94:0/2708333435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ccc1223f0 con 0x563ccc4ab3b0 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f93fb7fe640 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4bb290 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00d210 tx=0x7f93ec00d6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ecb191894922c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f93daffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f93daffd640 1 -- 172.21.15.94:0/2708333435 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec0125f0 con 0x563ccc4ab3b0 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f93daffd640 1 -- 172.21.15.94:0/2708333435 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f93ec012790 con 0x563ccc4ab3b0 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f9404a11740 1 -- 172.21.15.94:0/2708333435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93f4003b70 con 0x563ccc4ab3b0 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f93daffd640 1 -- 172.21.15.94:0/2708333435 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f93ec021050 con 0x563ccc4ab3b0 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f9404a11740 10 client.16923 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f9404a11740 1 -- 172.21.15.94:0/2708333435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563ccc1ec7d0 con 0x563ccc4ab3b0 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f93daffd640 1 -- 172.21.15.94:0/2708333435 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93f4003b70 con 0x563ccc4ab3b0 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f93daffd640 1 -- 172.21.15.94:0/2708333435 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f93ec013b40 con 0x563ccc4ab3b0 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f9404a11740 10 client.16923 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:23.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f9404a11740 10 client.16923 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:44:23.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f9404a11740 20 client.16923 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f9404a11740 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563ccc4c9010 0x563ccc4cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f9404a11740 4 client.16923 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.409+0000 7f9404a11740 1 -- 172.21.15.94:0/2708333435 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ccc3af930 con 0x563ccc4c9010 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.410+0000 7f93f3fff640 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563ccc4c9010 0x563ccc4cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.410+0000 7f93f3fff640 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563ccc4c9010 0x563ccc4cb470 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5ce4f7e4050bc07 server_cookie=7c2c45558c4892d4 in_seq=0 out_seq=0 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.410+0000 7f93daffd640 10 client.16923 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f93daffd640 1 -- 172.21.15.94:0/2708333435 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ccc3af930 con 0x563ccc4c9010 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f93daffd640 10 client.16923 handle_command_reply: tid=0 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 client.16923 shutdown 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 -- 172.21.15.94:0/2708333435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 msgr2=0x563ccc4bb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4bb290 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00d210 tx=0x7f93ec00d6d0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 -- 172.21.15.94:0/2708333435 shutdown_connections 2024-04-21T08:44:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563ccc4c9010 0x563ccc4cb470 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4abd60 0x563ccc4b8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f93daffd640 0 client.16923 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:44:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 --2- 172.21.15.94:0/2708333435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4bb290 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 -- 172.21.15.94:0/2708333435 >> 172.21.15.94:0/2708333435 conn(0x563ccc4a5ba0 msgr2=0x563ccc4a78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 -- 172.21.15.94:0/2708333435 shutdown_connections 2024-04-21T08:44:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 1 -- 172.21.15.94:0/2708333435 wait complete. 2024-04-21T08:44:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.411+0000 7f9404a11740 20 client.16923 trim_cache size 0 max 16384 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f9404a11740 1 Processor -- start 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f9404a11740 1 -- start start 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f9404a11740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4ab3b0 0x563ccc4c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f9404a11740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4abd60 0x563ccc4b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f9404a11740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4bad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f9404a11740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc1afb80 con 0x563ccc4abd60 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f9404a11740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc1ad450 con 0x563ccc4ab3b0 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f9404a11740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc1ab2a0 con 0x563ccc4b4e60 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f93f3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4abd60 0x563ccc4b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f93f3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4abd60 0x563ccc4b6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55404/0 (socket says 172.21.15.94:55404) 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f93f3fff640 1 -- 172.21.15.94:0/973019558 learned_addr learned my addr 172.21.15.94:0/973019558 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.424+0000 7f93fb7fe640 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4ab3b0 0x563ccc4c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93f3fff640 1 -- 172.21.15.94:0/973019558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 msgr2=0x563ccc4bad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93fbfff640 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4bad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93f3fff640 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4bad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93f3fff640 1 -- 172.21.15.94:0/973019558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4ab3b0 msgr2=0x563ccc4c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93f3fff640 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4ab3b0 0x563ccc4c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93f3fff640 1 -- 172.21.15.94:0/973019558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ccc4bbe40 con 0x563ccc4abd60 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93fb7fe640 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4ab3b0 0x563ccc4c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93fbfff640 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4bad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93f3fff640 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4abd60 0x563ccc4b6250 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009fb0 tx=0x7f93e40984e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1c44ce816240c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93f17fa640 1 -- 172.21.15.94:0/973019558 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e40d0040 con 0x563ccc4abd60 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93f17fa640 1 -- 172.21.15.94:0/973019558 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93e409a2a0 con 0x563ccc4abd60 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f93f17fa640 1 -- 172.21.15.94:0/973019558 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4020040 con 0x563ccc4abd60 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f9404a11740 1 -- 172.21.15.94:0/973019558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4abd60 msgr2=0x563ccc4b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f9404a11740 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4abd60 0x563ccc4b6250 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009fb0 tx=0x7f93e40984e0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f9404a11740 1 -- 172.21.15.94:0/973019558 shutdown_connections 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f9404a11740 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4b4e60 0x563ccc4bad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f9404a11740 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4abd60 0x563ccc4b6250 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.425+0000 7f9404a11740 1 --2- 172.21.15.94:0/973019558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4ab3b0 0x563ccc4c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 -- 172.21.15.94:0/973019558 >> 172.21.15.94:0/973019558 conn(0x563ccc4a5ba0 msgr2=0x563ccc4a4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 -- 172.21.15.94:0/973019558 shutdown_connections 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 -- 172.21.15.94:0/973019558 wait complete. 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 -- start start 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 Processor -- start 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ba510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4abd60 0x563ccc4baa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4b4e60 0x563ccc4bafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc1ab910 con 0x563ccc4ab3b0 2024-04-21T08:44:23.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc1acd80 con 0x563ccc4b4e60 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc1abfe0 con 0x563ccc4abd60 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f9404a11740 10 client.0 _ll_register_callbacks cb 0x7f93ec00bbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f93fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ba510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f93fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ba510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55416/0 (socket says 172.21.15.94:55416) 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f93fb7fe640 1 -- 172.21.15.94:0/3917463136 learned_addr learned my addr 172.21.15.94:0/3917463136 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.426+0000 7f93f3fff640 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4abd60 0x563ccc4baa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93fbfff640 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4b4e60 0x563ccc4bafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93fb7fe640 1 -- 172.21.15.94:0/3917463136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4abd60 msgr2=0x563ccc4baa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93fb7fe640 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4abd60 0x563ccc4baa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93fb7fe640 1 -- 172.21.15.94:0/3917463136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4b4e60 msgr2=0x563ccc4bafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93fb7fe640 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4b4e60 0x563ccc4bafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93fb7fe640 1 -- 172.21.15.94:0/3917463136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ccc52e8f0 con 0x563ccc4ab3b0 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93fb7fe640 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ba510 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f93ec01c450 tx=0x7f93ec00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ee2779b70ce777 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93daffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93daffd640 1 -- 172.21.15.94:0/3917463136 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec01c6e0 con 0x563ccc4ab3b0 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93daffd640 1 -- 172.21.15.94:0/3917463136 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f93ec01c950 con 0x563ccc4ab3b0 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f9404a11740 1 -- 172.21.15.94:0/3917463136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93e409a2a0 con 0x563ccc4ab3b0 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93daffd640 1 -- 172.21.15.94:0/3917463136 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f93ec01cb50 con 0x563ccc4ab3b0 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f9404a11740 10 client.16929 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f9404a11740 1 -- 172.21.15.94:0/3917463136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563ccc52fb60 con 0x563ccc4ab3b0 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93daffd640 1 -- 172.21.15.94:0/3917463136 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93e409a2a0 con 0x563ccc4ab3b0 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f93daffd640 1 -- 172.21.15.94:0/3917463136 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f93ec012660 con 0x563ccc4ab3b0 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f9404a11740 10 client.16929 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f9404a11740 10 client.16929 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f9404a11740 20 client.16929 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f9404a11740 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563ccc530290 0x563ccc5326b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f9404a11740 4 client.16929 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.427+0000 7f9404a11740 1 -- 172.21.15.94:0/3917463136 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ccc41dc10 con 0x563ccc530290 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f93f3fff640 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563ccc530290 0x563ccc5326b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f93f3fff640 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563ccc530290 0x563ccc5326b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d9c2ccf7a158fb1 server_cookie=9c0cde7a31936a86 in_seq=0 out_seq=0 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f93daffd640 10 client.16929 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f93daffd640 1 -- 172.21.15.94:0/3917463136 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ccc41dc10 con 0x563ccc530290 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f93daffd640 10 client.16929 handle_command_reply: tid=0 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 client.16929 shutdown 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 -- 172.21.15.94:0/3917463136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 msgr2=0x563ccc4ba510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ba510 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f93ec01c450 tx=0x7f93ec00dd10 comp rx=0 tx=0).stop 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 -- 172.21.15.94:0/3917463136 shutdown_connections 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563ccc530290 0x563ccc5326b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563ccc4abd60 0x563ccc4baa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f93daffd640 0 client.16929 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ccc4b4e60 0x563ccc4bafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 --2- 172.21.15.94:0/3917463136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563ccc4ab3b0 0x563ccc4ba510 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 -- 172.21.15.94:0/3917463136 >> 172.21.15.94:0/3917463136 conn(0x563ccc4a5ba0 msgr2=0x563ccc4a6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 -- 172.21.15.94:0/3917463136 shutdown_connections 2024-04-21T08:44:23.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.428+0000 7f9404a11740 1 -- 172.21.15.94:0/3917463136 wait complete. 2024-04-21T08:44:23.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9404a11740 20 client.16929 trim_cache size 0 max 16384 2024-04-21T08:44:23.436 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:44:23.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 -- 172.21.15.94:0/973719963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93e80a3d20 msgr2=0x7f93e80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93e80a3d20 0x7f93e80a61e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f93f404c000 tx=0x7f93f4048000 comp rx=0 tx=0).stop 2024-04-21T08:44:23.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 -- 172.21.15.94:0/973719963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93fc106170 msgr2=0x7f93fc199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93fc106170 0x7f93fc199ad0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f93e400d740 tx=0x7f93e400dc00 comp rx=0 tx=0).stop 2024-04-21T08:44:23.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 -- 172.21.15.94:0/973719963 shutdown_connections 2024-04-21T08:44:23.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f93e80a3d20 0x7f93e80a61e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93fc19a060 0x7f93fc1a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93fc106170 0x7f93fc199ad0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 --2- 172.21.15.94:0/973719963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93fc1010a0 0x7f93fc199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 -- 172.21.15.94:0/973719963 >> 172.21.15.94:0/973719963 conn(0x7f93fc077eb0 msgr2=0x7f93fc100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 -- 172.21.15.94:0/973719963 shutdown_connections 2024-04-21T08:44:23.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.429+0000 7f9401e25640 1 -- 172.21.15.94:0/973719963 wait complete. 2024-04-21T08:44:23.440 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:44:23.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.541+0000 7ff8eec70640 1 -- 172.21.15.94:0/382240266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8e81031d0 msgr2=0x7ff8e81035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.541+0000 7ff8eec70640 1 --2- 172.21.15.94:0/382240266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8e81031d0 0x7ff8e81035b0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009900 tx=0x7ff8dc02bd60 comp rx=0 tx=0).stop 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.541+0000 7ff8eec70640 1 -- 172.21.15.94:0/382240266 shutdown_connections 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.541+0000 7ff8eec70640 1 --2- 172.21.15.94:0/382240266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e810c510 0x7ff8e810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.541+0000 7ff8eec70640 1 --2- 172.21.15.94:0/382240266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8e8103af0 0x7ff8e810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.541+0000 7ff8eec70640 1 --2- 172.21.15.94:0/382240266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8e81031d0 0x7ff8e81035b0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.541+0000 7ff8eec70640 1 -- 172.21.15.94:0/382240266 >> 172.21.15.94:0/382240266 conn(0x7ff8e8077f50 msgr2=0x7ff8e8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.541+0000 7ff8eec70640 1 -- 172.21.15.94:0/382240266 shutdown_connections 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.541+0000 7ff8eec70640 1 -- 172.21.15.94:0/382240266 wait complete. 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eec70640 1 Processor -- start 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eec70640 1 -- start start 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eec70640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8e81031d0 0x7ff8e8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eec70640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e8103af0 0x7ff8e8198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eec70640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8e810c510 0x7ff8e819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eec70640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e81121a0 con 0x7ff8e81031d0 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eec70640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e8112020 con 0x7ff8e810c510 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eec70640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e8112320 con 0x7ff8e8103af0 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eca0c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8e81031d0 0x7ff8e8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eca0c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8e81031d0 0x7ff8e8198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55434/0 (socket says 172.21.15.94:55434) 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eca0c640 1 -- 172.21.15.94:0/1221528631 learned_addr learned my addr 172.21.15.94:0/1221528631 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e8103af0 0x7ff8e8198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8ed20d640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8e810c510 0x7ff8e819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.542+0000 7ff8eca0c640 1 -- 172.21.15.94:0/1221528631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e8103af0 msgr2=0x7ff8e8198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e8103af0 0x7ff8e8198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8eca0c640 1 -- 172.21.15.94:0/1221528631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8e810c510 msgr2=0x7ff8e819d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8e810c510 0x7ff8e819d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8eca0c640 1 -- 172.21.15.94:0/1221528631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e819d730 con 0x7ff8e81031d0 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e8103af0 0x7ff8e8198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8e81031d0 0x7ff8e8198740 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc005940 tx=0x7ff8dc02d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8e5ffb640 1 -- 172.21.15.94:0/1221528631 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc043020 con 0x7ff8e81031d0 2024-04-21T08:44:23.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8e5ffb640 1 -- 172.21.15.94:0/1221528631 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8dc02dd20 con 0x7ff8e81031d0 2024-04-21T08:44:23.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8eec70640 1 -- 172.21.15.94:0/1221528631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e819d9c0 con 0x7ff8e81031d0 2024-04-21T08:44:23.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8e5ffb640 1 -- 172.21.15.94:0/1221528631 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc003b80 con 0x7ff8e81031d0 2024-04-21T08:44:23.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.543+0000 7ff8eec70640 1 -- 172.21.15.94:0/1221528631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e81a52b0 con 0x7ff8e81031d0 2024-04-21T08:44:23.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8e5ffb640 1 -- 172.21.15.94:0/1221528631 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff8dc046020 con 0x7ff8e81031d0 2024-04-21T08:44:23.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8f185b740 1 Processor -- start 2024-04-21T08:44:23.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8f185b740 1 -- start start 2024-04-21T08:44:23.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 0x56100333a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333ae80 0x561003343430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561003343a00 0x561003345e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8f185b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56100329f1f0 con 0x561003343a00 2024-04-21T08:44:23.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8f185b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56100323c4b0 con 0x56100333a4d0 2024-04-21T08:44:23.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8f185b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56100329b1b0 con 0x56100333ae80 2024-04-21T08:44:23.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561003343a00 0x561003345e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561003343a00 0x561003345e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49678/0 (socket says 172.21.15.94:49678) 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8ed20d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 0x56100333a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8e7fff640 1 -- 172.21.15.94:0/503848508 learned_addr learned my addr 172.21.15.94:0/503848508 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.546+0000 7ff8e5ffb640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8d80a3d20 0x7ff8d80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8ed20d640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8d80a3d20 0x7ff8d80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/503848508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333ae80 0x561003343430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8e7fff640 1 -- 172.21.15.94:0/503848508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333ae80 msgr2=0x561003343430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/503848508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333ae80 0x561003343430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8e7fff640 1 -- 172.21.15.94:0/503848508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 msgr2=0x56100333a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/503848508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 0x56100333a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8e7fff640 1 -- 172.21.15.94:0/503848508 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56100322e1b0 con 0x561003343a00 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8e5ffb640 1 -- 172.21.15.94:0/1221528631 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff8dc0f2d90 con 0x7ff8e81031d0 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/503848508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333ae80 0x561003343430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/503848508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561003343a00 0x561003345e50 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009c20 tx=0x7ff8d400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5c193611884a34 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8ed20d640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8d80a3d20 0x7ff8d80a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0005fd0 tx=0x7ff8e0004ab0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8d2ffd640 1 -- 172.21.15.94:0/503848508 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4016070 con 0x561003343a00 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8d2ffd640 1 -- 172.21.15.94:0/503848508 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8d4011920 con 0x561003343a00 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.547+0000 7ff8d2ffd640 1 -- 172.21.15.94:0/503848508 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d40107b0 con 0x561003343a00 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 -- 172.21.15.94:0/503848508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561003343a00 msgr2=0x561003345e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 --2- 172.21.15.94:0/503848508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561003343a00 0x561003345e50 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009c20 tx=0x7ff8d400e6c0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 -- 172.21.15.94:0/503848508 shutdown_connections 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 --2- 172.21.15.94:0/503848508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333ae80 0x561003343430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 --2- 172.21.15.94:0/503848508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561003343a00 0x561003345e50 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 --2- 172.21.15.94:0/503848508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 0x56100333a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 -- 172.21.15.94:0/503848508 >> 172.21.15.94:0/503848508 conn(0x561003334ba0 msgr2=0x561003334f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 -- 172.21.15.94:0/503848508 shutdown_connections 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 -- 172.21.15.94:0/503848508 wait complete. 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 -- start start 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 Processor -- start 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 0x561003343290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003340620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.548+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561003343a00 0x561003340b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8f185b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x561003346970 con 0x56100333ae80 2024-04-21T08:44:23.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8f185b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561003258500 con 0x56100333a4d0 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8f185b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561003346af0 con 0x561003343a00 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8f185b740 10 client.0 _ll_register_callbacks cb 0x561003226b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003340620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003340620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49694/0 (socket says 172.21.15.94:49694) 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8e7fff640 1 -- 172.21.15.94:0/4156646855 learned_addr learned my addr 172.21.15.94:0/4156646855 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 0x561003343290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8ed20d640 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561003343a00 0x561003340b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8e7fff640 1 -- 172.21.15.94:0/4156646855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561003343a00 msgr2=0x561003340b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561003343a00 0x561003340b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8e7fff640 1 -- 172.21.15.94:0/4156646855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 msgr2=0x561003343290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 0x561003343290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8e7fff640 1 -- 172.21.15.94:0/4156646855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561002fb13f0 con 0x56100333ae80 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 0x561003343290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:23.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003340620 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400efd0 tx=0x7ff8d40058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2b7ec99a747567 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8c7fff640 1 -- 172.21.15.94:0/4156646855 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4016050 con 0x56100333ae80 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8c7fff640 1 -- 172.21.15.94:0/4156646855 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff8d4005040 con 0x56100333ae80 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8f185b740 1 -- 172.21.15.94:0/4156646855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8d4011920 con 0x56100333ae80 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8c7fff640 1 -- 172.21.15.94:0/4156646855 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8d4030050 con 0x56100333ae80 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8f185b740 10 client.16941 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.549+0000 7ff8f185b740 1 -- 172.21.15.94:0/4156646855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56100307b7d0 con 0x56100333ae80 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.550+0000 7ff8c7fff640 1 -- 172.21.15.94:0/4156646855 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8d4011920 con 0x56100333ae80 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.550+0000 7ff8c7fff640 1 -- 172.21.15.94:0/4156646855 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff8d402b030 con 0x56100333ae80 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.550+0000 7ff8f185b740 10 client.16941 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.550+0000 7ff8f185b740 10 client.16941 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.550+0000 7ff8f185b740 20 client.16941 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.550+0000 7ff8f185b740 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5610033539e0 0x561003355dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.550+0000 7ff8f185b740 4 client.16941 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.550+0000 7ff8f185b740 1 -- 172.21.15.94:0/4156646855 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56100323e930 con 0x5610033539e0 2024-04-21T08:44:23.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.550+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5610033539e0 0x561003355dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8c7fff640 10 client.16941 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5610033539e0 0x561003355dd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b7f89b56e55d360c server_cookie=1bc124f25bbd23b8 in_seq=0 out_seq=0 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8c7fff640 1 -- 172.21.15.94:0/4156646855 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56100323e930 con 0x5610033539e0 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8c7fff640 10 client.16941 handle_command_reply: tid=0 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8f185b740 1 client.16941 shutdown 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8f185b740 1 -- 172.21.15.94:0/4156646855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 msgr2=0x561003340620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8f185b740 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003340620 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400efd0 tx=0x7ff8d40058c0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8f185b740 1 -- 172.21.15.94:0/4156646855 shutdown_connections 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8f185b740 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561003343a00 0x561003340b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8f185b740 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5610033539e0 0x561003355dd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8f185b740 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003340620 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8f185b740 1 --2- 172.21.15.94:0/4156646855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56100333a4d0 0x561003343290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8c7fff640 0 client.16941 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:44:23.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.551+0000 7ff8f185b740 1 -- 172.21.15.94:0/4156646855 >> 172.21.15.94:0/4156646855 conn(0x561003334ba0 msgr2=0x5610033368f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.552+0000 7ff8f185b740 1 -- 172.21.15.94:0/4156646855 shutdown_connections 2024-04-21T08:44:23.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.552+0000 7ff8f185b740 1 -- 172.21.15.94:0/4156646855 wait complete. 2024-04-21T08:44:23.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.552+0000 7ff8f185b740 20 client.16941 trim_cache size 0 max 16384 2024-04-21T08:44:23.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8f185b740 1 Processor -- start 2024-04-21T08:44:23.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8f185b740 1 -- start start 2024-04-21T08:44:23.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333a4d0 0x56100334f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003342d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561003343a00 0x561003343290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8f185b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e81b1d20 con 0x56100333ae80 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8f185b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e81af5f0 con 0x561003343a00 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8f185b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e81ad440 con 0x56100333a4d0 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003342d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003342d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49704/0 (socket says 172.21.15.94:49704) 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8e7fff640 1 -- 172.21.15.94:0/3890355648 learned_addr learned my addr 172.21.15.94:0/3890355648 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8e7fff640 1 -- 172.21.15.94:0/3890355648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333a4d0 msgr2=0x56100334f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/3890355648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333a4d0 0x56100334f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8e7fff640 1 -- 172.21.15.94:0/3890355648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561003343a00 msgr2=0x561003343290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/3890355648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561003343a00 0x561003343290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8e7fff640 1 -- 172.21.15.94:0/3890355648 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561003340e80 con 0x56100333ae80 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/3890355648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003342d50 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400ebc0 tx=0x7ff8d4009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5771e60e0fa70a57 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8d27fc640 1 -- 172.21.15.94:0/3890355648 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4016070 con 0x56100333ae80 2024-04-21T08:44:23.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8d27fc640 1 -- 172.21.15.94:0/3890355648 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8d40107a0 con 0x56100333ae80 2024-04-21T08:44:23.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.565+0000 7ff8d27fc640 1 -- 172.21.15.94:0/3890355648 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4010c80 con 0x56100333ae80 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 -- 172.21.15.94:0/3890355648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 msgr2=0x561003342d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 --2- 172.21.15.94:0/3890355648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003342d50 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400ebc0 tx=0x7ff8d4009aa0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 -- 172.21.15.94:0/3890355648 shutdown_connections 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 --2- 172.21.15.94:0/3890355648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561003343a00 0x561003343290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 --2- 172.21.15.94:0/3890355648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003342d50 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 --2- 172.21.15.94:0/3890355648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333a4d0 0x56100334f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 -- 172.21.15.94:0/3890355648 >> 172.21.15.94:0/3890355648 conn(0x561003334ba0 msgr2=0x56100333bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 -- 172.21.15.94:0/3890355648 shutdown_connections 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 -- 172.21.15.94:0/3890355648 wait complete. 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 -- start start 2024-04-21T08:44:23.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.566+0000 7ff8f185b740 1 Processor -- start 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333a4d0 0x56100334bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003347300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8f185b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561003343a00 0x561003347840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8f185b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e81adab0 con 0x56100333ae80 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8f185b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e81aef20 con 0x561003343a00 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8f185b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e81ae180 con 0x56100333a4d0 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8f185b740 10 client.0 _ll_register_callbacks cb 0x7ff8d4011830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003347300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003347300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49710/0 (socket says 172.21.15.94:49710) 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8e7fff640 1 -- 172.21.15.94:0/3898680757 learned_addr learned my addr 172.21.15.94:0/3898680757 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8ed20d640 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561003343a00 0x561003347840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8e7fff640 1 -- 172.21.15.94:0/3898680757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333a4d0 msgr2=0x56100334bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333a4d0 0x56100334bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8e7fff640 1 -- 172.21.15.94:0/3898680757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561003343a00 msgr2=0x561003347840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561003343a00 0x561003347840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8e7fff640 1 -- 172.21.15.94:0/3898680757 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561003347d80 con 0x56100333ae80 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8ed20d640 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561003343a00 0x561003347840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8e7fff640 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003347300 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400efd0 tx=0x7ff8d40042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc5d308e22e5265 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8c7fff640 1 -- 172.21.15.94:0/3898680757 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4016040 con 0x56100333ae80 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8c7fff640 1 -- 172.21.15.94:0/3898680757 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff8d4004440 con 0x56100333ae80 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8f185b740 1 -- 172.21.15.94:0/3898680757 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8d40107a0 con 0x56100333ae80 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.567+0000 7ff8c7fff640 1 -- 172.21.15.94:0/3898680757 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8d4042050 con 0x56100333ae80 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8f185b740 10 client.16947 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8f185b740 1 -- 172.21.15.94:0/3898680757 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5610033b99a0 con 0x56100333ae80 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8c7fff640 1 -- 172.21.15.94:0/3898680757 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8d40107a0 con 0x56100333ae80 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8c7fff640 1 -- 172.21.15.94:0/3898680757 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff8d402b040 con 0x56100333ae80 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8f185b740 10 client.16947 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8f185b740 10 client.16947 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:44:23.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8f185b740 20 client.16947 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8f185b740 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5610033ba060 0x5610033bc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8f185b740 4 client.16947 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8f185b740 1 -- 172.21.15.94:0/3898680757 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5610032acc10 con 0x5610033ba060 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.568+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5610033ba060 0x5610033bc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8eca0c640 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5610033ba060 0x5610033bc450 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f32c467cf2e222b4 server_cookie=56cadbbee517a1d6 in_seq=0 out_seq=0 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8c7fff640 10 client.16947 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8c7fff640 1 -- 172.21.15.94:0/3898680757 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5610032acc10 con 0x5610033ba060 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8c7fff640 10 client.16947 handle_command_reply: tid=0 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 client.16947 shutdown 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 -- 172.21.15.94:0/3898680757 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 msgr2=0x561003347300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003347300 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400efd0 tx=0x7ff8d40042d0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 -- 172.21.15.94:0/3898680757 shutdown_connections 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x5610033ba060 0x5610033bc450 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561003343a00 0x561003347840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8c7fff640 0 client.16947 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:44:23.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56100333ae80 0x561003347300 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 --2- 172.21.15.94:0/3898680757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56100333a4d0 0x56100334bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 -- 172.21.15.94:0/3898680757 >> 172.21.15.94:0/3898680757 conn(0x561003334ba0 msgr2=0x56100333b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 -- 172.21.15.94:0/3898680757 shutdown_connections 2024-04-21T08:44:23.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 1 -- 172.21.15.94:0/3898680757 wait complete. 2024-04-21T08:44:23.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.569+0000 7ff8f185b740 20 client.16947 trim_cache size 0 max 16384 2024-04-21T08:44:23.575 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:44:23.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 -- 172.21.15.94:0/1221528631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8d80a3d20 msgr2=0x7ff8d80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8d80a3d20 0x7ff8d80a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0005fd0 tx=0x7ff8e0004ab0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 -- 172.21.15.94:0/1221528631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8e81031d0 msgr2=0x7ff8e8198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8e81031d0 0x7ff8e8198740 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc005940 tx=0x7ff8dc02d920 comp rx=0 tx=0).stop 2024-04-21T08:44:23.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 -- 172.21.15.94:0/1221528631 shutdown_connections 2024-04-21T08:44:23.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff8d80a3d20 0x7ff8d80a61e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e8103af0 0x7ff8e8198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8e810c510 0x7ff8e819d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 --2- 172.21.15.94:0/1221528631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8e81031d0 0x7ff8e8198740 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 -- 172.21.15.94:0/1221528631 >> 172.21.15.94:0/1221528631 conn(0x7ff8e8077f50 msgr2=0x7ff8e80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 -- 172.21.15.94:0/1221528631 shutdown_connections 2024-04-21T08:44:23.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.570+0000 7ff8eec70640 1 -- 172.21.15.94:0/1221528631 wait complete. 2024-04-21T08:44:23.580 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:44:23.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.683+0000 7f013c554640 1 -- 172.21.15.94:0/2099745207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01341010b0 msgr2=0x7f0134106be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.683+0000 7f013c554640 1 --2- 172.21.15.94:0/2099745207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01341010b0 0x7f0134106be0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f0124009930 tx=0x7f012402bc80 comp rx=0 tx=0).stop 2024-04-21T08:44:23.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.683+0000 7f013c554640 1 -- 172.21.15.94:0/2099745207 shutdown_connections 2024-04-21T08:44:23.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.683+0000 7f013c554640 1 --2- 172.21.15.94:0/2099745207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0134107310 0x7f013410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.683+0000 7f013c554640 1 --2- 172.21.15.94:0/2099745207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01341010b0 0x7f0134106be0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.683+0000 7f013c554640 1 --2- 172.21.15.94:0/2099745207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0134100700 0x7f0134100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.683+0000 7f013c554640 1 -- 172.21.15.94:0/2099745207 >> 172.21.15.94:0/2099745207 conn(0x7f0134077410 msgr2=0x7f01340fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.683+0000 7f013c554640 1 -- 172.21.15.94:0/2099745207 shutdown_connections 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.683+0000 7f013c554640 1 -- 172.21.15.94:0/2099745207 wait complete. 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f013c554640 1 Processor -- start 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f013c554640 1 -- start start 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f013c554640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0134100700 0x7f013419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f013c554640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0134107310 0x7f013419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f013c554640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0134198690 0x7f0134198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f013c554640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f01341121b0 con 0x7f0134107310 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f013c554640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0134112030 con 0x7f0134100700 2024-04-21T08:44:23.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f013c554640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0134112330 con 0x7f0134198690 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f0139aef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0134107310 0x7f013419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f0139aef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0134107310 0x7f013419ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49730/0 (socket says 172.21.15.94:49730) 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f0139aef640 1 -- 172.21.15.94:0/3164237151 learned_addr learned my addr 172.21.15.94:0/3164237151 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f013aaf1640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0134198690 0x7f0134198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f013a2f0640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0134100700 0x7f013419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f0139aef640 1 -- 172.21.15.94:0/3164237151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0134198690 msgr2=0x7f0134198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f0139aef640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0134198690 0x7f0134198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f0139aef640 1 -- 172.21.15.94:0/3164237151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0134100700 msgr2=0x7f013419e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f0139aef640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0134100700 0x7f013419e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f0139aef640 1 -- 172.21.15.94:0/3164237151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0134199290 con 0x7f0134107310 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.684+0000 7f0139aef640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0134107310 0x7f013419ea20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f0124009230 tx=0x7f01240045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.685+0000 7f012b7fe640 1 -- 172.21.15.94:0/3164237151 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124043020 con 0x7f0134107310 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.685+0000 7f012b7fe640 1 -- 172.21.15.94:0/3164237151 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f012402cb60 con 0x7f0134107310 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.685+0000 7f012b7fe640 1 -- 172.21.15.94:0/3164237151 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124003bf0 con 0x7f0134107310 2024-04-21T08:44:23.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.685+0000 7f013c554640 1 -- 172.21.15.94:0/3164237151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0134199520 con 0x7f0134107310 2024-04-21T08:44:23.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.685+0000 7f013c554640 1 -- 172.21.15.94:0/3164237151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01341a55d0 con 0x7f0134107310 2024-04-21T08:44:23.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.687+0000 7f012b7fe640 1 -- 172.21.15.94:0/3164237151 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f0124046020 con 0x7f0134107310 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.687+0000 7f013f217740 1 Processor -- start 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.687+0000 7f013f217740 1 -- start start 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.687+0000 7f013f217740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e09e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.687+0000 7f013f217740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09ee80 0x56463e0a7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.687+0000 7f013f217740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e0a7a00 0x56463e0a9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.687+0000 7f013f217740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56463e0031f0 con 0x56463e09e4d0 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.687+0000 7f013f217740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56463dfa04b0 con 0x56463e09ee80 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.687+0000 7f013f217740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56463dfff1b0 con 0x56463e0a7a00 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.687+0000 7f012b7fe640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f01200a3d20 0x7f01200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e09e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e09e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49736/0 (socket says 172.21.15.94:49736) 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 -- 172.21.15.94:0/3101504338 learned_addr learned my addr 172.21.15.94:0/3101504338 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f01200a3d20 0x7f01200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f012b7fe640 1 -- 172.21.15.94:0/3164237151 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f01240f2b00 con 0x7f0134107310 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f0139aef640 1 --2- 172.21.15.94:0/3101504338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e0a7a00 0x56463e0a9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013a2f0640 1 --2- 172.21.15.94:0/3101504338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09ee80 0x56463e0a7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 -- 172.21.15.94:0/3101504338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e0a7a00 msgr2=0x56463e0a9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 --2- 172.21.15.94:0/3101504338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e0a7a00 0x56463e0a9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 -- 172.21.15.94:0/3101504338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09ee80 msgr2=0x56463e0a7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 --2- 172.21.15.94:0/3101504338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09ee80 0x56463e0a7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 -- 172.21.15.94:0/3101504338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56463df921b0 con 0x56463e09e4d0 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 --2- 172.21.15.94:0/3101504338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e09e8b0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f012c012d80 tx=0x7f012c010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c206d1a795d5ea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f013aaf1640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f01200a3d20 0x7f01200a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f012c010c40 tx=0x7f012c00c000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f0128bfe640 1 -- 172.21.15.94:0/3101504338 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c024070 con 0x56463e09e4d0 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f0128bfe640 1 -- 172.21.15.94:0/3101504338 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f012c0125a0 con 0x56463e09e4d0 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.688+0000 7f0128bfe640 1 -- 172.21.15.94:0/3101504338 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c005280 con 0x56463e09e4d0 2024-04-21T08:44:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.689+0000 7f013f217740 1 -- 172.21.15.94:0/3101504338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 msgr2=0x56463e09e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.689+0000 7f013f217740 1 --2- 172.21.15.94:0/3101504338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e09e8b0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f012c012d80 tx=0x7f012c010740 comp rx=0 tx=0).stop 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.689+0000 7f013f217740 1 -- 172.21.15.94:0/3101504338 shutdown_connections 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.689+0000 7f013f217740 1 --2- 172.21.15.94:0/3101504338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e0a7a00 0x56463e0a9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.689+0000 7f013f217740 1 --2- 172.21.15.94:0/3101504338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09ee80 0x56463e0a7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.689+0000 7f013f217740 1 --2- 172.21.15.94:0/3101504338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e09e8b0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.689+0000 7f013f217740 1 -- 172.21.15.94:0/3101504338 >> 172.21.15.94:0/3101504338 conn(0x56463e098ba0 msgr2=0x56463e098f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.689+0000 7f013f217740 1 -- 172.21.15.94:0/3101504338 shutdown_connections 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.689+0000 7f013f217740 1 -- 172.21.15.94:0/3101504338 wait complete. 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.689+0000 7f013f217740 1 -- start start 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013f217740 1 Processor -- start 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013f217740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 0x56463e0a7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013f217740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09ee80 0x56463e0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013f217740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e0a7a00 0x56463e0a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013f217740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56463e0aa910 con 0x56463e09ee80 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013f217740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56463dfbc500 con 0x56463e09e4d0 2024-04-21T08:44:23.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013f217740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56463e0aaa90 con 0x56463e0a7a00 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013f217740 10 client.0 _ll_register_callbacks cb 0x56463df8ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f0139aef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09ee80 0x56463e0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f0139aef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09ee80 0x56463e0a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49750/0 (socket says 172.21.15.94:49750) 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f0139aef640 1 -- 172.21.15.94:0/2400197647 learned_addr learned my addr 172.21.15.94:0/2400197647 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013a2f0640 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 0x56463e0a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013aaf1640 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e0a7a00 0x56463e0a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f0139aef640 1 -- 172.21.15.94:0/2400197647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e0a7a00 msgr2=0x56463e0a4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f0139aef640 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e0a7a00 0x56463e0a4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f0139aef640 1 -- 172.21.15.94:0/2400197647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 msgr2=0x56463e0a7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f0139aef640 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 0x56463e0a7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f0139aef640 1 -- 172.21.15.94:0/2400197647 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56463dd153f0 con 0x56463e09ee80 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f013a2f0640 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 0x56463e0a7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:23.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f0139aef640 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09ee80 0x56463e0a46a0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f0124005ec0 tx=0x7f01240036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5119136f46fb4e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.690+0000 7f011dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f011dffb640 1 -- 172.21.15.94:0/2400197647 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01240dc060 con 0x56463e09ee80 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f011dffb640 1 -- 172.21.15.94:0/2400197647 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f01240a90b0 con 0x56463e09ee80 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f013f217740 1 -- 172.21.15.94:0/2400197647 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f012c0125a0 con 0x56463e09ee80 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f011dffb640 1 -- 172.21.15.94:0/2400197647 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f01240c3d10 con 0x56463e09ee80 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f013f217740 10 client.16959 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f013f217740 1 -- 172.21.15.94:0/2400197647 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56463dddf7d0 con 0x56463e09ee80 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f011dffb640 1 -- 172.21.15.94:0/2400197647 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f012c0125a0 con 0x56463e09ee80 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f011dffb640 1 -- 172.21.15.94:0/2400197647 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f01240d5070 con 0x56463e09ee80 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f013f217740 10 client.16959 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f013f217740 10 client.16959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f013f217740 20 client.16959 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f013f217740 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56463e0b7c00 0x56463e0ba030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f013f217740 4 client.16959 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f013f217740 1 -- 172.21.15.94:0/2400197647 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56463dfa2930 con 0x56463e0b7c00 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.691+0000 7f013a2f0640 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56463e0b7c00 0x56463e0ba030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.692+0000 7f013a2f0640 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56463e0b7c00 0x56463e0ba030 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c79edb75fe45ee0 server_cookie=8049dfca62e15d86 in_seq=0 out_seq=0 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.692+0000 7f011dffb640 10 client.16959 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.692+0000 7f011dffb640 1 -- 172.21.15.94:0/2400197647 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56463dfa2930 con 0x56463e0b7c00 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.692+0000 7f011dffb640 10 client.16959 handle_command_reply: tid=0 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.692+0000 7f013f217740 1 client.16959 shutdown 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 1 -- 172.21.15.94:0/2400197647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09ee80 msgr2=0x56463e0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09ee80 0x56463e0a46a0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f0124005ec0 tx=0x7f01240036f0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 1 -- 172.21.15.94:0/2400197647 shutdown_connections 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56463e0b7c00 0x56463e0ba030 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e0a7a00 0x56463e0a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f011dffb640 0 client.16959 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09ee80 0x56463e0a46a0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 1 --2- 172.21.15.94:0/2400197647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 0x56463e0a7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 1 -- 172.21.15.94:0/2400197647 >> 172.21.15.94:0/2400197647 conn(0x56463e098ba0 msgr2=0x56463e09a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 1 -- 172.21.15.94:0/2400197647 shutdown_connections 2024-04-21T08:44:23.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 1 -- 172.21.15.94:0/2400197647 wait complete. 2024-04-21T08:44:23.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.693+0000 7f013f217740 20 client.16959 trim_cache size 0 max 16384 2024-04-21T08:44:23.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013f217740 1 Processor -- start 2024-04-21T08:44:23.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013f217740 1 -- start start 2024-04-21T08:44:23.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013f217740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e0b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013f217740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 0x56463e0a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013f217740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e0a7a00 0x56463e0a7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013f217740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f01341b1b90 con 0x56463e09e4d0 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013f217740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f01341af460 con 0x56463e0a7a00 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013f217740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f01341ad2b0 con 0x56463e09ee80 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013a2f0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e0b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013a2f0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e0b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49760/0 (socket says 172.21.15.94:49760) 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013a2f0640 1 -- 172.21.15.94:0/3897083935 learned_addr learned my addr 172.21.15.94:0/3897083935 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f0139aef640 1 --2- 172.21.15.94:0/3897083935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 0x56463e0a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013a2f0640 1 -- 172.21.15.94:0/3897083935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 msgr2=0x56463e0a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013a2f0640 1 --2- 172.21.15.94:0/3897083935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 0x56463e0a6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013a2f0640 1 -- 172.21.15.94:0/3897083935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e0a7a00 msgr2=0x56463e0a7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.706+0000 7f013a2f0640 1 --2- 172.21.15.94:0/3897083935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e0a7a00 0x56463e0a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013a2f0640 1 -- 172.21.15.94:0/3897083935 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56463e0a4fb0 con 0x56463e09e4d0 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f0139aef640 1 --2- 172.21.15.94:0/3897083935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 0x56463e0a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:23.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013a2f0640 1 --2- 172.21.15.94:0/3897083935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e0b3800 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f0130009d60 tx=0x7f0130005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2141be38c1588b3e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f011ffff640 1 -- 172.21.15.94:0/3897083935 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130004220 con 0x56463e09e4d0 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f011ffff640 1 -- 172.21.15.94:0/3897083935 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0130024030 con 0x56463e09e4d0 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f011ffff640 1 -- 172.21.15.94:0/3897083935 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130004d60 con 0x56463e09e4d0 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013f217740 1 -- 172.21.15.94:0/3897083935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 msgr2=0x56463e0b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013f217740 1 --2- 172.21.15.94:0/3897083935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e0b3800 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f0130009d60 tx=0x7f0130005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013f217740 1 -- 172.21.15.94:0/3897083935 shutdown_connections 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013f217740 1 --2- 172.21.15.94:0/3897083935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 0x56463e0a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013f217740 1 --2- 172.21.15.94:0/3897083935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e0a7a00 0x56463e0a7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013f217740 1 --2- 172.21.15.94:0/3897083935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e09e4d0 0x56463e0b3800 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013f217740 1 -- 172.21.15.94:0/3897083935 >> 172.21.15.94:0/3897083935 conn(0x56463e098ba0 msgr2=0x56463e097c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013f217740 1 -- 172.21.15.94:0/3897083935 shutdown_connections 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.707+0000 7f013f217740 1 -- 172.21.15.94:0/3897083935 wait complete. 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013f217740 1 -- start start 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013f217740 1 Processor -- start 2024-04-21T08:44:23.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013f217740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 0x56463e0abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013f217740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 0x56463e0ac510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013f217740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e0a7a00 0x56463e117180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013f217740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f01341ad920 con 0x56463e0a7a00 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013f217740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f01341aed90 con 0x56463e09e4d0 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013f217740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f01341adff0 con 0x56463e09ee80 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013f217740 10 client.0 _ll_register_callbacks cb 0x7f01240baa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013aaf1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e0a7a00 0x56463e117180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013aaf1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e0a7a00 0x56463e117180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49772/0 (socket says 172.21.15.94:49772) 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013aaf1640 1 -- 172.21.15.94:0/1606937151 learned_addr learned my addr 172.21.15.94:0/1606937151 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f0139aef640 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 0x56463e0ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013aaf1640 1 -- 172.21.15.94:0/1606937151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 msgr2=0x56463e0ac510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013a2f0640 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 0x56463e0abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013aaf1640 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 0x56463e0ac510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013aaf1640 1 -- 172.21.15.94:0/1606937151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 msgr2=0x56463e0abfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013aaf1640 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 0x56463e0abfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013aaf1640 1 -- 172.21.15.94:0/1606937151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56463e117770 con 0x56463e0a7a00 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f0139aef640 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 0x56463e0ac510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013a2f0640 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 0x56463e0abfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f013aaf1640 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e0a7a00 0x56463e117180 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f012c012ac0 tx=0x7f012c009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3273cd97d7aeecfd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.708+0000 7f011dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f011dffb640 1 -- 172.21.15.94:0/1606937151 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c02c030 con 0x56463e0a7a00 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f011dffb640 1 -- 172.21.15.94:0/1606937151 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f012c01b040 con 0x56463e0a7a00 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f013f217740 1 -- 172.21.15.94:0/1606937151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0130024030 con 0x56463e0a7a00 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f011dffb640 1 -- 172.21.15.94:0/1606937151 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f012c024070 con 0x56463e0a7a00 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f013f217740 10 client.16965 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f013f217740 1 -- 172.21.15.94:0/1606937151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56463e11ea60 con 0x56463e0a7a00 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f011dffb640 1 -- 172.21.15.94:0/1606937151 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0130024030 con 0x56463e0a7a00 2024-04-21T08:44:23.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f011dffb640 1 -- 172.21.15.94:0/1606937151 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f012c00f040 con 0x56463e0a7a00 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f013f217740 10 client.16965 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f013f217740 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f013f217740 20 client.16965 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f013f217740 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56463e11ef50 0x56463e121340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f013f217740 4 client.16965 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f013f217740 1 -- 172.21.15.94:0/1606937151 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56463e010c10 con 0x56463e11ef50 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.709+0000 7f013a2f0640 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56463e11ef50 0x56463e121340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f011dffb640 10 client.16965 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013a2f0640 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56463e11ef50 0x56463e121340 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40c4684d3a26d64a server_cookie=dd2c5e65a3277f6 in_seq=0 out_seq=0 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f011dffb640 1 -- 172.21.15.94:0/1606937151 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56463e010c10 con 0x56463e11ef50 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f011dffb640 10 client.16965 handle_command_reply: tid=0 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 client.16965 shutdown 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 -- 172.21.15.94:0/1606937151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e0a7a00 msgr2=0x56463e117180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e0a7a00 0x56463e117180 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f012c012ac0 tx=0x7f012c009a00 comp rx=0 tx=0).stop 2024-04-21T08:44:23.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 -- 172.21.15.94:0/1606937151 shutdown_connections 2024-04-21T08:44:23.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x56463e11ef50 0x56463e121340 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56463e09ee80 0x56463e0ac510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56463e0a7a00 0x56463e117180 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 --2- 172.21.15.94:0/1606937151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56463e09e4d0 0x56463e0abfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 -- 172.21.15.94:0/1606937151 >> 172.21.15.94:0/1606937151 conn(0x56463e098ba0 msgr2=0x56463e097370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f011dffb640 0 client.16965 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:23.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 -- 172.21.15.94:0/1606937151 shutdown_connections 2024-04-21T08:44:23.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 1 -- 172.21.15.94:0/1606937151 wait complete. 2024-04-21T08:44:23.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.710+0000 7f013f217740 20 client.16965 trim_cache size 0 max 16384 2024-04-21T08:44:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:44:23.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 -- 172.21.15.94:0/3164237151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f01200a3d20 msgr2=0x7f01200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f01200a3d20 0x7f01200a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f012c010c40 tx=0x7f012c00c000 comp rx=0 tx=0).stop 2024-04-21T08:44:23.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 -- 172.21.15.94:0/3164237151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0134107310 msgr2=0x7f013419ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0134107310 0x7f013419ea20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f0124009230 tx=0x7f01240045e0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 -- 172.21.15.94:0/3164237151 shutdown_connections 2024-04-21T08:44:23.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f01200a3d20 0x7f01200a61e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0134198690 0x7f0134198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0134107310 0x7f013419ea20 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 --2- 172.21.15.94:0/3164237151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0134100700 0x7f013419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 -- 172.21.15.94:0/3164237151 >> 172.21.15.94:0/3164237151 conn(0x7f0134077410 msgr2=0x7f01340fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 -- 172.21.15.94:0/3164237151 shutdown_connections 2024-04-21T08:44:23.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:23.711+0000 7f013c554640 1 -- 172.21.15.94:0/3164237151 wait complete. 2024-04-21T08:44:23.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:23 smithi111 ceph-mon[31493]: pgmap v135: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:23.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:23 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2201569446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:24.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:23 smithi094 ceph-mon[27988]: pgmap v135: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:24.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:23 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2201569446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:24.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:23 smithi173 ceph-mon[31428]: pgmap v135: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:24.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:23 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2201569446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:24.722 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:44:24.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:24.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:44:24.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.831+0000 7fda8b6fd640 1 -- 172.21.15.94:0/3785576198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda84102190 msgr2=0x7fda84102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.831+0000 7fda8b6fd640 1 --2- 172.21.15.94:0/3785576198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda84102190 0x7fda84102570 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fda80009900 tx=0x7fda8002bc50 comp rx=0 tx=0).stop 2024-04-21T08:44:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.831+0000 7fda8b6fd640 1 -- 172.21.15.94:0/3785576198 shutdown_connections 2024-04-21T08:44:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.831+0000 7fda8b6fd640 1 --2- 172.21.15.94:0/3785576198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda84102b40 0x7fda84106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.831+0000 7fda8b6fd640 1 --2- 172.21.15.94:0/3785576198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda84107260 0x7fda8410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.831+0000 7fda8b6fd640 1 --2- 172.21.15.94:0/3785576198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda84102190 0x7fda84102570 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.831+0000 7fda8b6fd640 1 -- 172.21.15.94:0/3785576198 >> 172.21.15.94:0/3785576198 conn(0x7fda840793a0 msgr2=0x7fda840797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.831+0000 7fda8b6fd640 1 -- 172.21.15.94:0/3785576198 shutdown_connections 2024-04-21T08:44:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.831+0000 7fda8b6fd640 1 -- 172.21.15.94:0/3785576198 wait complete. 2024-04-21T08:44:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda8b6fd640 1 Processor -- start 2024-04-21T08:44:24.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda8b6fd640 1 -- start start 2024-04-21T08:44:24.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda8b6fd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda84102190 0x7fda8419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda8b6fd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda84107260 0x7fda8419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda8b6fd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda841985a0 0x7fda84198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda8b6fd640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fda84112220 con 0x7fda84102190 2024-04-21T08:44:24.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda8b6fd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fda841120a0 con 0x7fda841985a0 2024-04-21T08:44:24.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda8b6fd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fda841123a0 con 0x7fda84107260 2024-04-21T08:44:24.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda89499640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda84102190 0x7fda8419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda89499640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda84102190 0x7fda8419e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49782/0 (socket says 172.21.15.94:49782) 2024-04-21T08:44:24.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda89499640 1 -- 172.21.15.94:0/1490254322 learned_addr learned my addr 172.21.15.94:0/1490254322 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda89c9a640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda841985a0 0x7fda84198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda89499640 1 -- 172.21.15.94:0/1490254322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda84107260 msgr2=0x7fda8419e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda89499640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda84107260 0x7fda8419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda89499640 1 -- 172.21.15.94:0/1490254322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda841985a0 msgr2=0x7fda84198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.832+0000 7fda89499640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda841985a0 0x7fda84198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.833+0000 7fda89499640 1 -- 172.21.15.94:0/1490254322 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda841990c0 con 0x7fda84102190 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.833+0000 7fda89499640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda84102190 0x7fda8419e3f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fda80005940 tx=0x7fda8002c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.833+0000 7fda7a7fc640 1 -- 172.21.15.94:0/1490254322 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda80043020 con 0x7fda84102190 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.833+0000 7fda7a7fc640 1 -- 172.21.15.94:0/1490254322 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda8002cc80 con 0x7fda84102190 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.833+0000 7fda7a7fc640 1 -- 172.21.15.94:0/1490254322 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda80003c30 con 0x7fda84102190 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.833+0000 7fda8b6fd640 1 -- 172.21.15.94:0/1490254322 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda84199350 con 0x7fda84102190 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.833+0000 7fda8b6fd640 1 -- 172.21.15.94:0/1490254322 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda841a55a0 con 0x7fda84102190 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.835+0000 7fda7a7fc640 1 -- 172.21.15.94:0/1490254322 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fda80046020 con 0x7fda84102190 2024-04-21T08:44:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda8e3c9740 1 Processor -- start 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda8e3c9740 1 -- start start 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ebf3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda7a7fc640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda700a3d20 0x7fda700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ebfcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebfd270 0x55b89ebfd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b89eb57f00 con 0x55b89ebf37c0 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b89eaf5590 con 0x55b89ebf4170 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b89eb53ec0 con 0x55b89ebfd270 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89c9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ebf3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89c9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ebf3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49790/0 (socket says 172.21.15.94:49790) 2024-04-21T08:44:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89c9a640 1 -- 172.21.15.94:0/1868798205 learned_addr learned my addr 172.21.15.94:0/1868798205 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda7a7fc640 1 -- 172.21.15.94:0/1490254322 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fda800f2f80 con 0x7fda84102190 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89499640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda700a3d20 0x7fda700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda88c98640 1 --2- 172.21.15.94:0/1868798205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ebfcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89c9a640 1 -- 172.21.15.94:0/1868798205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebfd270 msgr2=0x55b89ebfd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89c9a640 1 --2- 172.21.15.94:0/1868798205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebfd270 0x55b89ebfd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89c9a640 1 -- 172.21.15.94:0/1868798205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 msgr2=0x55b89ebfcca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89c9a640 1 --2- 172.21.15.94:0/1868798205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ebfcca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89c9a640 1 -- 172.21.15.94:0/1868798205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b89e9327e0 con 0x55b89ebf37c0 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda88c98640 1 --2- 172.21.15.94:0/1868798205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ebfcca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89499640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda700a3d20 0x7fda700a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fda800bebe0 tx=0x7fda800057a0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.836+0000 7fda89c9a640 1 --2- 172.21.15.94:0/1868798205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ebf3ba0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fda7c008e60 tx=0x7fda7c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56493126655047bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda6f7fe640 1 -- 172.21.15.94:0/1868798205 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c015070 con 0x55b89ebf37c0 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda6f7fe640 1 -- 172.21.15.94:0/1868798205 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda7c004d20 con 0x55b89ebf37c0 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda6f7fe640 1 -- 172.21.15.94:0/1868798205 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c005460 con 0x55b89ebf37c0 2024-04-21T08:44:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1868798205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 msgr2=0x55b89ebf3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/1868798205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ebf3ba0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fda7c008e60 tx=0x7fda7c00f730 comp rx=0 tx=0).stop 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1868798205 shutdown_connections 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/1868798205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebfd270 0x55b89ebfd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/1868798205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ebfcca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/1868798205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ebf3ba0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1868798205 >> 172.21.15.94:0/1868798205 conn(0x55b89ebedfb0 msgr2=0x55b89ebee390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1868798205 shutdown_connections 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.837+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1868798205 wait complete. 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda8e3c9740 1 -- start start 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda8e3c9740 1 Processor -- start 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec00d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf4170 0x55b89ec01260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebfd270 0x55b89ec017a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b89ec041b0 con 0x55b89ebf37c0 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b89eb11580 con 0x55b89ebfd270 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b89ec04330 con 0x55b89ebf4170 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda8e3c9740 10 client.0 _ll_register_callbacks cb 0x55b89eb17650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda89c9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec00d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda89c9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec00d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49804/0 (socket says 172.21.15.94:49804) 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda89c9a640 1 -- 172.21.15.94:0/1014567746 learned_addr learned my addr 172.21.15.94:0/1014567746 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.838+0000 7fda88c98640 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf4170 0x55b89ec01260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda89c9a640 1 -- 172.21.15.94:0/1014567746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf4170 msgr2=0x55b89ec01260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda89c9a640 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf4170 0x55b89ec01260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda89c9a640 1 -- 172.21.15.94:0/1014567746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebfd270 msgr2=0x55b89ec017a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda89c9a640 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebfd270 0x55b89ec017a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda89c9a640 1 -- 172.21.15.94:0/1014567746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b89e992f70 con 0x55b89ebf37c0 2024-04-21T08:44:24.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda88c98640 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf4170 0x55b89ec01260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda89c9a640 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec00d20 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fda7c008520 tx=0x7fda7c008550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e2745610fdcbc55 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda6cff9640 1 -- 172.21.15.94:0/1014567746 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c015040 con 0x55b89ebf37c0 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda6cff9640 1 -- 172.21.15.94:0/1014567746 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fda7c0053a0 con 0x55b89ebf37c0 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1014567746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda7c004d20 con 0x55b89ebf37c0 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda6cff9640 1 -- 172.21.15.94:0/1014567746 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fda7c030050 con 0x55b89ebf37c0 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda8e3c9740 10 client.16977 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1014567746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b89e82a170 con 0x55b89ebf37c0 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda6cff9640 1 -- 172.21.15.94:0/1014567746 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda7c004d20 con 0x55b89ebf37c0 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda6cff9640 1 -- 172.21.15.94:0/1014567746 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fda7c027840 con 0x55b89ebf37c0 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda8e3c9740 10 client.16977 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda8e3c9740 10 client.16977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda8e3c9740 20 client.16977 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b89ec11660 0x55b89ec13a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda8e3c9740 4 client.16977 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.839+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1014567746 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b89eb65920 con 0x55b89ec11660 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.840+0000 7fda88c98640 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b89ec11660 0x55b89ec13a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.840+0000 7fda6cff9640 10 client.16977 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.840+0000 7fda88c98640 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b89ec11660 0x55b89ec13a50 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f6b83b2f97b0966 server_cookie=b733df38803b38f1 in_seq=0 out_seq=0 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda6cff9640 1 -- 172.21.15.94:0/1014567746 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b89eb65920 con 0x55b89ec11660 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda6cff9640 10 client.16977 handle_command_reply: tid=0 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 client.16977 shutdown 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1014567746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 msgr2=0x55b89ec00d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec00d20 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fda7c008520 tx=0x7fda7c008550 comp rx=0 tx=0).stop 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1014567746 shutdown_connections 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b89ec11660 0x55b89ec13a50 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf4170 0x55b89ec01260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda6cff9640 0 client.16977 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebfd270 0x55b89ec017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/1014567746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec00d20 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1014567746 >> 172.21.15.94:0/1014567746 conn(0x55b89ebedfb0 msgr2=0x55b89ebefd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1014567746 shutdown_connections 2024-04-21T08:44:24.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 1 -- 172.21.15.94:0/1014567746 wait complete. 2024-04-21T08:44:24.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.841+0000 7fda8e3c9740 20 client.16977 trim_cache size 0 max 16384 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.858+0000 7fda8e3c9740 1 Processor -- start 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.858+0000 7fda8e3c9740 1 -- start start 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.858+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec03450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.858+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ebfb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.858+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebfd270 0x55b89ec0d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.858+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fda841afab0 con 0x55b89ebf37c0 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.858+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fda841ad380 con 0x55b89ebf4170 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.858+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fda841ab1d0 con 0x55b89ebfd270 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda89c9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec03450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda89c9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec03450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49810/0 (socket says 172.21.15.94:49810) 2024-04-21T08:44:24.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda89c9a640 1 -- 172.21.15.94:0/2348934777 learned_addr learned my addr 172.21.15.94:0/2348934777 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda89c9a640 1 --2- 172.21.15.94:0/2348934777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebfd270 0x55b89ec0d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda89c9a640 1 -- 172.21.15.94:0/2348934777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebfd270 msgr2=0x55b89ec0d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda89c9a640 1 --2- 172.21.15.94:0/2348934777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebfd270 0x55b89ec0d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda88c98640 1 --2- 172.21.15.94:0/2348934777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ebfb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda89c9a640 1 -- 172.21.15.94:0/2348934777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 msgr2=0x55b89ebfb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda89c9a640 1 --2- 172.21.15.94:0/2348934777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ebfb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda89c9a640 1 -- 172.21.15.94:0/2348934777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b89ebee7f0 con 0x55b89ebf37c0 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda89c9a640 1 --2- 172.21.15.94:0/2348934777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec03450 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fda7c008270 tx=0x7fda7c008de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34df9824337d6196 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda6effd640 1 -- 172.21.15.94:0/2348934777 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c015070 con 0x55b89ebf37c0 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda6effd640 1 -- 172.21.15.94:0/2348934777 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda7c022070 con 0x55b89ebf37c0 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.859+0000 7fda6effd640 1 -- 172.21.15.94:0/2348934777 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c019930 con 0x55b89ebf37c0 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2348934777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 msgr2=0x55b89ec03450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/2348934777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec03450 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fda7c008270 tx=0x7fda7c008de0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2348934777 shutdown_connections 2024-04-21T08:44:24.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/2348934777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebfd270 0x55b89ec0d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/2348934777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ebfb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/2348934777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebf37c0 0x55b89ec03450 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2348934777 >> 172.21.15.94:0/2348934777 conn(0x55b89ebedfb0 msgr2=0x55b89ebefc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2348934777 shutdown_connections 2024-04-21T08:44:24.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2348934777 wait complete. 2024-04-21T08:44:24.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 -- start start 2024-04-21T08:44:24.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 Processor -- start 2024-04-21T08:44:24.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf37c0 0x55b89ec185f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ec18b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebfd270 0x55b89ec19070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fda841ab840 con 0x55b89ebfd270 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fda841accb0 con 0x55b89ebf4170 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.860+0000 7fda8e3c9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fda841abf10 con 0x55b89ebf37c0 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda8e3c9740 10 client.0 _ll_register_callbacks cb 0x7fda7c008d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda89c9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebfd270 0x55b89ec19070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda89c9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebfd270 0x55b89ec19070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49812/0 (socket says 172.21.15.94:49812) 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda89c9a640 1 -- 172.21.15.94:0/2736515026 learned_addr learned my addr 172.21.15.94:0/2736515026 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda89c9a640 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf37c0 0x55b89ec185f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda88c98640 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ec18b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda89c9a640 1 -- 172.21.15.94:0/2736515026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf37c0 msgr2=0x55b89ec185f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda89c9a640 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf37c0 0x55b89ec185f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda89c9a640 1 -- 172.21.15.94:0/2736515026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 msgr2=0x55b89ec18b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda89c9a640 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ec18b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda89c9a640 1 -- 172.21.15.94:0/2736515026 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b89ec768f0 con 0x55b89ebfd270 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda89c9a640 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebfd270 0x55b89ec19070 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00fdb0 tx=0x7fda7c0021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d92f8fb01e52c03 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda6cff9640 1 -- 172.21.15.94:0/2736515026 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c015070 con 0x55b89ebfd270 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda6cff9640 1 -- 172.21.15.94:0/2736515026 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fda7c0196d0 con 0x55b89ebfd270 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2736515026 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda7c022070 con 0x55b89ebfd270 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda6cff9640 1 -- 172.21.15.94:0/2736515026 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fda7c0198d0 con 0x55b89ebfd270 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda8e3c9740 10 client.16983 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.861+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2736515026 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b89ec77b70 con 0x55b89ebfd270 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda6cff9640 1 -- 172.21.15.94:0/2736515026 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda7c022070 con 0x55b89ebfd270 2024-04-21T08:44:24.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda6cff9640 1 -- 172.21.15.94:0/2736515026 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fda7c036a60 con 0x55b89ebfd270 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda8e3c9740 10 client.16983 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda8e3c9740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda8e3c9740 20 client.16983 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b89ec78000 0x55b89ec7a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda8e3c9740 4 client.16983 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2736515026 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b89eb5f790 con 0x55b89ec78000 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda88c98640 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b89ec78000 0x55b89ec7a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda88c98640 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b89ec78000 0x55b89ec7a420 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c17aaeadb4e5975 server_cookie=8b1ad34eefe4dc11 in_seq=0 out_seq=0 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.862+0000 7fda6cff9640 10 client.16983 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.863+0000 7fda6cff9640 1 -- 172.21.15.94:0/2736515026 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b89eb5f790 con 0x55b89ec78000 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.863+0000 7fda6cff9640 10 client.16983 handle_command_reply: tid=0 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 client.16983 shutdown 2024-04-21T08:44:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2736515026 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebfd270 msgr2=0x55b89ec19070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebfd270 0x55b89ec19070 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00fdb0 tx=0x7fda7c0021d0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2736515026 shutdown_connections 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b89ec78000 0x55b89ec7a420 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b89ebfd270 0x55b89ec19070 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b89ebf4170 0x55b89ec18b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda6cff9640 0 client.16983 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 --2- 172.21.15.94:0/2736515026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89ebf37c0 0x55b89ec185f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2736515026 >> 172.21.15.94:0/2736515026 conn(0x55b89ebedfb0 msgr2=0x55b89ebec780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2736515026 shutdown_connections 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 1 -- 172.21.15.94:0/2736515026 wait complete. 2024-04-21T08:44:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8e3c9740 20 client.16983 trim_cache size 0 max 16384 2024-04-21T08:44:24.871 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:44:24.871 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:44:24.871 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "823e0fdf-47d4-437f-b6ad-75ac1b137559", 2024-04-21T08:44:24.871 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:44:24.871 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8b6fd640 1 -- 172.21.15.94:0/1490254322 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda700a3d20 msgr2=0x7fda700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8b6fd640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda700a3d20 0x7fda700a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fda800bebe0 tx=0x7fda800057a0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8b6fd640 1 -- 172.21.15.94:0/1490254322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda84102190 msgr2=0x7fda8419e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.864+0000 7fda8b6fd640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda84102190 0x7fda8419e3f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fda80005940 tx=0x7fda8002c880 comp rx=0 tx=0).stop 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.865+0000 7fda8b6fd640 1 -- 172.21.15.94:0/1490254322 shutdown_connections 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.865+0000 7fda8b6fd640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda700a3d20 0x7fda700a61e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.865+0000 7fda8b6fd640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda84107260 0x7fda8419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.865+0000 7fda8b6fd640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda841985a0 0x7fda84198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.865+0000 7fda8b6fd640 1 --2- 172.21.15.94:0/1490254322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda84102190 0x7fda8419e3f0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.865+0000 7fda8b6fd640 1 -- 172.21.15.94:0/1490254322 >> 172.21.15.94:0/1490254322 conn(0x7fda840793a0 msgr2=0x7fda840fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.865+0000 7fda8b6fd640 1 -- 172.21.15.94:0/1490254322 shutdown_connections 2024-04-21T08:44:24.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.865+0000 7fda8b6fd640 1 -- 172.21.15.94:0/1490254322 wait complete. 2024-04-21T08:44:24.875 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:44:24.875 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:44:24.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.976+0000 7ffbc73b2640 1 -- 172.21.15.94:0/2057482587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbc010c500 msgr2=0x7ffbc010e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.976+0000 7ffbc73b2640 1 --2- 172.21.15.94:0/2057482587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbc010c500 0x7ffbc010e9a0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009ca0 tx=0x7ffbb802c230 comp rx=0 tx=0).stop 2024-04-21T08:44:24.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.976+0000 7ffbc73b2640 1 -- 172.21.15.94:0/2057482587 shutdown_connections 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.976+0000 7ffbc73b2640 1 --2- 172.21.15.94:0/2057482587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbc0103ae0 0x7ffbc010bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.976+0000 7ffbc73b2640 1 --2- 172.21.15.94:0/2057482587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbc010c500 0x7ffbc010e9a0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.976+0000 7ffbc73b2640 1 --2- 172.21.15.94:0/2057482587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbc01031c0 0x7ffbc01035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.976+0000 7ffbc73b2640 1 -- 172.21.15.94:0/2057482587 >> 172.21.15.94:0/2057482587 conn(0x7ffbc0078000 msgr2=0x7ffbc0078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.976+0000 7ffbc73b2640 1 -- 172.21.15.94:0/2057482587 shutdown_connections 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.976+0000 7ffbc73b2640 1 -- 172.21.15.94:0/2057482587 wait complete. 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc73b2640 1 Processor -- start 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc73b2640 1 -- start start 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc73b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbc01031c0 0x7ffbc01986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc73b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbc0103ae0 0x7ffbc0198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc73b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbc010c500 0x7ffbc019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc73b2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01121f0 con 0x7ffbc010c500 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc73b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc0112070 con 0x7ffbc01031c0 2024-04-21T08:44:24.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc73b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc0112370 con 0x7ffbc0103ae0 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc594f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbc010c500 0x7ffbc019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc594f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbc010c500 0x7ffbc019cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49822/0 (socket says 172.21.15.94:49822) 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc594f640 1 -- 172.21.15.94:0/3279580228 learned_addr learned my addr 172.21.15.94:0/3279580228 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc494d640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbc0103ae0 0x7ffbc0198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc514e640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbc01031c0 0x7ffbc01986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc594f640 1 -- 172.21.15.94:0/3279580228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbc0103ae0 msgr2=0x7ffbc0198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc594f640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbc0103ae0 0x7ffbc0198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc594f640 1 -- 172.21.15.94:0/3279580228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbc01031c0 msgr2=0x7ffbc01986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc594f640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbc01031c0 0x7ffbc01986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc594f640 1 -- 172.21.15.94:0/3279580228 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc019d650 con 0x7ffbc010c500 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.977+0000 7ffbc514e640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbc01031c0 0x7ffbc01986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:24.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.978+0000 7ffbc594f640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbc010c500 0x7ffbc019cf20 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7ffbb802c840 tx=0x7ffbb8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.978+0000 7ffbb67fc640 1 -- 172.21.15.94:0/3279580228 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8039040 con 0x7ffbc010c500 2024-04-21T08:44:24.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.978+0000 7ffbc73b2640 1 -- 172.21.15.94:0/3279580228 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc019d8e0 con 0x7ffbc010c500 2024-04-21T08:44:24.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.978+0000 7ffbb67fc640 1 -- 172.21.15.94:0/3279580228 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb8044070 con 0x7ffbc010c500 2024-04-21T08:44:24.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.978+0000 7ffbb67fc640 1 -- 172.21.15.94:0/3279580228 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8032a30 con 0x7ffbc010c500 2024-04-21T08:44:24.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.978+0000 7ffbc73b2640 1 -- 172.21.15.94:0/3279580228 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc01a51d0 con 0x7ffbc010c500 2024-04-21T08:44:24.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.980+0000 7ffbb67fc640 1 -- 172.21.15.94:0/3279580228 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ffbb8004580 con 0x7ffbc010c500 2024-04-21T08:44:24.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.980+0000 7ffbc9fa2740 1 Processor -- start 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc9fa2740 1 -- start start 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbb67fc640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbac0a3d20 0x7ffbac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 0x55b591ab9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ab9ea0 0x55b591ac2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ac2a80 0x55b591ac4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b591a1e1e0 con 0x55b591ab9ea0 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b5919bb390 con 0x55b591ab9580 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b591a1a1a0 con 0x55b591ac2a80 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbac0a3d20 0x7ffbac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbb67fc640 1 -- 172.21.15.94:0/3279580228 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ffbb80f32f0 con 0x7ffbc010c500 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ab9ea0 0x55b591ac2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ab9ea0 0x55b591ac2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49828/0 (socket says 172.21.15.94:49828) 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 -- 172.21.15.94:0/2506336298 learned_addr learned my addr 172.21.15.94:0/2506336298 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc514e640 1 --2- 172.21.15.94:0/2506336298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 0x55b591ab9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 -- 172.21.15.94:0/2506336298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ac2a80 msgr2=0x55b591ac4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 --2- 172.21.15.94:0/2506336298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ac2a80 0x55b591ac4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 -- 172.21.15.94:0/2506336298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 msgr2=0x55b591ab9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 --2- 172.21.15.94:0/2506336298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 0x55b591ab9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 -- 172.21.15.94:0/2506336298 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5917fa7d0 con 0x55b591ab9ea0 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc514e640 1 --2- 172.21.15.94:0/2506336298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 0x55b591ab9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbac0a3d20 0x7ffbac0a61e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0014670 tx=0x7ffbb000a940 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.981+0000 7ffbc494d640 1 --2- 172.21.15.94:0/2506336298 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ab9ea0 0x55b591ac2540 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0006aa0 tx=0x7ffbb0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927a5a41738ffd3a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbab7fe640 1 -- 172.21.15.94:0/2506336298 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0025070 con 0x55b591ab9ea0 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbab7fe640 1 -- 172.21.15.94:0/2506336298 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb001c070 con 0x55b591ab9ea0 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbab7fe640 1 -- 172.21.15.94:0/2506336298 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0020420 con 0x55b591ab9ea0 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/2506336298 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ab9ea0 msgr2=0x55b591ac2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/2506336298 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ab9ea0 0x55b591ac2540 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0006aa0 tx=0x7ffbb0019000 comp rx=0 tx=0).stop 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/2506336298 shutdown_connections 2024-04-21T08:44:24.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/2506336298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ac2a80 0x55b591ac4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/2506336298 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ab9ea0 0x55b591ac2540 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/2506336298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 0x55b591ab9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/2506336298 >> 172.21.15.94:0/2506336298 conn(0x55b591ab3cb0 msgr2=0x55b591ab4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/2506336298 shutdown_connections 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.982+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/2506336298 wait complete. 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc9fa2740 1 -- start start 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc9fa2740 1 Processor -- start 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 0x55b591abfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9ea0 0x55b591ac0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b591ac5810 con 0x55b591ac2a80 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b5919d72f0 con 0x55b591ab9ea0 2024-04-21T08:44:24.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b591ac5990 con 0x55b591ab9580 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc9fa2740 10 client.0 _ll_register_callbacks cb 0x55b5919e4580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc594f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc594f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49842/0 (socket says 172.21.15.94:49842) 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc594f640 1 -- 172.21.15.94:0/1385807670 learned_addr learned my addr 172.21.15.94:0/1385807670 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc494d640 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9ea0 0x55b591ac0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc514e640 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 0x55b591abfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc594f640 1 -- 172.21.15.94:0/1385807670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 msgr2=0x55b591abfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc594f640 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 0x55b591abfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc594f640 1 -- 172.21.15.94:0/1385807670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9ea0 msgr2=0x55b591ac0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc594f640 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9ea0 0x55b591ac0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.983+0000 7ffbc594f640 1 -- 172.21.15.94:0/1385807670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5917303f0 con 0x55b591ac2a80 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffbc594f640 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac0570 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80098d0 tx=0x7ffbb80beb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c049618fa225f7b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffba8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffba8ff9640 1 -- 172.21.15.94:0/1385807670 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80befc0 con 0x55b591ac2a80 2024-04-21T08:44:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffba8ff9640 1 -- 172.21.15.94:0/1385807670 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ffbb8008950 con 0x55b591ac2a80 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/1385807670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbb001c070 con 0x55b591ac2a80 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffba8ff9640 1 -- 172.21.15.94:0/1385807670 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffbb8008b50 con 0x55b591ac2a80 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffbc9fa2740 10 client.16995 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/1385807670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b5916f00f0 con 0x55b591ac2a80 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffba8ff9640 1 -- 172.21.15.94:0/1385807670 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb001c070 con 0x55b591ac2a80 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffba8ff9640 1 -- 172.21.15.94:0/1385807670 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ffbb80c1060 con 0x55b591ac2a80 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffbc9fa2740 10 client.16995 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffbc9fa2740 10 client.16995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffbc9fa2740 20 client.16995 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b591ad2a60 0x55b591ad4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffbc9fa2740 4 client.16995 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.984+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/1385807670 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b591a2bc00 con 0x55b591ad2a60 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.985+0000 7ffbc514e640 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b591ad2a60 0x55b591ad4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.985+0000 7ffba8ff9640 10 client.16995 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:24.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.985+0000 7ffbc514e640 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b591ad2a60 0x55b591ad4e50 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8edc2911f4aa7349 server_cookie=d8cdc99879a8be75 in_seq=0 out_seq=0 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffba8ff9640 1 -- 172.21.15.94:0/1385807670 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b591a2bc00 con 0x55b591ad2a60 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffba8ff9640 10 client.16995 handle_command_reply: tid=0 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 client.16995 shutdown 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/1385807670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 msgr2=0x55b591ac0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac0570 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80098d0 tx=0x7ffbb80beb00 comp rx=0 tx=0).stop 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/1385807670 shutdown_connections 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b591ad2a60 0x55b591ad4e50 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac0570 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9ea0 0x55b591ac0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffba8ff9640 0 client.16995 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/1385807670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 0x55b591abfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/1385807670 >> 172.21.15.94:0/1385807670 conn(0x55b591ab3cb0 msgr2=0x55b591ab5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/1385807670 shutdown_connections 2024-04-21T08:44:24.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/1385807670 wait complete. 2024-04-21T08:44:24.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:24.986+0000 7ffbc9fa2740 20 client.16995 trim_cache size 0 max 16384 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc9fa2740 1 Processor -- start 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc9fa2740 1 -- start start 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 0x55b591ac2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9ea0 0x55b591ace700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591abec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01b1b30 con 0x55b591ac2a80 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01af400 con 0x55b591ab9580 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01ad250 con 0x55b591ab9ea0 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc594f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591abec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc594f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591abec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49850/0 (socket says 172.21.15.94:49850) 2024-04-21T08:44:25.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc594f640 1 -- 172.21.15.94:0/408559595 learned_addr learned my addr 172.21.15.94:0/408559595 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.003+0000 7ffbc494d640 1 --2- 172.21.15.94:0/408559595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9ea0 0x55b591ace700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbc594f640 1 -- 172.21.15.94:0/408559595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9ea0 msgr2=0x55b591ace700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbc594f640 1 --2- 172.21.15.94:0/408559595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9ea0 0x55b591ace700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbc594f640 1 -- 172.21.15.94:0/408559595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 msgr2=0x55b591ac2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbc514e640 1 --2- 172.21.15.94:0/408559595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 0x55b591ac2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbc594f640 1 --2- 172.21.15.94:0/408559595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 0x55b591ac2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbc594f640 1 -- 172.21.15.94:0/408559595 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b591ab4ee0 con 0x55b591ac2a80 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbc494d640 1 --2- 172.21.15.94:0/408559595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9ea0 0x55b591ace700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbc594f640 1 --2- 172.21.15.94:0/408559595 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591abec50 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80f0c60 tx=0x7ffbb80bd4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8020a719bf1d2d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbaaffd640 1 -- 172.21.15.94:0/408559595 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80be920 con 0x55b591ac2a80 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbaaffd640 1 -- 172.21.15.94:0/408559595 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb80df070 con 0x55b591ac2a80 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbaaffd640 1 -- 172.21.15.94:0/408559595 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80c77a0 con 0x55b591ac2a80 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/408559595 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 msgr2=0x55b591abec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.004+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/408559595 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591abec50 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80f0c60 tx=0x7ffbb80bd4d0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/408559595 shutdown_connections 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/408559595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9ea0 0x55b591ace700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/408559595 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591abec50 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/408559595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9580 0x55b591ac2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/408559595 >> 172.21.15.94:0/408559595 conn(0x55b591ab3cb0 msgr2=0x55b591ab50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/408559595 shutdown_connections 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/408559595 wait complete. 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 -- start start 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 Processor -- start 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 0x55b591acad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9ea0 0x55b591ac61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01ad8c0 con 0x55b591ac2a80 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01aed30 con 0x55b591ab9ea0 2024-04-21T08:44:25.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01adf90 con 0x55b591ab9580 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc9fa2740 10 client.0 _ll_register_callbacks cb 0x7ffbb80bb260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc594f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc594f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49858/0 (socket says 172.21.15.94:49858) 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.005+0000 7ffbc594f640 1 -- 172.21.15.94:0/3993877343 learned_addr learned my addr 172.21.15.94:0/3993877343 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc514e640 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 0x55b591acad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc494d640 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9ea0 0x55b591ac61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc594f640 1 -- 172.21.15.94:0/3993877343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 msgr2=0x55b591acad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc594f640 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 0x55b591acad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc594f640 1 -- 172.21.15.94:0/3993877343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9ea0 msgr2=0x55b591ac61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc594f640 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9ea0 0x55b591ac61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc594f640 1 -- 172.21.15.94:0/3993877343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b591ac6c30 con 0x55b591ac2a80 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc514e640 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 0x55b591acad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc594f640 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac66f0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80f0820 tx=0x7ffbb80bec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0aac54d147170d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffba8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffba8ff9640 1 -- 172.21.15.94:0/3993877343 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80be920 con 0x55b591ac2a80 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffba8ff9640 1 -- 172.21.15.94:0/3993877343 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ffbb80c79d0 con 0x55b591ac2a80 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/3993877343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbb80df070 con 0x55b591ac2a80 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffba8ff9640 1 -- 172.21.15.94:0/3993877343 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffbb80c7c30 con 0x55b591ac2a80 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc9fa2740 10 client.17001 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/3993877343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b591b38c90 con 0x55b591ac2a80 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffba8ff9640 1 -- 172.21.15.94:0/3993877343 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb80df070 con 0x55b591ac2a80 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffba8ff9640 1 -- 172.21.15.94:0/3993877343 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ffbb80c1030 con 0x55b591ac2a80 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc9fa2740 10 client.17001 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc9fa2740 10 client.17001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc9fa2740 20 client.17001 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b591b39110 0x55b591b3b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc9fa2740 4 client.17001 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:44:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.006+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/3993877343 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b591a25a70 con 0x55b591b39110 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc514e640 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b591b39110 0x55b591b3b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc514e640 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b591b39110 0x55b591b3b500 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e89e3b263629743 server_cookie=5447012e32227ffd in_seq=0 out_seq=0 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffba8ff9640 10 client.17001 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffba8ff9640 1 -- 172.21.15.94:0/3993877343 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b591a25a70 con 0x55b591b39110 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffba8ff9640 10 client.17001 handle_command_reply: tid=0 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc9fa2740 1 client.17001 shutdown 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/3993877343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 msgr2=0x55b591ac66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac66f0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80f0820 tx=0x7ffbb80bec30 comp rx=0 tx=0).stop 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/3993877343 shutdown_connections 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b591b39110 0x55b591b3b500 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b591ac2a80 0x55b591ac66f0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b591ab9ea0 0x55b591ac61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffba8ff9640 0 client.17001 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:25.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc9fa2740 1 --2- 172.21.15.94:0/3993877343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b591ab9580 0x55b591acad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.007+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/3993877343 >> 172.21.15.94:0/3993877343 conn(0x55b591ab3cb0 msgr2=0x55b591ab50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:25.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/3993877343 shutdown_connections 2024-04-21T08:44:25.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc9fa2740 1 -- 172.21.15.94:0/3993877343 wait complete. 2024-04-21T08:44:25.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc9fa2740 20 client.17001 trim_cache size 0 max 16384 2024-04-21T08:44:25.015 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:44:25.015 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:44:25.015 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:44:25.015 INFO:teuthology.orchestra.run.smithi094.stdout: "823e0fdf-47d4-437f-b6ad-75ac1b137559": { 2024-04-21T08:44:25.015 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:44:25.016 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "823e0fdf-47d4-437f-b6ad-75ac1b137559", 2024-04-21T08:44:25.016 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:44:25.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:44:25.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:44:25.016 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 -- 172.21.15.94:0/3279580228 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbac0a3d20 msgr2=0x7ffbac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbac0a3d20 0x7ffbac0a61e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0014670 tx=0x7ffbb000a940 comp rx=0 tx=0).stop 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 -- 172.21.15.94:0/3279580228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbc010c500 msgr2=0x7ffbc019cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbc010c500 0x7ffbc019cf20 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7ffbb802c840 tx=0x7ffbb8003ba0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 -- 172.21.15.94:0/3279580228 shutdown_connections 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffbac0a3d20 0x7ffbac0a61e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbc0103ae0 0x7ffbc0198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbc010c500 0x7ffbc019cf20 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 --2- 172.21.15.94:0/3279580228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbc01031c0 0x7ffbc01986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 -- 172.21.15.94:0/3279580228 >> 172.21.15.94:0/3279580228 conn(0x7ffbc0078000 msgr2=0x7ffbc00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 -- 172.21.15.94:0/3279580228 shutdown_connections 2024-04-21T08:44:25.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:25.008+0000 7ffbc73b2640 1 -- 172.21.15.94:0/3279580228 wait complete. 2024-04-21T08:44:25.019 INFO:tasks.cephfs.filesystem:scrub status for tag:823e0fdf-47d4-437f-b6ad-75ac1b137559 - {'path': '/', 'tag': '823e0fdf-47d4-437f-b6ad-75ac1b137559', 'options': 'recursive,force'} 2024-04-21T08:44:25.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:25 smithi111 ceph-mon[31493]: pgmap v136: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:44:25.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:25 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:44:25.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:25 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:25.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:25 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:44:25.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:25 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:25 smithi094 ceph-mon[27988]: pgmap v136: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:44:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:25 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:44:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:25 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:26.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:25 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:44:26.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:25 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:25 smithi173 ceph-mon[31428]: pgmap v136: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:44:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:25 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:44:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:25 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:25 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:44:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:25 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:26.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:26 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:44:27.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:26 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:44:27.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:26 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:44:27.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:27 smithi111 ceph-mon[31493]: pgmap v137: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:44:28.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:27 smithi094 ceph-mon[27988]: pgmap v137: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:44:28.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:27 smithi173 ceph-mon[31428]: pgmap v137: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:44:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:29 smithi111 ceph-mon[31493]: pgmap v138: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:30.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:29 smithi094 ceph-mon[27988]: pgmap v138: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:30.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:29 smithi173 ceph-mon[31428]: pgmap v138: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:31 smithi111 ceph-mon[31493]: pgmap v139: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:44:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:31 smithi094 ceph-mon[27988]: pgmap v139: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:44:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:31 smithi173 ceph-mon[31428]: pgmap v139: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:44:33.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:33 smithi111 ceph-mon[31493]: pgmap v140: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:44:34.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:33 smithi094 ceph-mon[27988]: pgmap v140: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:44:34.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:33 smithi173 ceph-mon[31428]: pgmap v140: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:44:35.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:35 smithi111 ceph-mon[31493]: pgmap v141: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:44:36.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:35 smithi094 ceph-mon[27988]: pgmap v141: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:44:36.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:35 smithi173 ceph-mon[31428]: pgmap v141: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:44:37.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:37 smithi111 ceph-mon[31493]: pgmap v142: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:44:38.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:37 smithi094 ceph-mon[27988]: pgmap v142: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:44:38.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:37 smithi173 ceph-mon[31428]: pgmap v142: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:44:39.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:39 smithi111 ceph-mon[31493]: pgmap v143: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:44:40.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:39 smithi094 ceph-mon[27988]: pgmap v143: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:44:40.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:39 smithi173 ceph-mon[31428]: pgmap v143: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:44:41.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:41 smithi111 ceph-mon[31493]: pgmap v144: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:44:42.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:41 smithi094 ceph-mon[27988]: pgmap v144: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:44:42.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:41 smithi173 ceph-mon[31428]: pgmap v144: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:44:43.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:43 smithi111 ceph-mon[31493]: pgmap v145: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:44.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:43 smithi094 ceph-mon[27988]: pgmap v145: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:44.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:43 smithi173 ceph-mon[31428]: pgmap v145: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:45.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:45 smithi111 ceph-mon[31493]: pgmap v146: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:44:46.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:45 smithi094 ceph-mon[27988]: pgmap v146: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:44:46.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:45 smithi173 ceph-mon[31428]: pgmap v146: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:44:47.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:47 smithi111 ceph-mon[31493]: pgmap v147: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:44:48.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:47 smithi094 ceph-mon[27988]: pgmap v147: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:44:48.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:47 smithi173 ceph-mon[31428]: pgmap v147: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:44:49.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:49 smithi111 ceph-mon[31493]: pgmap v148: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:50.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:49 smithi094 ceph-mon[27988]: pgmap v148: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:50.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:49 smithi173 ceph-mon[31428]: pgmap v148: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:51.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:51 smithi111 ceph-mon[31493]: pgmap v149: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:44:52.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:51 smithi094 ceph-mon[27988]: pgmap v149: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:44:52.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:51 smithi173 ceph-mon[31428]: pgmap v149: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:44:53.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:53 smithi111 ceph-mon[31493]: pgmap v150: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:54.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:53 smithi094 ceph-mon[27988]: pgmap v150: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:54.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:53 smithi173 ceph-mon[31428]: pgmap v150: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:44:55.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:44:55.020 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:44:55.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.132+0000 7f63fdfe6640 1 -- 172.21.15.94:0/4233465539 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63f8108060 msgr2=0x7f63f8108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.132+0000 7f63fdfe6640 1 --2- 172.21.15.94:0/4233465539 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63f8108060 0x7f63f8108440 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009930 tx=0x7f63e802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:55.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.132+0000 7f63fdfe6640 1 -- 172.21.15.94:0/4233465539 shutdown_connections 2024-04-21T08:44:55.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.132+0000 7f63fdfe6640 1 --2- 172.21.15.94:0/4233465539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f806a5b0 0x7f63f806aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.132+0000 7f63fdfe6640 1 --2- 172.21.15.94:0/4233465539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63f806af70 0x7f63f810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.132+0000 7f63fdfe6640 1 --2- 172.21.15.94:0/4233465539 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63f8108060 0x7f63f8108440 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.132+0000 7f63fdfe6640 1 -- 172.21.15.94:0/4233465539 >> 172.21.15.94:0/4233465539 conn(0x7f63f8077c40 msgr2=0x7f63f8078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.133+0000 7f63fdfe6640 1 -- 172.21.15.94:0/4233465539 shutdown_connections 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.133+0000 7f63fdfe6640 1 -- 172.21.15.94:0/4233465539 wait complete. 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.133+0000 7f63fdfe6640 1 Processor -- start 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.133+0000 7f63fdfe6640 1 -- start start 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63fdfe6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f806a5b0 0x7f63f8101170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63fdfe6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63f806af70 0x7f63f81016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63fdfe6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63f8108060 0x7f63f8101bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63fdfe6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81121a0 con 0x7f63f806af70 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63fdfe6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63f8112020 con 0x7f63f8108060 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63fdfe6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63f8112320 con 0x7f63f806a5b0 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63f806af70 0x7f63f81016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63f806af70 0x7f63f81016b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47764/0 (socket says 172.21.15.94:47764) 2024-04-21T08:44:55.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f6ffd640 1 -- 172.21.15.94:0/2761677849 learned_addr learned my addr 172.21.15.94:0/2761677849 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f77fe640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f806a5b0 0x7f63f8101170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f6ffd640 1 -- 172.21.15.94:0/2761677849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f806a5b0 msgr2=0x7f63f8101170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f806a5b0 0x7f63f8101170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f7fff640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63f8108060 0x7f63f8101bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f6ffd640 1 -- 172.21.15.94:0/2761677849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63f8108060 msgr2=0x7f63f8101bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63f8108060 0x7f63f8101bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f6ffd640 1 -- 172.21.15.94:0/2761677849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63f81a6c20 con 0x7f63f806af70 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f77fe640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f806a5b0 0x7f63f8101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63f806af70 0x7f63f81016b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00d770 tx=0x7f63ec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f4ff9640 1 -- 172.21.15.94:0/2761677849 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec017070 con 0x7f63f806af70 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f4ff9640 1 -- 172.21.15.94:0/2761677849 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63ec0054d0 con 0x7f63f806af70 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63f4ff9640 1 -- 172.21.15.94:0/2761677849 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec004c30 con 0x7f63f806af70 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63fdfe6640 1 -- 172.21.15.94:0/2761677849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63f81a6f10 con 0x7f63f806af70 2024-04-21T08:44:55.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.134+0000 7f63fdfe6640 1 -- 172.21.15.94:0/2761677849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63f81a7380 con 0x7f63f806af70 2024-04-21T08:44:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f63f4ff9640 1 -- 172.21.15.94:0/2761677849 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f63ec004e10 con 0x7f63f806af70 2024-04-21T08:44:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f6400bd3740 1 Processor -- start 2024-04-21T08:44:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f6400bd3740 1 -- start start 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4580 0x558eb78a4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4ea0 0x558eb78ad540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78ada80 0x558eb78afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f6400bd3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x558eb78091e0 con 0x558eb78ada80 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f6400bd3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558eb77a6390 con 0x558eb78a4ea0 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f6400bd3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558eb78051a0 con 0x558eb78a4580 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f63f4ff9640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e00a3d20 0x7f63e00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f63f6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78ada80 0x558eb78afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f63f6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78ada80 0x558eb78afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47778/0 (socket says 172.21.15.94:47778) 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f63f6ffd640 1 -- 172.21.15.94:0/4282151807 learned_addr learned my addr 172.21.15.94:0/4282151807 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f63f77fe640 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4ea0 0x558eb78ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f63f7fff640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e00a3d20 0x7f63e00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f63f7fff640 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4580 0x558eb78a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.137+0000 7f63f4ff9640 1 -- 172.21.15.94:0/2761677849 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f63ec0d0730 con 0x7f63f806af70 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63f6ffd640 1 -- 172.21.15.94:0/4282151807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4580 msgr2=0x558eb78a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4580 0x558eb78a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63f6ffd640 1 -- 172.21.15.94:0/4282151807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4ea0 msgr2=0x558eb78ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4ea0 0x558eb78ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63f6ffd640 1 -- 172.21.15.94:0/4282151807 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eb75e57d0 con 0x558eb78ada80 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63f7fff640 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4580 0x558eb78a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63f77fe640 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4ea0 0x558eb78ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78ada80 0x558eb78afe70 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f63ec09c4b0 tx=0x7f63ec009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba1b2199ae3e93c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63f7fff640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e00a3d20 0x7f63e00a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f63f0005c40 tx=0x7f63f00093e0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63ddffb640 1 -- 172.21.15.94:0/4282151807 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec09c9f0 con 0x558eb78ada80 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63ddffb640 1 -- 172.21.15.94:0/4282151807 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63ec0a06a0 con 0x558eb78ada80 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.138+0000 7f63ddffb640 1 -- 172.21.15.94:0/4282151807 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec098960 con 0x558eb78ada80 2024-04-21T08:44:55.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 -- 172.21.15.94:0/4282151807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78ada80 msgr2=0x558eb78afe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78ada80 0x558eb78afe70 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f63ec09c4b0 tx=0x7f63ec009f90 comp rx=0 tx=0).stop 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 -- 172.21.15.94:0/4282151807 shutdown_connections 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78ada80 0x558eb78afe70 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4ea0 0x558eb78ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 --2- 172.21.15.94:0/4282151807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4580 0x558eb78a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 -- 172.21.15.94:0/4282151807 >> 172.21.15.94:0/4282151807 conn(0x558eb789ecb0 msgr2=0x558eb789f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 -- 172.21.15.94:0/4282151807 shutdown_connections 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 -- 172.21.15.94:0/4282151807 wait complete. 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 -- start start 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 Processor -- start 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78aaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4ea0 0x558eb78aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 0x558eb78ab500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x558eb78b0810 con 0x558eb78a4580 2024-04-21T08:44:55.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558eb77c22f0 con 0x558eb78ada80 2024-04-21T08:44:55.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.139+0000 7f6400bd3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558eb78b0990 con 0x558eb78a4ea0 2024-04-21T08:44:55.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f6400bd3740 10 client.0 _ll_register_callbacks cb 0x558eb77cf580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:55.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78aaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78aaa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47790/0 (socket says 172.21.15.94:47790) 2024-04-21T08:44:55.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f77fe640 1 -- 172.21.15.94:0/1656646555 learned_addr learned my addr 172.21.15.94:0/1656646555 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4ea0 0x558eb78aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f7fff640 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 0x558eb78ab500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f77fe640 1 -- 172.21.15.94:0/1656646555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4ea0 msgr2=0x558eb78aafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f77fe640 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4ea0 0x558eb78aafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f77fe640 1 -- 172.21.15.94:0/1656646555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 msgr2=0x558eb78ab500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f77fe640 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 0x558eb78ab500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f77fe640 1 -- 172.21.15.94:0/1656646555 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eb751b3f0 con 0x558eb78a4580 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4ea0 0x558eb78aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f7fff640 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 0x558eb78ab500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63f77fe640 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78aaa80 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009930 tx=0x7f63e802ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad893b45e302b36e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63d6ffd640 1 -- 172.21.15.94:0/1656646555 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8045070 con 0x558eb78a4580 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63d6ffd640 1 -- 172.21.15.94:0/1656646555 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f63e8005450 con 0x558eb78a4580 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f6400bd3740 1 -- 172.21.15.94:0/1656646555 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63ec0a06a0 con 0x558eb78a4580 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f63d6ffd640 1 -- 172.21.15.94:0/1656646555 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f63e8051050 con 0x558eb78a4580 2024-04-21T08:44:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f6400bd3740 10 client.17013 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.140+0000 7f6400bd3740 1 -- 172.21.15.94:0/1656646555 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558eb74db0f0 con 0x558eb78a4580 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f63d6ffd640 1 -- 172.21.15.94:0/1656646555 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63ec0a06a0 con 0x558eb78a4580 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f63d6ffd640 1 -- 172.21.15.94:0/1656646555 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f63e80035f0 con 0x558eb78a4580 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f6400bd3740 10 client.17013 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f6400bd3740 10 client.17013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f6400bd3740 20 client.17013 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f6400bd3740 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558eb78bd970 0x558eb78bfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f6400bd3740 4 client.17013 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f6400bd3740 1 -- 172.21.15.94:0/1656646555 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558eb7816c00 con 0x558eb78bd970 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558eb78bd970 0x558eb78bfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f63d6ffd640 10 client.17013 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.141+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558eb78bd970 0x558eb78bfe10 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcc3bf6d7b8dd117 server_cookie=97173ac00fc9a206 in_seq=0 out_seq=0 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f63d6ffd640 1 -- 172.21.15.94:0/1656646555 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558eb7816c00 con 0x558eb78bd970 2024-04-21T08:44:55.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f63d6ffd640 10 client.17013 handle_command_reply: tid=0 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f6400bd3740 1 client.17013 shutdown 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f6400bd3740 1 -- 172.21.15.94:0/1656646555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 msgr2=0x558eb78aaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f6400bd3740 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78aaa80 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009930 tx=0x7f63e802ca90 comp rx=0 tx=0).stop 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f6400bd3740 1 -- 172.21.15.94:0/1656646555 shutdown_connections 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f6400bd3740 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558eb78bd970 0x558eb78bfe10 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f6400bd3740 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4ea0 0x558eb78aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f6400bd3740 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 0x558eb78ab500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f6400bd3740 1 --2- 172.21.15.94:0/1656646555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78aaa80 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f63d6ffd640 0 client.17013 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.142+0000 7f6400bd3740 1 -- 172.21.15.94:0/1656646555 >> 172.21.15.94:0/1656646555 conn(0x558eb789ecb0 msgr2=0x558eb78a0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.143+0000 7f6400bd3740 1 -- 172.21.15.94:0/1656646555 shutdown_connections 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.143+0000 7f6400bd3740 1 -- 172.21.15.94:0/1656646555 wait complete. 2024-04-21T08:44:55.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.143+0000 7f6400bd3740 20 client.17013 trim_cache size 0 max 16384 2024-04-21T08:44:55.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f6400bd3740 1 Processor -- start 2024-04-21T08:44:55.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f6400bd3740 1 -- start start 2024-04-21T08:44:55.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4ea0 0x558eb78b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 0x558eb789dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f6400bd3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81b3ee0 con 0x558eb78a4580 2024-04-21T08:44:55.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f6400bd3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81b17b0 con 0x558eb78ada80 2024-04-21T08:44:55.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f6400bd3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81af600 con 0x558eb78a4ea0 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78ad1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47806/0 (socket says 172.21.15.94:47806) 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f77fe640 1 -- 172.21.15.94:0/1321341628 learned_addr learned my addr 172.21.15.94:0/1321341628 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f77fe640 1 -- 172.21.15.94:0/1321341628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4ea0 msgr2=0x558eb78b96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f77fe640 1 --2- 172.21.15.94:0/1321341628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4ea0 0x558eb78b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f77fe640 1 -- 172.21.15.94:0/1321341628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 msgr2=0x558eb789dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f7fff640 1 --2- 172.21.15.94:0/1321341628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 0x558eb789dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f77fe640 1 --2- 172.21.15.94:0/1321341628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 0x558eb789dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f77fe640 1 -- 172.21.15.94:0/1321341628 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eb789eae0 con 0x558eb78a4580 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f7fff640 1 --2- 172.21.15.94:0/1321341628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 0x558eb789dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.160+0000 7f63f77fe640 1 --2- 172.21.15.94:0/1321341628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78ad1b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f63e8005e10 tx=0x7f63e8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7138e05cd60555 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f63dd7fa640 1 -- 172.21.15.94:0/1321341628 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e804e070 con 0x558eb78a4580 2024-04-21T08:44:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f63dd7fa640 1 -- 172.21.15.94:0/1321341628 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e802c5d0 con 0x558eb78a4580 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f63dd7fa640 1 -- 172.21.15.94:0/1321341628 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8045040 con 0x558eb78a4580 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f6400bd3740 1 -- 172.21.15.94:0/1321341628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 msgr2=0x558eb78ad1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f6400bd3740 1 --2- 172.21.15.94:0/1321341628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78ad1b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f63e8005e10 tx=0x7f63e8005e40 comp rx=0 tx=0).stop 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f6400bd3740 1 -- 172.21.15.94:0/1321341628 shutdown_connections 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f6400bd3740 1 --2- 172.21.15.94:0/1321341628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78a4ea0 0x558eb78b96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f6400bd3740 1 --2- 172.21.15.94:0/1321341628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78ada80 0x558eb789dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f6400bd3740 1 --2- 172.21.15.94:0/1321341628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4580 0x558eb78ad1b0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f6400bd3740 1 -- 172.21.15.94:0/1321341628 >> 172.21.15.94:0/1321341628 conn(0x558eb789ecb0 msgr2=0x558eb789d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f6400bd3740 1 -- 172.21.15.94:0/1321341628 shutdown_connections 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f6400bd3740 1 -- 172.21.15.94:0/1321341628 wait complete. 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.161+0000 7f6400bd3740 1 -- start start 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f6400bd3740 1 Processor -- start 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4580 0x558eb78b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4ea0 0x558eb78b2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f6400bd3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78ada80 0x558eb7922fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f6400bd3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81afc70 con 0x558eb78a4ea0 2024-04-21T08:44:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f6400bd3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81b10e0 con 0x558eb78a4580 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f6400bd3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81b0340 con 0x558eb78ada80 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f6400bd3740 10 client.0 _ll_register_callbacks cb 0x7f63e8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4ea0 0x558eb78b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4ea0 0x558eb78b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47818/0 (socket says 172.21.15.94:47818) 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f6ffd640 1 -- 172.21.15.94:0/2240768404 learned_addr learned my addr 172.21.15.94:0/2240768404 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f77fe640 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4580 0x558eb78b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f6ffd640 1 -- 172.21.15.94:0/2240768404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78ada80 msgr2=0x558eb7922fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78ada80 0x558eb7922fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f6ffd640 1 -- 172.21.15.94:0/2240768404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4580 msgr2=0x558eb78b1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4580 0x558eb78b1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f6ffd640 1 -- 172.21.15.94:0/2240768404 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eb78afe70 con 0x558eb78a4ea0 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f77fe640 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4580 0x558eb78b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63f6ffd640 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4ea0 0x558eb78b2190 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f63ec098750 tx=0x7f63ec0a5080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3e062a13170bc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.162+0000 7f63d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f63d6ffd640 1 -- 172.21.15.94:0/2240768404 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec09be30 con 0x558eb78a4ea0 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f63d6ffd640 1 -- 172.21.15.94:0/2240768404 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f63ec09c650 con 0x558eb78a4ea0 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f6400bd3740 1 -- 172.21.15.94:0/2240768404 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63e802c5d0 con 0x558eb78a4ea0 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f63d6ffd640 1 -- 172.21.15.94:0/2240768404 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f63ec0a06a0 con 0x558eb78a4ea0 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f6400bd3740 10 client.17019 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f6400bd3740 1 -- 172.21.15.94:0/2240768404 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558eb79244c0 con 0x558eb78a4ea0 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f63d6ffd640 1 -- 172.21.15.94:0/2240768404 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e802c5d0 con 0x558eb78a4ea0 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f63d6ffd640 1 -- 172.21.15.94:0/2240768404 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f63ec098960 con 0x558eb78a4ea0 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f6400bd3740 10 client.17019 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f6400bd3740 10 client.17019 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f6400bd3740 20 client.17019 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f6400bd3740 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558eb7924850 0x558eb7926c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f6400bd3740 4 client.17019 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f6400bd3740 1 -- 172.21.15.94:0/2240768404 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558eb7810a70 con 0x558eb7924850 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.163+0000 7f63f77fe640 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558eb7924850 0x558eb7926c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f63d6ffd640 10 client.17019 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f63f77fe640 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558eb7924850 0x558eb7926c40 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7ac8647009a901 server_cookie=f216e50131d17fbf in_seq=0 out_seq=0 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f63d6ffd640 1 -- 172.21.15.94:0/2240768404 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558eb7810a70 con 0x558eb7924850 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f63d6ffd640 10 client.17019 handle_command_reply: tid=0 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 client.17019 shutdown 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 -- 172.21.15.94:0/2240768404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4ea0 msgr2=0x558eb78b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4ea0 0x558eb78b2190 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f63ec098750 tx=0x7f63ec0a5080 comp rx=0 tx=0).stop 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 -- 172.21.15.94:0/2240768404 shutdown_connections 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558eb7924850 0x558eb7926c40 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558eb78ada80 0x558eb7922fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558eb78a4ea0 0x558eb78b2190 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f63d6ffd640 0 client.17019 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:55.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 --2- 172.21.15.94:0/2240768404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eb78a4580 0x558eb78b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 -- 172.21.15.94:0/2240768404 >> 172.21.15.94:0/2240768404 conn(0x558eb789ecb0 msgr2=0x558eb789dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 -- 172.21.15.94:0/2240768404 shutdown_connections 2024-04-21T08:44:55.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 1 -- 172.21.15.94:0/2240768404 wait complete. 2024-04-21T08:44:55.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.164+0000 7f6400bd3740 20 client.17019 trim_cache size 0 max 16384 2024-04-21T08:44:55.171 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:44:55.171 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:44:55.171 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:44:55.172 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:44:55.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 -- 172.21.15.94:0/2761677849 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e00a3d20 msgr2=0x7f63e00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e00a3d20 0x7f63e00a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f63f0005c40 tx=0x7f63f00093e0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 -- 172.21.15.94:0/2761677849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63f806af70 msgr2=0x7f63f81016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63f806af70 0x7f63f81016b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00d770 tx=0x7f63ec00dc30 comp rx=0 tx=0).stop 2024-04-21T08:44:55.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 -- 172.21.15.94:0/2761677849 shutdown_connections 2024-04-21T08:44:55.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63e00a3d20 0x7f63e00a61e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63f8108060 0x7f63f8101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63f806af70 0x7f63f81016b0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 --2- 172.21.15.94:0/2761677849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f806a5b0 0x7f63f8101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 -- 172.21.15.94:0/2761677849 >> 172.21.15.94:0/2761677849 conn(0x7f63f8077c40 msgr2=0x7f63f80ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 -- 172.21.15.94:0/2761677849 shutdown_connections 2024-04-21T08:44:55.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.165+0000 7f63fdfe6640 1 -- 172.21.15.94:0/2761677849 wait complete. 2024-04-21T08:44:55.175 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:44:55.175 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.275+0000 7f484214b640 1 -- 172.21.15.94:0/2810872571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f483c102b40 msgr2=0x7f483c106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.275+0000 7f484214b640 1 --2- 172.21.15.94:0/2810872571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f483c102b40 0x7f483c106b80 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f4830009900 tx=0x7f483002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.275+0000 7f484214b640 1 -- 172.21.15.94:0/2810872571 shutdown_connections 2024-04-21T08:44:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.275+0000 7f484214b640 1 --2- 172.21.15.94:0/2810872571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f483c107260 0x7f483c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.275+0000 7f484214b640 1 --2- 172.21.15.94:0/2810872571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f483c102b40 0x7f483c106b80 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.275+0000 7f484214b640 1 --2- 172.21.15.94:0/2810872571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f483c102190 0x7f483c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.275+0000 7f484214b640 1 -- 172.21.15.94:0/2810872571 >> 172.21.15.94:0/2810872571 conn(0x7f483c0793e0 msgr2=0x7f483c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.275+0000 7f484214b640 1 -- 172.21.15.94:0/2810872571 shutdown_connections 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.275+0000 7f484214b640 1 -- 172.21.15.94:0/2810872571 wait complete. 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.275+0000 7f484214b640 1 Processor -- start 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f484214b640 1 -- start start 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f484214b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f483c102190 0x7f483c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f484214b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f483c107260 0x7f483c198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f484214b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f483c19d020 0x7f483c19d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f484214b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1121a0 con 0x7f483c19d020 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f484214b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f483c112020 con 0x7f483c102190 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f484214b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f483c112320 con 0x7f483c107260 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f483c19d020 0x7f483c19d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f483c19d020 0x7f483c19d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47838/0 (socket says 172.21.15.94:47838) 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483bfff640 1 -- 172.21.15.94:0/2942909778 learned_addr learned my addr 172.21.15.94:0/2942909778 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483affd640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f483c107260 0x7f483c198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483b7fe640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f483c102190 0x7f483c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483bfff640 1 -- 172.21.15.94:0/2942909778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f483c107260 msgr2=0x7f483c198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483bfff640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f483c107260 0x7f483c198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483bfff640 1 -- 172.21.15.94:0/2942909778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f483c102190 msgr2=0x7f483c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483bfff640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f483c102190 0x7f483c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483bfff640 1 -- 172.21.15.94:0/2942909778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f483c19db10 con 0x7f483c19d020 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483b7fe640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f483c102190 0x7f483c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f483bfff640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f483c19d020 0x7f483c19d4d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f483400eaa0 tx=0x7f483400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.276+0000 7f4838ff9640 1 -- 172.21.15.94:0/2942909778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483400cdb0 con 0x7f483c19d020 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.277+0000 7f484214b640 1 -- 172.21.15.94:0/2942909778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f483c19de00 con 0x7f483c19d020 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.277+0000 7f4838ff9640 1 -- 172.21.15.94:0/2942909778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4834005020 con 0x7f483c19d020 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.277+0000 7f4838ff9640 1 -- 172.21.15.94:0/2942909778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48340116d0 con 0x7f483c19d020 2024-04-21T08:44:55.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.277+0000 7f484214b640 1 -- 172.21.15.94:0/2942909778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f483c1a55c0 con 0x7f483c19d020 2024-04-21T08:44:55.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.278+0000 7f484214b640 1 -- 172.21.15.94:0/2942909778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f483c069230 con 0x7f483c19d020 2024-04-21T08:44:55.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.279+0000 7f4838ff9640 1 -- 172.21.15.94:0/2942909778 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f48340054a0 con 0x7f483c19d020 2024-04-21T08:44:55.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.280+0000 7f4838ff9640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f48240a3d20 0x7f48240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.280+0000 7f483b7fe640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f48240a3d20 0x7f48240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.280+0000 7f4838ff9640 1 -- 172.21.15.94:0/2942909778 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f48340d10b0 con 0x7f483c19d020 2024-04-21T08:44:55.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.280+0000 7f483b7fe640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f48240a3d20 0x7f48240a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f482c0058c0 tx=0x7f482c005d30 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.284+0000 7f4838ff9640 1 -- 172.21.15.94:0/2942909778 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4834016030 con 0x7f483c19d020 2024-04-21T08:44:55.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.499+0000 7f484214b640 1 -- 172.21.15.94:0/2942909778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f483c199ab0 con 0x7f483c19d020 2024-04-21T08:44:55.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.500+0000 7f4838ff9640 1 -- 172.21.15.94:0/2942909778 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f483c199ab0 con 0x7f483c19d020 2024-04-21T08:44:55.499 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:44:55.499 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:44:55.499 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:44:55.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 -- 172.21.15.94:0/2942909778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f48240a3d20 msgr2=0x7f48240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f48240a3d20 0x7f48240a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f482c0058c0 tx=0x7f482c005d30 comp rx=0 tx=0).stop 2024-04-21T08:44:55.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 -- 172.21.15.94:0/2942909778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f483c19d020 msgr2=0x7f483c19d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f483c19d020 0x7f483c19d4d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f483400eaa0 tx=0x7f483400ef60 comp rx=0 tx=0).stop 2024-04-21T08:44:55.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 -- 172.21.15.94:0/2942909778 shutdown_connections 2024-04-21T08:44:55.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f48240a3d20 0x7f48240a61e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f483c107260 0x7f483c198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f483c19d020 0x7f483c19d4d0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 --2- 172.21.15.94:0/2942909778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f483c102190 0x7f483c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 -- 172.21.15.94:0/2942909778 >> 172.21.15.94:0/2942909778 conn(0x7f483c0793e0 msgr2=0x7f483c1000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 -- 172.21.15.94:0/2942909778 shutdown_connections 2024-04-21T08:44:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.502+0000 7f484214b640 1 -- 172.21.15.94:0/2942909778 wait complete. 2024-04-21T08:44:55.512 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:44:55.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.616+0000 7feadb12b640 1 -- 172.21.15.94:0/131609882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fead4101b40 msgr2=0x7fead410dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.616+0000 7feadb12b640 1 --2- 172.21.15.94:0/131609882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fead4101b40 0x7fead410dc80 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7feac4009930 tx=0x7feac402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:55.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.616+0000 7feadb12b640 1 -- 172.21.15.94:0/131609882 shutdown_connections 2024-04-21T08:44:55.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.616+0000 7feadb12b640 1 --2- 172.21.15.94:0/131609882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fead41011a0 0x7fead4101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.616+0000 7feadb12b640 1 --2- 172.21.15.94:0/131609882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fead4101b40 0x7fead410dc80 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.616+0000 7feadb12b640 1 --2- 172.21.15.94:0/131609882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fead4107030 0x7fead4107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.616+0000 7feadb12b640 1 -- 172.21.15.94:0/131609882 >> 172.21.15.94:0/131609882 conn(0x7fead4076c50 msgr2=0x7fead4077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.616+0000 7feadb12b640 1 -- 172.21.15.94:0/131609882 shutdown_connections 2024-04-21T08:44:55.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.616+0000 7feadb12b640 1 -- 172.21.15.94:0/131609882 wait complete. 2024-04-21T08:44:55.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7feadb12b640 1 Processor -- start 2024-04-21T08:44:55.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7feadb12b640 1 -- start start 2024-04-21T08:44:55.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7feadb12b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fead41011a0 0x7fead4199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7feadb12b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fead4101b40 0x7fead4199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7feadb12b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fead4107030 0x7fead41a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7feadb12b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fead4111140 con 0x7fead41011a0 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7feadb12b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fead4110fc0 con 0x7fead4101b40 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7feadb12b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fead41112c0 con 0x7fead4107030 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7fead8ec7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fead41011a0 0x7fead4199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7fead8ec7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fead41011a0 0x7fead4199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47852/0 (socket says 172.21.15.94:47852) 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7fead8ec7640 1 -- 172.21.15.94:0/3825634023 learned_addr learned my addr 172.21.15.94:0/3825634023 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7fead3fff640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fead4101b40 0x7fead4199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7fead96c8640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fead4107030 0x7fead41a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7fead8ec7640 1 -- 172.21.15.94:0/3825634023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fead4107030 msgr2=0x7fead41a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7fead8ec7640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fead4107030 0x7fead41a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7fead8ec7640 1 -- 172.21.15.94:0/3825634023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fead4101b40 msgr2=0x7fead4199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7fead8ec7640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fead4101b40 0x7fead4199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.617+0000 7fead8ec7640 1 -- 172.21.15.94:0/3825634023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead41a4c00 con 0x7fead41011a0 2024-04-21T08:44:55.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.618+0000 7fead8ec7640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fead41011a0 0x7fead4199640 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7feacc00ba50 tx=0x7feacc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.618+0000 7fead1ffb640 1 -- 172.21.15.94:0/3825634023 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc019070 con 0x7fead41011a0 2024-04-21T08:44:55.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.618+0000 7feadb12b640 1 -- 172.21.15.94:0/3825634023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead41a4dd0 con 0x7fead41011a0 2024-04-21T08:44:55.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.618+0000 7fead1ffb640 1 -- 172.21.15.94:0/3825634023 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feacc00cbe0 con 0x7fead41011a0 2024-04-21T08:44:55.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.618+0000 7fead1ffb640 1 -- 172.21.15.94:0/3825634023 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc0075d0 con 0x7fead41011a0 2024-04-21T08:44:55.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.618+0000 7feadb12b640 1 -- 172.21.15.94:0/3825634023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead41a5230 con 0x7fead41011a0 2024-04-21T08:44:55.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.620+0000 7fead1ffb640 1 -- 172.21.15.94:0/3825634023 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7feacc01c070 con 0x7fead41011a0 2024-04-21T08:44:55.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead1ffb640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feac00a3d20 0x7feac00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7feadddfb740 1 Processor -- start 2024-04-21T08:44:55.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7feadddfb740 1 -- start start 2024-04-21T08:44:55.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead1ffb640 1 -- 172.21.15.94:0/3825634023 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7feacc0d00a0 con 0x7fead41011a0 2024-04-21T08:44:55.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead3fff640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feac00a3d20 0x7feac00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7feadddfb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7932a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7feadddfb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c7932ae80 0x555c79333430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7feadddfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c79333a00 0x555c79335e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7feadddfb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x555c7928f1f0 con 0x555c7932a4d0 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7feadddfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555c7922c4b0 con 0x555c79333a00 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7feadddfb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555c7928b1b0 con 0x555c7932ae80 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead8ec7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7932a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead8ec7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7932a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47862/0 (socket says 172.21.15.94:47862) 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead8ec7640 1 -- 172.21.15.94:0/2473975696 learned_addr learned my addr 172.21.15.94:0/2473975696 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead96c8640 1 --2- 172.21.15.94:0/2473975696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c79333a00 0x555c79335e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead3fff640 1 --2- 172.21.15.94:0/2473975696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c7932ae80 0x555c79333430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead3fff640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feac00a3d20 0x7feac00a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fead406b050 tx=0x7feabc0103d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead8ec7640 1 -- 172.21.15.94:0/2473975696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c7932ae80 msgr2=0x555c79333430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead8ec7640 1 --2- 172.21.15.94:0/2473975696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c7932ae80 0x555c79333430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead8ec7640 1 -- 172.21.15.94:0/2473975696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c79333a00 msgr2=0x555c79335e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead8ec7640 1 --2- 172.21.15.94:0/2473975696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c79333a00 0x555c79335e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.621+0000 7fead8ec7640 1 -- 172.21.15.94:0/2473975696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c7921e1b0 con 0x555c7932a4d0 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.622+0000 7fead8ec7640 1 --2- 172.21.15.94:0/2473975696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7932a8b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7feacc098280 tx=0x7feacc09bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80406f440bbea1f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.622+0000 7feacaffd640 1 -- 172.21.15.94:0/2473975696 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc09a120 con 0x555c7932a4d0 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.622+0000 7feacaffd640 1 -- 172.21.15.94:0/2473975696 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feacc0cd690 con 0x555c7932a4d0 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.622+0000 7feacaffd640 1 -- 172.21.15.94:0/2473975696 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc0bc0f0 con 0x555c7932a4d0 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.622+0000 7feadddfb740 1 -- 172.21.15.94:0/2473975696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 msgr2=0x555c7932a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.622+0000 7feadddfb740 1 --2- 172.21.15.94:0/2473975696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7932a8b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7feacc098280 tx=0x7feacc09bee0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 -- 172.21.15.94:0/2473975696 shutdown_connections 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 --2- 172.21.15.94:0/2473975696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c7932ae80 0x555c79333430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 --2- 172.21.15.94:0/2473975696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c79333a00 0x555c79335e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 --2- 172.21.15.94:0/2473975696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7932a8b0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 -- 172.21.15.94:0/2473975696 >> 172.21.15.94:0/2473975696 conn(0x555c79324ba0 msgr2=0x555c79324f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 -- 172.21.15.94:0/2473975696 shutdown_connections 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 -- 172.21.15.94:0/2473975696 wait complete. 2024-04-21T08:44:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 -- start start 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 Processor -- start 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c7932a4d0 0x555c79333350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c793306e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c79333a00 0x555c79330cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x555c79336970 con 0x555c7932ae80 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555c79248500 con 0x555c79333a00 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555c79336af0 con 0x555c7932a4d0 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7fead3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c793306e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7feadddfb740 10 client.0 _ll_register_callbacks cb 0x555c79216b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7fead3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c793306e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47866/0 (socket says 172.21.15.94:47866) 2024-04-21T08:44:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7fead3fff640 1 -- 172.21.15.94:0/1314143195 learned_addr learned my addr 172.21.15.94:0/1314143195 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.623+0000 7fead8ec7640 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c7932a4d0 0x555c79333350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7fead96c8640 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c79333a00 0x555c79330cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7fead3fff640 1 -- 172.21.15.94:0/1314143195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c7932a4d0 msgr2=0x555c79333350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7fead3fff640 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c7932a4d0 0x555c79333350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7fead3fff640 1 -- 172.21.15.94:0/1314143195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c79333a00 msgr2=0x555c79330cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7fead3fff640 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c79333a00 0x555c79330cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7fead3fff640 1 -- 172.21.15.94:0/1314143195 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c78fa13f0 con 0x555c7932ae80 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7fead3fff640 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c793306e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7feabc013860 tx=0x7feabc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84fab1b6b994249 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7feab3fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7feab3fff640 1 -- 172.21.15.94:0/1314143195 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc01f930 con 0x555c7932ae80 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7feab3fff640 1 -- 172.21.15.94:0/1314143195 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7feabc01fad0 con 0x555c7932ae80 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7feadddfb740 1 -- 172.21.15.94:0/1314143195 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feacc0cd690 con 0x555c7932ae80 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7feab3fff640 1 -- 172.21.15.94:0/1314143195 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7feabc01fd10 con 0x555c7932ae80 2024-04-21T08:44:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7feadddfb740 10 client.17037 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7feadddfb740 1 -- 172.21.15.94:0/1314143195 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555c7906b7d0 con 0x555c7932ae80 2024-04-21T08:44:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.624+0000 7feab3fff640 1 -- 172.21.15.94:0/1314143195 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feacc0cd690 con 0x555c7932ae80 2024-04-21T08:44:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.625+0000 7feab3fff640 1 -- 172.21.15.94:0/1314143195 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7feabc026070 con 0x555c7932ae80 2024-04-21T08:44:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.625+0000 7feadddfb740 10 client.17037 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.625+0000 7feadddfb740 10 client.17037 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:44:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.625+0000 7feadddfb740 20 client.17037 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.625+0000 7feadddfb740 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x555c79343c50 0x555c79346070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.625+0000 7feadddfb740 4 client.17037 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.625+0000 7feadddfb740 1 -- 172.21.15.94:0/1314143195 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c7922e930 con 0x555c79343c50 2024-04-21T08:44:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.625+0000 7fead8ec7640 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x555c79343c50 0x555c79346070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.625+0000 7fead8ec7640 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x555c79343c50 0x555c79346070 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3132cb6036c19f83 server_cookie=2e60d4cbf4e299d7 in_seq=0 out_seq=0 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.625+0000 7feab3fff640 10 client.17037 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feab3fff640 1 -- 172.21.15.94:0/1314143195 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c7922e930 con 0x555c79343c50 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feab3fff640 10 client.17037 handle_command_reply: tid=0 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 client.17037 shutdown 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 -- 172.21.15.94:0/1314143195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 msgr2=0x555c793306e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c793306e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7feabc013860 tx=0x7feabc004730 comp rx=0 tx=0).stop 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 -- 172.21.15.94:0/1314143195 shutdown_connections 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x555c79343c50 0x555c79346070 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c79333a00 0x555c79330cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feab3fff640 0 client.17037 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c793306e0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 --2- 172.21.15.94:0/1314143195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c7932a4d0 0x555c79333350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 -- 172.21.15.94:0/1314143195 >> 172.21.15.94:0/1314143195 conn(0x555c79324ba0 msgr2=0x555c793268f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 -- 172.21.15.94:0/1314143195 shutdown_connections 2024-04-21T08:44:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 1 -- 172.21.15.94:0/1314143195 wait complete. 2024-04-21T08:44:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.626+0000 7feadddfb740 20 client.17037 trim_cache size 0 max 16384 2024-04-21T08:44:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.639+0000 7feadddfb740 1 Processor -- start 2024-04-21T08:44:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.639+0000 7feadddfb740 1 -- start start 2024-04-21T08:44:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7feadddfb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7933f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7feadddfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c7932ae80 0x555c79332e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7feadddfb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c79333a00 0x555c79333350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7feadddfb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fead41b1db0 con 0x555c7932a4d0 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7feadddfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fead41af680 con 0x555c7932ae80 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7feadddfb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fead41ad4d0 con 0x555c79333a00 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7fead8ec7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7933f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7fead8ec7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7933f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47882/0 (socket says 172.21.15.94:47882) 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7fead8ec7640 1 -- 172.21.15.94:0/1577871018 learned_addr learned my addr 172.21.15.94:0/1577871018 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7fead96c8640 1 --2- 172.21.15.94:0/1577871018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c79333a00 0x555c79333350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7fead8ec7640 1 -- 172.21.15.94:0/1577871018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c79333a00 msgr2=0x555c79333350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7fead8ec7640 1 --2- 172.21.15.94:0/1577871018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c79333a00 0x555c79333350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7fead8ec7640 1 -- 172.21.15.94:0/1577871018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c7932ae80 msgr2=0x555c79332e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7fead8ec7640 1 --2- 172.21.15.94:0/1577871018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c7932ae80 0x555c79332e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7fead8ec7640 1 -- 172.21.15.94:0/1577871018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c793316f0 con 0x555c7932a4d0 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7fead8ec7640 1 --2- 172.21.15.94:0/1577871018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7933f830 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7feacc00ba20 tx=0x7feacc00e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5beb3cbb7214e595 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7feaca7fc640 1 -- 172.21.15.94:0/1577871018 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc0980c0 con 0x555c7932a4d0 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7feaca7fc640 1 -- 172.21.15.94:0/1577871018 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feacc0cd600 con 0x555c7932a4d0 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.640+0000 7feaca7fc640 1 -- 172.21.15.94:0/1577871018 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc0a4590 con 0x555c7932a4d0 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 -- 172.21.15.94:0/1577871018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 msgr2=0x555c7933f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 --2- 172.21.15.94:0/1577871018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7933f830 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7feacc00ba20 tx=0x7feacc00e040 comp rx=0 tx=0).stop 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 -- 172.21.15.94:0/1577871018 shutdown_connections 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 --2- 172.21.15.94:0/1577871018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c79333a00 0x555c79333350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 --2- 172.21.15.94:0/1577871018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c7932ae80 0x555c79332e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 --2- 172.21.15.94:0/1577871018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932a4d0 0x555c7933f830 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 -- 172.21.15.94:0/1577871018 >> 172.21.15.94:0/1577871018 conn(0x555c79324ba0 msgr2=0x555c7932bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 -- 172.21.15.94:0/1577871018 shutdown_connections 2024-04-21T08:44:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 -- 172.21.15.94:0/1577871018 wait complete. 2024-04-21T08:44:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 -- start start 2024-04-21T08:44:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.641+0000 7feadddfb740 1 Processor -- start 2024-04-21T08:44:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feadddfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c7932a4d0 0x555c7932d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feadddfb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c7932d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feadddfb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c79333a00 0x555c7932dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feadddfb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fead41adb40 con 0x555c7932ae80 2024-04-21T08:44:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feadddfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fead41aefb0 con 0x555c7932a4d0 2024-04-21T08:44:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feadddfb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fead41ae210 con 0x555c79333a00 2024-04-21T08:44:55.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feadddfb740 10 client.0 _ll_register_callbacks cb 0x7feabc013530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c7932d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c7932d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47894/0 (socket says 172.21.15.94:47894) 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead3fff640 1 -- 172.21.15.94:0/2967912285 learned_addr learned my addr 172.21.15.94:0/2967912285 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead8ec7640 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c7932a4d0 0x555c7932d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead96c8640 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c79333a00 0x555c7932dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead3fff640 1 -- 172.21.15.94:0/2967912285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c79333a00 msgr2=0x555c7932dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead3fff640 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c79333a00 0x555c7932dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead3fff640 1 -- 172.21.15.94:0/2967912285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c7932a4d0 msgr2=0x555c7932d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead3fff640 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c7932a4d0 0x555c7932d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead3fff640 1 -- 172.21.15.94:0/2967912285 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c793a92e0 con 0x555c7932ae80 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead8ec7640 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c7932a4d0 0x555c7932d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7fead3fff640 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c7932d980 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7feabc013830 tx=0x7feabc013380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8df4706320fb49 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feab3fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feab3fff640 1 -- 172.21.15.94:0/2967912285 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc00b970 con 0x555c7932ae80 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feab3fff640 1 -- 172.21.15.94:0/2967912285 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7feabc011ad0 con 0x555c7932ae80 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feadddfb740 1 -- 172.21.15.94:0/2967912285 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feacc0cd600 con 0x555c7932ae80 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feab3fff640 1 -- 172.21.15.94:0/2967912285 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7feabc011cf0 con 0x555c7932ae80 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feadddfb740 10 client.17043 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.642+0000 7feadddfb740 1 -- 172.21.15.94:0/2967912285 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555c793aa5c0 con 0x555c7932ae80 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.643+0000 7feab3fff640 1 -- 172.21.15.94:0/2967912285 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feacc0cd600 con 0x555c7932ae80 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.643+0000 7feab3fff640 1 -- 172.21.15.94:0/2967912285 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7feabc026040 con 0x555c7932ae80 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.643+0000 7feadddfb740 10 client.17043 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.643+0000 7feadddfb740 10 client.17043 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.643+0000 7feadddfb740 20 client.17043 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.643+0000 7feadddfb740 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x555c793aad60 0x555c793ad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.643+0000 7feadddfb740 4 client.17043 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.643+0000 7feadddfb740 1 -- 172.21.15.94:0/2967912285 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555c7929cc10 con 0x555c793aad60 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.643+0000 7fead8ec7640 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x555c793aad60 0x555c793ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feab3fff640 10 client.17043 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7fead8ec7640 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x555c793aad60 0x555c793ad180 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fd1e7acd43aa7dad server_cookie=40a44a6dc194d50e in_seq=0 out_seq=0 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feab3fff640 1 -- 172.21.15.94:0/2967912285 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555c7929cc10 con 0x555c793aad60 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feab3fff640 10 client.17043 handle_command_reply: tid=0 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 client.17043 shutdown 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 -- 172.21.15.94:0/2967912285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 msgr2=0x555c7932d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c7932d980 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7feabc013830 tx=0x7feabc013380 comp rx=0 tx=0).stop 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 -- 172.21.15.94:0/2967912285 shutdown_connections 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x555c793aad60 0x555c793ad180 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c79333a00 0x555c7932dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feab3fff640 0 client.17043 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:44:55.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555c7932ae80 0x555c7932d980 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 --2- 172.21.15.94:0/2967912285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c7932a4d0 0x555c7932d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 -- 172.21.15.94:0/2967912285 >> 172.21.15.94:0/2967912285 conn(0x555c79324ba0 msgr2=0x555c7932b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 -- 172.21.15.94:0/2967912285 shutdown_connections 2024-04-21T08:44:55.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 1 -- 172.21.15.94:0/2967912285 wait complete. 2024-04-21T08:44:55.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.644+0000 7feadddfb740 20 client.17043 trim_cache size 0 max 16384 2024-04-21T08:44:55.651 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:44:55.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 -- 172.21.15.94:0/3825634023 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feac00a3d20 msgr2=0x7feac00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feac00a3d20 0x7feac00a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fead406b050 tx=0x7feabc0103d0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 -- 172.21.15.94:0/3825634023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fead41011a0 msgr2=0x7fead4199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fead41011a0 0x7fead4199640 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7feacc00ba50 tx=0x7feacc00bf10 comp rx=0 tx=0).stop 2024-04-21T08:44:55.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 -- 172.21.15.94:0/3825634023 shutdown_connections 2024-04-21T08:44:55.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7feac00a3d20 0x7feac00a61e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fead4107030 0x7fead41a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fead4101b40 0x7fead4199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 --2- 172.21.15.94:0/3825634023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fead41011a0 0x7fead4199640 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 -- 172.21.15.94:0/3825634023 >> 172.21.15.94:0/3825634023 conn(0x7fead4076c50 msgr2=0x7fead40ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 -- 172.21.15.94:0/3825634023 shutdown_connections 2024-04-21T08:44:55.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.645+0000 7feadb12b640 1 -- 172.21.15.94:0/3825634023 wait complete. 2024-04-21T08:44:55.654 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:44:55.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.756+0000 7fd72c470640 1 -- 172.21.15.94:0/1961762032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd72406a820 msgr2=0x7fd72406ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.756+0000 7fd72c470640 1 --2- 172.21.15.94:0/1961762032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd72406a820 0x7fd72406ac00 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fd720009ca0 tx=0x7fd72002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.756+0000 7fd72c470640 1 -- 172.21.15.94:0/1961762032 shutdown_connections 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.756+0000 7fd72c470640 1 --2- 172.21.15.94:0/1961762032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd72406b1d0 0x7fd724106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.756+0000 7fd72c470640 1 --2- 172.21.15.94:0/1961762032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd724107270 0x7fd72410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.756+0000 7fd72c470640 1 --2- 172.21.15.94:0/1961762032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd72406a820 0x7fd72406ac00 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.756+0000 7fd72c470640 1 -- 172.21.15.94:0/1961762032 >> 172.21.15.94:0/1961762032 conn(0x7fd724077eb0 msgr2=0x7fd7240782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.756+0000 7fd72c470640 1 -- 172.21.15.94:0/1961762032 shutdown_connections 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.756+0000 7fd72c470640 1 -- 172.21.15.94:0/1961762032 wait complete. 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd72c470640 1 Processor -- start 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd72c470640 1 -- start start 2024-04-21T08:44:55.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd72c470640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd72406a820 0x7fd7241986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd72c470640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd724107270 0x7fd724198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd72c470640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd72419cf90 0x7fd72419d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd72c470640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241121b0 con 0x7fd724107270 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd72c470640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd724112030 con 0x7fd72419cf90 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd72c470640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd724112330 con 0x7fd72406a820 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd729a0b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd724107270 0x7fd724198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd729a0b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd724107270 0x7fd724198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47908/0 (socket says 172.21.15.94:47908) 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.757+0000 7fd729a0b640 1 -- 172.21.15.94:0/1986987707 learned_addr learned my addr 172.21.15.94:0/1986987707 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd72aa0d640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd72419cf90 0x7fd72419d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd729a0b640 1 -- 172.21.15.94:0/1986987707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd72406a820 msgr2=0x7fd7241986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd729a0b640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd72406a820 0x7fd7241986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd729a0b640 1 -- 172.21.15.94:0/1986987707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd72419cf90 msgr2=0x7fd72419d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd729a0b640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd72419cf90 0x7fd72419d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd729a0b640 1 -- 172.21.15.94:0/1986987707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72419da80 con 0x7fd724107270 2024-04-21T08:44:55.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd729a0b640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd724107270 0x7fd724198c10 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd714005fd0 tx=0x7fd71400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd71b7fe640 1 -- 172.21.15.94:0/1986987707 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714017070 con 0x7fd724107270 2024-04-21T08:44:55.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd72c470640 1 -- 172.21.15.94:0/1986987707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72419dd70 con 0x7fd724107270 2024-04-21T08:44:55.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd71b7fe640 1 -- 172.21.15.94:0/1986987707 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd714009c30 con 0x7fd724107270 2024-04-21T08:44:55.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd71b7fe640 1 -- 172.21.15.94:0/1986987707 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714003d30 con 0x7fd724107270 2024-04-21T08:44:55.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.758+0000 7fd72c470640 1 -- 172.21.15.94:0/1986987707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7241a55b0 con 0x7fd724107270 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.760+0000 7fd71b7fe640 1 -- 172.21.15.94:0/1986987707 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd714003ed0 con 0x7fd724107270 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd72f05c740 1 Processor -- start 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd72f05c740 1 -- start start 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aa34d0 0x55c857aa38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aac430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857aaee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd72f05c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c857a081f0 con 0x55c857aaca00 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd72f05c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c8579a54b0 con 0x55c857aa34d0 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd71b7fe640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd7100a3d20 0x7fd7100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd72f05c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c857a041b0 con 0x55c857aa3e80 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd729a0b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857aaee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd72aa0d640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd7100a3d20 0x7fd7100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd729a0b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857aaee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47918/0 (socket says 172.21.15.94:47918) 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd729a0b640 1 -- 172.21.15.94:0/2913958413 learned_addr learned my addr 172.21.15.94:0/2913958413 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd72aa0d640 1 --2- 172.21.15.94:0/2913958413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aa34d0 0x55c857aa38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd71b7fe640 1 -- 172.21.15.94:0/1986987707 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd7140d08e0 con 0x7fd724107270 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd729a0b640 1 -- 172.21.15.94:0/2913958413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 msgr2=0x55c857aac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd729a0b640 1 --2- 172.21.15.94:0/2913958413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd729a0b640 1 -- 172.21.15.94:0/2913958413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aa34d0 msgr2=0x55c857aa38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.761+0000 7fd729a0b640 1 --2- 172.21.15.94:0/2913958413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aa34d0 0x55c857aa38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.762+0000 7fd729a0b640 1 -- 172.21.15.94:0/2913958413 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8579971b0 con 0x55c857aaca00 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.762+0000 7fd729a0b640 1 --2- 172.21.15.94:0/2913958413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857aaee50 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd714098e70 tx=0x7fd71406bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a426fc6be672af24 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.762+0000 7fd72aa0d640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd7100a3d20 0x7fd7100a61e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd71c005d80 tx=0x7fd71c00ca70 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.762+0000 7fd718bfe640 1 -- 172.21.15.94:0/2913958413 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714009de0 con 0x55c857aaca00 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.762+0000 7fd718bfe640 1 -- 172.21.15.94:0/2913958413 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd71409c4a0 con 0x55c857aaca00 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.762+0000 7fd718bfe640 1 -- 172.21.15.94:0/2913958413 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714098b10 con 0x55c857aaca00 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.762+0000 7fd72f05c740 1 -- 172.21.15.94:0/2913958413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 msgr2=0x55c857aaee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.762+0000 7fd72f05c740 1 --2- 172.21.15.94:0/2913958413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857aaee50 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd714098e70 tx=0x7fd71406bd80 comp rx=0 tx=0).stop 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 -- 172.21.15.94:0/2913958413 shutdown_connections 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 --2- 172.21.15.94:0/2913958413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 --2- 172.21.15.94:0/2913958413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857aaee50 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 --2- 172.21.15.94:0/2913958413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aa34d0 0x55c857aa38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 -- 172.21.15.94:0/2913958413 >> 172.21.15.94:0/2913958413 conn(0x55c857a9dba0 msgr2=0x55c857a9df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 -- 172.21.15.94:0/2913958413 shutdown_connections 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 -- 172.21.15.94:0/2913958413 wait complete. 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 -- start start 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 Processor -- start 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857aac380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aa9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 0x55c857aa9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c857aaf970 con 0x55c857aa34d0 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c8579c1500 con 0x55c857aaca00 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c857aafaf0 con 0x55c857aa3e80 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72f05c740 10 client.0 _ll_register_callbacks cb 0x55c85798fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72a20c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857aac380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72a20c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857aac380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47932/0 (socket says 172.21.15.94:47932) 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72a20c640 1 -- 172.21.15.94:0/644860307 learned_addr learned my addr 172.21.15.94:0/644860307 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd729a0b640 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aa9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.763+0000 7fd72aa0d640 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 0x55c857aa9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72a20c640 1 -- 172.21.15.94:0/644860307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 msgr2=0x55c857aa9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72a20c640 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aa9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72a20c640 1 -- 172.21.15.94:0/644860307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 msgr2=0x55c857aa9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72a20c640 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 0x55c857aa9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72a20c640 1 -- 172.21.15.94:0/644860307 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c85771a3f0 con 0x55c857aa34d0 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72a20c640 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857aac380 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd720004af0 tx=0x7fd720005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3b71dcde756db5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd70dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd70dffb640 1 -- 172.21.15.94:0/644860307 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720041020 con 0x55c857aa34d0 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd70dffb640 1 -- 172.21.15.94:0/644860307 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd720032740 con 0x55c857aa34d0 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72f05c740 1 -- 172.21.15.94:0/644860307 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd71409c4a0 con 0x55c857aa34d0 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd70dffb640 1 -- 172.21.15.94:0/644860307 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd72004c050 con 0x55c857aa34d0 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72f05c740 10 client.17055 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72f05c740 1 -- 172.21.15.94:0/644860307 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c8577e47d0 con 0x55c857aa34d0 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd70dffb640 1 -- 172.21.15.94:0/644860307 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd71409c4a0 con 0x55c857aa34d0 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd70dffb640 1 -- 172.21.15.94:0/644860307 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd72003c030 con 0x55c857aa34d0 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72f05c740 10 client.17055 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72f05c740 10 client.17055 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72f05c740 20 client.17055 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.764+0000 7fd72f05c740 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c857abcce0 0x55c857abf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.765+0000 7fd72f05c740 4 client.17055 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.765+0000 7fd72f05c740 1 -- 172.21.15.94:0/644860307 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8579a7930 con 0x55c857abcce0 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.765+0000 7fd729a0b640 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c857abcce0 0x55c857abf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.765+0000 7fd70dffb640 10 client.17055 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.765+0000 7fd729a0b640 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c857abcce0 0x55c857abf130 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=31fa7bba1fb8a854 server_cookie=184909d66525f611 in_seq=0 out_seq=0 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd70dffb640 1 -- 172.21.15.94:0/644860307 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8579a7930 con 0x55c857abcce0 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd70dffb640 10 client.17055 handle_command_reply: tid=0 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 client.17055 shutdown 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 -- 172.21.15.94:0/644860307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 msgr2=0x55c857aac380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857aac380 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd720004af0 tx=0x7fd720005f60 comp rx=0 tx=0).stop 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 -- 172.21.15.94:0/644860307 shutdown_connections 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aa9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c857abcce0 0x55c857abf130 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 0x55c857aa9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd70dffb640 0 client.17055 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:44:55.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 --2- 172.21.15.94:0/644860307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857aac380 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 -- 172.21.15.94:0/644860307 >> 172.21.15.94:0/644860307 conn(0x55c857a9dba0 msgr2=0x55c857a9f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 -- 172.21.15.94:0/644860307 shutdown_connections 2024-04-21T08:44:55.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 1 -- 172.21.15.94:0/644860307 wait complete. 2024-04-21T08:44:55.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.766+0000 7fd72f05c740 20 client.17055 trim_cache size 0 max 16384 2024-04-21T08:44:55.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.779+0000 7fd72f05c740 1 Processor -- start 2024-04-21T08:44:55.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.779+0000 7fd72f05c740 1 -- start start 2024-04-21T08:44:55.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.779+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aa34d0 0x55c857ab8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.779+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857ac49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.779+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857ac4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.779+0000 7fd72f05c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241b1ca0 con 0x55c857aaca00 2024-04-21T08:44:55.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.779+0000 7fd72f05c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241af570 con 0x55c857aa34d0 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.779+0000 7fd72f05c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241ad3c0 con 0x55c857aa3e80 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd72aa0d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857ac4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd72aa0d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857ac4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47944/0 (socket says 172.21.15.94:47944) 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd72aa0d640 1 -- 172.21.15.94:0/469555577 learned_addr learned my addr 172.21.15.94:0/469555577 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd72a20c640 1 --2- 172.21.15.94:0/469555577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aa34d0 0x55c857ab8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd729a0b640 1 --2- 172.21.15.94:0/469555577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857ac49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd72aa0d640 1 -- 172.21.15.94:0/469555577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 msgr2=0x55c857ac49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd72aa0d640 1 --2- 172.21.15.94:0/469555577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857ac49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd72aa0d640 1 -- 172.21.15.94:0/469555577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aa34d0 msgr2=0x55c857ab8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd72aa0d640 1 --2- 172.21.15.94:0/469555577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aa34d0 0x55c857ab8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd72aa0d640 1 -- 172.21.15.94:0/469555577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c857a9d3f0 con 0x55c857aaca00 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd729a0b640 1 --2- 172.21.15.94:0/469555577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857ac49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd72aa0d640 1 --2- 172.21.15.94:0/469555577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857ac4f30 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00cf10 tx=0x7fd71c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4428d33a17e50947 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd70ffff640 1 -- 172.21.15.94:0/469555577 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c026070 con 0x55c857aaca00 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd70ffff640 1 -- 172.21.15.94:0/469555577 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd71c02bb10 con 0x55c857aaca00 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.780+0000 7fd70ffff640 1 -- 172.21.15.94:0/469555577 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c012af0 con 0x55c857aaca00 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 -- 172.21.15.94:0/469555577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 msgr2=0x55c857ac4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 --2- 172.21.15.94:0/469555577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857ac4f30 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00cf10 tx=0x7fd71c004730 comp rx=0 tx=0).stop 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 -- 172.21.15.94:0/469555577 shutdown_connections 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 --2- 172.21.15.94:0/469555577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857ac49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 --2- 172.21.15.94:0/469555577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aaca00 0x55c857ac4f30 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 --2- 172.21.15.94:0/469555577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aa34d0 0x55c857ab8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 -- 172.21.15.94:0/469555577 >> 172.21.15.94:0/469555577 conn(0x55c857a9dba0 msgr2=0x55c857a9ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 -- 172.21.15.94:0/469555577 shutdown_connections 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 -- 172.21.15.94:0/469555577 wait complete. 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 -- start start 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 Processor -- start 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857ab5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.781+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aa6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72f05c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 0x55c857aa6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72f05c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241ada30 con 0x55c857aa34d0 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72f05c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241aeea0 con 0x55c857aaca00 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72f05c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241ae100 con 0x55c857aa3e80 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72f05c740 10 client.0 _ll_register_callbacks cb 0x7fd720032380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72a20c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857ab5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72a20c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857ab5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47952/0 (socket says 172.21.15.94:47952) 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72a20c640 1 -- 172.21.15.94:0/578817987 learned_addr learned my addr 172.21.15.94:0/578817987 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd729a0b640 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aa6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72aa0d640 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 0x55c857aa6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72a20c640 1 -- 172.21.15.94:0/578817987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 msgr2=0x55c857aa6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72a20c640 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aa6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72a20c640 1 -- 172.21.15.94:0/578817987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 msgr2=0x55c857aa6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72a20c640 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 0x55c857aa6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72a20c640 1 -- 172.21.15.94:0/578817987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c857aa6e50 con 0x55c857aa34d0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72aa0d640 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 0x55c857aa6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72a20c640 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857ab5130 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd720002230 tx=0x7fd720004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5f2eabbc6644ff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd70dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd70dffb640 1 -- 172.21.15.94:0/578817987 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720041070 con 0x55c857aa34d0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72f05c740 1 -- 172.21.15.94:0/578817987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd71c02bb10 con 0x55c857aa34d0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd70dffb640 1 -- 172.21.15.94:0/578817987 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd720034440 con 0x55c857aa34d0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd70dffb640 1 -- 172.21.15.94:0/578817987 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd720039040 con 0x55c857aa34d0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72f05c740 10 client.17061 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd72f05c740 1 -- 172.21.15.94:0/578817987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c857b23270 con 0x55c857aa34d0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.782+0000 7fd70dffb640 1 -- 172.21.15.94:0/578817987 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd71c02bb10 con 0x55c857aa34d0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd70dffb640 1 -- 172.21.15.94:0/578817987 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd72003c040 con 0x55c857aa34d0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd72f05c740 10 client.17061 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd72f05c740 10 client.17061 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd72f05c740 20 client.17061 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd72f05c740 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c857b23a10 0x55c857b25e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd72f05c740 4 client.17061 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd72f05c740 1 -- 172.21.15.94:0/578817987 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c857a15c10 con 0x55c857b23a10 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd729a0b640 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c857b23a10 0x55c857b25e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd70dffb640 10 client.17061 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd729a0b640 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c857b23a10 0x55c857b25e30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be61604ddc709105 server_cookie=63906bd4aeee974c in_seq=0 out_seq=0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.783+0000 7fd70dffb640 1 -- 172.21.15.94:0/578817987 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c857a15c10 con 0x55c857b23a10 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd70dffb640 10 client.17061 handle_command_reply: tid=0 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 client.17061 shutdown 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 -- 172.21.15.94:0/578817987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 msgr2=0x55c857ab5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857ab5130 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd720002230 tx=0x7fd720004230 comp rx=0 tx=0).stop 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 -- 172.21.15.94:0/578817987 shutdown_connections 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c857aa3e80 0x55c857aa6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c857b23a10 0x55c857b25e30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c857aaca00 0x55c857aa6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 --2- 172.21.15.94:0/578817987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c857aa34d0 0x55c857ab5130 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd70dffb640 0 client.17061 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 -- 172.21.15.94:0/578817987 >> 172.21.15.94:0/578817987 conn(0x55c857a9dba0 msgr2=0x55c857a9f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 -- 172.21.15.94:0/578817987 shutdown_connections 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 1 -- 172.21.15.94:0/578817987 wait complete. 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72f05c740 20 client.17061 trim_cache size 0 max 16384 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 -- 172.21.15.94:0/1986987707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd7100a3d20 msgr2=0x7fd7100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd7100a3d20 0x7fd7100a61e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd71c005d80 tx=0x7fd71c00ca70 comp rx=0 tx=0).stop 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 -- 172.21.15.94:0/1986987707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd724107270 msgr2=0x7fd724198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd724107270 0x7fd724198c10 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd714005fd0 tx=0x7fd71400dde0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 -- 172.21.15.94:0/1986987707 shutdown_connections 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd7100a3d20 0x7fd7100a61e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd72419cf90 0x7fd72419d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd724107270 0x7fd724198c10 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 --2- 172.21.15.94:0/1986987707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd72406a820 0x7fd7241986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 -- 172.21.15.94:0/1986987707 >> 172.21.15.94:0/1986987707 conn(0x7fd724077eb0 msgr2=0x7fd724108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 -- 172.21.15.94:0/1986987707 shutdown_connections 2024-04-21T08:44:55.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.784+0000 7fd72c470640 1 -- 172.21.15.94:0/1986987707 wait complete. 2024-04-21T08:44:55.794 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:55 smithi094 ceph-mon[27988]: pgmap v151: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:44:55.794 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:55 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2942909778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:55.795 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:44:55.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.896+0000 7fd82f631640 1 -- 172.21.15.94:0/1834067815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd828101a50 msgr2=0x7fd828105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.896+0000 7fd82f631640 1 --2- 172.21.15.94:0/1834067815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd828101a50 0x7fd828105a90 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd824009900 tx=0x7fd82402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:55.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.896+0000 7fd82f631640 1 -- 172.21.15.94:0/1834067815 shutdown_connections 2024-04-21T08:44:55.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.896+0000 7fd82f631640 1 --2- 172.21.15.94:0/1834067815 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd828106170 0x7fd82810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.896+0000 7fd82f631640 1 --2- 172.21.15.94:0/1834067815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd828101a50 0x7fd828105a90 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.896+0000 7fd82f631640 1 --2- 172.21.15.94:0/1834067815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8281010a0 0x7fd828101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.896+0000 7fd82f631640 1 -- 172.21.15.94:0/1834067815 >> 172.21.15.94:0/1834067815 conn(0x7fd828077eb0 msgr2=0x7fd8280782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.896+0000 7fd82f631640 1 -- 172.21.15.94:0/1834067815 shutdown_connections 2024-04-21T08:44:55.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.896+0000 7fd82f631640 1 -- 172.21.15.94:0/1834067815 wait complete. 2024-04-21T08:44:55.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82f631640 1 Processor -- start 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82f631640 1 -- start start 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82f631640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8281010a0 0x7fd82810e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82f631640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd828106170 0x7fd82810ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82f631640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd82810f150 0x7fd8281a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82f631640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd828112290 con 0x7fd82810f150 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82f631640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd828112110 con 0x7fd8281010a0 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82f631640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd828112410 con 0x7fd828106170 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82dbce640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd82810f150 0x7fd8281a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82dbce640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd82810f150 0x7fd8281a6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47984/0 (socket says 172.21.15.94:47984) 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82dbce640 1 -- 172.21.15.94:0/2128968783 learned_addr learned my addr 172.21.15.94:0/2128968783 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8281010a0 0x7fd82810e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd828106170 0x7fd82810ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82dbce640 1 -- 172.21.15.94:0/2128968783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd828106170 msgr2=0x7fd82810ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82dbce640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd828106170 0x7fd82810ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82dbce640 1 -- 172.21.15.94:0/2128968783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8281010a0 msgr2=0x7fd82810e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82dbce640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8281010a0 0x7fd82810e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.897+0000 7fd82dbce640 1 -- 172.21.15.94:0/2128968783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8281a7160 con 0x7fd82810f150 2024-04-21T08:44:55.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.898+0000 7fd82dbce640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd82810f150 0x7fd8281a6a60 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd82000b970 tx=0x7fd82000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.898+0000 7fd81e7fc640 1 -- 172.21.15.94:0/2128968783 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd820019070 con 0x7fd82810f150 2024-04-21T08:44:55.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.898+0000 7fd82f631640 1 -- 172.21.15.94:0/2128968783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8281a7450 con 0x7fd82810f150 2024-04-21T08:44:55.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.898+0000 7fd81e7fc640 1 -- 172.21.15.94:0/2128968783 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd820014440 con 0x7fd82810f150 2024-04-21T08:44:55.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.898+0000 7fd81e7fc640 1 -- 172.21.15.94:0/2128968783 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8200075d0 con 0x7fd82810f150 2024-04-21T08:44:55.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.898+0000 7fd82f631640 1 -- 172.21.15.94:0/2128968783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8281a7970 con 0x7fd82810f150 2024-04-21T08:44:55.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd81e7fc640 1 -- 172.21.15.94:0/2128968783 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd82001c070 con 0x7fd82810f150 2024-04-21T08:44:55.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd83221e740 1 Processor -- start 2024-04-21T08:44:55.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd83221e740 1 -- start start 2024-04-21T08:44:55.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd83221e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9043304d0 0x55c9043308b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd83221e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904330e80 0x55c904339430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd83221e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904339a00 0x55c90433be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd83221e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c9042951f0 con 0x55c904330e80 2024-04-21T08:44:55.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd83221e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c9042324b0 con 0x55c904339a00 2024-04-21T08:44:55.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd83221e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c9042911b0 con 0x55c9043304d0 2024-04-21T08:44:55.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82d3cd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904330e80 0x55c904339430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82d3cd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904330e80 0x55c904339430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47996/0 (socket says 172.21.15.94:47996) 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82d3cd640 1 -- 172.21.15.94:0/3888526494 learned_addr learned my addr 172.21.15.94:0/3888526494 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd81e7fc640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd8140a3d50 0x7fd8140a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/3888526494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904339a00 0x55c90433be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82dbce640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd8140a3d50 0x7fd8140a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/3888526494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9043304d0 0x55c9043308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82d3cd640 1 -- 172.21.15.94:0/3888526494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9043304d0 msgr2=0x55c9043308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/3888526494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9043304d0 0x55c9043308b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82d3cd640 1 -- 172.21.15.94:0/3888526494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904339a00 msgr2=0x55c90433be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/3888526494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904339a00 0x55c90433be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82d3cd640 1 -- 172.21.15.94:0/3888526494 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9042241b0 con 0x55c904330e80 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.901+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/3888526494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9043304d0 0x55c9043308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.902+0000 7fd81e7fc640 1 -- 172.21.15.94:0/2128968783 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd8200cff10 con 0x7fd82810f150 2024-04-21T08:44:55.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.902+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/3888526494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904330e80 0x55c904339430 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd818009a70 tx=0x7fd818009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8794db2cdd3176fa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.902+0000 7fd82dbce640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd8140a3d50 0x7fd8140a6210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd82009b990 tx=0x7fd82006a640 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.902+0000 7fd8137fe640 1 -- 172.21.15.94:0/3888526494 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818017070 con 0x55c904330e80 2024-04-21T08:44:55.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.902+0000 7fd8137fe640 1 -- 172.21.15.94:0/3888526494 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd818013470 con 0x55c904330e80 2024-04-21T08:44:55.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.902+0000 7fd8137fe640 1 -- 172.21.15.94:0/3888526494 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818012630 con 0x55c904330e80 2024-04-21T08:44:55.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.902+0000 7fd83221e740 1 -- 172.21.15.94:0/3888526494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904330e80 msgr2=0x55c904339430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 --2- 172.21.15.94:0/3888526494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904330e80 0x55c904339430 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd818009a70 tx=0x7fd818009f30 comp rx=0 tx=0).stop 2024-04-21T08:44:55.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 -- 172.21.15.94:0/3888526494 shutdown_connections 2024-04-21T08:44:55.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 --2- 172.21.15.94:0/3888526494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904339a00 0x55c90433be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 --2- 172.21.15.94:0/3888526494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904330e80 0x55c904339430 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 --2- 172.21.15.94:0/3888526494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9043304d0 0x55c9043308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 -- 172.21.15.94:0/3888526494 >> 172.21.15.94:0/3888526494 conn(0x55c90432aba0 msgr2=0x55c90432af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 -- 172.21.15.94:0/3888526494 shutdown_connections 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 -- 172.21.15.94:0/3888526494 wait complete. 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 -- start start 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 Processor -- start 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c9043392e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c904330e80 0x55c904336670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904339a00 0x55c904336bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c90433c970 con 0x55c9043304d0 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c90424e500 con 0x55c904339a00 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c90433caf0 con 0x55c904330e80 2024-04-21T08:44:55.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd83221e740 10 client.0 _ll_register_callbacks cb 0x55c90421cb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.903+0000 7fd82cbcc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c9043392e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd82cbcc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c9043392e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48004/0 (socket says 172.21.15.94:48004) 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd82cbcc640 1 -- 172.21.15.94:0/900227649 learned_addr learned my addr 172.21.15.94:0/900227649 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c904330e80 0x55c904336670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd82cbcc640 1 -- 172.21.15.94:0/900227649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c904330e80 msgr2=0x55c904336670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c904330e80 0x55c904336670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd82cbcc640 1 -- 172.21.15.94:0/900227649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904339a00 msgr2=0x55c904336bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904339a00 0x55c904336bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd82cbcc640 1 -- 172.21.15.94:0/900227649 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c903fa73f0 con 0x55c9043304d0 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c9043392e0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd824009230 tx=0x7fd824009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe07a75b5914ea7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd810ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd810ff9640 1 -- 172.21.15.94:0/900227649 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824045070 con 0x55c9043304d0 2024-04-21T08:44:55.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd810ff9640 1 -- 172.21.15.94:0/900227649 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd82402cab0 con 0x55c9043304d0 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd83221e740 1 -- 172.21.15.94:0/900227649 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd818013470 con 0x55c9043304d0 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd810ff9640 1 -- 172.21.15.94:0/900227649 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd824005040 con 0x55c9043304d0 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd83221e740 10 client.17073 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd83221e740 1 -- 172.21.15.94:0/900227649 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c9040717d0 con 0x55c9043304d0 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.904+0000 7fd810ff9640 1 -- 172.21.15.94:0/900227649 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd818013470 con 0x55c9043304d0 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.905+0000 7fd810ff9640 1 -- 172.21.15.94:0/900227649 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd824004ce0 con 0x55c9043304d0 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.905+0000 7fd83221e740 10 client.17073 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.905+0000 7fd83221e740 10 client.17073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.905+0000 7fd83221e740 20 client.17073 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.905+0000 7fd83221e740 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55c904349a90 0x55c90434bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.905+0000 7fd83221e740 4 client.17073 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.905+0000 7fd83221e740 1 -- 172.21.15.94:0/900227649 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c904234930 con 0x55c904349a90 2024-04-21T08:44:55.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.905+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55c904349a90 0x55c90434bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.905+0000 7fd810ff9640 10 client.17073 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:55.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.905+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55c904349a90 0x55c90434bec0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6343df586d1e543 server_cookie=75da8fe9e8d418c6 in_seq=0 out_seq=0 2024-04-21T08:44:55.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.906+0000 7fd810ff9640 1 -- 172.21.15.94:0/900227649 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c904234930 con 0x55c904349a90 2024-04-21T08:44:55.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.906+0000 7fd810ff9640 10 client.17073 handle_command_reply: tid=0 2024-04-21T08:44:55.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.906+0000 7fd83221e740 1 client.17073 shutdown 2024-04-21T08:44:55.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.906+0000 7fd83221e740 1 -- 172.21.15.94:0/900227649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 msgr2=0x55c9043392e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.906+0000 7fd83221e740 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c9043392e0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd824009230 tx=0x7fd824009670 comp rx=0 tx=0).stop 2024-04-21T08:44:55.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.906+0000 7fd83221e740 1 -- 172.21.15.94:0/900227649 shutdown_connections 2024-04-21T08:44:55.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.906+0000 7fd83221e740 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55c904349a90 0x55c90434bec0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.907+0000 7fd83221e740 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c904330e80 0x55c904336670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.907+0000 7fd83221e740 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904339a00 0x55c904336bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.907+0000 7fd810ff9640 0 client.17073 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:55.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.907+0000 7fd83221e740 1 --2- 172.21.15.94:0/900227649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c9043392e0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.907+0000 7fd83221e740 1 -- 172.21.15.94:0/900227649 >> 172.21.15.94:0/900227649 conn(0x55c90432aba0 msgr2=0x55c90432c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.907+0000 7fd83221e740 1 -- 172.21.15.94:0/900227649 shutdown_connections 2024-04-21T08:44:55.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.907+0000 7fd83221e740 1 -- 172.21.15.94:0/900227649 wait complete. 2024-04-21T08:44:55.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.907+0000 7fd83221e740 20 client.17073 trim_cache size 0 max 16384 2024-04-21T08:44:55.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd83221e740 1 Processor -- start 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd83221e740 1 -- start start 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd83221e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c904345830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd83221e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904330e80 0x55c904338da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd83221e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c904339a00 0x55c9043392e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd83221e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281b4010 con 0x55c9043304d0 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd83221e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281b18e0 con 0x55c904330e80 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd83221e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281af730 con 0x55c904339a00 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd82cbcc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c904345830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd82cbcc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c904345830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48018/0 (socket says 172.21.15.94:48018) 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd82cbcc640 1 -- 172.21.15.94:0/2658349282 learned_addr learned my addr 172.21.15.94:0/2658349282 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd82cbcc640 1 -- 172.21.15.94:0/2658349282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c904339a00 msgr2=0x55c9043392e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/2658349282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c904339a00 0x55c9043392e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd82cbcc640 1 -- 172.21.15.94:0/2658349282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904330e80 msgr2=0x55c904338da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/2658349282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904330e80 0x55c904338da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/2658349282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904330e80 0x55c904338da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd82cbcc640 1 -- 172.21.15.94:0/2658349282 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c904330c20 con 0x55c9043304d0 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/2658349282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c904345830 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fd8240035f0 tx=0x7fd824004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9714616599bd7e5a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd812ffd640 1 -- 172.21.15.94:0/2658349282 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824045070 con 0x55c9043304d0 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd812ffd640 1 -- 172.21.15.94:0/2658349282 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd824005690 con 0x55c9043304d0 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.920+0000 7fd812ffd640 1 -- 172.21.15.94:0/2658349282 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824004920 con 0x55c9043304d0 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 -- 172.21.15.94:0/2658349282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 msgr2=0x55c904345830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 --2- 172.21.15.94:0/2658349282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c904345830 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fd8240035f0 tx=0x7fd824004610 comp rx=0 tx=0).stop 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 -- 172.21.15.94:0/2658349282 shutdown_connections 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 --2- 172.21.15.94:0/2658349282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c904339a00 0x55c9043392e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 --2- 172.21.15.94:0/2658349282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904330e80 0x55c904338da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 --2- 172.21.15.94:0/2658349282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c9043304d0 0x55c904345830 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 -- 172.21.15.94:0/2658349282 >> 172.21.15.94:0/2658349282 conn(0x55c90432aba0 msgr2=0x55c904329c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 -- 172.21.15.94:0/2658349282 shutdown_connections 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 -- 172.21.15.94:0/2658349282 wait complete. 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 -- start start 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.921+0000 7fd83221e740 1 Processor -- start 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd83221e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9043304d0 0x55c904342030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd83221e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904330e80 0x55c90433d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd83221e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904339a00 0x55c90433d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd83221e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281afda0 con 0x55c904339a00 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd83221e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281b1210 con 0x55c904330e80 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd83221e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281b0470 con 0x55c9043304d0 2024-04-21T08:44:55.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd83221e740 10 client.0 _ll_register_callbacks cb 0x7fd8240040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd82cbcc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904339a00 0x55c90433d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd82cbcc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904339a00 0x55c90433d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48030/0 (socket says 172.21.15.94:48030) 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd82cbcc640 1 -- 172.21.15.94:0/475312434 learned_addr learned my addr 172.21.15.94:0/475312434 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904330e80 0x55c90433d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd82cbcc640 1 -- 172.21.15.94:0/475312434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9043304d0 msgr2=0x55c904342030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9043304d0 0x55c904342030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd82cbcc640 1 -- 172.21.15.94:0/475312434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904330e80 msgr2=0x55c90433d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904330e80 0x55c90433d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd82cbcc640 1 -- 172.21.15.94:0/475312434 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c90433e020 con 0x55c904339a00 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd82cbcc640 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904339a00 0x55c90433d8e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fd824003860 tx=0x7fd82402cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886a1082351da30c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd810ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd810ff9640 1 -- 172.21.15.94:0/475312434 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824045070 con 0x55c904339a00 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd810ff9640 1 -- 172.21.15.94:0/475312434 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd8240053e0 con 0x55c904339a00 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.922+0000 7fd83221e740 1 -- 172.21.15.94:0/475312434 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c9042a2c10 con 0x55c904339a00 2024-04-21T08:44:55.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd810ff9640 1 -- 172.21.15.94:0/475312434 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd8240048d0 con 0x55c904339a00 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd83221e740 10 client.17079 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd83221e740 1 -- 172.21.15.94:0/475312434 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c9043aff30 con 0x55c904339a00 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd810ff9640 1 -- 172.21.15.94:0/475312434 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c9042a2c10 con 0x55c904339a00 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd810ff9640 1 -- 172.21.15.94:0/475312434 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd82405ebf0 con 0x55c904339a00 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd83221e740 10 client.17079 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd83221e740 10 client.17079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd83221e740 20 client.17079 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd83221e740 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55c9043b04a0 0x55c9043b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd83221e740 4 client.17079 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd83221e740 1 -- 172.21.15.94:0/475312434 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c90429ca80 con 0x55c9043b04a0 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.923+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55c9043b04a0 0x55c9043b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd82d3cd640 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55c9043b04a0 0x55c9043b28c0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f60c7c5f8366fd9 server_cookie=aa136f049b702072 in_seq=0 out_seq=0 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd810ff9640 10 client.17079 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:55.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd810ff9640 1 -- 172.21.15.94:0/475312434 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c90429ca80 con 0x55c9043b04a0 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd810ff9640 10 client.17079 handle_command_reply: tid=0 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 client.17079 shutdown 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 -- 172.21.15.94:0/475312434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904339a00 msgr2=0x55c90433d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904339a00 0x55c90433d8e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fd824003860 tx=0x7fd82402cdc0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 -- 172.21.15.94:0/475312434 shutdown_connections 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55c9043b04a0 0x55c9043b28c0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c904339a00 0x55c90433d8e0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c904330e80 0x55c90433d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd810ff9640 0 client.17079 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 --2- 172.21.15.94:0/475312434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9043304d0 0x55c904342030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 -- 172.21.15.94:0/475312434 >> 172.21.15.94:0/475312434 conn(0x55c90432aba0 msgr2=0x55c90432bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 -- 172.21.15.94:0/475312434 shutdown_connections 2024-04-21T08:44:55.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 1 -- 172.21.15.94:0/475312434 wait complete. 2024-04-21T08:44:55.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.924+0000 7fd83221e740 20 client.17079 trim_cache size 0 max 16384 2024-04-21T08:44:55.931 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 -- 172.21.15.94:0/2128968783 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd8140a3d50 msgr2=0x7fd8140a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd8140a3d50 0x7fd8140a6210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd82009b990 tx=0x7fd82006a640 comp rx=0 tx=0).stop 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 -- 172.21.15.94:0/2128968783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd82810f150 msgr2=0x7fd8281a6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd82810f150 0x7fd8281a6a60 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd82000b970 tx=0x7fd82000bd70 comp rx=0 tx=0).stop 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 -- 172.21.15.94:0/2128968783 shutdown_connections 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd8140a3d50 0x7fd8140a6210 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd828106170 0x7fd82810ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd82810f150 0x7fd8281a6a60 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 --2- 172.21.15.94:0/2128968783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8281010a0 0x7fd82810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 -- 172.21.15.94:0/2128968783 >> 172.21.15.94:0/2128968783 conn(0x7fd828077eb0 msgr2=0x7fd8280ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 -- 172.21.15.94:0/2128968783 shutdown_connections 2024-04-21T08:44:55.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:55.925+0000 7fd82f631640 1 -- 172.21.15.94:0/2128968783 wait complete. 2024-04-21T08:44:55.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:55 smithi111 ceph-mon[31493]: pgmap v151: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:44:55.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:55 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2942909778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:56.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:55 smithi173 ceph-mon[31428]: pgmap v151: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:44:56.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:55 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2942909778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:56.935 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:44:56.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:56.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:44:57.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.055+0000 7fd051d2e640 1 -- 172.21.15.94:0/1145456913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04c1010a0 msgr2=0x7fd04c101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.055+0000 7fd051d2e640 1 --2- 172.21.15.94:0/1145456913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04c1010a0 0x7fd04c101480 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009900 tx=0x7fd03c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.055+0000 7fd051d2e640 1 -- 172.21.15.94:0/1145456913 shutdown_connections 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.055+0000 7fd051d2e640 1 --2- 172.21.15.94:0/1145456913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd04c101a50 0x7fd04c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.055+0000 7fd051d2e640 1 --2- 172.21.15.94:0/1145456913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c106170 0x7fd04c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.055+0000 7fd051d2e640 1 --2- 172.21.15.94:0/1145456913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04c1010a0 0x7fd04c101480 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.055+0000 7fd051d2e640 1 -- 172.21.15.94:0/1145456913 >> 172.21.15.94:0/1145456913 conn(0x7fd04c077eb0 msgr2=0x7fd04c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.055+0000 7fd051d2e640 1 -- 172.21.15.94:0/1145456913 shutdown_connections 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.055+0000 7fd051d2e640 1 -- 172.21.15.94:0/1145456913 wait complete. 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd051d2e640 1 Processor -- start 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd051d2e640 1 -- start start 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd051d2e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd04c1010a0 0x7fd04c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd051d2e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c106170 0x7fd04c198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd051d2e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04c19cfa0 0x7fd04c19d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd051d2e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c1110b0 con 0x7fd04c19cfa0 2024-04-21T08:44:57.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd051d2e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c110f30 con 0x7fd04c106170 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd051d2e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c111230 con 0x7fd04c1010a0 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd04bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04c19cfa0 0x7fd04c19d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd04bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04c19cfa0 0x7fd04c19d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48040/0 (socket says 172.21.15.94:48040) 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.056+0000 7fd04bfff640 1 -- 172.21.15.94:0/4262806487 learned_addr learned my addr 172.21.15.94:0/4262806487 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd04b7fe640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd04c1010a0 0x7fd04c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd04affd640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c106170 0x7fd04c198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd04bfff640 1 -- 172.21.15.94:0/4262806487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd04c1010a0 msgr2=0x7fd04c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd04bfff640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd04c1010a0 0x7fd04c1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd04bfff640 1 -- 172.21.15.94:0/4262806487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c106170 msgr2=0x7fd04c198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd04bfff640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c106170 0x7fd04c198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd04bfff640 1 -- 172.21.15.94:0/4262806487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04c19da90 con 0x7fd04c19cfa0 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd04bfff640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04c19cfa0 0x7fd04c19d450 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fd04400eb10 tx=0x7fd04400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd048ff9640 1 -- 172.21.15.94:0/4262806487 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd044004b20 con 0x7fd04c19cfa0 2024-04-21T08:44:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd051d2e640 1 -- 172.21.15.94:0/4262806487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd04c19dd80 con 0x7fd04c19cfa0 2024-04-21T08:44:57.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd048ff9640 1 -- 172.21.15.94:0/4262806487 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd044004cc0 con 0x7fd04c19cfa0 2024-04-21T08:44:57.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd048ff9640 1 -- 172.21.15.94:0/4262806487 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd044010670 con 0x7fd04c19cfa0 2024-04-21T08:44:57.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.057+0000 7fd051d2e640 1 -- 172.21.15.94:0/4262806487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd04c1a5570 con 0x7fd04c19cfa0 2024-04-21T08:44:57.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.059+0000 7fd0549f3740 1 Processor -- start 2024-04-21T08:44:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.059+0000 7fd048ff9640 1 -- 172.21.15.94:0/4262806487 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd044005020 con 0x7fd04c19cfa0 2024-04-21T08:44:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd0549f3740 1 -- start start 2024-04-21T08:44:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c1290 0x564ea26c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c9e10 0x564ea26cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd0549f3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564ea2624f00 con 0x564ea26c08e0 2024-04-21T08:44:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd0549f3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564ea25c2590 con 0x564ea26c9e10 2024-04-21T08:44:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd0549f3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564ea2620ec0 con 0x564ea26c1290 2024-04-21T08:44:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48056/0 (socket says 172.21.15.94:48056) 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04affd640 1 -- 172.21.15.94:0/679438842 learned_addr learned my addr 172.21.15.94:0/679438842 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04affd640 1 --2- 172.21.15.94:0/679438842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c9e10 0x564ea26cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04b7fe640 1 --2- 172.21.15.94:0/679438842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c1290 0x564ea26c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd048ff9640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd0340a3d20 0x7fd0340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04affd640 1 -- 172.21.15.94:0/679438842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c1290 msgr2=0x564ea26c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04affd640 1 --2- 172.21.15.94:0/679438842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c1290 0x564ea26c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04affd640 1 -- 172.21.15.94:0/679438842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c9e10 msgr2=0x564ea26cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04affd640 1 --2- 172.21.15.94:0/679438842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c9e10 0x564ea26cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04affd640 1 -- 172.21.15.94:0/679438842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ea23ff7e0 con 0x564ea26c08e0 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04bfff640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd0340a3d20 0x7fd0340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.060+0000 7fd04affd640 1 --2- 172.21.15.94:0/679438842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c0cc0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fd04000dfd0 tx=0x7fd04000d650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ecc31f224964b50 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd048ff9640 1 -- 172.21.15.94:0/4262806487 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd0440d1250 con 0x7fd04c19cfa0 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd031ffb640 1 -- 172.21.15.94:0/679438842 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040004030 con 0x564ea26c08e0 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd031ffb640 1 -- 172.21.15.94:0/679438842 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0400041d0 con 0x564ea26c08e0 2024-04-21T08:44:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd031ffb640 1 -- 172.21.15.94:0/679438842 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0400116e0 con 0x564ea26c08e0 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd0549f3740 1 -- 172.21.15.94:0/679438842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 msgr2=0x564ea26c0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd0549f3740 1 --2- 172.21.15.94:0/679438842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c0cc0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fd04000dfd0 tx=0x7fd04000d650 comp rx=0 tx=0).stop 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd04bfff640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd0340a3d20 0x7fd0340a61e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd0440ce670 tx=0x7fd0440bd040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd0549f3740 1 -- 172.21.15.94:0/679438842 shutdown_connections 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd0549f3740 1 --2- 172.21.15.94:0/679438842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c1290 0x564ea26c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd0549f3740 1 --2- 172.21.15.94:0/679438842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c9e10 0x564ea26cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd0549f3740 1 --2- 172.21.15.94:0/679438842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c0cc0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.061+0000 7fd0549f3740 1 -- 172.21.15.94:0/679438842 >> 172.21.15.94:0/679438842 conn(0x564ea26bafb0 msgr2=0x564ea26bb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 1 -- 172.21.15.94:0/679438842 shutdown_connections 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 1 -- 172.21.15.94:0/679438842 wait complete. 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 1 -- start start 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 1 Processor -- start 2024-04-21T08:44:57.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c1290 0x564ea26c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c9e10 0x564ea26c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564ea26ccd80 con 0x564ea26c08e0 2024-04-21T08:44:57.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564ea25de580 con 0x564ea26c9e10 2024-04-21T08:44:57.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564ea26ccf00 con 0x564ea26c1290 2024-04-21T08:44:57.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd0549f3740 10 client.0 _ll_register_callbacks cb 0x564ea25e4650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:57.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd04affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd04affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48064/0 (socket says 172.21.15.94:48064) 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.062+0000 7fd04affd640 1 -- 172.21.15.94:0/3457490841 learned_addr learned my addr 172.21.15.94:0/3457490841 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd04affd640 1 -- 172.21.15.94:0/3457490841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c1290 msgr2=0x564ea26c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd04affd640 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c1290 0x564ea26c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd04affd640 1 -- 172.21.15.94:0/3457490841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c9e10 msgr2=0x564ea26c78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd04affd640 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c9e10 0x564ea26c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd04affd640 1 -- 172.21.15.94:0/3457490841 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ea245ff70 con 0x564ea26c08e0 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd04affd640 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c6e40 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd04000d620 tx=0x7fd040004150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ce2eda1196d3cc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd02affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd02affd640 1 -- 172.21.15.94:0/3457490841 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040004480 con 0x564ea26c08e0 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd02affd640 1 -- 172.21.15.94:0/3457490841 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd04001a050 con 0x564ea26c08e0 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd0549f3740 1 -- 172.21.15.94:0/3457490841 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0400041d0 con 0x564ea26c08e0 2024-04-21T08:44:57.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd02affd640 1 -- 172.21.15.94:0/3457490841 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd040003800 con 0x564ea26c08e0 2024-04-21T08:44:57.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd0549f3740 10 client.17091 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:57.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd0549f3740 1 -- 172.21.15.94:0/3457490841 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564ea22f7170 con 0x564ea26c08e0 2024-04-21T08:44:57.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.063+0000 7fd02affd640 1 -- 172.21.15.94:0/3457490841 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0400041d0 con 0x564ea26c08e0 2024-04-21T08:44:57.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.064+0000 7fd02affd640 1 -- 172.21.15.94:0/3457490841 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd0400049c0 con 0x564ea26c08e0 2024-04-21T08:44:57.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.064+0000 7fd0549f3740 10 client.17091 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:57.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.064+0000 7fd0549f3740 10 client.17091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:57.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.064+0000 7fd0549f3740 20 client.17091 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:57.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.064+0000 7fd0549f3740 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564ea26da340 0x564ea26dc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.064+0000 7fd0549f3740 4 client.17091 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:57.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.064+0000 7fd0549f3740 1 -- 172.21.15.94:0/3457490841 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ea2632920 con 0x564ea26da340 2024-04-21T08:44:57.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.064+0000 7fd04b7fe640 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564ea26da340 0x564ea26dc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.064+0000 7fd04b7fe640 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564ea26da340 0x564ea26dc730 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10f0e1afa91e5e42 server_cookie=c8a8f4e8762d2306 in_seq=0 out_seq=0 2024-04-21T08:44:57.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.064+0000 7fd02affd640 10 client.17091 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:57.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd02affd640 1 -- 172.21.15.94:0/3457490841 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ea2632920 con 0x564ea26da340 2024-04-21T08:44:57.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd02affd640 10 client.17091 handle_command_reply: tid=0 2024-04-21T08:44:57.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 client.17091 shutdown 2024-04-21T08:44:57.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 -- 172.21.15.94:0/3457490841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 msgr2=0x564ea26c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c6e40 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd04000d620 tx=0x7fd040004150 comp rx=0 tx=0).stop 2024-04-21T08:44:57.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 -- 172.21.15.94:0/3457490841 shutdown_connections 2024-04-21T08:44:57.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564ea26da340 0x564ea26dc730 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c1290 0x564ea26c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c9e10 0x564ea26c78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd02affd640 0 client.17091 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:57.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 --2- 172.21.15.94:0/3457490841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26c6e40 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 -- 172.21.15.94:0/3457490841 >> 172.21.15.94:0/3457490841 conn(0x564ea26bafb0 msgr2=0x564ea26bcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 -- 172.21.15.94:0/3457490841 shutdown_connections 2024-04-21T08:44:57.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.065+0000 7fd0549f3740 1 -- 172.21.15.94:0/3457490841 wait complete. 2024-04-21T08:44:57.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.066+0000 7fd0549f3740 20 client.17091 trim_cache size 0 max 16384 2024-04-21T08:44:57.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd0549f3740 1 Processor -- start 2024-04-21T08:44:57.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd0549f3740 1 -- start start 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26d5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c1290 0x564ea26c6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c9e10 0x564ea26d2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd0549f3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c1b1c30 con 0x564ea26c08e0 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd0549f3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c1af500 con 0x564ea26c1290 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd0549f3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c1ad350 con 0x564ea26c9e10 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd04affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26d5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd04affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26d5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48072/0 (socket says 172.21.15.94:48072) 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd04affd640 1 -- 172.21.15.94:0/1215693861 learned_addr learned my addr 172.21.15.94:0/1215693861 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd04affd640 1 -- 172.21.15.94:0/1215693861 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c9e10 msgr2=0x564ea26d2530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd04affd640 1 --2- 172.21.15.94:0/1215693861 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c9e10 0x564ea26d2530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd04affd640 1 -- 172.21.15.94:0/1215693861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c1290 msgr2=0x564ea26c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd04affd640 1 --2- 172.21.15.94:0/1215693861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c1290 0x564ea26c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.083+0000 7fd04affd640 1 -- 172.21.15.94:0/1215693861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ea26b6880 con 0x564ea26c08e0 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd04affd640 1 --2- 172.21.15.94:0/1215693861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26d5c40 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fd040003ef0 tx=0x7fd04000d780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51e3b4db3198a0e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0317fa640 1 -- 172.21.15.94:0/1215693861 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04003a070 con 0x564ea26c08e0 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0317fa640 1 -- 172.21.15.94:0/1215693861 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd040011600 con 0x564ea26c08e0 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0317fa640 1 -- 172.21.15.94:0/1215693861 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040004680 con 0x564ea26c08e0 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0549f3740 1 -- 172.21.15.94:0/1215693861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 msgr2=0x564ea26d5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0549f3740 1 --2- 172.21.15.94:0/1215693861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26d5c40 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fd040003ef0 tx=0x7fd04000d780 comp rx=0 tx=0).stop 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0549f3740 1 -- 172.21.15.94:0/1215693861 shutdown_connections 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0549f3740 1 --2- 172.21.15.94:0/1215693861 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c9e10 0x564ea26d2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0549f3740 1 --2- 172.21.15.94:0/1215693861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c1290 0x564ea26c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0549f3740 1 --2- 172.21.15.94:0/1215693861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c08e0 0x564ea26d5c40 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0549f3740 1 -- 172.21.15.94:0/1215693861 >> 172.21.15.94:0/1215693861 conn(0x564ea26bafb0 msgr2=0x564ea26c6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0549f3740 1 -- 172.21.15.94:0/1215693861 shutdown_connections 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.084+0000 7fd0549f3740 1 -- 172.21.15.94:0/1215693861 wait complete. 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd0549f3740 1 -- start start 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd0549f3740 1 Processor -- start 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c08e0 0x564ea26c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c1290 0x564ea26e16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd0549f3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c9e10 0x564ea26e1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd0549f3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c1ad9c0 con 0x564ea26c1290 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd0549f3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c1aee30 con 0x564ea26c08e0 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd0549f3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c1ae090 con 0x564ea26c9e10 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd0549f3740 10 client.0 _ll_register_callbacks cb 0x7fd040013400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd04b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c1290 0x564ea26e16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd04b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c1290 0x564ea26e16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48086/0 (socket says 172.21.15.94:48086) 2024-04-21T08:44:57.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd04b7fe640 1 -- 172.21.15.94:0/4290236412 learned_addr learned my addr 172.21.15.94:0/4290236412 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd04affd640 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c08e0 0x564ea26c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd04b7fe640 1 -- 172.21.15.94:0/4290236412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c9e10 msgr2=0x564ea26e1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd04b7fe640 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c9e10 0x564ea26e1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd04b7fe640 1 -- 172.21.15.94:0/4290236412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c08e0 msgr2=0x564ea26c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd04b7fe640 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c08e0 0x564ea26c4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.085+0000 7fd04b7fe640 1 -- 172.21.15.94:0/4290236412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ea27436f0 con 0x564ea26c1290 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd04b7fe640 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c1290 0x564ea26e16e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0092a0 tx=0x7fd03c02c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a35cb1cba7cfa55 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd02affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd02affd640 1 -- 172.21.15.94:0/4290236412 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c004030 con 0x564ea26c1290 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd02affd640 1 -- 172.21.15.94:0/4290236412 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd03c0041d0 con 0x564ea26c1290 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd0549f3740 1 -- 172.21.15.94:0/4290236412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd040011600 con 0x564ea26c1290 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd02affd640 1 -- 172.21.15.94:0/4290236412 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd03c0043d0 con 0x564ea26c1290 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd0549f3740 10 client.17097 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:57.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd0549f3740 1 -- 172.21.15.94:0/4290236412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564ea2744920 con 0x564ea26c1290 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd02affd640 1 -- 172.21.15.94:0/4290236412 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd040011600 con 0x564ea26c1290 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd02affd640 1 -- 172.21.15.94:0/4290236412 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd03c045250 con 0x564ea26c1290 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd0549f3740 10 client.17097 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd0549f3740 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd0549f3740 20 client.17097 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd0549f3740 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564ea27450d0 0x564ea27474c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd0549f3740 4 client.17097 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.086+0000 7fd0549f3740 1 -- 172.21.15.94:0/4290236412 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564ea262c790 con 0x564ea27450d0 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.087+0000 7fd04affd640 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564ea27450d0 0x564ea27474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.087+0000 7fd02affd640 10 client.17097 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.087+0000 7fd04affd640 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564ea27450d0 0x564ea27474c0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbf08070e542c83a server_cookie=992c5785779d05c0 in_seq=0 out_seq=0 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.088+0000 7fd02affd640 1 -- 172.21.15.94:0/4290236412 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564ea262c790 con 0x564ea27450d0 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.088+0000 7fd02affd640 10 client.17097 handle_command_reply: tid=0 2024-04-21T08:44:57.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.088+0000 7fd0549f3740 1 client.17097 shutdown 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.088+0000 7fd0549f3740 1 -- 172.21.15.94:0/4290236412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c1290 msgr2=0x564ea26e16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.088+0000 7fd0549f3740 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c1290 0x564ea26e16e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0092a0 tx=0x7fd03c02c1f0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.088+0000 7fd0549f3740 1 -- 172.21.15.94:0/4290236412 shutdown_connections 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.088+0000 7fd0549f3740 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564ea27450d0 0x564ea27474c0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.088+0000 7fd0549f3740 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ea26c9e10 0x564ea26e1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd02affd640 0 client.17097 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd0549f3740 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564ea26c1290 0x564ea26e16e0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd0549f3740 1 --2- 172.21.15.94:0/4290236412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564ea26c08e0 0x564ea26c4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd0549f3740 1 -- 172.21.15.94:0/4290236412 >> 172.21.15.94:0/4290236412 conn(0x564ea26bafb0 msgr2=0x564ea26c7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd0549f3740 1 -- 172.21.15.94:0/4290236412 shutdown_connections 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd0549f3740 1 -- 172.21.15.94:0/4290236412 wait complete. 2024-04-21T08:44:57.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd0549f3740 20 client.17097 trim_cache size 0 max 16384 2024-04-21T08:44:57.096 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:44:57.096 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:44:57.096 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "229bcaf2-30f8-4925-9c30-29d3069b90e0", 2024-04-21T08:44:57.096 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:44:57.096 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:44:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 -- 172.21.15.94:0/4262806487 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd0340a3d20 msgr2=0x7fd0340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd0340a3d20 0x7fd0340a61e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd0440ce670 tx=0x7fd0440bd040 comp rx=0 tx=0).stop 2024-04-21T08:44:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 -- 172.21.15.94:0/4262806487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04c19cfa0 msgr2=0x7fd04c19d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04c19cfa0 0x7fd04c19d450 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fd04400eb10 tx=0x7fd04400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 -- 172.21.15.94:0/4262806487 shutdown_connections 2024-04-21T08:44:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd0340a3d20 0x7fd0340a61e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04c19cfa0 0x7fd04c19d450 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c106170 0x7fd04c198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 --2- 172.21.15.94:0/4262806487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd04c1010a0 0x7fd04c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 -- 172.21.15.94:0/4262806487 >> 172.21.15.94:0/4262806487 conn(0x7fd04c077eb0 msgr2=0x7fd04c107340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 -- 172.21.15.94:0/4262806487 shutdown_connections 2024-04-21T08:44:57.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.089+0000 7fd051d2e640 1 -- 172.21.15.94:0/4262806487 wait complete. 2024-04-21T08:44:57.099 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:44:57.100 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:44:57.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.201+0000 7f2287055640 1 -- 172.21.15.94:0/2000548182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2280101a40 msgr2=0x7f2280105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.201+0000 7f2287055640 1 --2- 172.21.15.94:0/2000548182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2280101a40 0x7f2280105a80 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f226c009900 tx=0x7f226c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:57.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.201+0000 7f2287055640 1 -- 172.21.15.94:0/2000548182 shutdown_connections 2024-04-21T08:44:57.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.201+0000 7f2287055640 1 --2- 172.21.15.94:0/2000548182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2280106160 0x7f228010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.201+0000 7f2287055640 1 --2- 172.21.15.94:0/2000548182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2280101a40 0x7f2280105a80 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.201+0000 7f2287055640 1 --2- 172.21.15.94:0/2000548182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2280101090 0x7f2280101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.201+0000 7f2287055640 1 -- 172.21.15.94:0/2000548182 >> 172.21.15.94:0/2000548182 conn(0x7f22800771d0 msgr2=0x7f22800775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.201+0000 7f2287055640 1 -- 172.21.15.94:0/2000548182 shutdown_connections 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.201+0000 7f2287055640 1 -- 172.21.15.94:0/2000548182 wait complete. 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2287055640 1 Processor -- start 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2287055640 1 -- start start 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2287055640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2280101090 0x7f2280199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2287055640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2280106160 0x7f2280199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2287055640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228019a010 0x7f22801a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2287055640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f22801110a0 con 0x7f2280101090 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2287055640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2280110f20 con 0x7f228019a010 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2287055640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2280111220 con 0x7f2280106160 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2284df1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2280101090 0x7f2280199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2284df1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2280101090 0x7f2280199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48110/0 (socket says 172.21.15.94:48110) 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2284df1640 1 -- 172.21.15.94:0/2776091066 learned_addr learned my addr 172.21.15.94:0/2776091066 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f22855f2640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228019a010 0x7f22801a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f227ffff640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2280106160 0x7f2280199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2284df1640 1 -- 172.21.15.94:0/2776091066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2280106160 msgr2=0x7f2280199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2284df1640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2280106160 0x7f2280199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2284df1640 1 -- 172.21.15.94:0/2776091066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228019a010 msgr2=0x7f22801a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2284df1640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228019a010 0x7f22801a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f2284df1640 1 -- 172.21.15.94:0/2776091066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22801a4dc0 con 0x7f2280101090 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.202+0000 7f227ffff640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2280106160 0x7f2280199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.203+0000 7f2284df1640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2280101090 0x7f2280199540 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f227400d770 tx=0x7f227400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.203+0000 7f227dffb640 1 -- 172.21.15.94:0/2776091066 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274017070 con 0x7f2280101090 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.203+0000 7f227dffb640 1 -- 172.21.15.94:0/2776091066 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22740054d0 con 0x7f2280101090 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.203+0000 7f2287055640 1 -- 172.21.15.94:0/2776091066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22801a50b0 con 0x7f2280101090 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.203+0000 7f227dffb640 1 -- 172.21.15.94:0/2776091066 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274004c30 con 0x7f2280101090 2024-04-21T08:44:57.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.203+0000 7f2287055640 1 -- 172.21.15.94:0/2776091066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22801a55b0 con 0x7f2280101090 2024-04-21T08:44:57.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.205+0000 7f227dffb640 1 -- 172.21.15.94:0/2776091066 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f2274004e10 con 0x7f2280101090 2024-04-21T08:44:57.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.205+0000 7f2289c46740 1 Processor -- start 2024-04-21T08:44:57.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.205+0000 7f227dffb640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22700a3d20 0x7f22700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.205+0000 7f2289c46740 1 -- start start 2024-04-21T08:44:57.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f22855f2640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22700a3d20 0x7f22700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2289c46740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20580 0x55b21bb20960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2289c46740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20ea0 0x55b21bb29540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f227dffb640 1 -- 172.21.15.94:0/2776091066 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f22740d0760 con 0x7f2280101090 2024-04-21T08:44:57.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2289c46740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 0x55b21bb2be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2289c46740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b21ba851e0 con 0x55b21bb20580 2024-04-21T08:44:57.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2289c46740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b21ba22390 con 0x55b21bb29a80 2024-04-21T08:44:57.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2289c46740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b21ba811a0 con 0x55b21bb20ea0 2024-04-21T08:44:57.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2284df1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20580 0x55b21bb20960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2284df1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20580 0x55b21bb20960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48122/0 (socket says 172.21.15.94:48122) 2024-04-21T08:44:57.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2284df1640 1 -- 172.21.15.94:0/3465180343 learned_addr learned my addr 172.21.15.94:0/3465180343 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:57.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f227ffff640 1 --2- 172.21.15.94:0/3465180343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20ea0 0x55b21bb29540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f22855f2640 1 --2- 172.21.15.94:0/3465180343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 0x55b21bb2be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2284df1640 1 -- 172.21.15.94:0/3465180343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20ea0 msgr2=0x55b21bb29540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2284df1640 1 --2- 172.21.15.94:0/3465180343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20ea0 0x55b21bb29540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2284df1640 1 -- 172.21.15.94:0/3465180343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 msgr2=0x55b21bb2be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2284df1640 1 --2- 172.21.15.94:0/3465180343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 0x55b21bb2be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2284df1640 1 -- 172.21.15.94:0/3465180343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b21b8617d0 con 0x55b21bb20580 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f22855f2640 1 --2- 172.21.15.94:0/3465180343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 0x55b21bb2be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f22855f2640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22700a3d20 0x7f22700a61e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2278005fd0 tx=0x7f22780107a0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f2284df1640 1 --2- 172.21.15.94:0/3465180343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20580 0x55b21bb20960 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f2274098cf0 tx=0x7f227409cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b40de83b29f52c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f226affd640 1 -- 172.21.15.94:0/3465180343 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274020040 con 0x55b21bb20580 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f226affd640 1 -- 172.21.15.94:0/3465180343 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22740a4f90 con 0x55b21bb20580 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.206+0000 7f226affd640 1 -- 172.21.15.94:0/3465180343 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22740d0540 con 0x55b21bb20580 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.207+0000 7f2289c46740 1 -- 172.21.15.94:0/3465180343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20580 msgr2=0x55b21bb20960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.207+0000 7f2289c46740 1 --2- 172.21.15.94:0/3465180343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20580 0x55b21bb20960 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f2274098cf0 tx=0x7f227409cae0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.207+0000 7f2289c46740 1 -- 172.21.15.94:0/3465180343 shutdown_connections 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.207+0000 7f2289c46740 1 --2- 172.21.15.94:0/3465180343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20ea0 0x55b21bb29540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.207+0000 7f2289c46740 1 --2- 172.21.15.94:0/3465180343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 0x55b21bb2be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.207+0000 7f2289c46740 1 --2- 172.21.15.94:0/3465180343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20580 0x55b21bb20960 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.207+0000 7f2289c46740 1 -- 172.21.15.94:0/3465180343 >> 172.21.15.94:0/3465180343 conn(0x55b21bb1acb0 msgr2=0x55b21bb1b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.207+0000 7f2289c46740 1 -- 172.21.15.94:0/3465180343 shutdown_connections 2024-04-21T08:44:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.207+0000 7f2289c46740 1 -- 172.21.15.94:0/3465180343 wait complete. 2024-04-21T08:44:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.207+0000 7f2289c46740 1 -- start start 2024-04-21T08:44:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f2289c46740 1 Processor -- start 2024-04-21T08:44:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f2289c46740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb26b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f2289c46740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb20ea0 0x55b21bb27060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f2289c46740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb275a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f2289c46740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b21bb2c870 con 0x55b21bb29a80 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f2289c46740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b21ba3e2f0 con 0x55b21bb20ea0 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f2289c46740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b21bb2c9f0 con 0x55b21bb20580 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f2289c46740 10 client.0 _ll_register_callbacks cb 0x55b21ba4b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f22855f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb275a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f22855f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb275a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48134/0 (socket says 172.21.15.94:48134) 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f22855f2640 1 -- 172.21.15.94:0/434502347 learned_addr learned my addr 172.21.15.94:0/434502347 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f2284df1640 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb26b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f22855f2640 1 -- 172.21.15.94:0/434502347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 msgr2=0x55b21bb26b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f22855f2640 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb26b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f22855f2640 1 -- 172.21.15.94:0/434502347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb20ea0 msgr2=0x55b21bb27060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f22855f2640 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb20ea0 0x55b21bb27060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f22855f2640 1 -- 172.21.15.94:0/434502347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b21b7973f0 con 0x55b21bb29a80 2024-04-21T08:44:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f22855f2640 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb275a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f2278011c70 tx=0x7f2278004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a022c6d715d03099 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.208+0000 7f225ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f225ffff640 1 -- 172.21.15.94:0/434502347 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278020620 con 0x55b21bb29a80 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f225ffff640 1 -- 172.21.15.94:0/434502347 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f22780207c0 con 0x55b21bb29a80 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f2289c46740 1 -- 172.21.15.94:0/434502347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22740a4f90 con 0x55b21bb29a80 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f225ffff640 1 -- 172.21.15.94:0/434502347 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f227802f050 con 0x55b21bb29a80 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f2289c46740 10 client.17109 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f2289c46740 1 -- 172.21.15.94:0/434502347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b21b7570f0 con 0x55b21bb29a80 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f225ffff640 1 -- 172.21.15.94:0/434502347 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22740a4f90 con 0x55b21bb29a80 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f225ffff640 1 -- 172.21.15.94:0/434502347 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2278028960 con 0x55b21bb29a80 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f2289c46740 10 client.17109 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f2289c46740 10 client.17109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f2289c46740 20 client.17109 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f2289c46740 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b21bb39af0 0x55b21bb3bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f2289c46740 4 client.17109 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f2289c46740 1 -- 172.21.15.94:0/434502347 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b21ba92c00 con 0x55b21bb39af0 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.209+0000 7f2284df1640 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b21bb39af0 0x55b21bb3bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.210+0000 7f225ffff640 10 client.17109 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.210+0000 7f2284df1640 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b21bb39af0 0x55b21bb3bf60 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a253360900b6ed4c server_cookie=48124d02b2b3dabf in_seq=0 out_seq=0 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.210+0000 7f225ffff640 1 -- 172.21.15.94:0/434502347 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b21ba92c00 con 0x55b21bb39af0 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.210+0000 7f225ffff640 10 client.17109 handle_command_reply: tid=0 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.210+0000 7f2289c46740 1 client.17109 shutdown 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 1 -- 172.21.15.94:0/434502347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 msgr2=0x55b21bb275a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb275a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f2278011c70 tx=0x7f2278004ab0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 1 -- 172.21.15.94:0/434502347 shutdown_connections 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b21bb39af0 0x55b21bb3bf60 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb275a0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb20ea0 0x55b21bb27060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f225ffff640 0 client.17109 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 1 --2- 172.21.15.94:0/434502347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb26b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 1 -- 172.21.15.94:0/434502347 >> 172.21.15.94:0/434502347 conn(0x55b21bb1acb0 msgr2=0x55b21bb1ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 1 -- 172.21.15.94:0/434502347 shutdown_connections 2024-04-21T08:44:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 1 -- 172.21.15.94:0/434502347 wait complete. 2024-04-21T08:44:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.211+0000 7f2289c46740 20 client.17109 trim_cache size 0 max 16384 2024-04-21T08:44:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f2289c46740 1 Processor -- start 2024-04-21T08:44:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f2289c46740 1 -- start start 2024-04-21T08:44:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f2289c46740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb29250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f2289c46740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb20ea0 0x55b21bb25a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f2289c46740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb35730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f2289c46740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f22801b1d50 con 0x55b21bb29a80 2024-04-21T08:44:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f2289c46740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22801af620 con 0x55b21bb20ea0 2024-04-21T08:44:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f2289c46740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f22801ad470 con 0x55b21bb20580 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f22855f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb35730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f22855f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb35730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48150/0 (socket says 172.21.15.94:48150) 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f22855f2640 1 -- 172.21.15.94:0/212569163 learned_addr learned my addr 172.21.15.94:0/212569163 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f2284df1640 1 --2- 172.21.15.94:0/212569163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb29250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f22855f2640 1 -- 172.21.15.94:0/212569163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 msgr2=0x55b21bb29250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.228+0000 7f22855f2640 1 --2- 172.21.15.94:0/212569163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb29250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f22855f2640 1 -- 172.21.15.94:0/212569163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb20ea0 msgr2=0x55b21bb25a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f22855f2640 1 --2- 172.21.15.94:0/212569163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb20ea0 0x55b21bb25a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f22855f2640 1 -- 172.21.15.94:0/212569163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b21bb20cd0 con 0x55b21bb29a80 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f22855f2640 1 --2- 172.21.15.94:0/212569163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb35730 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2278009730 tx=0x7f2278009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8ac2ae81bc5357 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f226a7fc640 1 -- 172.21.15.94:0/212569163 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278003b70 con 0x55b21bb29a80 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f226a7fc640 1 -- 172.21.15.94:0/212569163 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2278009c40 con 0x55b21bb29a80 2024-04-21T08:44:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f226a7fc640 1 -- 172.21.15.94:0/212569163 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278004440 con 0x55b21bb29a80 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f2289c46740 1 -- 172.21.15.94:0/212569163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 msgr2=0x55b21bb35730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f2289c46740 1 --2- 172.21.15.94:0/212569163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb35730 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2278009730 tx=0x7f2278009fc0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f2289c46740 1 -- 172.21.15.94:0/212569163 shutdown_connections 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f2289c46740 1 --2- 172.21.15.94:0/212569163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb29a80 0x55b21bb35730 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f2289c46740 1 --2- 172.21.15.94:0/212569163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb20ea0 0x55b21bb25a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f2289c46740 1 --2- 172.21.15.94:0/212569163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb29250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f2289c46740 1 -- 172.21.15.94:0/212569163 >> 172.21.15.94:0/212569163 conn(0x55b21bb1acb0 msgr2=0x55b21bb27370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f2289c46740 1 -- 172.21.15.94:0/212569163 shutdown_connections 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.229+0000 7f2289c46740 1 -- 172.21.15.94:0/212569163 wait complete. 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f2289c46740 1 -- start start 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f2289c46740 1 Processor -- start 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f2289c46740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb98060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f2289c46740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20ea0 0x55b21bb985a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f2289c46740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 0x55b21bba30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f2289c46740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f22801adae0 con 0x55b21bb20ea0 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f2289c46740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22801aef50 con 0x55b21bb29a80 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f2289c46740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f22801ae1b0 con 0x55b21bb20580 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f2289c46740 10 client.0 _ll_register_callbacks cb 0x7f2278009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f227ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20ea0 0x55b21bb985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f227ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20ea0 0x55b21bb985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48152/0 (socket says 172.21.15.94:48152) 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f227ffff640 1 -- 172.21.15.94:0/829214072 learned_addr learned my addr 172.21.15.94:0/829214072 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f2284df1640 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb98060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.230+0000 7f22855f2640 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 0x55b21bba30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f227ffff640 1 -- 172.21.15.94:0/829214072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 msgr2=0x55b21bb98060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f227ffff640 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb98060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f227ffff640 1 -- 172.21.15.94:0/829214072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 msgr2=0x55b21bba30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:57.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f227ffff640 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 0x55b21bba30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f227ffff640 1 -- 172.21.15.94:0/829214072 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b21bba3620 con 0x55b21bb20ea0 2024-04-21T08:44:57.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f2284df1640 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb98060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:57.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f227ffff640 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20ea0 0x55b21bb985a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f228006a3c0 tx=0x7f226c02ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96df4f1a6209d376 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:57.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f225ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:44:57.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f225ffff640 1 -- 172.21.15.94:0/829214072 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c02c5d0 con 0x55b21bb20ea0 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f225ffff640 1 -- 172.21.15.94:0/829214072 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f226c005030 con 0x55b21bb20ea0 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f2289c46740 1 -- 172.21.15.94:0/829214072 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2278009c40 con 0x55b21bb20ea0 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f225ffff640 1 -- 172.21.15.94:0/829214072 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f226c050050 con 0x55b21bb20ea0 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f2289c46740 10 client.17115 fetch_fsmap learned FSMap version 24 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f2289c46740 1 -- 172.21.15.94:0/829214072 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b21bba4810 con 0x55b21bb20ea0 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f225ffff640 1 -- 172.21.15.94:0/829214072 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2278009c40 con 0x55b21bb20ea0 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f225ffff640 1 -- 172.21.15.94:0/829214072 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f226c0035f0 con 0x55b21bb20ea0 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f2289c46740 10 client.17115 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f2289c46740 10 client.17115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f2289c46740 20 client.17115 populate_metadata read hostname 'smithi094' 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f2289c46740 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b21bba5010 0x55b21bba7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f2289c46740 4 client.17115 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:44:57.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.231+0000 7f2289c46740 1 -- 172.21.15.94:0/829214072 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b21ba8ca70 con 0x55b21bba5010 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.232+0000 7f2284df1640 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b21bba5010 0x55b21bba7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.232+0000 7f2284df1640 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b21bba5010 0x55b21bba7430 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=214c3e6ba3fd7ea server_cookie=d7eef40a31f75885 in_seq=0 out_seq=0 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.232+0000 7f225ffff640 10 client.17115 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.232+0000 7f225ffff640 1 -- 172.21.15.94:0/829214072 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b21ba8ca70 con 0x55b21bba5010 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.232+0000 7f225ffff640 10 client.17115 handle_command_reply: tid=0 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.232+0000 7f2289c46740 1 client.17115 shutdown 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.232+0000 7f2289c46740 1 -- 172.21.15.94:0/829214072 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20ea0 msgr2=0x55b21bb985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.232+0000 7f2289c46740 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20ea0 0x55b21bb985a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f228006a3c0 tx=0x7f226c02ca10 comp rx=0 tx=0).stop 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2289c46740 1 -- 172.21.15.94:0/829214072 shutdown_connections 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2289c46740 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b21bba5010 0x55b21bba7430 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2289c46740 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b21bb29a80 0x55b21bba30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2289c46740 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b21bb20ea0 0x55b21bb985a0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f225ffff640 0 client.17115 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2289c46740 1 --2- 172.21.15.94:0/829214072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b21bb20580 0x55b21bb98060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2289c46740 1 -- 172.21.15.94:0/829214072 >> 172.21.15.94:0/829214072 conn(0x55b21bb1acb0 msgr2=0x55b21bb194e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2289c46740 1 -- 172.21.15.94:0/829214072 shutdown_connections 2024-04-21T08:44:57.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2289c46740 1 -- 172.21.15.94:0/829214072 wait complete. 2024-04-21T08:44:57.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2289c46740 20 client.17115 trim_cache size 0 max 16384 2024-04-21T08:44:57.240 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:44:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:44:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:44:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "229bcaf2-30f8-4925-9c30-29d3069b90e0": { 2024-04-21T08:44:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:44:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "229bcaf2-30f8-4925-9c30-29d3069b90e0", 2024-04-21T08:44:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:44:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:44:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:44:57.242 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:44:57.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 -- 172.21.15.94:0/2776091066 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22700a3d20 msgr2=0x7f22700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22700a3d20 0x7f22700a61e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2278005fd0 tx=0x7f22780107a0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 -- 172.21.15.94:0/2776091066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2280101090 msgr2=0x7f2280199540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:57.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2280101090 0x7f2280199540 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f227400d770 tx=0x7f227400dc30 comp rx=0 tx=0).stop 2024-04-21T08:44:57.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 -- 172.21.15.94:0/2776091066 shutdown_connections 2024-04-21T08:44:57.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22700a3d20 0x7f22700a61e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2280106160 0x7f2280199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228019a010 0x7f22801a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 --2- 172.21.15.94:0/2776091066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2280101090 0x7f2280199540 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:57.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 -- 172.21.15.94:0/2776091066 >> 172.21.15.94:0/2776091066 conn(0x7f22800771d0 msgr2=0x7f22800fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:57.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 -- 172.21.15.94:0/2776091066 shutdown_connections 2024-04-21T08:44:57.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:44:57.233+0000 7f2287055640 1 -- 172.21.15.94:0/2776091066 wait complete. 2024-04-21T08:44:57.246 INFO:tasks.cephfs.filesystem:scrub status for tag:229bcaf2-30f8-4925-9c30-29d3069b90e0 - {'path': '/', 'tag': '229bcaf2-30f8-4925-9c30-29d3069b90e0', 'options': 'recursive,force'} 2024-04-21T08:44:57.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:57 smithi111 ceph-mon[31493]: pgmap v152: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:44:57.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:57 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:44:57.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:57 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:57.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:57 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:44:57.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:57 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:57 smithi094 ceph-mon[27988]: pgmap v152: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:44:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:57 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:44:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:57 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:57 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:44:58.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:57 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:58.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:57 smithi173 ceph-mon[31428]: pgmap v152: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:44:58.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:57 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:44:58.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:57 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:58.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:57 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:44:58.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:57 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:59.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:44:59 smithi111 ceph-mon[31493]: pgmap v153: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:00.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:44:59 smithi094 ceph-mon[27988]: pgmap v153: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:00.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:44:59 smithi173 ceph-mon[31428]: pgmap v153: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:01 smithi111 ceph-mon[31493]: pgmap v154: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:45:01.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:01 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:45:02.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:01 smithi094 ceph-mon[27988]: pgmap v154: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:45:02.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:01 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:45:02.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:01 smithi173 ceph-mon[31428]: pgmap v154: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:45:02.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:01 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:45:03.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:03 smithi111 ceph-mon[31493]: pgmap v155: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:45:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:03 smithi094 ceph-mon[27988]: pgmap v155: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:45:04.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:03 smithi173 ceph-mon[31428]: pgmap v155: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:45:05.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:05 smithi111 ceph-mon[31493]: pgmap v156: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-21T08:45:06.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:05 smithi094 ceph-mon[27988]: pgmap v156: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-21T08:45:06.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:05 smithi173 ceph-mon[31428]: pgmap v156: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-21T08:45:06.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:06 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:07.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:06 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:07.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:06 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:07.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:07 smithi111 ceph-mon[31493]: pgmap v157: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:45:07.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:07 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:07.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:07 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:07.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:07 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:45:08.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:07 smithi094 ceph-mon[27988]: pgmap v157: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:45:08.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:07 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:08.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:07 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:08.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:07 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:45:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:07 smithi173 ceph-mon[31428]: pgmap v157: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:45:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:07 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:07 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:08.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:07 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:45:09.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:09 smithi111 ceph-mon[31493]: pgmap v158: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:45:10.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:09 smithi094 ceph-mon[27988]: pgmap v158: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:45:10.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:09 smithi173 ceph-mon[31428]: pgmap v158: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:45:11.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:11 smithi111 ceph-mon[31493]: pgmap v159: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-21T08:45:12.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:11 smithi094 ceph-mon[27988]: pgmap v159: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-21T08:45:12.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:11 smithi173 ceph-mon[31428]: pgmap v159: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-21T08:45:13.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:13 smithi111 ceph-mon[31493]: pgmap v160: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:45:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:13 smithi094 ceph-mon[27988]: pgmap v160: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:45:14.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:13 smithi173 ceph-mon[31428]: pgmap v160: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T08:45:15.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:15 smithi111 ceph-mon[31493]: pgmap v161: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:45:16.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:15 smithi094 ceph-mon[27988]: pgmap v161: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:45:16.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:15 smithi173 ceph-mon[31428]: pgmap v161: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T08:45:17.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:17 smithi111 ceph-mon[31493]: pgmap v162: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:45:18.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:17 smithi094 ceph-mon[27988]: pgmap v162: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:45:18.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:17 smithi173 ceph-mon[31428]: pgmap v162: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:45:19.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:19 smithi111 ceph-mon[31493]: pgmap v163: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:45:20.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:19 smithi094 ceph-mon[27988]: pgmap v163: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:45:20.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:19 smithi173 ceph-mon[31428]: pgmap v163: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:45:21.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:21 smithi111 ceph-mon[31493]: pgmap v164: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:45:22.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:21 smithi094 ceph-mon[27988]: pgmap v164: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:45:22.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:21 smithi173 ceph-mon[31428]: pgmap v164: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:45:23.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:23 smithi111 ceph-mon[31493]: pgmap v165: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:24.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:23 smithi094 ceph-mon[27988]: pgmap v165: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:24.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:23 smithi173 ceph-mon[31428]: pgmap v165: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:25.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:25 smithi111 ceph-mon[31493]: pgmap v166: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:45:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:25 smithi094 ceph-mon[27988]: pgmap v166: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:45:26.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:25 smithi173 ceph-mon[31428]: pgmap v166: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:45:27.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:45:27.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:45:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.363+0000 7f7f240c1640 1 -- 172.21.15.94:0/2360247043 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f1c106f40 msgr2=0x7f7f1c06a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.363+0000 7f7f240c1640 1 --2- 172.21.15.94:0/2360247043 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f1c106f40 0x7f7f1c06a5b0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7f18009900 tx=0x7f7f1802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.363+0000 7f7f240c1640 1 -- 172.21.15.94:0/2360247043 shutdown_connections 2024-04-21T08:45:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.363+0000 7f7f240c1640 1 --2- 172.21.15.94:0/2360247043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c06aaf0 0x7f7f1c06af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.363+0000 7f7f240c1640 1 --2- 172.21.15.94:0/2360247043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1c104930 0x7f7f1c104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.363+0000 7f7f240c1640 1 --2- 172.21.15.94:0/2360247043 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f1c106f40 0x7f7f1c06a5b0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.363+0000 7f7f240c1640 1 -- 172.21.15.94:0/2360247043 >> 172.21.15.94:0/2360247043 conn(0x7f7f1c077c40 msgr2=0x7f7f1c078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.363+0000 7f7f240c1640 1 -- 172.21.15.94:0/2360247043 shutdown_connections 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.363+0000 7f7f240c1640 1 -- 172.21.15.94:0/2360247043 wait complete. 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f240c1640 1 Processor -- start 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f240c1640 1 -- start start 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f240c1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1c06aaf0 0x7f7f1c1a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f240c1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c104930 0x7f7f1c1a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f240c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f1c106f40 0x7f7f1c19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f240c1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c1186f0 con 0x7f7f1c106f40 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f240c1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c118570 con 0x7f7f1c06aaf0 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f240c1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c118870 con 0x7f7f1c104930 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f2265e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f1c106f40 0x7f7f1c19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f2265e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f1c106f40 0x7f7f1c19eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35136/0 (socket says 172.21.15.94:35136) 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f2265e640 1 -- 172.21.15.94:0/2403191999 learned_addr learned my addr 172.21.15.94:0/2403191999 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f2165c640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c104930 0x7f7f1c1a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.364+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1c06aaf0 0x7f7f1c1a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f2265e640 1 -- 172.21.15.94:0/2403191999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c104930 msgr2=0x7f7f1c1a4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f2265e640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c104930 0x7f7f1c1a4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f2265e640 1 -- 172.21.15.94:0/2403191999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1c06aaf0 msgr2=0x7f7f1c1a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f2265e640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1c06aaf0 0x7f7f1c1a4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f2265e640 1 -- 172.21.15.94:0/2403191999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1c19f230 con 0x7f7f1c106f40 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1c06aaf0 0x7f7f1c1a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f2265e640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f1c106f40 0x7f7f1c19eab0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400cad0 tx=0x7f7f1400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f12ffd640 1 -- 172.21.15.94:0/2403191999 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1400ae20 con 0x7f7f1c106f40 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f12ffd640 1 -- 172.21.15.94:0/2403191999 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f14005020 con 0x7f7f1c106f40 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f240c1640 1 -- 172.21.15.94:0/2403191999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1c19f520 con 0x7f7f1c106f40 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f12ffd640 1 -- 172.21.15.94:0/2403191999 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1400f6b0 con 0x7f7f1c106f40 2024-04-21T08:45:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.365+0000 7f7f240c1640 1 -- 172.21.15.94:0/2403191999 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f1c1ab7c0 con 0x7f7f1c106f40 2024-04-21T08:45:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.367+0000 7f7f12ffd640 1 -- 172.21.15.94:0/2403191999 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f7f140051d0 con 0x7f7f1c106f40 2024-04-21T08:45:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.367+0000 7f7f26d8c740 1 Processor -- start 2024-04-21T08:45:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f26d8c740 1 -- start start 2024-04-21T08:45:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c42960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f12ffd640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f080a3d20 0x7f7f080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 0x564c91c4b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c4ba80 0x564c91c4de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564c91ba71e0 con 0x564c91c4ba80 2024-04-21T08:45:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564c91b44390 con 0x564c91c42580 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564c91ba31a0 con 0x564c91c42ea0 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f2265e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c4ba80 0x564c91c4de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f080a3d20 0x7f7f080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f2265e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c4ba80 0x564c91c4de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35144/0 (socket says 172.21.15.94:35144) 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f2265e640 1 -- 172.21.15.94:0/1656782901 learned_addr learned my addr 172.21.15.94:0/1656782901 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f2165c640 1 --2- 172.21.15.94:0/1656782901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c42960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f12ffd640 1 -- 172.21.15.94:0/2403191999 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f7f140d0c20 con 0x7f7f1c106f40 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f2165c640 1 --2- 172.21.15.94:0/1656782901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 0x564c91c4b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f2265e640 1 -- 172.21.15.94:0/1656782901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 msgr2=0x564c91c4b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f2265e640 1 --2- 172.21.15.94:0/1656782901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 0x564c91c4b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f2265e640 1 -- 172.21.15.94:0/1656782901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 msgr2=0x564c91c42960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.368+0000 7f7f2265e640 1 --2- 172.21.15.94:0/1656782901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c42960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.369+0000 7f7f2265e640 1 -- 172.21.15.94:0/1656782901 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c919837d0 con 0x564c91c4ba80 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.369+0000 7f7f2165c640 1 --2- 172.21.15.94:0/1656782901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 0x564c91c4b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.369+0000 7f7f2265e640 1 --2- 172.21.15.94:0/1656782901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c4ba80 0x564c91c4de70 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400a6e0 tx=0x7f7f14005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76404bc5155db93 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.369+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f080a3d20 0x7f7f080a61e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7f18009230 tx=0x7f7f18005a40 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.369+0000 7f7f07fff640 1 -- 172.21.15.94:0/1656782901 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1401d070 con 0x564c91c4ba80 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.369+0000 7f7f07fff640 1 -- 172.21.15.94:0/1656782901 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f140a6310 con 0x564c91c4ba80 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.369+0000 7f7f07fff640 1 -- 172.21.15.94:0/1656782901 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f140ce210 con 0x564c91c4ba80 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 -- 172.21.15.94:0/1656782901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c4ba80 msgr2=0x564c91c4de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/1656782901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c4ba80 0x564c91c4de70 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400a6e0 tx=0x7f7f14005890 comp rx=0 tx=0).stop 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 -- 172.21.15.94:0/1656782901 shutdown_connections 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/1656782901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 0x564c91c4b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/1656782901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c4ba80 0x564c91c4de70 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/1656782901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c42960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 -- 172.21.15.94:0/1656782901 >> 172.21.15.94:0/1656782901 conn(0x564c91c3ccb0 msgr2=0x564c91c3d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 -- 172.21.15.94:0/1656782901 shutdown_connections 2024-04-21T08:45:27.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 -- 172.21.15.94:0/1656782901 wait complete. 2024-04-21T08:45:27.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 -- start start 2024-04-21T08:45:27.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.370+0000 7f7f26d8c740 1 Processor -- start 2024-04-21T08:45:27.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c48b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c49060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 0x564c91c495a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564c91c4e7b0 con 0x564c91c42ea0 2024-04-21T08:45:27.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564c91b602f0 con 0x564c91c42580 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564c91c4e930 con 0x564c91c4ba80 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f26d8c740 10 client.0 _ll_register_callbacks cb 0x564c91b6d580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f2165c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c49060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f2165c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c49060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35146/0 (socket says 172.21.15.94:35146) 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f2165c640 1 -- 172.21.15.94:0/2344955475 learned_addr learned my addr 172.21.15.94:0/2344955475 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c48b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f2265e640 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 0x564c91c495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f2165c640 1 -- 172.21.15.94:0/2344955475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 msgr2=0x564c91c495a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f2165c640 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 0x564c91c495a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f2165c640 1 -- 172.21.15.94:0/2344955475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 msgr2=0x564c91c48b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f2165c640 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c48b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f2165c640 1 -- 172.21.15.94:0/2344955475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c918b93f0 con 0x564c91c42ea0 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f2165c640 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c49060 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0126b0 tx=0x7f7f0c012b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeac567f2e16be12 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f057fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f057fa640 1 -- 172.21.15.94:0/2344955475 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c011d70 con 0x564c91c42ea0 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f057fa640 1 -- 172.21.15.94:0/2344955475 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f7f0c004030 con 0x564c91c42ea0 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f26d8c740 1 -- 172.21.15.94:0/2344955475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f140a6310 con 0x564c91c42ea0 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.371+0000 7f7f057fa640 1 -- 172.21.15.94:0/2344955475 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7f0c012e20 con 0x564c91c42ea0 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f26d8c740 10 client.17127 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f26d8c740 1 -- 172.21.15.94:0/2344955475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564c918790f0 con 0x564c91c42ea0 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f057fa640 1 -- 172.21.15.94:0/2344955475 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f140a6310 con 0x564c91c42ea0 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f057fa640 1 -- 172.21.15.94:0/2344955475 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7f0c01c070 con 0x564c91c42ea0 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f26d8c740 10 client.17127 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f26d8c740 10 client.17127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f26d8c740 20 client.17127 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564c91c5ba70 0x564c91c5df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f26d8c740 4 client.17127 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f26d8c740 1 -- 172.21.15.94:0/2344955475 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c91bb4c00 con 0x564c91c5ba70 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.372+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564c91c5ba70 0x564c91c5df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.373+0000 7f7f057fa640 10 client.17127 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.373+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564c91c5ba70 0x564c91c5df10 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32d7f0d9b2aba2d2 server_cookie=3fc4dd66ac34797c in_seq=0 out_seq=0 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.373+0000 7f7f057fa640 1 -- 172.21.15.94:0/2344955475 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c91bb4c00 con 0x564c91c5ba70 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.373+0000 7f7f057fa640 10 client.17127 handle_command_reply: tid=0 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.373+0000 7f7f26d8c740 1 client.17127 shutdown 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.373+0000 7f7f26d8c740 1 -- 172.21.15.94:0/2344955475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 msgr2=0x564c91c49060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.373+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c49060 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0126b0 tx=0x7f7f0c012b70 comp rx=0 tx=0).stop 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.374+0000 7f7f26d8c740 1 -- 172.21.15.94:0/2344955475 shutdown_connections 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.374+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564c91c5ba70 0x564c91c5df10 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.374+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 0x564c91c495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.374+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c49060 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.374+0000 7f7f057fa640 0 client.17127 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.374+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/2344955475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c48b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.374+0000 7f7f26d8c740 1 -- 172.21.15.94:0/2344955475 >> 172.21.15.94:0/2344955475 conn(0x564c91c3ccb0 msgr2=0x564c91c3ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.374+0000 7f7f26d8c740 1 -- 172.21.15.94:0/2344955475 shutdown_connections 2024-04-21T08:45:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.374+0000 7f7f26d8c740 1 -- 172.21.15.94:0/2344955475 wait complete. 2024-04-21T08:45:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.374+0000 7f7f26d8c740 20 client.17127 trim_cache size 0 max 16384 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f26d8c740 1 Processor -- start 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f26d8c740 1 -- start start 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c4b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c57670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 0x564c91c491f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c1b8360 con 0x564c91c42ea0 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c1b5c30 con 0x564c91c42580 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c1b3a80 con 0x564c91c4ba80 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f2165c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c57670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f2165c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c57670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35156/0 (socket says 172.21.15.94:35156) 2024-04-21T08:45:27.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.391+0000 7f7f2165c640 1 -- 172.21.15.94:0/1378486241 learned_addr learned my addr 172.21.15.94:0/1378486241 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f2165c640 1 -- 172.21.15.94:0/1378486241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 msgr2=0x564c91c491f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f2265e640 1 --2- 172.21.15.94:0/1378486241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 0x564c91c491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/1378486241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c4b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f2165c640 1 --2- 172.21.15.94:0/1378486241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 0x564c91c491f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f2165c640 1 -- 172.21.15.94:0/1378486241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 msgr2=0x564c91c4b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f2165c640 1 --2- 172.21.15.94:0/1378486241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c4b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f2165c640 1 -- 172.21.15.94:0/1378486241 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c91c4e1a0 con 0x564c91c42ea0 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f2265e640 1 --2- 172.21.15.94:0/1378486241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 0x564c91c491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f2165c640 1 --2- 172.21.15.94:0/1378486241 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c57670 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00bd60 tx=0x7f7f0c00bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a031e1a1d7dced server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f077fe640 1 -- 172.21.15.94:0/1378486241 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c031030 con 0x564c91c42ea0 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f077fe640 1 -- 172.21.15.94:0/1378486241 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f0c003fc0 con 0x564c91c42ea0 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.392+0000 7f7f077fe640 1 -- 172.21.15.94:0/1378486241 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c000e60 con 0x564c91c42ea0 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 -- 172.21.15.94:0/1378486241 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 msgr2=0x564c91c57670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/1378486241 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c57670 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00bd60 tx=0x7f7f0c00bd90 comp rx=0 tx=0).stop 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 -- 172.21.15.94:0/1378486241 shutdown_connections 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/1378486241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c4ba80 0x564c91c491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/1378486241 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42ea0 0x564c91c57670 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/1378486241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c42580 0x564c91c4b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 -- 172.21.15.94:0/1378486241 >> 172.21.15.94:0/1378486241 conn(0x564c91c3ccb0 msgr2=0x564c91c48ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 -- 172.21.15.94:0/1378486241 shutdown_connections 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 -- 172.21.15.94:0/1378486241 wait complete. 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 -- start start 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 Processor -- start 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42580 0x564c91c4f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 0x564c91c4fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c4ba80 0x564c91c50080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c1b40f0 con 0x564c91c42580 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c1b5560 con 0x564c91c4ba80 2024-04-21T08:45:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c1b47c0 con 0x564c91c42ea0 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.393+0000 7f7f26d8c740 10 client.0 _ll_register_callbacks cb 0x7f7f0c00bef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f21e5d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42580 0x564c91c4f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f21e5d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42580 0x564c91c4f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35172/0 (socket says 172.21.15.94:35172) 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f21e5d640 1 -- 172.21.15.94:0/3685451320 learned_addr learned my addr 172.21.15.94:0/3685451320 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f2165c640 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 0x564c91c4fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f2265e640 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c4ba80 0x564c91c50080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f21e5d640 1 -- 172.21.15.94:0/3685451320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 msgr2=0x564c91c4fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 0x564c91c4fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f21e5d640 1 -- 172.21.15.94:0/3685451320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c4ba80 msgr2=0x564c91c50080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c4ba80 0x564c91c50080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f21e5d640 1 -- 172.21.15.94:0/3685451320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c91cc10c0 con 0x564c91c42580 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f21e5d640 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42580 0x564c91c4f600 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f7f180045a0 tx=0x7f7f18009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20f3162beadba0c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f057fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f057fa640 1 -- 172.21.15.94:0/3685451320 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f18055070 con 0x564c91c42580 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f057fa640 1 -- 172.21.15.94:0/3685451320 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f7f180511d0 con 0x564c91c42580 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f26d8c740 1 -- 172.21.15.94:0/3685451320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f0c003fc0 con 0x564c91c42580 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f057fa640 1 -- 172.21.15.94:0/3685451320 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7f18061050 con 0x564c91c42580 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f26d8c740 10 client.17133 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f26d8c740 1 -- 172.21.15.94:0/3685451320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564c91cc2360 con 0x564c91c42580 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.394+0000 7f7f057fa640 1 -- 172.21.15.94:0/3685451320 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f0c003fc0 con 0x564c91c42580 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f057fa640 1 -- 172.21.15.94:0/3685451320 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7f18004b40 con 0x564c91c42580 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f26d8c740 10 client.17133 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f26d8c740 10 client.17133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f26d8c740 20 client.17133 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564c91cc2a60 0x564c91cc4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f26d8c740 4 client.17133 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f26d8c740 1 -- 172.21.15.94:0/3685451320 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564c91baea70 con 0x564c91cc2a60 2024-04-21T08:45:27.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f2165c640 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564c91cc2a60 0x564c91cc4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f057fa640 10 client.17133 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f2165c640 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564c91cc2a60 0x564c91cc4e80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b37460cced2fa786 server_cookie=ff1f2130acff82fe in_seq=0 out_seq=0 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.395+0000 7f7f057fa640 1 -- 172.21.15.94:0/3685451320 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564c91baea70 con 0x564c91cc2a60 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f057fa640 10 client.17133 handle_command_reply: tid=0 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 client.17133 shutdown 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 -- 172.21.15.94:0/3685451320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42580 msgr2=0x564c91c4f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42580 0x564c91c4f600 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f7f180045a0 tx=0x7f7f18009260 comp rx=0 tx=0).stop 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 -- 172.21.15.94:0/3685451320 shutdown_connections 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564c91cc2a60 0x564c91cc4e80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c91c42ea0 0x564c91c4fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c91c4ba80 0x564c91c50080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f057fa640 0 client.17133 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 --2- 172.21.15.94:0/3685451320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564c91c42580 0x564c91c4f600 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 -- 172.21.15.94:0/3685451320 >> 172.21.15.94:0/3685451320 conn(0x564c91c3ccb0 msgr2=0x564c91c3bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 -- 172.21.15.94:0/3685451320 shutdown_connections 2024-04-21T08:45:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 1 -- 172.21.15.94:0/3685451320 wait complete. 2024-04-21T08:45:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f26d8c740 20 client.17133 trim_cache size 0 max 16384 2024-04-21T08:45:27.403 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:45:27.403 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:45:27.403 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:45:27.403 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:45:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f240c1640 1 -- 172.21.15.94:0/2403191999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f080a3d20 msgr2=0x7f7f080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f240c1640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f080a3d20 0x7f7f080a61e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7f18009230 tx=0x7f7f18005a40 comp rx=0 tx=0).stop 2024-04-21T08:45:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f240c1640 1 -- 172.21.15.94:0/2403191999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f1c106f40 msgr2=0x7f7f1c19eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f240c1640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f1c106f40 0x7f7f1c19eab0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400cad0 tx=0x7f7f1400cf90 comp rx=0 tx=0).stop 2024-04-21T08:45:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f240c1640 1 -- 172.21.15.94:0/2403191999 shutdown_connections 2024-04-21T08:45:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f240c1640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f080a3d20 0x7f7f080a61e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f240c1640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c104930 0x7f7f1c1a4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f240c1640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f1c106f40 0x7f7f1c19eab0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f240c1640 1 --2- 172.21.15.94:0/2403191999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1c06aaf0 0x7f7f1c1a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.396+0000 7f7f240c1640 1 -- 172.21.15.94:0/2403191999 >> 172.21.15.94:0/2403191999 conn(0x7f7f1c077c40 msgr2=0x7f7f1c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.397+0000 7f7f240c1640 1 -- 172.21.15.94:0/2403191999 shutdown_connections 2024-04-21T08:45:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.397+0000 7f7f240c1640 1 -- 172.21.15.94:0/2403191999 wait complete. 2024-04-21T08:45:27.407 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:45:27.408 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:27.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.506+0000 7ff79d975640 1 -- 172.21.15.94:0/3448868198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7981021a0 msgr2=0x7ff798102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.506+0000 7ff79d975640 1 --2- 172.21.15.94:0/3448868198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7981021a0 0x7ff798102580 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff788009900 tx=0x7ff78802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:27.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.506+0000 7ff79d975640 1 -- 172.21.15.94:0/3448868198 shutdown_connections 2024-04-21T08:45:27.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.506+0000 7ff79d975640 1 --2- 172.21.15.94:0/3448868198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff798107270 0x7ff79810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.506+0000 7ff79d975640 1 --2- 172.21.15.94:0/3448868198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff798102b50 0x7ff798106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.506+0000 7ff79d975640 1 --2- 172.21.15.94:0/3448868198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7981021a0 0x7ff798102580 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.506+0000 7ff79d975640 1 -- 172.21.15.94:0/3448868198 >> 172.21.15.94:0/3448868198 conn(0x7ff7980793a0 msgr2=0x7ff7980797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.506+0000 7ff79d975640 1 -- 172.21.15.94:0/3448868198 shutdown_connections 2024-04-21T08:45:27.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.506+0000 7ff79d975640 1 -- 172.21.15.94:0/3448868198 wait complete. 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff79d975640 1 Processor -- start 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff79d975640 1 -- start start 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff79d975640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7981021a0 0x7ff798198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff79d975640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff798107270 0x7ff798198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff79d975640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff79819cfe0 0x7ff79819d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff79d975640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff7981121e0 con 0x7ff79819cfe0 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff79d975640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff798112060 con 0x7ff798107270 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff79d975640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff798112360 con 0x7ff7981021a0 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff7977fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff79819cfe0 0x7ff79819d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff7977fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff79819cfe0 0x7ff79819d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35182/0 (socket says 172.21.15.94:35182) 2024-04-21T08:45:27.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff7977fe640 1 -- 172.21.15.94:0/1339844731 learned_addr learned my addr 172.21.15.94:0/1339844731 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff78ffff640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff798107270 0x7ff798198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff796ffd640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7981021a0 0x7ff798198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff7977fe640 1 -- 172.21.15.94:0/1339844731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7981021a0 msgr2=0x7ff798198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff7977fe640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7981021a0 0x7ff798198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff7977fe640 1 -- 172.21.15.94:0/1339844731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff798107270 msgr2=0x7ff798198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff7977fe640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff798107270 0x7ff798198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff7977fe640 1 -- 172.21.15.94:0/1339844731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79819dad0 con 0x7ff79819cfe0 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.507+0000 7ff78ffff640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff798107270 0x7ff798198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.508+0000 7ff7977fe640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff79819cfe0 0x7ff79819d490 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff79000eb10 tx=0x7ff79000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.508+0000 7ff79c973640 1 -- 172.21.15.94:0/1339844731 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790004b20 con 0x7ff79819cfe0 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.508+0000 7ff79c973640 1 -- 172.21.15.94:0/1339844731 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff790004cc0 con 0x7ff79819cfe0 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.508+0000 7ff79c973640 1 -- 172.21.15.94:0/1339844731 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790010670 con 0x7ff79819cfe0 2024-04-21T08:45:27.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.508+0000 7ff79d975640 1 -- 172.21.15.94:0/1339844731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79819ddc0 con 0x7ff79819cfe0 2024-04-21T08:45:27.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.508+0000 7ff79d975640 1 -- 172.21.15.94:0/1339844731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7981a55b0 con 0x7ff79819cfe0 2024-04-21T08:45:27.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.509+0000 7ff79d975640 1 -- 172.21.15.94:0/1339844731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff798069230 con 0x7ff79819cfe0 2024-04-21T08:45:27.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.510+0000 7ff79c973640 1 -- 172.21.15.94:0/1339844731 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff790005000 con 0x7ff79819cfe0 2024-04-21T08:45:27.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.510+0000 7ff79c973640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7840a3d20 0x7ff7840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.510+0000 7ff796ffd640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7840a3d20 0x7ff7840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.510+0000 7ff79c973640 1 -- 172.21.15.94:0/1339844731 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff7900d1400 con 0x7ff79819cfe0 2024-04-21T08:45:27.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.511+0000 7ff796ffd640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7840a3d20 0x7ff7840a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff788000c00 tx=0x7ff78802c6b0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.514+0000 7ff79c973640 1 -- 172.21.15.94:0/1339844731 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff790053ff0 con 0x7ff79819cfe0 2024-04-21T08:45:27.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.733+0000 7ff79d975640 1 -- 172.21.15.94:0/1339844731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff798199a70 con 0x7ff79819cfe0 2024-04-21T08:45:27.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.733+0000 7ff79c973640 1 -- 172.21.15.94:0/1339844731 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7ff79009d6c0 con 0x7ff79819cfe0 2024-04-21T08:45:27.733 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:45:27.733 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:45:27.734 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:45:27.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 -- 172.21.15.94:0/1339844731 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7840a3d20 msgr2=0x7ff7840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7840a3d20 0x7ff7840a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff788000c00 tx=0x7ff78802c6b0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 -- 172.21.15.94:0/1339844731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff79819cfe0 msgr2=0x7ff79819d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff79819cfe0 0x7ff79819d490 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff79000eb10 tx=0x7ff79000c2d0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 -- 172.21.15.94:0/1339844731 shutdown_connections 2024-04-21T08:45:27.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7840a3d20 0x7ff7840a61e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff79819cfe0 0x7ff79819d490 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff798107270 0x7ff798198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 --2- 172.21.15.94:0/1339844731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7981021a0 0x7ff798198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 -- 172.21.15.94:0/1339844731 >> 172.21.15.94:0/1339844731 conn(0x7ff7980793a0 msgr2=0x7ff7980fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 -- 172.21.15.94:0/1339844731 shutdown_connections 2024-04-21T08:45:27.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.735+0000 7ff79d975640 1 -- 172.21.15.94:0/1339844731 wait complete. 2024-04-21T08:45:27.747 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:45:27.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.848+0000 7fe591a8c640 1 -- 172.21.15.94:0/1993400119 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe58c101a50 msgr2=0x7fe58c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.848+0000 7fe591a8c640 1 --2- 172.21.15.94:0/1993400119 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe58c101a50 0x7fe58c105a90 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fe580009900 tx=0x7fe58002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:27.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.848+0000 7fe591a8c640 1 -- 172.21.15.94:0/1993400119 shutdown_connections 2024-04-21T08:45:27.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.848+0000 7fe591a8c640 1 --2- 172.21.15.94:0/1993400119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe58c106170 0x7fe58c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.848+0000 7fe591a8c640 1 --2- 172.21.15.94:0/1993400119 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe58c101a50 0x7fe58c105a90 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.848+0000 7fe591a8c640 1 --2- 172.21.15.94:0/1993400119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe58c1010a0 0x7fe58c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.848+0000 7fe591a8c640 1 -- 172.21.15.94:0/1993400119 >> 172.21.15.94:0/1993400119 conn(0x7fe58c077eb0 msgr2=0x7fe58c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.848+0000 7fe591a8c640 1 -- 172.21.15.94:0/1993400119 shutdown_connections 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.848+0000 7fe591a8c640 1 -- 172.21.15.94:0/1993400119 wait complete. 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.848+0000 7fe591a8c640 1 Processor -- start 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe591a8c640 1 -- start start 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe591a8c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe58c1010a0 0x7fe58c1994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe591a8c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe58c106170 0x7fe58c199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe591a8c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe58c199fa0 0x7fe58c1a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe591a8c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c1110b0 con 0x7fe58c199fa0 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe591a8c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c110f30 con 0x7fe58c1010a0 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe591a8c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c111230 con 0x7fe58c106170 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe58c1010a0 0x7fe58c1994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe58affd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe58c106170 0x7fe58c199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe58bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe58c199fa0 0x7fe58c1a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe58affd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe58c106170 0x7fe58c199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:33046/0 (socket says 172.21.15.94:33046) 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe58c1010a0 0x7fe58c1994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:42538/0 (socket says 172.21.15.94:42538) 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe58bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe58c199fa0 0x7fe58c1a4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35202/0 (socket says 172.21.15.94:35202) 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe58b7fe640 1 -- 172.21.15.94:0/1341231009 learned_addr learned my addr 172.21.15.94:0/1341231009 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.849+0000 7fe58bfff640 1 -- 172.21.15.94:0/1341231009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe58c106170 msgr2=0x7fe58c199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.850+0000 7fe58bfff640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe58c106170 0x7fe58c199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.850+0000 7fe58bfff640 1 -- 172.21.15.94:0/1341231009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe58c1010a0 msgr2=0x7fe58c1994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.850+0000 7fe58bfff640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe58c1010a0 0x7fe58c1994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.850+0000 7fe58bfff640 1 -- 172.21.15.94:0/1341231009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58c1a4d50 con 0x7fe58c199fa0 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.850+0000 7fe58bfff640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe58c199fa0 0x7fe58c1a4550 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe58400ca30 tx=0x7fe58400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.850+0000 7fe588ff9640 1 -- 172.21.15.94:0/1341231009 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe584007d90 con 0x7fe58c199fa0 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.850+0000 7fe591a8c640 1 -- 172.21.15.94:0/1341231009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58c197320 con 0x7fe58c199fa0 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.850+0000 7fe588ff9640 1 -- 172.21.15.94:0/1341231009 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe584005020 con 0x7fe58c199fa0 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.850+0000 7fe588ff9640 1 -- 172.21.15.94:0/1341231009 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe58400f670 con 0x7fe58c199fa0 2024-04-21T08:45:27.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.850+0000 7fe591a8c640 1 -- 172.21.15.94:0/1341231009 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe58c197840 con 0x7fe58c199fa0 2024-04-21T08:45:27.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe588ff9640 1 -- 172.21.15.94:0/1341231009 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fe5840051d0 con 0x7fe58c199fa0 2024-04-21T08:45:27.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe594680740 1 Processor -- start 2024-04-21T08:45:27.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe594680740 1 -- start start 2024-04-21T08:45:27.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe594680740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 0x556e821f88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe594680740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e82201430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe594680740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e82203e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe594680740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556e8215d1f0 con 0x556e82201a00 2024-04-21T08:45:27.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe594680740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556e820fa4b0 con 0x556e821f84d0 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe594680740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556e821591b0 con 0x556e821f8e80 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e82203e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e82203e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35210/0 (socket says 172.21.15.94:35210) 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58affd640 1 -- 172.21.15.94:0/3486082514 learned_addr learned my addr 172.21.15.94:0/3486082514 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe588ff9640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe5740a3d20 0x7fe5740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58affd640 1 --2- 172.21.15.94:0/3486082514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 0x556e821f88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58b7fe640 1 --2- 172.21.15.94:0/3486082514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e82201430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58bfff640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe5740a3d20 0x7fe5740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58affd640 1 -- 172.21.15.94:0/3486082514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 msgr2=0x556e82201430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58affd640 1 --2- 172.21.15.94:0/3486082514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e82201430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58affd640 1 -- 172.21.15.94:0/3486082514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 msgr2=0x556e821f88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58affd640 1 --2- 172.21.15.94:0/3486082514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 0x556e821f88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.853+0000 7fe58affd640 1 -- 172.21.15.94:0/3486082514 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e820ec1b0 con 0x556e82201a00 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.854+0000 7fe588ff9640 1 -- 172.21.15.94:0/1341231009 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe5840d0d00 con 0x7fe58c199fa0 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.854+0000 7fe58affd640 1 --2- 172.21.15.94:0/3486082514 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e82203e50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fe58002c680 tx=0x7fe58002c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a455d12b6af1419 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.854+0000 7fe58bfff640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe5740a3d20 0x7fe5740a61e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe58409c900 tx=0x7fe5840bd040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.854+0000 7fe571ffb640 1 -- 172.21.15.94:0/3486082514 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe58002c9e0 con 0x556e82201a00 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.854+0000 7fe571ffb640 1 -- 172.21.15.94:0/3486082514 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5800054f0 con 0x556e82201a00 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.854+0000 7fe571ffb640 1 -- 172.21.15.94:0/3486082514 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe580004d30 con 0x556e82201a00 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 -- 172.21.15.94:0/3486082514 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 msgr2=0x556e82203e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 --2- 172.21.15.94:0/3486082514 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e82203e50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fe58002c680 tx=0x7fe58002c610 comp rx=0 tx=0).stop 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 -- 172.21.15.94:0/3486082514 shutdown_connections 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 --2- 172.21.15.94:0/3486082514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e82201430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 --2- 172.21.15.94:0/3486082514 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e82203e50 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 --2- 172.21.15.94:0/3486082514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 0x556e821f88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 -- 172.21.15.94:0/3486082514 >> 172.21.15.94:0/3486082514 conn(0x556e821f2ba0 msgr2=0x556e821f2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 -- 172.21.15.94:0/3486082514 shutdown_connections 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 -- 172.21.15.94:0/3486082514 wait complete. 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 -- start start 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 Processor -- start 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f84d0 0x556e82201250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.855+0000 7fe594680740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e821fe5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe594680740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e82201a00 0x556e821feb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe594680740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556e82204910 con 0x556e821f84d0 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe594680740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556e82116500 con 0x556e82201a00 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe594680740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556e82204a90 con 0x556e821f8e80 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe594680740 10 client.0 _ll_register_callbacks cb 0x556e820e4b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe58affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f84d0 0x556e82201250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe58affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f84d0 0x556e82201250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35212/0 (socket says 172.21.15.94:35212) 2024-04-21T08:45:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe58affd640 1 -- 172.21.15.94:0/2506156682 learned_addr learned my addr 172.21.15.94:0/2506156682 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe58b7fe640 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e821fe5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe58affd640 1 -- 172.21.15.94:0/2506156682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 msgr2=0x556e821fe5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe58affd640 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e821fe5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe58affd640 1 -- 172.21.15.94:0/2506156682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e82201a00 msgr2=0x556e821feb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe58affd640 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e82201a00 0x556e821feb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe58affd640 1 -- 172.21.15.94:0/2506156682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e81e6f3f0 con 0x556e821f84d0 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe58affd640 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f84d0 0x556e82201250 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fe580001eb0 tx=0x7fe5800049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93777a24c443c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe56affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe56affd640 1 -- 172.21.15.94:0/2506156682 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe58002cc60 con 0x556e821f84d0 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe56affd640 1 -- 172.21.15.94:0/2506156682 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe58003c5d0 con 0x556e821f84d0 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe594680740 1 -- 172.21.15.94:0/2506156682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5800054f0 con 0x556e821f84d0 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe56affd640 1 -- 172.21.15.94:0/2506156682 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe580060050 con 0x556e821f84d0 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe594680740 10 client.17151 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe594680740 1 -- 172.21.15.94:0/2506156682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556e81f397d0 con 0x556e821f84d0 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.856+0000 7fe56affd640 1 -- 172.21.15.94:0/2506156682 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5800054f0 con 0x556e821f84d0 2024-04-21T08:45:27.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.857+0000 7fe56affd640 1 -- 172.21.15.94:0/2506156682 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe58005b030 con 0x556e821f84d0 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.857+0000 7fe594680740 10 client.17151 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.857+0000 7fe594680740 10 client.17151 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.857+0000 7fe594680740 20 client.17151 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.857+0000 7fe594680740 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x556e82211970 0x556e82213d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.857+0000 7fe594680740 4 client.17151 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.857+0000 7fe594680740 1 -- 172.21.15.94:0/2506156682 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e820fc930 con 0x556e82211970 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.857+0000 7fe58b7fe640 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x556e82211970 0x556e82213d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.857+0000 7fe56affd640 10 client.17151 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.857+0000 7fe58b7fe640 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x556e82211970 0x556e82213d60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ef72319da669b7c6 server_cookie=63d27c81108faeed in_seq=0 out_seq=0 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe56affd640 1 -- 172.21.15.94:0/2506156682 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e820fc930 con 0x556e82211970 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe56affd640 10 client.17151 handle_command_reply: tid=0 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 client.17151 shutdown 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 -- 172.21.15.94:0/2506156682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f84d0 msgr2=0x556e82201250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f84d0 0x556e82201250 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fe580001eb0 tx=0x7fe5800049d0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 -- 172.21.15.94:0/2506156682 shutdown_connections 2024-04-21T08:45:27.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x556e82211970 0x556e82213d60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e821fe5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e82201a00 0x556e821feb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe56affd640 0 client.17151 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:45:27.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 --2- 172.21.15.94:0/2506156682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f84d0 0x556e82201250 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 -- 172.21.15.94:0/2506156682 >> 172.21.15.94:0/2506156682 conn(0x556e821f2ba0 msgr2=0x556e821f48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 -- 172.21.15.94:0/2506156682 shutdown_connections 2024-04-21T08:45:27.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 1 -- 172.21.15.94:0/2506156682 wait complete. 2024-04-21T08:45:27.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.858+0000 7fe594680740 20 client.17151 trim_cache size 0 max 16384 2024-04-21T08:45:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.871+0000 7fe594680740 1 Processor -- start 2024-04-21T08:45:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.871+0000 7fe594680740 1 -- start start 2024-04-21T08:45:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe594680740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 0x556e8220d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe594680740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f8e80 0x556e821f99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe594680740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e82201a00 0x556e82200d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe594680740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c1afa80 con 0x556e821f8e80 2024-04-21T08:45:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe594680740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c1ad350 con 0x556e821f84d0 2024-04-21T08:45:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe594680740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c1ab1a0 con 0x556e82201a00 2024-04-21T08:45:27.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f8e80 0x556e821f99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f8e80 0x556e821f99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35220/0 (socket says 172.21.15.94:35220) 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe58b7fe640 1 -- 172.21.15.94:0/1851532252 learned_addr learned my addr 172.21.15.94:0/1851532252 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe58b7fe640 1 -- 172.21.15.94:0/1851532252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e82201a00 msgr2=0x556e82200d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe58b7fe640 1 --2- 172.21.15.94:0/1851532252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e82201a00 0x556e82200d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe58b7fe640 1 -- 172.21.15.94:0/1851532252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 msgr2=0x556e8220d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe58b7fe640 1 --2- 172.21.15.94:0/1851532252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 0x556e8220d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe58b7fe640 1 -- 172.21.15.94:0/1851532252 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e821f8c20 con 0x556e821f8e80 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe58b7fe640 1 --2- 172.21.15.94:0/1851532252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f8e80 0x556e821f99c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x556e821f9280 tx=0x7fe57c00afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb62d2e3bc8bc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe5717fa640 1 -- 172.21.15.94:0/1851532252 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c021020 con 0x556e821f8e80 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe5717fa640 1 -- 172.21.15.94:0/1851532252 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe57c00f420 con 0x556e821f8e80 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.872+0000 7fe5717fa640 1 -- 172.21.15.94:0/1851532252 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c01c600 con 0x556e821f8e80 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 -- 172.21.15.94:0/1851532252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f8e80 msgr2=0x556e821f99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 --2- 172.21.15.94:0/1851532252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f8e80 0x556e821f99c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x556e821f9280 tx=0x7fe57c00afd0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 -- 172.21.15.94:0/1851532252 shutdown_connections 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 --2- 172.21.15.94:0/1851532252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e82201a00 0x556e82200d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 --2- 172.21.15.94:0/1851532252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e821f8e80 0x556e821f99c0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 --2- 172.21.15.94:0/1851532252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 0x556e8220d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 -- 172.21.15.94:0/1851532252 >> 172.21.15.94:0/1851532252 conn(0x556e821f2ba0 msgr2=0x556e821f92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 -- 172.21.15.94:0/1851532252 shutdown_connections 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 -- 172.21.15.94:0/1851532252 wait complete. 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 -- start start 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.873+0000 7fe594680740 1 Processor -- start 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe594680740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 0x556e8220a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe594680740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e821fafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe594680740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e821fb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe594680740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c1ab810 con 0x556e82201a00 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe594680740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c1acc80 con 0x556e821f84d0 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe594680740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c1abee0 con 0x556e821f8e80 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe594680740 10 client.0 _ll_register_callbacks cb 0x7fe58002c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe58affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e821fb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe58affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e821fb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35224/0 (socket says 172.21.15.94:35224) 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe58affd640 1 -- 172.21.15.94:0/4050291106 learned_addr learned my addr 172.21.15.94:0/4050291106 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe58affd640 1 -- 172.21.15.94:0/4050291106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 msgr2=0x556e821fafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe58affd640 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e821fafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe58affd640 1 -- 172.21.15.94:0/4050291106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 msgr2=0x556e8220a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe58affd640 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 0x556e8220a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe58affd640 1 -- 172.21.15.94:0/4050291106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e821fbd90 con 0x556e82201a00 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe58affd640 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e821fb560 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe5800096c0 tx=0x7fe580004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4465af854e371833 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe56affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:27.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe56affd640 1 -- 172.21.15.94:0/4050291106 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe58002c890 con 0x556e82201a00 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe56affd640 1 -- 172.21.15.94:0/4050291106 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe5800055d0 con 0x556e82201a00 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe594680740 1 -- 172.21.15.94:0/4050291106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe57c00f420 con 0x556e82201a00 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe56affd640 1 -- 172.21.15.94:0/4050291106 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe5800040d0 con 0x556e82201a00 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe594680740 10 client.17157 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.874+0000 7fe594680740 1 -- 172.21.15.94:0/4050291106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556e82277c40 con 0x556e82201a00 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe56affd640 1 -- 172.21.15.94:0/4050291106 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe57c00f420 con 0x556e82201a00 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe56affd640 1 -- 172.21.15.94:0/4050291106 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe58005b040 con 0x556e82201a00 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe594680740 10 client.17157 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe594680740 10 client.17157 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe594680740 20 client.17157 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe594680740 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x556e82278300 0x556e8227a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe594680740 4 client.17157 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe594680740 1 -- 172.21.15.94:0/4050291106 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e8216ac10 con 0x556e82278300 2024-04-21T08:45:27.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe58b7fe640 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x556e82278300 0x556e8227a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe56affd640 10 client.17157 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.875+0000 7fe58b7fe640 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x556e82278300 0x556e8227a720 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a4608f326d32b040 server_cookie=26818093663b4fef in_seq=0 out_seq=0 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe56affd640 1 -- 172.21.15.94:0/4050291106 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e8216ac10 con 0x556e82278300 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe56affd640 10 client.17157 handle_command_reply: tid=0 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 client.17157 shutdown 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 -- 172.21.15.94:0/4050291106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 msgr2=0x556e821fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e821fb560 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe5800096c0 tx=0x7fe580004530 comp rx=0 tx=0).stop 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 -- 172.21.15.94:0/4050291106 shutdown_connections 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x556e82278300 0x556e8227a720 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e821f8e80 0x556e821fafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe56affd640 0 client.17157 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556e82201a00 0x556e821fb560 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 --2- 172.21.15.94:0/4050291106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e821f84d0 0x556e8220a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 -- 172.21.15.94:0/4050291106 >> 172.21.15.94:0/4050291106 conn(0x556e821f2ba0 msgr2=0x556e821f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 -- 172.21.15.94:0/4050291106 shutdown_connections 2024-04-21T08:45:27.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 1 -- 172.21.15.94:0/4050291106 wait complete. 2024-04-21T08:45:27.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe594680740 20 client.17157 trim_cache size 0 max 16384 2024-04-21T08:45:27.883 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:45:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe591a8c640 1 -- 172.21.15.94:0/1341231009 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe5740a3d20 msgr2=0x7fe5740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe591a8c640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe5740a3d20 0x7fe5740a61e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe58409c900 tx=0x7fe5840bd040 comp rx=0 tx=0).stop 2024-04-21T08:45:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe591a8c640 1 -- 172.21.15.94:0/1341231009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe58c199fa0 msgr2=0x7fe58c1a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe591a8c640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe58c199fa0 0x7fe58c1a4550 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe58400ca30 tx=0x7fe58400cef0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe591a8c640 1 -- 172.21.15.94:0/1341231009 shutdown_connections 2024-04-21T08:45:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe591a8c640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe5740a3d20 0x7fe5740a61e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.876+0000 7fe591a8c640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe58c106170 0x7fe58c199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.877+0000 7fe591a8c640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe58c199fa0 0x7fe58c1a4550 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.877+0000 7fe591a8c640 1 --2- 172.21.15.94:0/1341231009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe58c1010a0 0x7fe58c1994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.877+0000 7fe591a8c640 1 -- 172.21.15.94:0/1341231009 >> 172.21.15.94:0/1341231009 conn(0x7fe58c077eb0 msgr2=0x7fe58c100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.877+0000 7fe591a8c640 1 -- 172.21.15.94:0/1341231009 shutdown_connections 2024-04-21T08:45:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.877+0000 7fe591a8c640 1 -- 172.21.15.94:0/1341231009 wait complete. 2024-04-21T08:45:27.886 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:27 smithi094 ceph-mon[27988]: pgmap v167: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:45:27.887 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:45:27.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:27 smithi111 ceph-mon[31493]: pgmap v167: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:45:27.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.988+0000 7fa55ce3f640 1 -- 172.21.15.94:0/2017272713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa558107260 msgr2=0x7fa55810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.988+0000 7fa55ce3f640 1 --2- 172.21.15.94:0/2017272713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa558107260 0x7fa55810ee30 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fa550009ca0 tx=0x7fa55002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.988+0000 7fa55ce3f640 1 -- 172.21.15.94:0/2017272713 shutdown_connections 2024-04-21T08:45:27.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.988+0000 7fa55ce3f640 1 --2- 172.21.15.94:0/2017272713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa558107260 0x7fa55810ee30 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.988+0000 7fa55ce3f640 1 --2- 172.21.15.94:0/2017272713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa558102b40 0x7fa558106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.988+0000 7fa55ce3f640 1 --2- 172.21.15.94:0/2017272713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa558102190 0x7fa558102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.988+0000 7fa55ce3f640 1 -- 172.21.15.94:0/2017272713 >> 172.21.15.94:0/2017272713 conn(0x7fa5580fd990 msgr2=0x7fa5580ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.988+0000 7fa55ce3f640 1 -- 172.21.15.94:0/2017272713 shutdown_connections 2024-04-21T08:45:27.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.988+0000 7fa55ce3f640 1 -- 172.21.15.94:0/2017272713 wait complete. 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa55ce3f640 1 Processor -- start 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa55ce3f640 1 -- start start 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa55ce3f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa558102190 0x7fa558074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa55ce3f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa558107260 0x7fa558074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa55ce3f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa558074cd0 0x7fa5581a6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa55ce3f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581121d0 con 0x7fa558107260 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa55ce3f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa558112050 con 0x7fa558102190 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa55ce3f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa558112350 con 0x7fa558074cd0 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa54ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa558107260 0x7fa558074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa54ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa558107260 0x7fa558074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35232/0 (socket says 172.21.15.94:35232) 2024-04-21T08:45:27.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa54ffff640 1 -- 172.21.15.94:0/1563329151 learned_addr learned my addr 172.21.15.94:0/1563329151 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa556d9d640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa558074cd0 0x7fa5581a6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa55659c640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa558102190 0x7fa558074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa54ffff640 1 -- 172.21.15.94:0/1563329151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa558074cd0 msgr2=0x7fa5581a6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa54ffff640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa558074cd0 0x7fa5581a6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa54ffff640 1 -- 172.21.15.94:0/1563329151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa558102190 msgr2=0x7fa558074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa54ffff640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa558102190 0x7fa558074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.989+0000 7fa54ffff640 1 -- 172.21.15.94:0/1563329151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5581a7000 con 0x7fa558107260 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.990+0000 7fa54ffff640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa558107260 0x7fa558074790 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fa54000c970 tx=0x7fa54000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.990+0000 7fa54f7fe640 1 -- 172.21.15.94:0/1563329151 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540014070 con 0x7fa558107260 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.990+0000 7fa54f7fe640 1 -- 172.21.15.94:0/1563329151 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa540004440 con 0x7fa558107260 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.990+0000 7fa55ce3f640 1 -- 172.21.15.94:0/1563329151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5581a72f0 con 0x7fa558107260 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.990+0000 7fa54f7fe640 1 -- 172.21.15.94:0/1563329151 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540003e50 con 0x7fa558107260 2024-04-21T08:45:27.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.990+0000 7fa55ce3f640 1 -- 172.21.15.94:0/1563329151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5581a7840 con 0x7fa558107260 2024-04-21T08:45:28.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.992+0000 7fa54f7fe640 1 -- 172.21.15.94:0/1563329151 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa540017070 con 0x7fa558107260 2024-04-21T08:45:28.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.992+0000 7fa55fa2f740 1 Processor -- start 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.992+0000 7fa55fa2f740 1 -- start start 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54f7fe640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa5440a3ef0 0x7fa5440a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9af4d0 0x55812b9af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9b8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 0x55812b9bae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55812b9141f0 con 0x55812b9afe80 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55812b8b14b0 con 0x55812b9af4d0 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55812b9101b0 con 0x55812b9b8a00 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa556d9d640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa5440a3ef0 0x7fa5440a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9b8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35242/0 (socket says 172.21.15.94:35242) 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54ffff640 1 -- 172.21.15.94:0/1006115538 learned_addr learned my addr 172.21.15.94:0/1006115538 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54f7fe640 1 -- 172.21.15.94:0/1563329151 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fa5400d0760 con 0x7fa558107260 2024-04-21T08:45:28.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa55659c640 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9af4d0 0x55812b9af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa556d9d640 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 0x55812b9bae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54ffff640 1 -- 172.21.15.94:0/1006115538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 msgr2=0x55812b9bae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54ffff640 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 0x55812b9bae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54ffff640 1 -- 172.21.15.94:0/1006115538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9af4d0 msgr2=0x55812b9af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54ffff640 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9af4d0 0x55812b9af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54ffff640 1 -- 172.21.15.94:0/1006115538 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55812b8a31b0 con 0x55812b9afe80 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa55659c640 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9af4d0 0x55812b9af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa556d9d640 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 0x55812b9bae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54ffff640 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9b8430 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa54009c020 tx=0x7fa5400038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf4825b3362df47 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa556d9d640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa5440a3ef0 0x7fa5440a63b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa550005cc0 tx=0x7fa5500093f0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54cff9640 1 -- 172.21.15.94:0/1006115538 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54009ca20 con 0x55812b9afe80 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54cff9640 1 -- 172.21.15.94:0/1006115538 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5400a5e50 con 0x55812b9afe80 2024-04-21T08:45:28.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.993+0000 7fa54cff9640 1 -- 172.21.15.94:0/1006115538 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400a9d80 con 0x55812b9afe80 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.994+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1006115538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 msgr2=0x55812b9b8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.994+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9b8430 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa54009c020 tx=0x7fa5400038b0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.994+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1006115538 shutdown_connections 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.994+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 0x55812b9bae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.994+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9b8430 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.994+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/1006115538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9af4d0 0x55812b9af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.994+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1006115538 >> 172.21.15.94:0/1006115538 conn(0x55812b9a9ba0 msgr2=0x55812b9a9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.994+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1006115538 shutdown_connections 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.994+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1006115538 wait complete. 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.994+0000 7fa55fa2f740 1 -- start start 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55fa2f740 1 Processor -- start 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9af4d0 0x55812b9b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 0x55812b9b5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 0x55812b9b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55812b9bb970 con 0x55812b9af4d0 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55812b8cd500 con 0x55812b9afe80 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55812b9bbaf0 con 0x55812b9b8a00 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9af4d0 0x55812b9b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55fa2f740 10 client.0 _ll_register_callbacks cb 0x55812b89bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9af4d0 0x55812b9b82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35256/0 (socket says 172.21.15.94:35256) 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55659c640 1 -- 172.21.15.94:0/3746733095 learned_addr learned my addr 172.21.15.94:0/3746733095 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa54ffff640 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 0x55812b9b5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa556d9d640 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 0x55812b9b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55659c640 1 -- 172.21.15.94:0/3746733095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 msgr2=0x55812b9b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55659c640 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 0x55812b9b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55659c640 1 -- 172.21.15.94:0/3746733095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 msgr2=0x55812b9b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55659c640 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 0x55812b9b5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55659c640 1 -- 172.21.15.94:0/3746733095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55812b6263f0 con 0x55812b9af4d0 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa54ffff640 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 0x55812b9b5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa55659c640 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9af4d0 0x55812b9b82d0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fa5480113e0 tx=0x7fa5480118a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce322689443aa298 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa53dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.995+0000 7fa53dffb640 1 -- 172.21.15.94:0/3746733095 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa548005220 con 0x55812b9af4d0 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa53dffb640 1 -- 172.21.15.94:0/3746733095 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fa5480053c0 con 0x55812b9af4d0 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3746733095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5400a5e50 con 0x55812b9af4d0 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa53dffb640 1 -- 172.21.15.94:0/3746733095 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa548024050 con 0x55812b9af4d0 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa55fa2f740 10 client.17169 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3746733095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55812b6f07d0 con 0x55812b9af4d0 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa53dffb640 1 -- 172.21.15.94:0/3746733095 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5400a5e50 con 0x55812b9af4d0 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa53dffb640 1 -- 172.21.15.94:0/3746733095 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa5480128d0 con 0x55812b9af4d0 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa55fa2f740 10 client.17169 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa55fa2f740 10 client.17169 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:45:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa55fa2f740 20 client.17169 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55812b9c8bd0 0x55812b9cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa55fa2f740 4 client.17169 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3746733095 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55812b8b3930 con 0x55812b9c8bd0 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.996+0000 7fa54ffff640 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55812b9c8bd0 0x55812b9cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.997+0000 7fa53dffb640 10 client.17169 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.997+0000 7fa54ffff640 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55812b9c8bd0 0x55812b9cafc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=addab7a1ad3378b4 server_cookie=6fcc0d0abba9df74 in_seq=0 out_seq=0 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.997+0000 7fa53dffb640 1 -- 172.21.15.94:0/3746733095 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55812b8b3930 con 0x55812b9c8bd0 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.997+0000 7fa53dffb640 10 client.17169 handle_command_reply: tid=0 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.997+0000 7fa55fa2f740 1 client.17169 shutdown 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.997+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3746733095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9af4d0 msgr2=0x55812b9b82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.997+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9af4d0 0x55812b9b82d0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fa5480113e0 tx=0x7fa5480118a0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.998+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3746733095 shutdown_connections 2024-04-21T08:45:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.998+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9b8a00 0x55812b9b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.998+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55812b9c8bd0 0x55812b9cafc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.998+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 0x55812b9b5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.998+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/3746733095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9af4d0 0x55812b9b82d0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.998+0000 7fa53dffb640 0 client.17169 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:45:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.998+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3746733095 >> 172.21.15.94:0/3746733095 conn(0x55812b9a9ba0 msgr2=0x55812b9ab8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.998+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3746733095 shutdown_connections 2024-04-21T08:45:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.998+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3746733095 wait complete. 2024-04-21T08:45:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:27.998+0000 7fa55fa2f740 20 client.17169 trim_cache size 0 max 16384 2024-04-21T08:45:28.016 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:27 smithi173 ceph-mon[31428]: pgmap v167: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:45:28.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa55fa2f740 1 Processor -- start 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa55fa2f740 1 -- start start 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 0x55812b9c4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9a8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9b8a00 0x55812b9b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581b3ee0 con 0x55812b9afe80 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581b17b0 con 0x55812b9b8a00 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581af600 con 0x55812b9af4d0 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa54ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9a8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa54ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9a8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35258/0 (socket says 172.21.15.94:35258) 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa54ffff640 1 -- 172.21.15.94:0/3415198095 learned_addr learned my addr 172.21.15.94:0/3415198095 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa55659c640 1 --2- 172.21.15.94:0/3415198095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 0x55812b9c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa54ffff640 1 -- 172.21.15.94:0/3415198095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 msgr2=0x55812b9c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa54ffff640 1 --2- 172.21.15.94:0/3415198095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 0x55812b9c4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa54ffff640 1 -- 172.21.15.94:0/3415198095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9b8a00 msgr2=0x55812b9b7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa54ffff640 1 --2- 172.21.15.94:0/3415198095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9b8a00 0x55812b9b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa54ffff640 1 -- 172.21.15.94:0/3415198095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55812b9b4610 con 0x55812b9afe80 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa55659c640 1 --2- 172.21.15.94:0/3415198095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 0x55812b9c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa54ffff640 1 --2- 172.21.15.94:0/3415198095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9a8a80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fa54009a870 tx=0x7fa5400045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1348fdbfed1dc24 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa53ffff640 1 -- 172.21.15.94:0/3415198095 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400b0070 con 0x55812b9afe80 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa53ffff640 1 -- 172.21.15.94:0/3415198095 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa54009a620 con 0x55812b9afe80 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.011+0000 7fa53ffff640 1 -- 172.21.15.94:0/3415198095 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400a4b00 con 0x55812b9afe80 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3415198095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 msgr2=0x55812b9a8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/3415198095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9a8a80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fa54009a870 tx=0x7fa5400045f0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3415198095 shutdown_connections 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/3415198095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9b8a00 0x55812b9b7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/3415198095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9afe80 0x55812b9a8a80 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/3415198095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 0x55812b9c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3415198095 >> 172.21.15.94:0/3415198095 conn(0x55812b9a9ba0 msgr2=0x55812b9a8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3415198095 shutdown_connections 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 -- 172.21.15.94:0/3415198095 wait complete. 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 -- start start 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.012+0000 7fa55fa2f740 1 Processor -- start 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 0x55812b9c12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 0x55812b9bc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa55fa2f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9b8a00 0x55812b9bcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581afc70 con 0x55812b9b8a00 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581b10e0 con 0x55812b9afe80 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa55fa2f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581b0340 con 0x55812b9af4d0 2024-04-21T08:45:28.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa55fa2f740 10 client.0 _ll_register_callbacks cb 0x7fa54800bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa556d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9b8a00 0x55812b9bcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa556d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9b8a00 0x55812b9bcd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35274/0 (socket says 172.21.15.94:35274) 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa556d9d640 1 -- 172.21.15.94:0/1284751135 learned_addr learned my addr 172.21.15.94:0/1284751135 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa55659c640 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 0x55812b9c12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa54ffff640 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 0x55812b9bc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa556d9d640 1 -- 172.21.15.94:0/1284751135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 msgr2=0x55812b9c12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa556d9d640 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 0x55812b9c12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa556d9d640 1 -- 172.21.15.94:0/1284751135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 msgr2=0x55812b9bc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa556d9d640 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 0x55812b9bc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa556d9d640 1 -- 172.21.15.94:0/1284751135 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55812ba2e0e0 con 0x55812b9b8a00 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa54ffff640 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 0x55812b9bc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa556d9d640 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9b8a00 0x55812b9bcd30 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fa550009ca0 tx=0x7fa5500095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c167b9a5fc08b18 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa53dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.013+0000 7fa53dffb640 1 -- 172.21.15.94:0/1284751135 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550039040 con 0x55812b9b8a00 2024-04-21T08:45:28.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa53dffb640 1 -- 172.21.15.94:0/1284751135 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fa550003d80 con 0x55812b9b8a00 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1284751135 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa54009a620 con 0x55812b9b8a00 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa53dffb640 1 -- 172.21.15.94:0/1284751135 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa550004040 con 0x55812b9b8a00 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa55fa2f740 10 client.17175 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1284751135 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55812ba2f280 con 0x55812b9b8a00 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa53dffb640 1 -- 172.21.15.94:0/1284751135 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa54009a620 con 0x55812b9b8a00 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa53dffb640 1 -- 172.21.15.94:0/1284751135 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa55004d7b0 con 0x55812b9b8a00 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa55fa2f740 10 client.17175 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa55fa2f740 10 client.17175 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa55fa2f740 20 client.17175 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55812ba2fa50 0x55812ba31e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa55fa2f740 4 client.17175 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1284751135 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55812b921c10 con 0x55812ba2fa50 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.014+0000 7fa55659c640 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55812ba2fa50 0x55812ba31e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa53dffb640 10 client.17175 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:45:28.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55659c640 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55812ba2fa50 0x55812ba31e70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=687d0e25b264fac3 server_cookie=25804280791ba469 in_seq=0 out_seq=0 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa53dffb640 1 -- 172.21.15.94:0/1284751135 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55812b921c10 con 0x55812ba2fa50 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa53dffb640 10 client.17175 handle_command_reply: tid=0 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 client.17175 shutdown 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1284751135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9b8a00 msgr2=0x55812b9bcd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9b8a00 0x55812b9bcd30 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fa550009ca0 tx=0x7fa5500095c0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1284751135 shutdown_connections 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55812ba2fa50 0x55812ba31e70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55812b9b8a00 0x55812b9bcd30 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55812b9afe80 0x55812b9bc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa53dffb640 0 client.17175 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 --2- 172.21.15.94:0/1284751135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55812b9af4d0 0x55812b9c12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1284751135 >> 172.21.15.94:0/1284751135 conn(0x55812b9a9ba0 msgr2=0x55812b9b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1284751135 shutdown_connections 2024-04-21T08:45:28.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 1 -- 172.21.15.94:0/1284751135 wait complete. 2024-04-21T08:45:28.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.015+0000 7fa55fa2f740 20 client.17175 trim_cache size 0 max 16384 2024-04-21T08:45:28.026 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:45:28.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 -- 172.21.15.94:0/1563329151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa5440a3ef0 msgr2=0x7fa5440a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa5440a3ef0 0x7fa5440a63b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa550005cc0 tx=0x7fa5500093f0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 -- 172.21.15.94:0/1563329151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa558107260 msgr2=0x7fa558074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa558107260 0x7fa558074790 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fa54000c970 tx=0x7fa54000ce30 comp rx=0 tx=0).stop 2024-04-21T08:45:28.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 -- 172.21.15.94:0/1563329151 shutdown_connections 2024-04-21T08:45:28.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa5440a3ef0 0x7fa5440a63b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa558074cd0 0x7fa5581a6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa558107260 0x7fa558074790 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 --2- 172.21.15.94:0/1563329151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa558102190 0x7fa558074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 -- 172.21.15.94:0/1563329151 >> 172.21.15.94:0/1563329151 conn(0x7fa5580fd990 msgr2=0x7fa5580ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 -- 172.21.15.94:0/1563329151 shutdown_connections 2024-04-21T08:45:28.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.016+0000 7fa55ce3f640 1 -- 172.21.15.94:0/1563329151 wait complete. 2024-04-21T08:45:28.028 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:45:28.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.129+0000 7fb4e6952640 1 -- 172.21.15.94:0/1855781565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4e0103af0 msgr2=0x7fb4e010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.129+0000 7fb4e6952640 1 --2- 172.21.15.94:0/1855781565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4e0103af0 0x7fb4e010bfd0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009900 tx=0x7fb4d402bd60 comp rx=0 tx=0).stop 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.129+0000 7fb4e6952640 1 -- 172.21.15.94:0/1855781565 shutdown_connections 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.129+0000 7fb4e6952640 1 --2- 172.21.15.94:0/1855781565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4e010c510 0x7fb4e010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.129+0000 7fb4e6952640 1 --2- 172.21.15.94:0/1855781565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4e0103af0 0x7fb4e010bfd0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.129+0000 7fb4e6952640 1 --2- 172.21.15.94:0/1855781565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4e01031d0 0x7fb4e01035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.129+0000 7fb4e6952640 1 -- 172.21.15.94:0/1855781565 >> 172.21.15.94:0/1855781565 conn(0x7fb4e0077f50 msgr2=0x7fb4e0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.129+0000 7fb4e6952640 1 -- 172.21.15.94:0/1855781565 shutdown_connections 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.129+0000 7fb4e6952640 1 -- 172.21.15.94:0/1855781565 wait complete. 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4e6952640 1 Processor -- start 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4e6952640 1 -- start start 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4e6952640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4e01031d0 0x7fb4e01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4e6952640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4e0103af0 0x7fb4e0198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4e6952640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4e010c510 0x7fb4e019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4e6952640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e01121a0 con 0x7fb4e01031d0 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4e6952640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e0112020 con 0x7fb4e0103af0 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4e6952640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e0112320 con 0x7fb4e010c510 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4dffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4e01031d0 0x7fb4e01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4dffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4e01031d0 0x7fb4e01986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35294/0 (socket says 172.21.15.94:35294) 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4dffff640 1 -- 172.21.15.94:0/120668710 learned_addr learned my addr 172.21.15.94:0/120668710 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4e4eef640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4e010c510 0x7fb4e019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4e0103af0 0x7fb4e0198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4dffff640 1 -- 172.21.15.94:0/120668710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4e010c510 msgr2=0x7fb4e019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4dffff640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4e010c510 0x7fb4e019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4dffff640 1 -- 172.21.15.94:0/120668710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4e0103af0 msgr2=0x7fb4e0198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4dffff640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4e0103af0 0x7fb4e0198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4dffff640 1 -- 172.21.15.94:0/120668710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e019d6b0 con 0x7fb4e01031d0 2024-04-21T08:45:28.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4e4eef640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4e010c510 0x7fb4e019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:28.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4e0103af0 0x7fb4e0198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:28.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.130+0000 7fb4dffff640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4e01031d0 0x7fb4e01986c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000c920 tx=0x7fb4d000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.131+0000 7fb4dd7fa640 1 -- 172.21.15.94:0/120668710 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0007c70 con 0x7fb4e01031d0 2024-04-21T08:45:28.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.131+0000 7fb4dd7fa640 1 -- 172.21.15.94:0/120668710 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d0007e10 con 0x7fb4e01031d0 2024-04-21T08:45:28.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.131+0000 7fb4e6952640 1 -- 172.21.15.94:0/120668710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4e019d9a0 con 0x7fb4e01031d0 2024-04-21T08:45:28.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.131+0000 7fb4dd7fa640 1 -- 172.21.15.94:0/120668710 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0018820 con 0x7fb4e01031d0 2024-04-21T08:45:28.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.131+0000 7fb4e6952640 1 -- 172.21.15.94:0/120668710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4e01a5230 con 0x7fb4e01031d0 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4dd7fa640 1 -- 172.21.15.94:0/120668710 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fb4d000f450 con 0x7fb4e01031d0 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e9625740 1 Processor -- start 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e9625740 1 -- start start 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980aa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e9625740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b0980021f0 con 0x55b0980a18e0 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e9625740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b097f9f4b0 con 0x55b0980aae10 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e9625740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b097ffe1b0 con 0x55b0980a2290 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e4eef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e4eef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980a1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35306/0 (socket says 172.21.15.94:35306) 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4e4eef640 1 -- 172.21.15.94:0/3030692970 learned_addr learned my addr 172.21.15.94:0/3030692970 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:28.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4dffff640 1 --2- 172.21.15.94:0/3030692970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/3030692970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.133+0000 7fb4dd7fa640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb4c80a3d20 0x7fb4c80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4e4eef640 1 -- 172.21.15.94:0/3030692970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 msgr2=0x55b0980aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4e4eef640 1 --2- 172.21.15.94:0/3030692970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980aa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4e4eef640 1 -- 172.21.15.94:0/3030692970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 msgr2=0x55b0980ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4e4eef640 1 --2- 172.21.15.94:0/3030692970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4e4eef640 1 -- 172.21.15.94:0/3030692970 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b097f911b0 con 0x55b0980a18e0 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4dd7fa640 1 -- 172.21.15.94:0/120668710 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb4d00d0e10 con 0x7fb4e01031d0 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4e4eef640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb4c80a3d20 0x7fb4c80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4e4eef640 1 --2- 172.21.15.94:0/3030692970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980a1cc0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8008ed0 tx=0x7fb4d800d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3b535dc4d20171 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4c67fc640 1 -- 172.21.15.94:0/3030692970 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8011e60 con 0x55b0980a18e0 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4c67fc640 1 -- 172.21.15.94:0/3030692970 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d801ad90 con 0x55b0980a18e0 2024-04-21T08:45:28.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4c67fc640 1 -- 172.21.15.94:0/3030692970 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8019b90 con 0x55b0980a18e0 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.134+0000 7fb4e4eef640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb4c80a3d20 0x7fb4c80a61e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb4d801e000 tx=0x7fb4d801f000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 -- 172.21.15.94:0/3030692970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 msgr2=0x55b0980a1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 --2- 172.21.15.94:0/3030692970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980a1cc0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8008ed0 tx=0x7fb4d800d710 comp rx=0 tx=0).stop 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 -- 172.21.15.94:0/3030692970 shutdown_connections 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 --2- 172.21.15.94:0/3030692970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980aa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 --2- 172.21.15.94:0/3030692970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 --2- 172.21.15.94:0/3030692970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980a1cc0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 -- 172.21.15.94:0/3030692970 >> 172.21.15.94:0/3030692970 conn(0x55b09809bfb0 msgr2=0x55b09809c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 -- 172.21.15.94:0/3030692970 shutdown_connections 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 -- 172.21.15.94:0/3030692970 wait complete. 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 -- start start 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.135+0000 7fb4e9625740 1 Processor -- start 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980aa670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980a7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4e9625740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b0980add80 con 0x55b0980a18e0 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4e9625740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b097fbb500 con 0x55b0980aae10 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4e9625740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b0980adf00 con 0x55b0980a2290 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4e9625740 10 client.0 _ll_register_callbacks cb 0x55b097f89b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4dffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980aa670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4dffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980aa670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35316/0 (socket says 172.21.15.94:35316) 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4dffff640 1 -- 172.21.15.94:0/2155661933 learned_addr learned my addr 172.21.15.94:0/2155661933 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980a7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4e4eef640 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4dffff640 1 -- 172.21.15.94:0/2155661933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 msgr2=0x55b0980a7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4dffff640 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980a7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4dffff640 1 -- 172.21.15.94:0/2155661933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 msgr2=0x55b0980a7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4dffff640 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980a7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4dffff640 1 -- 172.21.15.94:0/2155661933 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b097d143f0 con 0x55b0980a18e0 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4dffff640 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980aa670 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0098dd0 tx=0x7fb4d0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf6d64daf074fb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/2155661933 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d000de20 con 0x55b0980a18e0 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/2155661933 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb4d00ce3a0 con 0x55b0980a18e0 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4e9625740 1 -- 172.21.15.94:0/2155661933 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4d801ad90 con 0x55b0980a18e0 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/2155661933 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb4d00ce5a0 con 0x55b0980a18e0 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.136+0000 7fb4e9625740 10 client.17187 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.137+0000 7fb4e9625740 1 -- 172.21.15.94:0/2155661933 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b097dde7d0 con 0x55b0980a18e0 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.137+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/2155661933 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d801ad90 con 0x55b0980a18e0 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.137+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/2155661933 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb4d00870b0 con 0x55b0980a18e0 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.137+0000 7fb4e9625740 10 client.17187 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.137+0000 7fb4e9625740 10 client.17187 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:28.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.137+0000 7fb4e9625740 20 client.17187 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.137+0000 7fb4e9625740 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b0980bafe0 0x55b0980bd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.137+0000 7fb4e9625740 4 client.17187 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.137+0000 7fb4e9625740 1 -- 172.21.15.94:0/2155661933 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b097fa1930 con 0x55b0980bafe0 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.137+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b0980bafe0 0x55b0980bd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.138+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b0980bafe0 0x55b0980bd410 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72d0f84e1f796e90 server_cookie=1a6eb51fcab01804 in_seq=0 out_seq=0 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.138+0000 7fb4bf7fe640 10 client.17187 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.138+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/2155661933 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b097fa1930 con 0x55b0980bafe0 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.138+0000 7fb4bf7fe640 10 client.17187 handle_command_reply: tid=0 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.138+0000 7fb4e9625740 1 client.17187 shutdown 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.138+0000 7fb4e9625740 1 -- 172.21.15.94:0/2155661933 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 msgr2=0x55b0980aa670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.138+0000 7fb4e9625740 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980aa670 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0098dd0 tx=0x7fb4d0020000 comp rx=0 tx=0).stop 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.138+0000 7fb4e9625740 1 -- 172.21.15.94:0/2155661933 shutdown_connections 2024-04-21T08:45:28.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.138+0000 7fb4e9625740 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b0980bafe0 0x55b0980bd410 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.139+0000 7fb4e9625740 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980a7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.139+0000 7fb4e9625740 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980a7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.139+0000 7fb4bf7fe640 0 client.17187 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:28.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.139+0000 7fb4e9625740 1 --2- 172.21.15.94:0/2155661933 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980aa670 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.139+0000 7fb4e9625740 1 -- 172.21.15.94:0/2155661933 >> 172.21.15.94:0/2155661933 conn(0x55b09809bfb0 msgr2=0x55b09809dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.139+0000 7fb4e9625740 1 -- 172.21.15.94:0/2155661933 shutdown_connections 2024-04-21T08:45:28.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.139+0000 7fb4e9625740 1 -- 172.21.15.94:0/2155661933 wait complete. 2024-04-21T08:45:28.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.139+0000 7fb4e9625740 20 client.17187 trim_cache size 0 max 16384 2024-04-21T08:45:28.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4e9625740 1 Processor -- start 2024-04-21T08:45:28.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4e9625740 1 -- start start 2024-04-21T08:45:28.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980aa130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980aa670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4e9625740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e01afb90 con 0x55b0980a18e0 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4e9625740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e01ad460 con 0x55b0980aae10 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4e9625740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e01ab2b0 con 0x55b0980a2290 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4dffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4dffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980b6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35320/0 (socket says 172.21.15.94:35320) 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4dffff640 1 -- 172.21.15.94:0/2349246826 learned_addr learned my addr 172.21.15.94:0/2349246826 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/2349246826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980aa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4dffff640 1 -- 172.21.15.94:0/2349246826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 msgr2=0x55b0980aa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4dffff640 1 --2- 172.21.15.94:0/2349246826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980aa130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4dffff640 1 -- 172.21.15.94:0/2349246826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 msgr2=0x55b0980aa670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4dffff640 1 --2- 172.21.15.94:0/2349246826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980aa670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4dffff640 1 -- 172.21.15.94:0/2349246826 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0980a0b40 con 0x55b0980a18e0 2024-04-21T08:45:28.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/2349246826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980aa130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4dffff640 1 --2- 172.21.15.94:0/2349246826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980b6c40 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0099340 tx=0x7fb4d00076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddadf2f3179e48c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.152+0000 7fb4c5ffb640 1 -- 172.21.15.94:0/2349246826 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d00a55e0 con 0x55b0980a18e0 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4c5ffb640 1 -- 172.21.15.94:0/2349246826 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d009c970 con 0x55b0980a18e0 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4c5ffb640 1 -- 172.21.15.94:0/2349246826 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d009c0f0 con 0x55b0980a18e0 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4e9625740 1 -- 172.21.15.94:0/2349246826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 msgr2=0x55b0980b6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4e9625740 1 --2- 172.21.15.94:0/2349246826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980b6c40 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0099340 tx=0x7fb4d00076c0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4e9625740 1 -- 172.21.15.94:0/2349246826 shutdown_connections 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4e9625740 1 --2- 172.21.15.94:0/2349246826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a2290 0x55b0980aa130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4e9625740 1 --2- 172.21.15.94:0/2349246826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980aa670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4e9625740 1 --2- 172.21.15.94:0/2349246826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a18e0 0x55b0980b6c40 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4e9625740 1 -- 172.21.15.94:0/2349246826 >> 172.21.15.94:0/2349246826 conn(0x55b09809bfb0 msgr2=0x55b098096c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4e9625740 1 -- 172.21.15.94:0/2349246826 shutdown_connections 2024-04-21T08:45:28.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4e9625740 1 -- 172.21.15.94:0/2349246826 wait complete. 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.153+0000 7fb4e9625740 1 -- start start 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4e9625740 1 Processor -- start 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a18e0 0x55b0980b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a2290 0x55b0980ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4e9625740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980aee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4e9625740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e01ab920 con 0x55b0980a2290 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4e9625740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e01acd90 con 0x55b0980aae10 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4e9625740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e01abff0 con 0x55b0980a18e0 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4e9625740 10 client.0 _ll_register_callbacks cb 0x7fb4d000cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4df7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a2290 0x55b0980ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4df7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a2290 0x55b0980ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35324/0 (socket says 172.21.15.94:35324) 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4df7fe640 1 -- 172.21.15.94:0/3358085937 learned_addr learned my addr 172.21.15.94:0/3358085937 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:28.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4dffff640 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a18e0 0x55b0980b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4df7fe640 1 -- 172.21.15.94:0/3358085937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a18e0 msgr2=0x55b0980b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4e4eef640 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a18e0 0x55b0980b3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4df7fe640 1 -- 172.21.15.94:0/3358085937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 msgr2=0x55b0980aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980aee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4df7fe640 1 -- 172.21.15.94:0/3358085937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0980af600 con 0x55b0980a2290 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4dffff640 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a18e0 0x55b0980b3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4e4eef640 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4df7fe640 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a2290 0x55b0980ae8e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4005ab0 tx=0x7fb4d4004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbd683d991944ba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.154+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/3358085937 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4048070 con 0x55b0980a2290 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/3358085937 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb4d402cbe0 con 0x55b0980a2290 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4e9625740 1 -- 172.21.15.94:0/3358085937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4d009c970 con 0x55b0980a2290 2024-04-21T08:45:28.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/3358085937 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb4d4054050 con 0x55b0980a2290 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4e9625740 10 client.17193 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4e9625740 1 -- 172.21.15.94:0/3358085937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b098121500 con 0x55b0980a2290 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/3358085937 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d009c970 con 0x55b0980a2290 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/3358085937 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb4d4044500 con 0x55b0980a2290 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4e9625740 10 client.17193 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4e9625740 10 client.17193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4e9625740 20 client.17193 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4e9625740 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b098121cb0 0x55b0981240d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4e9625740 4 client.17193 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4e9625740 1 -- 172.21.15.94:0/3358085937 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b09800fc10 con 0x55b098121cb0 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.155+0000 7fb4dffff640 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b098121cb0 0x55b0981240d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4bf7fe640 10 client.17193 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4dffff640 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b098121cb0 0x55b0981240d0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ed5fae64940d830 server_cookie=c7bdd457608b4385 in_seq=0 out_seq=0 2024-04-21T08:45:28.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4bf7fe640 1 -- 172.21.15.94:0/3358085937 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b09800fc10 con 0x55b098121cb0 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4bf7fe640 10 client.17193 handle_command_reply: tid=0 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 client.17193 shutdown 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 -- 172.21.15.94:0/3358085937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a2290 msgr2=0x55b0980ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a2290 0x55b0980ae8e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4005ab0 tx=0x7fb4d4004590 comp rx=0 tx=0).stop 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 -- 172.21.15.94:0/3358085937 shutdown_connections 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b098121cb0 0x55b0981240d0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b0980aae10 0x55b0980aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b0980a2290 0x55b0980ae8e0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4bf7fe640 0 client.17193 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 --2- 172.21.15.94:0/3358085937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0980a18e0 0x55b0980b3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 -- 172.21.15.94:0/3358085937 >> 172.21.15.94:0/3358085937 conn(0x55b09809bfb0 msgr2=0x55b0980a2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 -- 172.21.15.94:0/3358085937 shutdown_connections 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 1 -- 172.21.15.94:0/3358085937 wait complete. 2024-04-21T08:45:28.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.156+0000 7fb4e9625740 20 client.17193 trim_cache size 0 max 16384 2024-04-21T08:45:28.164 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 -- 172.21.15.94:0/120668710 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb4c80a3d20 msgr2=0x7fb4c80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb4c80a3d20 0x7fb4c80a61e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb4d801e000 tx=0x7fb4d801f000 comp rx=0 tx=0).stop 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 -- 172.21.15.94:0/120668710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4e01031d0 msgr2=0x7fb4e01986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4e01031d0 0x7fb4e01986c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000c920 tx=0x7fb4d000cde0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 -- 172.21.15.94:0/120668710 shutdown_connections 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb4c80a3d20 0x7fb4c80a61e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4e010c510 0x7fb4e019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4e0103af0 0x7fb4e0198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 --2- 172.21.15.94:0/120668710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4e01031d0 0x7fb4e01986c0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 -- 172.21.15.94:0/120668710 >> 172.21.15.94:0/120668710 conn(0x7fb4e0077f50 msgr2=0x7fb4e00fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 -- 172.21.15.94:0/120668710 shutdown_connections 2024-04-21T08:45:28.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:28.157+0000 7fb4e6952640 1 -- 172.21.15.94:0/120668710 wait complete. 2024-04-21T08:45:28.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:28 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1339844731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:29.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:28 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1339844731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:29.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:28 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1339844731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:29.168 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:45:29.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:29.169 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:45:29.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.282+0000 7f39bb29a640 1 -- 172.21.15.94:0/1576240877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39b410c520 msgr2=0x7f39b410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.282+0000 7f39bb29a640 1 --2- 172.21.15.94:0/1576240877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39b410c520 0x7f39b410e9c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f39ac009ca0 tx=0x7f39ac02c230 comp rx=0 tx=0).stop 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.282+0000 7f39bb29a640 1 -- 172.21.15.94:0/1576240877 shutdown_connections 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.282+0000 7f39bb29a640 1 --2- 172.21.15.94:0/1576240877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39b410c520 0x7f39b410e9c0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.282+0000 7f39bb29a640 1 --2- 172.21.15.94:0/1576240877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b4103b00 0x7f39b410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.282+0000 7f39bb29a640 1 --2- 172.21.15.94:0/1576240877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b41031e0 0x7f39b41035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.282+0000 7f39bb29a640 1 -- 172.21.15.94:0/1576240877 >> 172.21.15.94:0/1576240877 conn(0x7f39b40771d0 msgr2=0x7f39b40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.282+0000 7f39bb29a640 1 -- 172.21.15.94:0/1576240877 shutdown_connections 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.282+0000 7f39bb29a640 1 -- 172.21.15.94:0/1576240877 wait complete. 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39bb29a640 1 Processor -- start 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39bb29a640 1 -- start start 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39bb29a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b41031e0 0x7f39b4198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39bb29a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39b4103b00 0x7f39b4198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39bb29a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b410c520 0x7f39b419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39bb29a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41121b0 con 0x7f39b4103b00 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39bb29a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4112030 con 0x7f39b410c520 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39bb29a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4112330 con 0x7f39b41031e0 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39b8835640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39b4103b00 0x7f39b4198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39b8835640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39b4103b00 0x7f39b4198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35334/0 (socket says 172.21.15.94:35334) 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39b8835640 1 -- 172.21.15.94:0/3126200993 learned_addr learned my addr 172.21.15.94:0/3126200993 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39b9837640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b410c520 0x7f39b419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.283+0000 7f39b9036640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b41031e0 0x7f39b4198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39b8835640 1 -- 172.21.15.94:0/3126200993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b41031e0 msgr2=0x7f39b4198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39b8835640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b41031e0 0x7f39b4198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39b8835640 1 -- 172.21.15.94:0/3126200993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b410c520 msgr2=0x7f39b419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39b8835640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b410c520 0x7f39b419cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39b8835640 1 -- 172.21.15.94:0/3126200993 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b419d630 con 0x7f39b4103b00 2024-04-21T08:45:29.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39b9036640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b41031e0 0x7f39b4198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:29.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39b8835640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39b4103b00 0x7f39b4198bd0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c920 tx=0x7f39a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39aa7fc640 1 -- 172.21.15.94:0/3126200993 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4007c70 con 0x7f39b4103b00 2024-04-21T08:45:29.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39aa7fc640 1 -- 172.21.15.94:0/3126200993 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39a4007e10 con 0x7f39b4103b00 2024-04-21T08:45:29.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39bb29a640 1 -- 172.21.15.94:0/3126200993 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b419d920 con 0x7f39b4103b00 2024-04-21T08:45:29.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39aa7fc640 1 -- 172.21.15.94:0/3126200993 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4018820 con 0x7f39b4103b00 2024-04-21T08:45:29.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.284+0000 7f39bb29a640 1 -- 172.21.15.94:0/3126200993 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b41a51b0 con 0x7f39b4103b00 2024-04-21T08:45:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.286+0000 7f39aa7fc640 1 -- 172.21.15.94:0/3126200993 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f39a400f450 con 0x7f39b4103b00 2024-04-21T08:45:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39bdf6e740 1 Processor -- start 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39bdf6e740 1 -- start start 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39aa7fc640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f39a00a3d20 0x7f39a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914777740 0x564914777b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914780c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914781250 0x5649147816d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5649146dbf00 con 0x5649147780f0 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564914679590 con 0x564914781250 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5649146d7ec0 con 0x564914777740 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b9837640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f39a00a3d20 0x7f39a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b8835640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914780c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39aa7fc640 1 -- 172.21.15.94:0/3126200993 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f39a40d0c90 con 0x7f39b4103b00 2024-04-21T08:45:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b8835640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914780c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35342/0 (socket says 172.21.15.94:35342) 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b8835640 1 -- 172.21.15.94:0/2101878727 learned_addr learned my addr 172.21.15.94:0/2101878727 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b9837640 1 --2- 172.21.15.94:0/2101878727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914781250 0x5649147816d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b9036640 1 --2- 172.21.15.94:0/2101878727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914777740 0x564914777b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b8835640 1 -- 172.21.15.94:0/2101878727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914777740 msgr2=0x564914777b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b8835640 1 --2- 172.21.15.94:0/2101878727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914777740 0x564914777b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b8835640 1 -- 172.21.15.94:0/2101878727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914781250 msgr2=0x5649147816d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b8835640 1 --2- 172.21.15.94:0/2101878727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914781250 0x5649147816d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b8835640 1 -- 172.21.15.94:0/2101878727 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649144b67e0 con 0x5649147780f0 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.287+0000 7f39b9837640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f39a00a3d20 0x7f39a00a61e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f39ac004af0 tx=0x7f39ac003b70 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f39b8835640 1 --2- 172.21.15.94:0/2101878727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914780c30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f39a4099220 tx=0x7f39a409c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc71e62961e929 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f399f7fe640 1 -- 172.21.15.94:0/2101878727 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4079c90 con 0x5649147780f0 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f399f7fe640 1 -- 172.21.15.94:0/2101878727 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39a409cf50 con 0x5649147780f0 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f399f7fe640 1 -- 172.21.15.94:0/2101878727 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a407b410 con 0x5649147780f0 2024-04-21T08:45:29.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2101878727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 msgr2=0x564914780c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/2101878727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914780c30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f39a4099220 tx=0x7f39a409c450 comp rx=0 tx=0).stop 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2101878727 shutdown_connections 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/2101878727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914781250 0x5649147816d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/2101878727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914780c30 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/2101878727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914777740 0x564914777b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2101878727 >> 172.21.15.94:0/2101878727 conn(0x564914771fb0 msgr2=0x564914772390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2101878727 shutdown_connections 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.288+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2101878727 wait complete. 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39bdf6e740 1 -- start start 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39bdf6e740 1 Processor -- start 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914777740 0x564914784d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914785280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914781250 0x5649147857c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5649147881d0 con 0x5649147780f0 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564914695580 con 0x564914777740 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564914788350 con 0x564914781250 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39bdf6e740 10 client.0 _ll_register_callbacks cb 0x56491469b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39b8835640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914785280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39b8835640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914785280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35350/0 (socket says 172.21.15.94:35350) 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39b8835640 1 -- 172.21.15.94:0/3045266550 learned_addr learned my addr 172.21.15.94:0/3045266550 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.289+0000 7f39b9837640 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914781250 0x5649147857c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39b9036640 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914777740 0x564914784d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39b8835640 1 -- 172.21.15.94:0/3045266550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914781250 msgr2=0x5649147857c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39b8835640 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914781250 0x5649147857c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39b8835640 1 -- 172.21.15.94:0/3045266550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914777740 msgr2=0x564914784d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39b8835640 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914777740 0x564914784d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39b8835640 1 -- 172.21.15.94:0/3045266550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564914516f70 con 0x5649147780f0 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39b9036640 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914777740 0x564914784d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39b8835640 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914785280 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f39a400fe10 tx=0x7f39a40a5520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca9346e9ec1ca64 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f399cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f399cff9640 1 -- 172.21.15.94:0/3045266550 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a407ba20 con 0x5649147780f0 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f399cff9640 1 -- 172.21.15.94:0/3045266550 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f39a4098ec0 con 0x5649147780f0 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39bdf6e740 1 -- 172.21.15.94:0/3045266550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39a409cf50 con 0x5649147780f0 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f399cff9640 1 -- 172.21.15.94:0/3045266550 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f39a407b3c0 con 0x5649147780f0 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39bdf6e740 10 client.17205 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f39bdf6e740 1 -- 172.21.15.94:0/3045266550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5649143ae170 con 0x5649147780f0 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f399cff9640 1 -- 172.21.15.94:0/3045266550 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39a409cf50 con 0x5649147780f0 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.290+0000 7f399cff9640 1 -- 172.21.15.94:0/3045266550 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f39a40b2070 con 0x5649147780f0 2024-04-21T08:45:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.291+0000 7f39bdf6e740 10 client.17205 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.291+0000 7f39bdf6e740 10 client.17205 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.291+0000 7f39bdf6e740 20 client.17205 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.291+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564914795660 0x564914797a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.291+0000 7f39bdf6e740 4 client.17205 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.291+0000 7f39bdf6e740 1 -- 172.21.15.94:0/3045266550 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649146e9920 con 0x564914795660 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.291+0000 7f39b9036640 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564914795660 0x564914797a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.291+0000 7f399cff9640 10 client.17205 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.291+0000 7f39b9036640 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564914795660 0x564914797a50 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d74122958d89ed00 server_cookie=ff47d0598a642002 in_seq=0 out_seq=0 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f399cff9640 1 -- 172.21.15.94:0/3045266550 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649146e9920 con 0x564914795660 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f399cff9640 10 client.17205 handle_command_reply: tid=0 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 client.17205 shutdown 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 -- 172.21.15.94:0/3045266550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 msgr2=0x564914785280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914785280 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f39a400fe10 tx=0x7f39a40a5520 comp rx=0 tx=0).stop 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 -- 172.21.15.94:0/3045266550 shutdown_connections 2024-04-21T08:45:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564914795660 0x564914797a50 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914781250 0x5649147857c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914785280 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f399cff9640 0 client.17205 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:29.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/3045266550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914777740 0x564914784d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 -- 172.21.15.94:0/3045266550 >> 172.21.15.94:0/3045266550 conn(0x564914771fb0 msgr2=0x564914773d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 -- 172.21.15.94:0/3045266550 shutdown_connections 2024-04-21T08:45:29.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 1 -- 172.21.15.94:0/3045266550 wait complete. 2024-04-21T08:45:29.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.292+0000 7f39bdf6e740 20 client.17205 trim_cache size 0 max 16384 2024-04-21T08:45:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39bdf6e740 1 Processor -- start 2024-04-21T08:45:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39bdf6e740 1 -- start start 2024-04-21T08:45:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914777740 0x564914787470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914782640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914781250 0x564914790cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41afa00 con 0x5649147780f0 2024-04-21T08:45:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41ad2d0 con 0x564914777740 2024-04-21T08:45:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41ab120 con 0x564914781250 2024-04-21T08:45:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39b8835640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914782640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39b8835640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914782640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35352/0 (socket says 172.21.15.94:35352) 2024-04-21T08:45:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39b8835640 1 -- 172.21.15.94:0/921025205 learned_addr learned my addr 172.21.15.94:0/921025205 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39b8835640 1 -- 172.21.15.94:0/921025205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914781250 msgr2=0x564914790cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39b8835640 1 --2- 172.21.15.94:0/921025205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914781250 0x564914790cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39b8835640 1 -- 172.21.15.94:0/921025205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914777740 msgr2=0x564914787470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39b8835640 1 --2- 172.21.15.94:0/921025205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914777740 0x564914787470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39b8835640 1 -- 172.21.15.94:0/921025205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564914776970 con 0x5649147780f0 2024-04-21T08:45:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f39b8835640 1 --2- 172.21.15.94:0/921025205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914782640 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c8f0 tx=0x7f39a409c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d766dc49e68f06a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f399effd640 1 -- 172.21.15.94:0/921025205 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a407ba20 con 0x5649147780f0 2024-04-21T08:45:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f399effd640 1 -- 172.21.15.94:0/921025205 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39a409c850 con 0x5649147780f0 2024-04-21T08:45:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.310+0000 7f399effd640 1 -- 172.21.15.94:0/921025205 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a40a5030 con 0x5649147780f0 2024-04-21T08:45:29.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 -- 172.21.15.94:0/921025205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 msgr2=0x564914782640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/921025205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914782640 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c8f0 tx=0x7f39a409c450 comp rx=0 tx=0).stop 2024-04-21T08:45:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 -- 172.21.15.94:0/921025205 shutdown_connections 2024-04-21T08:45:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/921025205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914781250 0x564914790cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/921025205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5649147780f0 0x564914782640 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/921025205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564914777740 0x564914787470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 -- 172.21.15.94:0/921025205 >> 172.21.15.94:0/921025205 conn(0x564914771fb0 msgr2=0x5649147905e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 -- 172.21.15.94:0/921025205 shutdown_connections 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 -- 172.21.15.94:0/921025205 wait complete. 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 -- start start 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.311+0000 7f39bdf6e740 1 Processor -- start 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914777740 0x564914798d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649147780f0 0x564914799270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39bdf6e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564914781250 0x5649147f4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41ab790 con 0x564914781250 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41acc00 con 0x5649147780f0 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39bdf6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41abe60 con 0x564914777740 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39bdf6e740 10 client.0 _ll_register_callbacks cb 0x7f39a409c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39b9837640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564914781250 0x5649147f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39b9837640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564914781250 0x5649147f4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35360/0 (socket says 172.21.15.94:35360) 2024-04-21T08:45:29.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39b9837640 1 -- 172.21.15.94:0/2615030164 learned_addr learned my addr 172.21.15.94:0/2615030164 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39b9036640 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914777740 0x564914798d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39b9837640 1 -- 172.21.15.94:0/2615030164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914777740 msgr2=0x564914798d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39b9837640 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914777740 0x564914798d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39b9837640 1 -- 172.21.15.94:0/2615030164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649147780f0 msgr2=0x564914799270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39b9837640 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649147780f0 0x564914799270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39b9837640 1 -- 172.21.15.94:0/2615030164 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649147f4e50 con 0x564914781250 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f39b9837640 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564914781250 0x5649147f4810 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f39ac037040 tx=0x7f39ac0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78861747f4ae56af server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f399cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f399cff9640 1 -- 172.21.15.94:0/2615030164 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac039040 con 0x564914781250 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2615030164 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39a409c850 con 0x564914781250 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.312+0000 7f399cff9640 1 -- 172.21.15.94:0/2615030164 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f39ac034850 con 0x564914781250 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f399cff9640 1 -- 172.21.15.94:0/2615030164 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f39ac05e050 con 0x564914781250 2024-04-21T08:45:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f39bdf6e740 10 client.17211 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2615030164 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5649147fc230 con 0x564914781250 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f399cff9640 1 -- 172.21.15.94:0/2615030164 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39a409c850 con 0x564914781250 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f399cff9640 1 -- 172.21.15.94:0/2615030164 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f39ac03f030 con 0x564914781250 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f39bdf6e740 10 client.17211 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f39bdf6e740 10 client.17211 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f39bdf6e740 20 client.17211 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5649147fca30 0x5649147fee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f39bdf6e740 4 client.17211 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2615030164 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5649146e3790 con 0x5649147fca30 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.313+0000 7f39b9036640 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5649147fca30 0x5649147fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.314+0000 7f39b9036640 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5649147fca30 0x5649147fee50 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a2d325f11d87f25 server_cookie=a908789cb22dad50 in_seq=0 out_seq=0 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.314+0000 7f399cff9640 10 client.17211 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f399cff9640 1 -- 172.21.15.94:0/2615030164 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5649146e3790 con 0x5649147fca30 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f399cff9640 10 client.17211 handle_command_reply: tid=0 2024-04-21T08:45:29.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 client.17211 shutdown 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2615030164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564914781250 msgr2=0x5649147f4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564914781250 0x5649147f4810 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f39ac037040 tx=0x7f39ac0095c0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2615030164 shutdown_connections 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5649147fca30 0x5649147fee50 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564914781250 0x5649147f4810 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f399cff9640 0 client.17211 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649147780f0 0x564914799270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 --2- 172.21.15.94:0/2615030164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564914777740 0x564914798d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2615030164 >> 172.21.15.94:0/2615030164 conn(0x564914771fb0 msgr2=0x564914785570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2615030164 shutdown_connections 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 1 -- 172.21.15.94:0/2615030164 wait complete. 2024-04-21T08:45:29.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.315+0000 7f39bdf6e740 20 client.17211 trim_cache size 0 max 16384 2024-04-21T08:45:29.327 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:45:29.327 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:45:29.327 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "9c083425-fdae-4c7e-b19d-415cd69bbcd9", 2024-04-21T08:45:29.327 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:45:29.327 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:45:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 -- 172.21.15.94:0/3126200993 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f39a00a3d20 msgr2=0x7f39a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f39a00a3d20 0x7f39a00a61e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f39ac004af0 tx=0x7f39ac003b70 comp rx=0 tx=0).stop 2024-04-21T08:45:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 -- 172.21.15.94:0/3126200993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39b4103b00 msgr2=0x7f39b4198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39b4103b00 0x7f39b4198bd0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c920 tx=0x7f39a400cde0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 -- 172.21.15.94:0/3126200993 shutdown_connections 2024-04-21T08:45:29.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f39a00a3d20 0x7f39a00a61e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b410c520 0x7f39b419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39b4103b00 0x7f39b4198bd0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 --2- 172.21.15.94:0/3126200993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b41031e0 0x7f39b4198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 -- 172.21.15.94:0/3126200993 >> 172.21.15.94:0/3126200993 conn(0x7f39b40771d0 msgr2=0x7f39b40fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 -- 172.21.15.94:0/3126200993 shutdown_connections 2024-04-21T08:45:29.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.316+0000 7f39bb29a640 1 -- 172.21.15.94:0/3126200993 wait complete. 2024-04-21T08:45:29.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:45:29.329 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:45:29.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 -- 172.21.15.94:0/2489246570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac0106160 msgr2=0x7f1ac010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 --2- 172.21.15.94:0/2489246570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac0106160 0x7f1ac010dd30 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009900 tx=0x7f1ab002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:29.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 -- 172.21.15.94:0/2489246570 shutdown_connections 2024-04-21T08:45:29.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 --2- 172.21.15.94:0/2489246570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac0101a40 0x7f1ac0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 --2- 172.21.15.94:0/2489246570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac0106160 0x7f1ac010dd30 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 --2- 172.21.15.94:0/2489246570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ac0101090 0x7f1ac0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 -- 172.21.15.94:0/2489246570 >> 172.21.15.94:0/2489246570 conn(0x7f1ac00771d0 msgr2=0x7f1ac00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 -- 172.21.15.94:0/2489246570 shutdown_connections 2024-04-21T08:45:29.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 -- 172.21.15.94:0/2489246570 wait complete. 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 Processor -- start 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.430+0000 7f1ac6d75640 1 -- start start 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1ac6d75640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ac0101090 0x7f1ac0197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1ac6d75640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac0106160 0x7f1ac0197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1ac6d75640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac0197da0 0x7f1ac01a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1ac6d75640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac01110d0 con 0x7f1ac0106160 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1ac6d75640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac0110f50 con 0x7f1ac0101090 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1ac6d75640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac0111250 con 0x7f1ac0197da0 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac0106160 0x7f1ac0197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac0106160 0x7f1ac0197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35392/0 (socket says 172.21.15.94:35392) 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abffff640 1 -- 172.21.15.94:0/4277096178 learned_addr learned my addr 172.21.15.94:0/4277096178 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1ac4b11640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ac0101090 0x7f1ac0197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1ac5312640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac0197da0 0x7f1ac01a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abffff640 1 -- 172.21.15.94:0/4277096178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac0197da0 msgr2=0x7f1ac01a69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abffff640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac0197da0 0x7f1ac01a69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abffff640 1 -- 172.21.15.94:0/4277096178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ac0101090 msgr2=0x7f1ac0197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abffff640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ac0101090 0x7f1ac0197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abffff640 1 -- 172.21.15.94:0/4277096178 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac01a6f60 con 0x7f1ac0106160 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1ac5312640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac0197da0 0x7f1ac01a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abffff640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac0106160 0x7f1ac0197860 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800ea20 tx=0x7f1aa800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abdffb640 1 -- 172.21.15.94:0/4277096178 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800cd20 con 0x7f1ac0106160 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abdffb640 1 -- 172.21.15.94:0/4277096178 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1aa80054d0 con 0x7f1ac0106160 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1ac6d75640 1 -- 172.21.15.94:0/4277096178 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac01a7250 con 0x7f1ac0106160 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.431+0000 7f1abdffb640 1 -- 172.21.15.94:0/4277096178 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8010430 con 0x7f1ac0106160 2024-04-21T08:45:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.432+0000 7f1ac6d75640 1 -- 172.21.15.94:0/4277096178 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac01a7750 con 0x7f1ac0106160 2024-04-21T08:45:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.433+0000 7f1abdffb640 1 -- 172.21.15.94:0/4277096178 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1aa8004fb0 con 0x7f1ac0106160 2024-04-21T08:45:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac95fd740 1 Processor -- start 2024-04-21T08:45:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1abdffb640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1aac0a3d20 0x7f1aac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac95fd740 1 -- start start 2024-04-21T08:45:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac5312640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1aac0a3d20 0x7f1aac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211ac960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1abdffb640 1 -- 172.21.15.94:0/4277096178 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f1aa80d1060 con 0x7f1ac0106160 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211b5a80 0x5563211b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5563211111e0 con 0x5563211ac580 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5563210ae390 con 0x5563211acea0 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55632110d1a0 con 0x5563211b5a80 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac4b11640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac4b11640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211ac960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35408/0 (socket says 172.21.15.94:35408) 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1ac4b11640 1 -- 172.21.15.94:0/3948939464 learned_addr learned my addr 172.21.15.94:0/3948939464 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.434+0000 7f1abffff640 1 --2- 172.21.15.94:0/3948939464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ac4b11640 1 -- 172.21.15.94:0/3948939464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211b5a80 msgr2=0x5563211b7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ac4b11640 1 --2- 172.21.15.94:0/3948939464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211b5a80 0x5563211b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ac4b11640 1 -- 172.21.15.94:0/3948939464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 msgr2=0x5563211b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ac4b11640 1 --2- 172.21.15.94:0/3948939464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ac4b11640 1 -- 172.21.15.94:0/3948939464 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556320eed7d0 con 0x5563211ac580 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ac5312640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1aac0a3d20 0x7f1aac0a61e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0036040 tx=0x7f1ab00040b0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ac4b11640 1 --2- 172.21.15.94:0/3948939464 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211ac960 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800af40 tx=0x7f1ab800f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba19c32efa913fd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ab6ffd640 1 -- 172.21.15.94:0/3948939464 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab801d070 con 0x5563211ac580 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ab6ffd640 1 -- 172.21.15.94:0/3948939464 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ab8010cf0 con 0x5563211ac580 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ab6ffd640 1 -- 172.21.15.94:0/3948939464 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8018ce0 con 0x5563211ac580 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ac95fd740 1 -- 172.21.15.94:0/3948939464 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 msgr2=0x5563211ac960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.435+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/3948939464 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211ac960 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800af40 tx=0x7f1ab800f850 comp rx=0 tx=0).stop 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 -- 172.21.15.94:0/3948939464 shutdown_connections 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/3948939464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211b5a80 0x5563211b7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/3948939464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/3948939464 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211ac960 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 -- 172.21.15.94:0/3948939464 >> 172.21.15.94:0/3948939464 conn(0x5563211a6cb0 msgr2=0x5563211a7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 -- 172.21.15.94:0/3948939464 shutdown_connections 2024-04-21T08:45:29.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 -- 172.21.15.94:0/3948939464 wait complete. 2024-04-21T08:45:29.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 -- start start 2024-04-21T08:45:29.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 Processor -- start 2024-04-21T08:45:29.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211ac580 0x5563211b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211b3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5563211b8870 con 0x5563211b5a80 2024-04-21T08:45:29.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5563210ca2f0 con 0x5563211acea0 2024-04-21T08:45:29.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5563211b89f0 con 0x5563211ac580 2024-04-21T08:45:29.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac95fd740 10 client.0 _ll_register_callbacks cb 0x5563210d7580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:29.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac5312640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac5312640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35424/0 (socket says 172.21.15.94:35424) 2024-04-21T08:45:29.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac5312640 1 -- 172.21.15.94:0/852518167 learned_addr learned my addr 172.21.15.94:0/852518167 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:29.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.436+0000 7f1ac4b11640 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211ac580 0x5563211b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1abffff640 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1ac5312640 1 -- 172.21.15.94:0/852518167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211ac580 msgr2=0x5563211b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1ac5312640 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211ac580 0x5563211b2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1ac5312640 1 -- 172.21.15.94:0/852518167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 msgr2=0x5563211b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1ac5312640 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1ac5312640 1 -- 172.21.15.94:0/852518167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556320e233f0 con 0x5563211b5a80 2024-04-21T08:45:29.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1ac5312640 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211b3560 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0004540 tx=0x7f1ab0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935ef32cafe47312 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1a9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:29.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1a9ffff640 1 -- 172.21.15.94:0/852518167 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0047040 con 0x5563211b5a80 2024-04-21T08:45:29.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1a9ffff640 1 -- 172.21.15.94:0/852518167 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f1ab0005040 con 0x5563211b5a80 2024-04-21T08:45:29.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1ac95fd740 1 -- 172.21.15.94:0/852518167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ab8010cf0 con 0x5563211b5a80 2024-04-21T08:45:29.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1a9ffff640 1 -- 172.21.15.94:0/852518167 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1ab0059050 con 0x5563211b5a80 2024-04-21T08:45:29.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1ac95fd740 10 client.17223 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:29.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1ac95fd740 1 -- 172.21.15.94:0/852518167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556320de30f0 con 0x5563211b5a80 2024-04-21T08:45:29.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1a9ffff640 1 -- 172.21.15.94:0/852518167 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ab8010cf0 con 0x5563211b5a80 2024-04-21T08:45:29.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.437+0000 7f1a9ffff640 1 -- 172.21.15.94:0/852518167 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1ab00035f0 con 0x5563211b5a80 2024-04-21T08:45:29.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.438+0000 7f1ac95fd740 10 client.17223 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:29.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.438+0000 7f1ac95fd740 10 client.17223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:29.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.438+0000 7f1ac95fd740 20 client.17223 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:29.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.438+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5563211c5af0 0x5563211c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.438+0000 7f1ac95fd740 4 client.17223 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:29.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.438+0000 7f1ac95fd740 1 -- 172.21.15.94:0/852518167 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55632111ec00 con 0x5563211c5af0 2024-04-21T08:45:29.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.438+0000 7f1ac4b11640 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5563211c5af0 0x5563211c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.438+0000 7f1a9ffff640 10 client.17223 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:29.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.438+0000 7f1ac4b11640 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5563211c5af0 0x5563211c7f40 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd345446f84380ca server_cookie=23ae7ee31b5f7e84 in_seq=0 out_seq=0 2024-04-21T08:45:29.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1a9ffff640 1 -- 172.21.15.94:0/852518167 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55632111ec00 con 0x5563211c5af0 2024-04-21T08:45:29.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1a9ffff640 10 client.17223 handle_command_reply: tid=0 2024-04-21T08:45:29.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 client.17223 shutdown 2024-04-21T08:45:29.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 -- 172.21.15.94:0/852518167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 msgr2=0x5563211b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211b3560 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0004540 tx=0x7f1ab0009260 comp rx=0 tx=0).stop 2024-04-21T08:45:29.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 -- 172.21.15.94:0/852518167 shutdown_connections 2024-04-21T08:45:29.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5563211c5af0 0x5563211c7f40 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211b3560 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1a9ffff640 0 client.17223 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:29.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/852518167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211ac580 0x5563211b2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 -- 172.21.15.94:0/852518167 >> 172.21.15.94:0/852518167 conn(0x5563211a6cb0 msgr2=0x5563211a8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 -- 172.21.15.94:0/852518167 shutdown_connections 2024-04-21T08:45:29.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 1 -- 172.21.15.94:0/852518167 wait complete. 2024-04-21T08:45:29.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.439+0000 7f1ac95fd740 20 client.17223 trim_cache size 0 max 16384 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.456+0000 7f1ac95fd740 1 Processor -- start 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.456+0000 7f1ac95fd740 1 -- start start 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.456+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211ac580 0x5563211b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.456+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211c1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac01b3e80 con 0x5563211b5a80 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac01b1750 con 0x5563211acea0 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac01af5a0 con 0x5563211ac580 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac5312640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac5312640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211c1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35432/0 (socket says 172.21.15.94:35432) 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac5312640 1 -- 172.21.15.94:0/3644496488 learned_addr learned my addr 172.21.15.94:0/3644496488 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:29.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1abffff640 1 --2- 172.21.15.94:0/3644496488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac5312640 1 -- 172.21.15.94:0/3644496488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211ac580 msgr2=0x5563211b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac5312640 1 --2- 172.21.15.94:0/3644496488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211ac580 0x5563211b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac5312640 1 -- 172.21.15.94:0/3644496488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 msgr2=0x5563211b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac5312640 1 --2- 172.21.15.94:0/3644496488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac5312640 1 -- 172.21.15.94:0/3644496488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563211b7e70 con 0x5563211b5a80 2024-04-21T08:45:29.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1abffff640 1 --2- 172.21.15.94:0/3644496488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:29.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ac5312640 1 --2- 172.21.15.94:0/3644496488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211c1730 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0004650 tx=0x7f1ab002c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a98707911b3801 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ab67fc640 1 -- 172.21.15.94:0/3644496488 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0047040 con 0x5563211b5a80 2024-04-21T08:45:29.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ab67fc640 1 -- 172.21.15.94:0/3644496488 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ab0050070 con 0x5563211b5a80 2024-04-21T08:45:29.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.457+0000 7f1ab67fc640 1 -- 172.21.15.94:0/3644496488 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0003ee0 con 0x5563211b5a80 2024-04-21T08:45:29.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 -- 172.21.15.94:0/3644496488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 msgr2=0x5563211c1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/3644496488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211c1730 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0004650 tx=0x7f1ab002c330 comp rx=0 tx=0).stop 2024-04-21T08:45:29.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 -- 172.21.15.94:0/3644496488 shutdown_connections 2024-04-21T08:45:29.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/3644496488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211b5a80 0x5563211c1730 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/3644496488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211acea0 0x5563211b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/3644496488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211ac580 0x5563211b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 -- 172.21.15.94:0/3644496488 >> 172.21.15.94:0/3644496488 conn(0x5563211a6cb0 msgr2=0x5563211a8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 -- 172.21.15.94:0/3644496488 shutdown_connections 2024-04-21T08:45:29.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 -- 172.21.15.94:0/3644496488 wait complete. 2024-04-21T08:45:29.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 -- start start 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.458+0000 7f1ac95fd740 1 Processor -- start 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211acea0 0x5563211b9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac95fd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211b5a80 0x55632122b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac01afc10 con 0x5563211ac580 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac01b1080 con 0x5563211b5a80 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac95fd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac01b02e0 con 0x5563211acea0 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac95fd740 10 client.0 _ll_register_callbacks cb 0x7f1ab0044040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac4b11640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac4b11640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211b99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35440/0 (socket says 172.21.15.94:35440) 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac4b11640 1 -- 172.21.15.94:0/2742134937 learned_addr learned my addr 172.21.15.94:0/2742134937 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac4b11640 1 -- 172.21.15.94:0/2742134937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211acea0 msgr2=0x5563211b9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac4b11640 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211acea0 0x5563211b9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac4b11640 1 -- 172.21.15.94:0/2742134937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211b5a80 msgr2=0x55632122b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac4b11640 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211b5a80 0x55632122b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac4b11640 1 -- 172.21.15.94:0/2742134937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563211ba820 con 0x5563211ac580 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac4b11640 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211b99e0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f1ab801b040 tx=0x7f1ab800fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6afc6c021af737 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1a9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1a9ffff640 1 -- 172.21.15.94:0/2742134937 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8028e60 con 0x5563211ac580 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1a9ffff640 1 -- 172.21.15.94:0/2742134937 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f1ab8004be0 con 0x5563211ac580 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac95fd740 1 -- 172.21.15.94:0/2742134937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ab0050070 con 0x5563211ac580 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1a9ffff640 1 -- 172.21.15.94:0/2742134937 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1ab8038050 con 0x5563211ac580 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac95fd740 10 client.17229 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1ac95fd740 1 -- 172.21.15.94:0/2742134937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55632122c6d0 con 0x5563211ac580 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.459+0000 7f1a9ffff640 1 -- 172.21.15.94:0/2742134937 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ab0050070 con 0x5563211ac580 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1a9ffff640 1 -- 172.21.15.94:0/2742134937 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1ab8010b50 con 0x5563211ac580 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1ac95fd740 10 client.17229 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:29.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1ac95fd740 10 client.17229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1ac95fd740 20 client.17229 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55632122cb20 0x55632122ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1ac95fd740 4 client.17229 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1ac95fd740 1 -- 172.21.15.94:0/2742134937 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556321118a70 con 0x55632122cb20 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1abffff640 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55632122cb20 0x55632122ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1a9ffff640 10 client.17229 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1abffff640 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55632122cb20 0x55632122ef10 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9125987c31e7e845 server_cookie=f9bf42ee108cf51c in_seq=0 out_seq=0 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1a9ffff640 1 -- 172.21.15.94:0/2742134937 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556321118a70 con 0x55632122cb20 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1a9ffff640 10 client.17229 handle_command_reply: tid=0 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.460+0000 7f1ac95fd740 1 client.17229 shutdown 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 1 -- 172.21.15.94:0/2742134937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 msgr2=0x5563211b99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211b99e0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f1ab801b040 tx=0x7f1ab800fe10 comp rx=0 tx=0).stop 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 1 -- 172.21.15.94:0/2742134937 shutdown_connections 2024-04-21T08:45:29.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55632122cb20 0x55632122ef10 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563211acea0 0x5563211b9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1a9ffff640 0 client.17229 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:29.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5563211b5a80 0x55632122b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 1 --2- 172.21.15.94:0/2742134937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5563211ac580 0x5563211b99e0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 1 -- 172.21.15.94:0/2742134937 >> 172.21.15.94:0/2742134937 conn(0x5563211a6cb0 msgr2=0x5563211b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 1 -- 172.21.15.94:0/2742134937 shutdown_connections 2024-04-21T08:45:29.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 1 -- 172.21.15.94:0/2742134937 wait complete. 2024-04-21T08:45:29.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac95fd740 20 client.17229 trim_cache size 0 max 16384 2024-04-21T08:45:29.470 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:45:29.470 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:45:29.471 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:45:29.471 INFO:teuthology.orchestra.run.smithi094.stdout: "9c083425-fdae-4c7e-b19d-415cd69bbcd9": { 2024-04-21T08:45:29.471 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:45:29.471 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "9c083425-fdae-4c7e-b19d-415cd69bbcd9", 2024-04-21T08:45:29.471 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:45:29.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:45:29.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:45:29.471 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:45:29.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 -- 172.21.15.94:0/4277096178 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1aac0a3d20 msgr2=0x7f1aac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1aac0a3d20 0x7f1aac0a61e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0036040 tx=0x7f1ab00040b0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 -- 172.21.15.94:0/4277096178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac0106160 msgr2=0x7f1ac0197860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac0106160 0x7f1ac0197860 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800ea20 tx=0x7f1aa800eee0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 -- 172.21.15.94:0/4277096178 shutdown_connections 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1aac0a3d20 0x7f1aac0a61e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac0197da0 0x7f1ac01a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac0106160 0x7f1ac0197860 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 --2- 172.21.15.94:0/4277096178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ac0101090 0x7f1ac0197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 -- 172.21.15.94:0/4277096178 >> 172.21.15.94:0/4277096178 conn(0x7f1ac00771d0 msgr2=0x7f1ac00fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.461+0000 7f1ac6d75640 1 -- 172.21.15.94:0/4277096178 shutdown_connections 2024-04-21T08:45:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:29.462+0000 7f1ac6d75640 1 -- 172.21.15.94:0/4277096178 wait complete. 2024-04-21T08:45:29.473 INFO:tasks.cephfs.filesystem:scrub status for tag:9c083425-fdae-4c7e-b19d-415cd69bbcd9 - {'path': '/', 'tag': '9c083425-fdae-4c7e-b19d-415cd69bbcd9', 'options': 'recursive,force'} 2024-04-21T08:45:29.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:29 smithi094 ceph-mon[27988]: pgmap v168: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:29.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:29 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:45:29.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:29 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:29.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:29 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:45:29.760 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:29 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:29.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:29 smithi111 ceph-mon[31493]: pgmap v168: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:29.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:29 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:45:29.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:29 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:29.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:29 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:45:29.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:29 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:30.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:29 smithi173 ceph-mon[31428]: pgmap v168: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:29 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:45:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:29 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:29 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:45:30.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:29 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:31.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:31 smithi111 ceph-mon[31493]: pgmap v169: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:45:31.938 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:31 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:45:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:31 smithi094 ceph-mon[27988]: pgmap v169: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:45:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:31 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:45:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:31 smithi173 ceph-mon[31428]: pgmap v169: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:45:32.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:31 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:45:33.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:33 smithi111 ceph-mon[31493]: pgmap v170: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T08:45:34.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:33 smithi094 ceph-mon[27988]: pgmap v170: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T08:45:34.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:33 smithi173 ceph-mon[31428]: pgmap v170: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T08:45:35.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:35 smithi111 ceph-mon[31493]: pgmap v171: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:45:36.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:35 smithi094 ceph-mon[27988]: pgmap v171: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:45:36.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:35 smithi173 ceph-mon[31428]: pgmap v171: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:45:38.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:37 smithi111 ceph-mon[31493]: pgmap v172: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:45:38.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:37 smithi094 ceph-mon[27988]: pgmap v172: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:45:38.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:37 smithi173 ceph-mon[31428]: pgmap v172: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:45:40.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:39 smithi111 ceph-mon[31493]: pgmap v173: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:45:40.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:39 smithi094 ceph-mon[27988]: pgmap v173: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:45:40.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:39 smithi173 ceph-mon[31428]: pgmap v173: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:45:42.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:41 smithi111 ceph-mon[31493]: pgmap v174: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:45:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:41 smithi094 ceph-mon[27988]: pgmap v174: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:45:42.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:41 smithi173 ceph-mon[31428]: pgmap v174: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:45:44.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:43 smithi111 ceph-mon[31493]: pgmap v175: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:45:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:43 smithi094 ceph-mon[27988]: pgmap v175: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:45:44.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:43 smithi173 ceph-mon[31428]: pgmap v175: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:45:46.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:45 smithi111 ceph-mon[31493]: pgmap v176: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:45:46.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:45 smithi094 ceph-mon[27988]: pgmap v176: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:45:46.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:45 smithi173 ceph-mon[31428]: pgmap v176: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:45:48.063 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:47 smithi173 ceph-mon[31428]: pgmap v177: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:45:48.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:47 smithi111 ceph-mon[31493]: pgmap v177: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:45:48.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:47 smithi094 ceph-mon[27988]: pgmap v177: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:45:50.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:49 smithi111 ceph-mon[31493]: pgmap v178: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:50.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:49 smithi094 ceph-mon[27988]: pgmap v178: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:50.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:49 smithi173 ceph-mon[31428]: pgmap v178: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:52.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:51 smithi111 ceph-mon[31493]: pgmap v179: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:45:52.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:51 smithi094 ceph-mon[27988]: pgmap v179: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:45:52.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:51 smithi173 ceph-mon[31428]: pgmap v179: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:45:54.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:53 smithi111 ceph-mon[31493]: pgmap v180: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:54.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:53 smithi094 ceph-mon[27988]: pgmap v180: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:54.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:53 smithi173 ceph-mon[31428]: pgmap v180: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:45:56.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:55 smithi111 ceph-mon[31493]: pgmap v181: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:45:56.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:55 smithi094 ceph-mon[27988]: pgmap v181: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:45:56.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:55 smithi173 ceph-mon[31428]: pgmap v181: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:45:58.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:57 smithi111 ceph-mon[31493]: pgmap v182: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:45:58.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:57 smithi094 ceph-mon[27988]: pgmap v182: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:45:58.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:57 smithi173 ceph-mon[31428]: pgmap v182: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:45:59.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:45:59.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:45:59.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.586+0000 7fc2464eb640 1 -- 172.21.15.94:0/2473843748 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc240103ae0 msgr2=0x7fc24010bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.586+0000 7fc2464eb640 1 --2- 172.21.15.94:0/2473843748 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc240103ae0 0x7fc24010bfc0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fc234009900 tx=0x7fc23402bd60 comp rx=0 tx=0).stop 2024-04-21T08:45:59.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 -- 172.21.15.94:0/2473843748 shutdown_connections 2024-04-21T08:45:59.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 --2- 172.21.15.94:0/2473843748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc24010c500 0x7fc24010e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 --2- 172.21.15.94:0/2473843748 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc240103ae0 0x7fc24010bfc0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 --2- 172.21.15.94:0/2473843748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2401031c0 0x7fc2401035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 -- 172.21.15.94:0/2473843748 >> 172.21.15.94:0/2473843748 conn(0x7fc240078000 msgr2=0x7fc240078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 -- 172.21.15.94:0/2473843748 shutdown_connections 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 -- 172.21.15.94:0/2473843748 wait complete. 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 Processor -- start 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 -- start start 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2401031c0 0x7fc240198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.587+0000 7fc2464eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc240103ae0 0x7fc240198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc2464eb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc24010c500 0x7fc24019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc2464eb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc240112170 con 0x7fc240103ae0 2024-04-21T08:45:59.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc2464eb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc240111ff0 con 0x7fc2401031c0 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc2464eb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2401122f0 con 0x7fc24010c500 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc240103ae0 0x7fc240198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc240103ae0 0x7fc240198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44692/0 (socket says 172.21.15.94:44692) 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23f7fe640 1 -- 172.21.15.94:0/1916886 learned_addr learned my addr 172.21.15.94:0/1916886 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc244a88640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc24010c500 0x7fc24019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23ffff640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2401031c0 0x7fc240198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23f7fe640 1 -- 172.21.15.94:0/1916886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc24010c500 msgr2=0x7fc24019cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23f7fe640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc24010c500 0x7fc24019cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23f7fe640 1 -- 172.21.15.94:0/1916886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2401031c0 msgr2=0x7fc240198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23f7fe640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2401031c0 0x7fc240198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23f7fe640 1 -- 172.21.15.94:0/1916886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24019d680 con 0x7fc240103ae0 2024-04-21T08:45:59.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc244a88640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc24010c500 0x7fc24019cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:59.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23f7fe640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc240103ae0 0x7fc240198bd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc234005940 tx=0x7fc23402d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:59.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23d7fa640 1 -- 172.21.15.94:0/1916886 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234043020 con 0x7fc240103ae0 2024-04-21T08:45:59.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23d7fa640 1 -- 172.21.15.94:0/1916886 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc23402dd20 con 0x7fc240103ae0 2024-04-21T08:45:59.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc23d7fa640 1 -- 172.21.15.94:0/1916886 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234003b80 con 0x7fc240103ae0 2024-04-21T08:45:59.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc2464eb640 1 -- 172.21.15.94:0/1916886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24019d910 con 0x7fc240103ae0 2024-04-21T08:45:59.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.588+0000 7fc2464eb640 1 -- 172.21.15.94:0/1916886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2401a5200 con 0x7fc240103ae0 2024-04-21T08:45:59.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc23d7fa640 1 -- 172.21.15.94:0/1916886 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fc234046020 con 0x7fc240103ae0 2024-04-21T08:45:59.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc2491b9740 1 Processor -- start 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc2491b9740 1 -- start start 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d2a2b0 0x565090d2a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2abd0 0x565090d37950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc23d7fa640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc2280a3d20 0x7fc2280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d37e90 0x565090d3a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc2491b9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x565090c8f1e0 con 0x565090d2a2b0 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc2491b9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x565090c2c390 con 0x565090d2abd0 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc2491b9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x565090c8b1a0 con 0x565090d37e90 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc244a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d2a2b0 0x565090d2a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc23f7fe640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc2280a3d20 0x7fc2280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc244a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d2a2b0 0x565090d2a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44706/0 (socket says 172.21.15.94:44706) 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.591+0000 7fc244a88640 1 -- 172.21.15.94:0/518620024 learned_addr learned my addr 172.21.15.94:0/518620024 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc23ffff640 1 --2- 172.21.15.94:0/518620024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2abd0 0x565090d37950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc23d7fa640 1 -- 172.21.15.94:0/1916886 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc2340f2d90 con 0x7fc240103ae0 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc244a88640 1 --2- 172.21.15.94:0/518620024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d37e90 0x565090d3a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc244a88640 1 -- 172.21.15.94:0/518620024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d37e90 msgr2=0x565090d3a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc244a88640 1 --2- 172.21.15.94:0/518620024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d37e90 0x565090d3a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc244a88640 1 -- 172.21.15.94:0/518620024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2abd0 msgr2=0x565090d37950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc244a88640 1 --2- 172.21.15.94:0/518620024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2abd0 0x565090d37950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc244a88640 1 -- 172.21.15.94:0/518620024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565090a6b7d0 con 0x565090d2a2b0 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc23f7fe640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc2280a3d20 0x7fc2280a61e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2340f0700 tx=0x7fc2340a9590 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc244a88640 1 --2- 172.21.15.94:0/518620024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d2a2b0 0x565090d2a690 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc238009fc0 tx=0x7fc238009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf44b96314bea4b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc2267fc640 1 -- 172.21.15.94:0/518620024 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23801a030 con 0x565090d2a2b0 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc2267fc640 1 -- 172.21.15.94:0/518620024 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc238014de0 con 0x565090d2a2b0 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.592+0000 7fc2267fc640 1 -- 172.21.15.94:0/518620024 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238005340 con 0x565090d2a2b0 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 -- 172.21.15.94:0/518620024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d2a2b0 msgr2=0x565090d2a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 --2- 172.21.15.94:0/518620024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d2a2b0 0x565090d2a690 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc238009fc0 tx=0x7fc238009910 comp rx=0 tx=0).stop 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 -- 172.21.15.94:0/518620024 shutdown_connections 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 --2- 172.21.15.94:0/518620024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d37e90 0x565090d3a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 --2- 172.21.15.94:0/518620024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2abd0 0x565090d37950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 --2- 172.21.15.94:0/518620024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d2a2b0 0x565090d2a690 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 -- 172.21.15.94:0/518620024 >> 172.21.15.94:0/518620024 conn(0x565090d24cb0 msgr2=0x565090d25090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 -- 172.21.15.94:0/518620024 shutdown_connections 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 -- 172.21.15.94:0/518620024 wait complete. 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 -- start start 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.593+0000 7fc2491b9740 1 Processor -- start 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d34e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d353d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d35910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc2491b9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x565090d3ac50 con 0x565090d37e90 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc2491b9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x565090c482f0 con 0x565090d2a2b0 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc2491b9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x565090d3add0 con 0x565090d2abd0 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc2491b9740 10 client.0 _ll_register_callbacks cb 0x565090c55580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc244a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d35910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc244a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d35910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44718/0 (socket says 172.21.15.94:44718) 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc244a88640 1 -- 172.21.15.94:0/3764198147 learned_addr learned my addr 172.21.15.94:0/3764198147 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc244a88640 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d34e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc23ffff640 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d353d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc244a88640 1 -- 172.21.15.94:0/3764198147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 msgr2=0x565090d353d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc244a88640 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d353d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc244a88640 1 -- 172.21.15.94:0/3764198147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 msgr2=0x565090d34e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc244a88640 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d34e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc244a88640 1 -- 172.21.15.94:0/3764198147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650909a13f0 con 0x565090d37e90 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc244a88640 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d35910 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc238005fd0 tx=0x7fc238012ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78a93c3b7af703d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc21f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3764198147 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23801a050 con 0x565090d37e90 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3764198147 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc238012b40 con 0x565090d37e90 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.594+0000 7fc2491b9740 1 -- 172.21.15.94:0/3764198147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc238014de0 con 0x565090d37e90 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3764198147 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc238035050 con 0x565090d37e90 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc2491b9740 10 client.17241 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc2491b9740 1 -- 172.21.15.94:0/3764198147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5650909610f0 con 0x565090d37e90 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3764198147 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc238014de0 con 0x565090d37e90 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3764198147 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc23802d020 con 0x565090d37e90 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc2491b9740 10 client.17241 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc2491b9740 10 client.17241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc2491b9740 20 client.17241 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:59.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565090d47e20 0x565090d4a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc2491b9740 4 client.17241 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc2491b9740 1 -- 172.21.15.94:0/3764198147 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565090c9cc00 con 0x565090d47e20 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.595+0000 7fc23ffff640 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565090d47e20 0x565090d4a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.596+0000 7fc21f7fe640 10 client.17241 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.596+0000 7fc23ffff640 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565090d47e20 0x565090d4a210 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=376d5084f5ce1199 server_cookie=1ce1ad50d4ecebb1 in_seq=0 out_seq=0 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.596+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3764198147 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565090c9cc00 con 0x565090d47e20 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.596+0000 7fc21f7fe640 10 client.17241 handle_command_reply: tid=0 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.596+0000 7fc2491b9740 1 client.17241 shutdown 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 1 -- 172.21.15.94:0/3764198147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 msgr2=0x565090d35910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d35910 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc238005fd0 tx=0x7fc238012ea0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 1 -- 172.21.15.94:0/3764198147 shutdown_connections 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565090d47e20 0x565090d4a210 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d353d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d35910 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3764198147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d34e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc21f7fe640 0 client.17241 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:59.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 1 -- 172.21.15.94:0/3764198147 >> 172.21.15.94:0/3764198147 conn(0x565090d24cb0 msgr2=0x565090d26a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:59.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 1 -- 172.21.15.94:0/3764198147 shutdown_connections 2024-04-21T08:45:59.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 1 -- 172.21.15.94:0/3764198147 wait complete. 2024-04-21T08:45:59.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.597+0000 7fc2491b9740 20 client.17241 trim_cache size 0 max 16384 2024-04-21T08:45:59.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.614+0000 7fc2491b9740 1 Processor -- start 2024-04-21T08:45:59.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.614+0000 7fc2491b9740 1 -- start start 2024-04-21T08:45:59.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.614+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.614+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d43b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.614+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d23bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.614+0000 7fc2491b9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc2401afa50 con 0x565090d37e90 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.614+0000 7fc2491b9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc2401ad320 con 0x565090d2a2b0 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.614+0000 7fc2491b9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2401ab170 con 0x565090d2abd0 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc244a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d23bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc244a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d23bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44720/0 (socket says 172.21.15.94:44720) 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc244a88640 1 -- 172.21.15.94:0/865577817 learned_addr learned my addr 172.21.15.94:0/865577817 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc23ffff640 1 --2- 172.21.15.94:0/865577817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d43b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc244a88640 1 --2- 172.21.15.94:0/865577817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc244a88640 1 -- 172.21.15.94:0/865577817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 msgr2=0x565090d43b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:59.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc244a88640 1 --2- 172.21.15.94:0/865577817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d43b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc244a88640 1 -- 172.21.15.94:0/865577817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 msgr2=0x565090d375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:59.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc244a88640 1 --2- 172.21.15.94:0/865577817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d375c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc244a88640 1 -- 172.21.15.94:0/865577817 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565090d2aa00 con 0x565090d37e90 2024-04-21T08:45:59.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc244a88640 1 --2- 172.21.15.94:0/865577817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d23bf0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fc238005f40 tx=0x7fc238005520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88047a3f9bdc2beb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:59.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc225ffb640 1 -- 172.21.15.94:0/865577817 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23801a030 con 0x565090d37e90 2024-04-21T08:45:59.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc225ffb640 1 -- 172.21.15.94:0/865577817 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc238024040 con 0x565090d37e90 2024-04-21T08:45:59.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.615+0000 7fc225ffb640 1 -- 172.21.15.94:0/865577817 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238009580 con 0x565090d37e90 2024-04-21T08:45:59.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.616+0000 7fc2491b9740 1 -- 172.21.15.94:0/865577817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 msgr2=0x565090d23bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.616+0000 7fc2491b9740 1 --2- 172.21.15.94:0/865577817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d23bf0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fc238005f40 tx=0x7fc238005520 comp rx=0 tx=0).stop 2024-04-21T08:45:59.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.616+0000 7fc2491b9740 1 -- 172.21.15.94:0/865577817 shutdown_connections 2024-04-21T08:45:59.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.616+0000 7fc2491b9740 1 --2- 172.21.15.94:0/865577817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d43b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.616+0000 7fc2491b9740 1 --2- 172.21.15.94:0/865577817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d23bf0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.616+0000 7fc2491b9740 1 --2- 172.21.15.94:0/865577817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.616+0000 7fc2491b9740 1 -- 172.21.15.94:0/865577817 >> 172.21.15.94:0/865577817 conn(0x565090d24cb0 msgr2=0x565090d234e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.616+0000 7fc2491b9740 1 -- 172.21.15.94:0/865577817 shutdown_connections 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.616+0000 7fc2491b9740 1 -- 172.21.15.94:0/865577817 wait complete. 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc2491b9740 1 -- start start 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc2491b9740 1 Processor -- start 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d3bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d3c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc2491b9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d3c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc2491b9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc2401ab7e0 con 0x565090d37e90 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc2491b9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc2401acc50 con 0x565090d2a2b0 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc2491b9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2401abeb0 con 0x565090d2abd0 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc2491b9740 10 client.0 _ll_register_callbacks cb 0x7fc23000b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc244a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d3c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc244a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d3c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44732/0 (socket says 172.21.15.94:44732) 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc244a88640 1 -- 172.21.15.94:0/3839588816 learned_addr learned my addr 172.21.15.94:0/3839588816 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc23ffff640 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d3c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.617+0000 7fc244a88640 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d3bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc244a88640 1 -- 172.21.15.94:0/3839588816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 msgr2=0x565090d3c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc244a88640 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d3c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc244a88640 1 -- 172.21.15.94:0/3839588816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 msgr2=0x565090d3bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc244a88640 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d3bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc244a88640 1 -- 172.21.15.94:0/3839588816 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565090dad2a0 con 0x565090d37e90 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc244a88640 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d3c550 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fc238012760 tx=0x7fc238005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579cb965062da6a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc21f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3839588816 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23803e040 con 0x565090d37e90 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3839588816 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc238012b40 con 0x565090d37e90 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc2491b9740 1 -- 172.21.15.94:0/3839588816 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc238024040 con 0x565090d37e90 2024-04-21T08:45:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3839588816 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc23801a040 con 0x565090d37e90 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc2491b9740 10 client.17247 fetch_fsmap learned FSMap version 24 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc2491b9740 1 -- 172.21.15.94:0/3839588816 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x565090dae590 con 0x565090d37e90 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.618+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3839588816 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc238024040 con 0x565090d37e90 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.619+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3839588816 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc238044070 con 0x565090d37e90 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.619+0000 7fc2491b9740 10 client.17247 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.619+0000 7fc2491b9740 10 client.17247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.619+0000 7fc2491b9740 20 client.17247 populate_metadata read hostname 'smithi094' 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.619+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565090daec20 0x565090db1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.619+0000 7fc2491b9740 4 client.17247 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.619+0000 7fc2491b9740 1 -- 172.21.15.94:0/3839588816 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565090c96a70 con 0x565090daec20 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.619+0000 7fc23ffff640 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565090daec20 0x565090db1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.619+0000 7fc23ffff640 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565090daec20 0x565090db1040 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c3d6d29389e3987 server_cookie=66d3dfb5a4ba3caf in_seq=0 out_seq=0 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.619+0000 7fc21f7fe640 10 client.17247 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc21f7fe640 1 -- 172.21.15.94:0/3839588816 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x565090c96a70 con 0x565090daec20 2024-04-21T08:45:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc21f7fe640 10 client.17247 handle_command_reply: tid=0 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 client.17247 shutdown 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 -- 172.21.15.94:0/3839588816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 msgr2=0x565090d3c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d3c550 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fc238012760 tx=0x7fc238005cf0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 -- 172.21.15.94:0/3839588816 shutdown_connections 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565090daec20 0x565090db1040 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565090d2abd0 0x565090d3c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565090d37e90 0x565090d3c550 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 --2- 172.21.15.94:0/3839588816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565090d2a2b0 0x565090d3bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc21f7fe640 0 client.17247 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 -- 172.21.15.94:0/3839588816 >> 172.21.15.94:0/3839588816 conn(0x565090d24cb0 msgr2=0x565090d356e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 -- 172.21.15.94:0/3839588816 shutdown_connections 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 1 -- 172.21.15.94:0/3839588816 wait complete. 2024-04-21T08:45:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.620+0000 7fc2491b9740 20 client.17247 trim_cache size 0 max 16384 2024-04-21T08:45:59.626 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:45:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:45:59.626 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:45:59.626 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:45:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 -- 172.21.15.94:0/1916886 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc2280a3d20 msgr2=0x7fc2280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc2280a3d20 0x7fc2280a61e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2340f0700 tx=0x7fc2340a9590 comp rx=0 tx=0).stop 2024-04-21T08:45:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 -- 172.21.15.94:0/1916886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc240103ae0 msgr2=0x7fc240198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc240103ae0 0x7fc240198bd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc234005940 tx=0x7fc23402d920 comp rx=0 tx=0).stop 2024-04-21T08:45:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 -- 172.21.15.94:0/1916886 shutdown_connections 2024-04-21T08:45:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc2280a3d20 0x7fc2280a61e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc24010c500 0x7fc24019cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc240103ae0 0x7fc240198bd0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 --2- 172.21.15.94:0/1916886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2401031c0 0x7fc240198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 -- 172.21.15.94:0/1916886 >> 172.21.15.94:0/1916886 conn(0x7fc240078000 msgr2=0x7fc2400fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 -- 172.21.15.94:0/1916886 shutdown_connections 2024-04-21T08:45:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.621+0000 7fc2464eb640 1 -- 172.21.15.94:0/1916886 wait complete. 2024-04-21T08:45:59.632 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:45:59.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:59.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 -- 172.21.15.94:0/667087804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00102180 msgr2=0x7fbe00102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 --2- 172.21.15.94:0/667087804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00102180 0x7fbe00102560 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8009ca0 tx=0x7fbdf802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 -- 172.21.15.94:0/667087804 shutdown_connections 2024-04-21T08:45:59.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 --2- 172.21.15.94:0/667087804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe00107250 0x7fbe0010ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 --2- 172.21.15.94:0/667087804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe00102b30 0x7fbe00106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 --2- 172.21.15.94:0/667087804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00102180 0x7fbe00102560 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 -- 172.21.15.94:0/667087804 >> 172.21.15.94:0/667087804 conn(0x7fbe00079350 msgr2=0x7fbe00079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:59.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 -- 172.21.15.94:0/667087804 shutdown_connections 2024-04-21T08:45:59.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 -- 172.21.15.94:0/667087804 wait complete. 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 Processor -- start 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.732+0000 7fbe07398640 1 -- start start 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe07398640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00102180 0x7fbe001986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe07398640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe00107250 0x7fbe00198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe07398640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe0019cfa0 0x7fbe0019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe07398640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbe001121c0 con 0x7fbe00102180 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe07398640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbe00112040 con 0x7fbe0019cfa0 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe07398640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbe00112340 con 0x7fbe00107250 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe05134640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00102180 0x7fbe001986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe05134640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00102180 0x7fbe001986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44752/0 (socket says 172.21.15.94:44752) 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe05134640 1 -- 172.21.15.94:0/3307479677 learned_addr learned my addr 172.21.15.94:0/3307479677 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe05935640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe0019cfa0 0x7fbe0019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe04933640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe00107250 0x7fbe00198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe05134640 1 -- 172.21.15.94:0/3307479677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe00107250 msgr2=0x7fbe00198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe05134640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe00107250 0x7fbe00198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe05134640 1 -- 172.21.15.94:0/3307479677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe0019cfa0 msgr2=0x7fbe0019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe05134640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe0019cfa0 0x7fbe0019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe05134640 1 -- 172.21.15.94:0/3307479677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0019da90 con 0x7fbe00102180 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe05134640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00102180 0x7fbe001986e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8005e30 tx=0x7fbdf8034780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbdf67fc640 1 -- 172.21.15.94:0/3307479677 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8039040 con 0x7fbe00102180 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbdf67fc640 1 -- 172.21.15.94:0/3307479677 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdf8044070 con 0x7fbe00102180 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbdf67fc640 1 -- 172.21.15.94:0/3307479677 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8003e60 con 0x7fbe00102180 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe07398640 1 -- 172.21.15.94:0/3307479677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0019dd20 con 0x7fbe00102180 2024-04-21T08:45:59.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.733+0000 7fbe07398640 1 -- 172.21.15.94:0/3307479677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe001a5480 con 0x7fbe00102180 2024-04-21T08:45:59.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.734+0000 7fbe07398640 1 -- 172.21.15.94:0/3307479677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe00068de0 con 0x7fbe00102180 2024-04-21T08:45:59.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.735+0000 7fbdf67fc640 1 -- 172.21.15.94:0/3307479677 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fbdf80324c0 con 0x7fbe00102180 2024-04-21T08:45:59.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.736+0000 7fbdf67fc640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbdec0a3d20 0x7fbdec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:59.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.736+0000 7fbe04933640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbdec0a3d20 0x7fbdec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:59.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.736+0000 7fbdf67fc640 1 -- 172.21.15.94:0/3307479677 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fbdf80f3840 con 0x7fbe00102180 2024-04-21T08:45:59.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.737+0000 7fbe04933640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbdec0a3d20 0x7fbdec0a61e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe00101ab0 tx=0x7fbdf000a2d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:59.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.740+0000 7fbdf67fc640 1 -- 172.21.15.94:0/3307479677 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbdf80bfc10 con 0x7fbe00102180 2024-04-21T08:45:59.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.959+0000 7fbe07398640 1 -- 172.21.15.94:0/3307479677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe00199a30 con 0x7fbe00102180 2024-04-21T08:45:59.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.959+0000 7fbdf67fc640 1 -- 172.21.15.94:0/3307479677 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7fbdf80bf510 con 0x7fbe00102180 2024-04-21T08:45:59.959 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:45:59.960 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:45:59.960 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:45:59.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.961+0000 7fbe07398640 1 -- 172.21.15.94:0/3307479677 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbdec0a3d20 msgr2=0x7fbdec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.961+0000 7fbe07398640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbdec0a3d20 0x7fbdec0a61e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe00101ab0 tx=0x7fbdf000a2d0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.961+0000 7fbe07398640 1 -- 172.21.15.94:0/3307479677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00102180 msgr2=0x7fbe001986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.961+0000 7fbe07398640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00102180 0x7fbe001986e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8005e30 tx=0x7fbdf8034780 comp rx=0 tx=0).stop 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.961+0000 7fbe07398640 1 -- 172.21.15.94:0/3307479677 shutdown_connections 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.961+0000 7fbe07398640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fbdec0a3d20 0x7fbdec0a61e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.961+0000 7fbe07398640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe00107250 0x7fbe00198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.962+0000 7fbe07398640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe0019cfa0 0x7fbe0019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.962+0000 7fbe07398640 1 --2- 172.21.15.94:0/3307479677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00102180 0x7fbe001986e0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.962+0000 7fbe07398640 1 -- 172.21.15.94:0/3307479677 >> 172.21.15.94:0/3307479677 conn(0x7fbe00079350 msgr2=0x7fbe00100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.962+0000 7fbe07398640 1 -- 172.21.15.94:0/3307479677 shutdown_connections 2024-04-21T08:45:59.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:45:59.962+0000 7fbe07398640 1 -- 172.21.15.94:0/3307479677 wait complete. 2024-04-21T08:45:59.973 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:46:00.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.075+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2259450679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3b8101090 msgr2=0x7fd3b8101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.075+0000 7fd3be6c8640 1 --2- 172.21.15.94:0/2259450679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3b8101090 0x7fd3b8101470 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009900 tx=0x7fd3a802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:00.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.075+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2259450679 shutdown_connections 2024-04-21T08:46:00.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.075+0000 7fd3be6c8640 1 --2- 172.21.15.94:0/2259450679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3b8106160 0x7fd3b810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.075+0000 7fd3be6c8640 1 --2- 172.21.15.94:0/2259450679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b8101a40 0x7fd3b8105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.075+0000 7fd3be6c8640 1 --2- 172.21.15.94:0/2259450679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3b8101090 0x7fd3b8101470 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.075+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2259450679 >> 172.21.15.94:0/2259450679 conn(0x7fd3b8077f50 msgr2=0x7fd3b8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.075+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2259450679 shutdown_connections 2024-04-21T08:46:00.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.075+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2259450679 wait complete. 2024-04-21T08:46:00.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3be6c8640 1 Processor -- start 2024-04-21T08:46:00.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3be6c8640 1 -- start start 2024-04-21T08:46:00.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3be6c8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3b8101090 0x7fd3b81995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3be6c8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b8106160 0x7fd3b8199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3be6c8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3b819a0c0 0x7fd3b81a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3be6c8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b81110d0 con 0x7fd3b819a0c0 2024-04-21T08:46:00.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3be6c8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b8110f50 con 0x7fd3b8106160 2024-04-21T08:46:00.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3be6c8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b8111250 con 0x7fd3b8101090 2024-04-21T08:46:00.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3bcc65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3b819a0c0 0x7fd3b81a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3bcc65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3b819a0c0 0x7fd3b81a4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44768/0 (socket says 172.21.15.94:44768) 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3bcc65640 1 -- 172.21.15.94:0/2051924424 learned_addr learned my addr 172.21.15.94:0/2051924424 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3b77fe640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b8106160 0x7fd3b8199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.076+0000 7fd3b7fff640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3b8101090 0x7fd3b81995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3bcc65640 1 -- 172.21.15.94:0/2051924424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3b8101090 msgr2=0x7fd3b81995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3b8101090 0x7fd3b81995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3bcc65640 1 -- 172.21.15.94:0/2051924424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b8106160 msgr2=0x7fd3b8199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b8106160 0x7fd3b8199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3bcc65640 1 -- 172.21.15.94:0/2051924424 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b81a4e40 con 0x7fd3b819a0c0 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3b819a0c0 0x7fd3b81a4670 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fd3b000eb10 tx=0x7fd3b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3b57fa640 1 -- 172.21.15.94:0/2051924424 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b0004b20 con 0x7fd3b819a0c0 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3b57fa640 1 -- 172.21.15.94:0/2051924424 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b0004cc0 con 0x7fd3b819a0c0 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2051924424 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b81a5130 con 0x7fd3b819a0c0 2024-04-21T08:46:00.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3b57fa640 1 -- 172.21.15.94:0/2051924424 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b0010670 con 0x7fd3b819a0c0 2024-04-21T08:46:00.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.077+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2051924424 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b81a5680 con 0x7fd3b819a0c0 2024-04-21T08:46:00.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.079+0000 7fd3b57fa640 1 -- 172.21.15.94:0/2051924424 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd3b0005020 con 0x7fd3b819a0c0 2024-04-21T08:46:00.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3c139a740 1 Processor -- start 2024-04-21T08:46:00.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3c139a740 1 -- start start 2024-04-21T08:46:00.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3b57fa640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd3a00a3d20 0x7fd3a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216994d0 0x55fd216998b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd21699e80 0x55fd216a2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3c139a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55fd215fe1f0 con 0x55fd216a2a00 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3c139a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fd2159b4b0 con 0x55fd216994d0 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3c139a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fd215fa1b0 con 0x55fd21699e80 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3b77fe640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd3a00a3d20 0x7fd3a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3b57fa640 1 -- 172.21.15.94:0/2051924424 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd3b00d10c0 con 0x7fd3b819a0c0 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3b7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216994d0 0x55fd216998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3b7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216994d0 0x55fd216998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:43632/0 (socket says 172.21.15.94:43632) 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3b7fff640 1 -- 172.21.15.94:0/665993816 learned_addr learned my addr 172.21.15.94:0/665993816 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/665993816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.080+0000 7fd3b77fe640 1 --2- 172.21.15.94:0/665993816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd21699e80 0x55fd216a2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd3bcc65640 1 -- 172.21.15.94:0/665993816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd21699e80 msgr2=0x55fd216a2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/665993816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd21699e80 0x55fd216a2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd3bcc65640 1 -- 172.21.15.94:0/665993816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216994d0 msgr2=0x55fd216998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/665993816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216994d0 0x55fd216998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd3bcc65640 1 -- 172.21.15.94:0/665993816 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd2158d1b0 con 0x55fd216a2a00 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd3b7fff640 1 --2- 172.21.15.94:0/665993816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216994d0 0x55fd216998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd3b77fe640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd3a00a3d20 0x7fd3a00a61e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8069c30 tx=0x7fd3ac0128d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/665993816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216a4e50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00ce430 tx=0x7fd3b000cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2cded558ffc1fd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd39e7fc640 1 -- 172.21.15.94:0/665993816 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b004b1c0 con 0x55fd216a2a00 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd39e7fc640 1 -- 172.21.15.94:0/665993816 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b009d380 con 0x55fd216a2a00 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.081+0000 7fd39e7fc640 1 -- 172.21.15.94:0/665993816 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b004afe0 con 0x55fd216a2a00 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.082+0000 7fd3c139a740 1 -- 172.21.15.94:0/665993816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 msgr2=0x55fd216a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.082+0000 7fd3c139a740 1 --2- 172.21.15.94:0/665993816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216a4e50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00ce430 tx=0x7fd3b000cee0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.082+0000 7fd3c139a740 1 -- 172.21.15.94:0/665993816 shutdown_connections 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.082+0000 7fd3c139a740 1 --2- 172.21.15.94:0/665993816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd21699e80 0x55fd216a2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.082+0000 7fd3c139a740 1 --2- 172.21.15.94:0/665993816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216a4e50 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.082+0000 7fd3c139a740 1 --2- 172.21.15.94:0/665993816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216994d0 0x55fd216998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.082+0000 7fd3c139a740 1 -- 172.21.15.94:0/665993816 >> 172.21.15.94:0/665993816 conn(0x55fd21693ba0 msgr2=0x55fd21693f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.082+0000 7fd3c139a740 1 -- 172.21.15.94:0/665993816 shutdown_connections 2024-04-21T08:46:00.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.082+0000 7fd3c139a740 1 -- 172.21.15.94:0/665993816 wait complete. 2024-04-21T08:46:00.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.082+0000 7fd3c139a740 1 -- start start 2024-04-21T08:46:00.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3c139a740 1 Processor -- start 2024-04-21T08:46:00.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216994d0 0x55fd216a2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd21699e80 0x55fd2169f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 0x55fd2169fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3c139a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55fd216a9d20 con 0x55fd216994d0 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3c139a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fd215b7500 con 0x55fd216a2a00 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3c139a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fd216a9ea0 con 0x55fd21699e80 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3c139a740 10 client.0 _ll_register_callbacks cb 0x55fd21585b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3b7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216994d0 0x55fd216a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3b7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216994d0 0x55fd216a2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44794/0 (socket says 172.21.15.94:44794) 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3b7fff640 1 -- 172.21.15.94:0/2877853932 learned_addr learned my addr 172.21.15.94:0/2877853932 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 0x55fd2169fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3b77fe640 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd21699e80 0x55fd2169f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3b7fff640 1 -- 172.21.15.94:0/2877853932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd21699e80 msgr2=0x55fd2169f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3b7fff640 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd21699e80 0x55fd2169f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3b7fff640 1 -- 172.21.15.94:0/2877853932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 msgr2=0x55fd2169fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3b7fff640 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 0x55fd2169fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.083+0000 7fd3b7fff640 1 -- 172.21.15.94:0/2877853932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd213103f0 con 0x55fd216994d0 2024-04-21T08:46:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.084+0000 7fd3b7fff640 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216994d0 0x55fd216a2290 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8036040 tx=0x7fd3a802ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e6b7470c7bb256 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.084+0000 7fd3977fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:00.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.084+0000 7fd3977fe640 1 -- 172.21.15.94:0/2877853932 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8005030 con 0x55fd216994d0 2024-04-21T08:46:00.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.084+0000 7fd3977fe640 1 -- 172.21.15.94:0/2877853932 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd3a80051d0 con 0x55fd216994d0 2024-04-21T08:46:00.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.084+0000 7fd3c139a740 1 -- 172.21.15.94:0/2877853932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3b009d380 con 0x55fd216994d0 2024-04-21T08:46:00.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.084+0000 7fd3977fe640 1 -- 172.21.15.94:0/2877853932 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd3a80053d0 con 0x55fd216994d0 2024-04-21T08:46:00.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.084+0000 7fd3c139a740 10 client.17265 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:00.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.084+0000 7fd3c139a740 1 -- 172.21.15.94:0/2877853932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fd213da7d0 con 0x55fd216994d0 2024-04-21T08:46:00.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.084+0000 7fd3977fe640 1 -- 172.21.15.94:0/2877853932 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b009d380 con 0x55fd216994d0 2024-04-21T08:46:00.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.084+0000 7fd3977fe640 1 -- 172.21.15.94:0/2877853932 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd3a8003710 con 0x55fd216994d0 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.085+0000 7fd3c139a740 10 client.17265 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.085+0000 7fd3c139a740 10 client.17265 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.085+0000 7fd3c139a740 20 client.17265 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.085+0000 7fd3c139a740 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55fd216b6f30 0x55fd216b9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.085+0000 7fd3c139a740 4 client.17265 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.085+0000 7fd3c139a740 1 -- 172.21.15.94:0/2877853932 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd2159d930 con 0x55fd216b6f30 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.085+0000 7fd3b77fe640 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55fd216b6f30 0x55fd216b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.085+0000 7fd3b77fe640 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55fd216b6f30 0x55fd216b9320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9a3ba2e2199966b9 server_cookie=9d69a5701034d108 in_seq=0 out_seq=0 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.085+0000 7fd3977fe640 10 client.17265 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3977fe640 1 -- 172.21.15.94:0/2877853932 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd2159d930 con 0x55fd216b6f30 2024-04-21T08:46:00.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3977fe640 10 client.17265 handle_command_reply: tid=0 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 client.17265 shutdown 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 -- 172.21.15.94:0/2877853932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216994d0 msgr2=0x55fd216a2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216994d0 0x55fd216a2290 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8036040 tx=0x7fd3a802ca10 comp rx=0 tx=0).stop 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 -- 172.21.15.94:0/2877853932 shutdown_connections 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55fd216b6f30 0x55fd216b9320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd21699e80 0x55fd2169f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 0x55fd2169fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3977fe640 0 client.17265 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 --2- 172.21.15.94:0/2877853932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216994d0 0x55fd216a2290 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 -- 172.21.15.94:0/2877853932 >> 172.21.15.94:0/2877853932 conn(0x55fd21693ba0 msgr2=0x55fd216958f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 -- 172.21.15.94:0/2877853932 shutdown_connections 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.086+0000 7fd3c139a740 1 -- 172.21.15.94:0/2877853932 wait complete. 2024-04-21T08:46:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.087+0000 7fd3c139a740 20 client.17265 trim_cache size 0 max 16384 2024-04-21T08:46:00.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.104+0000 7fd3c139a740 1 Processor -- start 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.104+0000 7fd3c139a740 1 -- start start 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 0x55fd2169ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd21699e80 0x55fd216beb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216bf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3c139a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b81afb70 con 0x55fd216a2a00 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3c139a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b81ad440 con 0x55fd21699e80 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3c139a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b81ab290 con 0x55fd216994d0 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3bcc65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216bf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3bcc65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216bf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44806/0 (socket says 172.21.15.94:44806) 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3bcc65640 1 -- 172.21.15.94:0/2116283526 learned_addr learned my addr 172.21.15.94:0/2116283526 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3b77fe640 1 --2- 172.21.15.94:0/2116283526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd21699e80 0x55fd216beb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3bcc65640 1 -- 172.21.15.94:0/2116283526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 msgr2=0x55fd2169ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/2116283526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 0x55fd2169ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3bcc65640 1 -- 172.21.15.94:0/2116283526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd21699e80 msgr2=0x55fd216beb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/2116283526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd21699e80 0x55fd216beb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3bcc65640 1 -- 172.21.15.94:0/2116283526 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd216987f0 con 0x55fd216a2a00 2024-04-21T08:46:00.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/2116283526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216bf090 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00049d0 tx=0x7fd3b0000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e31d59d9ac3f76 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd39dffb640 1 -- 172.21.15.94:0/2116283526 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b00bb070 con 0x55fd216a2a00 2024-04-21T08:46:00.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd39dffb640 1 -- 172.21.15.94:0/2116283526 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b009d380 con 0x55fd216a2a00 2024-04-21T08:46:00.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.105+0000 7fd39dffb640 1 -- 172.21.15.94:0/2116283526 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b00ce840 con 0x55fd216a2a00 2024-04-21T08:46:00.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.106+0000 7fd3c139a740 1 -- 172.21.15.94:0/2116283526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 msgr2=0x55fd216bf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.106+0000 7fd3c139a740 1 --2- 172.21.15.94:0/2116283526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216bf090 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00049d0 tx=0x7fd3b0000f40 comp rx=0 tx=0).stop 2024-04-21T08:46:00.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.106+0000 7fd3c139a740 1 -- 172.21.15.94:0/2116283526 shutdown_connections 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.106+0000 7fd3c139a740 1 --2- 172.21.15.94:0/2116283526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd216a2a00 0x55fd216bf090 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.106+0000 7fd3c139a740 1 --2- 172.21.15.94:0/2116283526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd21699e80 0x55fd216beb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.106+0000 7fd3c139a740 1 --2- 172.21.15.94:0/2116283526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 0x55fd2169ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.106+0000 7fd3c139a740 1 -- 172.21.15.94:0/2116283526 >> 172.21.15.94:0/2116283526 conn(0x55fd21693ba0 msgr2=0x55fd21692c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.106+0000 7fd3c139a740 1 -- 172.21.15.94:0/2116283526 shutdown_connections 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.106+0000 7fd3c139a740 1 -- 172.21.15.94:0/2116283526 wait complete. 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3c139a740 1 -- start start 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3c139a740 1 Processor -- start 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 0x55fd216af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd21699e80 0x55fd2169be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3c139a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 0x55fd2169c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3c139a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b81ab900 con 0x55fd21699e80 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3c139a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b81acd70 con 0x55fd216a2a00 2024-04-21T08:46:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3c139a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b81abfd0 con 0x55fd216994d0 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3c139a740 10 client.0 _ll_register_callbacks cb 0x7fd3a80043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3bcc65640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 0x55fd2169c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3b7fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 0x55fd216af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3bcc65640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 0x55fd2169c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:43662/0 (socket says 172.21.15.94:43662) 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3b7fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 0x55fd216af2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:56048/0 (socket says 172.21.15.94:56048) 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3bcc65640 1 -- 172.21.15.94:0/17309447 learned_addr learned my addr 172.21.15.94:0/17309447 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.107+0000 7fd3b77fe640 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd21699e80 0x55fd2169be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3bcc65640 1 -- 172.21.15.94:0/17309447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 msgr2=0x55fd216af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 0x55fd216af2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3bcc65640 1 -- 172.21.15.94:0/17309447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd21699e80 msgr2=0x55fd2169be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd21699e80 0x55fd2169be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3bcc65640 1 -- 172.21.15.94:0/17309447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd2169c910 con 0x55fd216a2a00 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3b7fff640 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 0x55fd216af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3bcc65640 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 0x55fd2169c3d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3b009ce40 tx=0x7fd3b00ce4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2f6651f50fc24f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3977fe640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T08:46:00.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3977fe640 1 -- 172.21.15.94:0/17309447 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b00bb070 con 0x55fd216a2a00 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3c139a740 1 -- 172.21.15.94:0/17309447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3b009d380 con 0x55fd216a2a00 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3977fe640 1 -- 172.21.15.94:0/17309447 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd3b009cee0 con 0x55fd216a2a00 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3977fe640 1 -- 172.21.15.94:0/17309447 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd3b00ce820 con 0x55fd216a2a00 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3c139a740 10 client.26003 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3c139a740 1 -- 172.21.15.94:0/17309447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fd2171d0e0 con 0x55fd216a2a00 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.108+0000 7fd3977fe640 1 -- 172.21.15.94:0/17309447 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b009d380 con 0x55fd216a2a00 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3977fe640 1 -- 172.21.15.94:0/17309447 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd3b00a5460 con 0x55fd216a2a00 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3c139a740 10 client.26003 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3c139a740 10 client.26003 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3c139a740 20 client.26003 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3c139a740 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55fd2171d310 0x55fd2171f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3c139a740 4 client.26003 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:46:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3c139a740 1 -- 172.21.15.94:0/17309447 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd2160bc10 con 0x55fd2171d310 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3b7fff640 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55fd2171d310 0x55fd2171f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3b7fff640 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55fd2171d310 0x55fd2171f700 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=86b04c57a6092d7e server_cookie=26944d060a49009b in_seq=0 out_seq=0 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3977fe640 10 client.26003 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3977fe640 1 -- 172.21.15.94:0/17309447 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd2160bc10 con 0x55fd2171d310 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3977fe640 10 client.26003 handle_command_reply: tid=0 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.109+0000 7fd3c139a740 1 client.26003 shutdown 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 1 -- 172.21.15.94:0/17309447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 msgr2=0x55fd2169c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 0x55fd2169c3d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3b009ce40 tx=0x7fd3b00ce4e0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 1 -- 172.21.15.94:0/17309447 shutdown_connections 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55fd2171d310 0x55fd2171f700 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd216a2a00 0x55fd2169c3d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fd21699e80 0x55fd2169be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 1 --2- 172.21.15.94:0/17309447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fd216994d0 0x55fd216af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3977fe640 0 client.26003 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:46:00.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 1 -- 172.21.15.94:0/17309447 >> 172.21.15.94:0/17309447 conn(0x55fd21693ba0 msgr2=0x55fd21692370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 1 -- 172.21.15.94:0/17309447 shutdown_connections 2024-04-21T08:46:00.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 1 -- 172.21.15.94:0/17309447 wait complete. 2024-04-21T08:46:00.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3c139a740 20 client.26003 trim_cache size 0 max 16384 2024-04-21T08:46:00.116 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2051924424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd3a00a3d20 msgr2=0x7fd3a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3be6c8640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd3a00a3d20 0x7fd3a00a61e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8069c30 tx=0x7fd3ac0128d0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2051924424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3b819a0c0 msgr2=0x7fd3b81a4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.110+0000 7fd3be6c8640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3b819a0c0 0x7fd3b81a4670 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fd3b000eb10 tx=0x7fd3b000c2d0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.111+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2051924424 shutdown_connections 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.111+0000 7fd3be6c8640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd3a00a3d20 0x7fd3a00a61e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.111+0000 7fd3be6c8640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3b819a0c0 0x7fd3b81a4670 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.111+0000 7fd3be6c8640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b8106160 0x7fd3b8199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.111+0000 7fd3be6c8640 1 --2- 172.21.15.94:0/2051924424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3b8101090 0x7fd3b81995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.111+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2051924424 >> 172.21.15.94:0/2051924424 conn(0x7fd3b8077f50 msgr2=0x7fd3b80ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.111+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2051924424 shutdown_connections 2024-04-21T08:46:00.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.111+0000 7fd3be6c8640 1 -- 172.21.15.94:0/2051924424 wait complete. 2024-04-21T08:46:00.121 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:45:59 smithi094 ceph-mon[27988]: pgmap v183: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:00.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:46:00.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:45:59 smithi111 ceph-mon[31493]: pgmap v183: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:00.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.222+0000 7ffaae08c640 1 -- 172.21.15.94:0/3855458614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffaa8102190 msgr2=0x7ffaa8102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.222+0000 7ffaae08c640 1 --2- 172.21.15.94:0/3855458614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffaa8102190 0x7ffaa8102570 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7ffa98009900 tx=0x7ffa9802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:00.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.222+0000 7ffaae08c640 1 -- 172.21.15.94:0/3855458614 shutdown_connections 2024-04-21T08:46:00.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.222+0000 7ffaae08c640 1 --2- 172.21.15.94:0/3855458614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaa8107260 0x7ffaa810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.222+0000 7ffaae08c640 1 --2- 172.21.15.94:0/3855458614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaa8102b40 0x7ffaa8106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.222+0000 7ffaae08c640 1 --2- 172.21.15.94:0/3855458614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffaa8102190 0x7ffaa8102570 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.222+0000 7ffaae08c640 1 -- 172.21.15.94:0/3855458614 >> 172.21.15.94:0/3855458614 conn(0x7ffaa80793a0 msgr2=0x7ffaa80797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.222+0000 7ffaae08c640 1 -- 172.21.15.94:0/3855458614 shutdown_connections 2024-04-21T08:46:00.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.222+0000 7ffaae08c640 1 -- 172.21.15.94:0/3855458614 wait complete. 2024-04-21T08:46:00.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaae08c640 1 Processor -- start 2024-04-21T08:46:00.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaae08c640 1 -- start start 2024-04-21T08:46:00.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaae08c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaa8102190 0x7ffaa8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaae08c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffaa8107260 0x7ffaa8198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaae08c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaa819d020 0x7ffaa819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaae08c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa8112220 con 0x7ffaa8107260 2024-04-21T08:46:00.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaae08c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81120a0 con 0x7ffaa819d020 2024-04-21T08:46:00.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaae08c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81123a0 con 0x7ffaa8102190 2024-04-21T08:46:00.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaa6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffaa8107260 0x7ffaa8198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaa6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffaa8107260 0x7ffaa8198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44830/0 (socket says 172.21.15.94:44830) 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaa6ffd640 1 -- 172.21.15.94:0/4143414444 learned_addr learned my addr 172.21.15.94:0/4143414444 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaa7fff640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaa819d020 0x7ffaa819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.223+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaa8102190 0x7ffaa8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaa6ffd640 1 -- 172.21.15.94:0/4143414444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaa8102190 msgr2=0x7ffaa8198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaa6ffd640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaa8102190 0x7ffaa8198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaa6ffd640 1 -- 172.21.15.94:0/4143414444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaa819d020 msgr2=0x7ffaa819d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaa6ffd640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaa819d020 0x7ffaa819d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaa6ffd640 1 -- 172.21.15.94:0/4143414444 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa819db10 con 0x7ffaa8107260 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaa6ffd640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffaa8107260 0x7ffaa8198ca0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c00c970 tx=0x7ffa9c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaa4ff9640 1 -- 172.21.15.94:0/4143414444 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c014070 con 0x7ffaa8107260 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaae08c640 1 -- 172.21.15.94:0/4143414444 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa819de00 con 0x7ffaa8107260 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaa4ff9640 1 -- 172.21.15.94:0/4143414444 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9c004440 con 0x7ffaa8107260 2024-04-21T08:46:00.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaa4ff9640 1 -- 172.21.15.94:0/4143414444 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c003e50 con 0x7ffaa8107260 2024-04-21T08:46:00.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.224+0000 7ffaae08c640 1 -- 172.21.15.94:0/4143414444 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa81a55f0 con 0x7ffaa8107260 2024-04-21T08:46:00.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffab0c84740 1 Processor -- start 2024-04-21T08:46:00.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffab0c84740 1 -- start start 2024-04-21T08:46:00.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffaa4ff9640 1 -- 172.21.15.94:0/4143414444 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ffa9c017070 con 0x7ffaa8107260 2024-04-21T08:46:00.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005d14d0 0x55d5005d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005da430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005daa00 0x55d5005dce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffab0c84740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d5005361f0 con 0x55d5005d1e80 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffab0c84740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d5004d34b0 con 0x55d5005d14d0 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffab0c84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d5005321b0 con 0x55d5005daa00 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffaa77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005da430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffaa77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005da430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44834/0 (socket says 172.21.15.94:44834) 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffaa77fe640 1 -- 172.21.15.94:0/776105802 learned_addr learned my addr 172.21.15.94:0/776105802 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffaa7fff640 1 --2- 172.21.15.94:0/776105802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005d14d0 0x55d5005d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffaa6ffd640 1 --2- 172.21.15.94:0/776105802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005daa00 0x55d5005dce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.227+0000 7ffaa77fe640 1 -- 172.21.15.94:0/776105802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005daa00 msgr2=0x55d5005dce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/776105802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005daa00 0x55d5005dce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffaa77fe640 1 -- 172.21.15.94:0/776105802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005d14d0 msgr2=0x55d5005d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/776105802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005d14d0 0x55d5005d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffaa77fe640 1 -- 172.21.15.94:0/776105802 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5004c51b0 con 0x55d5005d1e80 2024-04-21T08:46:00.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffaa4ff9640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffa900a3ec0 0x7ffa900a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffaa7fff640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffa900a3ec0 0x7ffa900a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/776105802 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005da430 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ffa98009900 tx=0x7ffa980040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559155da1af2bd62 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffaa4ff9640 1 -- 172.21.15.94:0/4143414444 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ffa9c0d09b0 con 0x7ffaa8107260 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffa8dffb640 1 -- 172.21.15.94:0/776105802 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98045020 con 0x55d5005d1e80 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffa8dffb640 1 -- 172.21.15.94:0/776105802 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9802cc90 con 0x55d5005d1e80 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.228+0000 7ffa8dffb640 1 -- 172.21.15.94:0/776105802 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa980046c0 con 0x55d5005d1e80 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 -- 172.21.15.94:0/776105802 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 msgr2=0x55d5005da430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffaa7fff640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffa900a3ec0 0x7ffa900a6380 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000a510 tx=0x7ffaa000d040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 --2- 172.21.15.94:0/776105802 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005da430 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ffa98009900 tx=0x7ffa980040f0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 -- 172.21.15.94:0/776105802 shutdown_connections 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 --2- 172.21.15.94:0/776105802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005daa00 0x55d5005dce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 --2- 172.21.15.94:0/776105802 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005da430 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 --2- 172.21.15.94:0/776105802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005d14d0 0x55d5005d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 -- 172.21.15.94:0/776105802 >> 172.21.15.94:0/776105802 conn(0x55d5005cbba0 msgr2=0x55d5005cbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 -- 172.21.15.94:0/776105802 shutdown_connections 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 -- 172.21.15.94:0/776105802 wait complete. 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 -- start start 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 Processor -- start 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005d14d0 0x55d5005da1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d1e80 0x55d5005d7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005daa00 0x55d5005d7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.229+0000 7ffab0c84740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d5005dd910 con 0x55d5005daa00 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffab0c84740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d5004ef500 con 0x55d5005d14d0 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffab0c84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d5005dda90 con 0x55d5005d1e80 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffab0c84740 10 client.0 _ll_register_callbacks cb 0x55d5004bdb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005daa00 0x55d5005d7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005daa00 0x55d5005d7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44838/0 (socket says 172.21.15.94:44838) 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa7fff640 1 -- 172.21.15.94:0/2193850917 learned_addr learned my addr 172.21.15.94:0/2193850917 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa6ffd640 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d1e80 0x55d5005d7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa7fff640 1 -- 172.21.15.94:0/2193850917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d1e80 msgr2=0x55d5005d7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005d14d0 0x55d5005da1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa7fff640 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d1e80 0x55d5005d7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa7fff640 1 -- 172.21.15.94:0/2193850917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005d14d0 msgr2=0x55d5005da1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa7fff640 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005d14d0 0x55d5005da1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa7fff640 1 -- 172.21.15.94:0/2193850917 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5002483f0 con 0x55d5005daa00 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa6ffd640 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d1e80 0x55d5005d7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffaa7fff640 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005daa00 0x55d5005d7a90 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0018a90 tx=0x7ffaa0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b9738a21bc540f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffa86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffa86ffd640 1 -- 172.21.15.94:0/2193850917 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0028060 con 0x55d5005daa00 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffa86ffd640 1 -- 172.21.15.94:0/2193850917 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ffaa0023440 con 0x55d5005daa00 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffab0c84740 1 -- 172.21.15.94:0/2193850917 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa9802cc90 con 0x55d5005daa00 2024-04-21T08:46:00.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffa86ffd640 1 -- 172.21.15.94:0/2193850917 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffaa002f050 con 0x55d5005daa00 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffab0c84740 10 client.17283 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffab0c84740 1 -- 172.21.15.94:0/2193850917 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d5003127d0 con 0x55d5005daa00 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.230+0000 7ffa86ffd640 1 -- 172.21.15.94:0/2193850917 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9802cc90 con 0x55d5005daa00 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.231+0000 7ffa86ffd640 1 -- 172.21.15.94:0/2193850917 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ffaa00165b0 con 0x55d5005daa00 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.231+0000 7ffab0c84740 10 client.17283 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.231+0000 7ffab0c84740 10 client.17283 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.231+0000 7ffab0c84740 20 client.17283 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.231+0000 7ffab0c84740 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55d5005eaab0 0x55d5005ecf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.231+0000 7ffab0c84740 4 client.17283 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.231+0000 7ffab0c84740 1 -- 172.21.15.94:0/2193850917 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5004d5930 con 0x55d5005eaab0 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.231+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55d5005eaab0 0x55d5005ecf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffa86ffd640 10 client.17283 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55d5005eaab0 0x55d5005ecf00 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ecd6135e01d0c72 server_cookie=6d4fe3a67a97ba00 in_seq=0 out_seq=0 2024-04-21T08:46:00.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffa86ffd640 1 -- 172.21.15.94:0/2193850917 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5004d5930 con 0x55d5005eaab0 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffa86ffd640 10 client.17283 handle_command_reply: tid=0 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffab0c84740 1 client.17283 shutdown 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffab0c84740 1 -- 172.21.15.94:0/2193850917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005daa00 msgr2=0x55d5005d7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffab0c84740 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005daa00 0x55d5005d7a90 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0018a90 tx=0x7ffaa0004a40 comp rx=0 tx=0).stop 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffab0c84740 1 -- 172.21.15.94:0/2193850917 shutdown_connections 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffab0c84740 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d1e80 0x55d5005d7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffab0c84740 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55d5005eaab0 0x55d5005ecf00 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffab0c84740 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005daa00 0x55d5005d7a90 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffab0c84740 1 --2- 172.21.15.94:0/2193850917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005d14d0 0x55d5005da1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffa86ffd640 0 client.17283 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.232+0000 7ffab0c84740 1 -- 172.21.15.94:0/2193850917 >> 172.21.15.94:0/2193850917 conn(0x55d5005cbba0 msgr2=0x55d5005cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.233+0000 7ffab0c84740 1 -- 172.21.15.94:0/2193850917 shutdown_connections 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.233+0000 7ffab0c84740 1 -- 172.21.15.94:0/2193850917 wait complete. 2024-04-21T08:46:00.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.233+0000 7ffab0c84740 20 client.17283 trim_cache size 0 max 16384 2024-04-21T08:46:00.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffab0c84740 1 Processor -- start 2024-04-21T08:46:00.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffab0c84740 1 -- start start 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 0x55d5005e67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d14d0 0x55d5005caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d1e80 0x55d5005d9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffab0c84740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81b1ca0 con 0x55d5005d14d0 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffab0c84740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81af570 con 0x55d5005cbba0 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffab0c84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81ad3c0 con 0x55d5005d1e80 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffaa6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d14d0 0x55d5005caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffaa6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d14d0 0x55d5005caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44842/0 (socket says 172.21.15.94:44842) 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffaa6ffd640 1 -- 172.21.15.94:0/3749382840 learned_addr learned my addr 172.21.15.94:0/3749382840 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/3749382840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 0x55d5005e67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffaa6ffd640 1 -- 172.21.15.94:0/3749382840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d1e80 msgr2=0x55d5005d9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffaa6ffd640 1 --2- 172.21.15.94:0/3749382840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d1e80 0x55d5005d9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffaa6ffd640 1 -- 172.21.15.94:0/3749382840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 msgr2=0x55d5005e67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffaa6ffd640 1 --2- 172.21.15.94:0/3749382840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 0x55d5005e67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffaa6ffd640 1 -- 172.21.15.94:0/3749382840 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5005dd2b0 con 0x55d5005d14d0 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.246+0000 7ffaa6ffd640 1 --2- 172.21.15.94:0/3749382840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d14d0 0x55d5005caa80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c000c00 tx=0x7ffa9c0038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63aa57bbfa243ed2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffa8d7fa640 1 -- 172.21.15.94:0/3749382840 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c0d0e60 con 0x55d5005d14d0 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffa8d7fa640 1 -- 172.21.15.94:0/3749382840 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9c09c570 con 0x55d5005d14d0 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffa8d7fa640 1 -- 172.21.15.94:0/3749382840 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c098be0 con 0x55d5005d14d0 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffab0c84740 1 -- 172.21.15.94:0/3749382840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d14d0 msgr2=0x55d5005caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffab0c84740 1 --2- 172.21.15.94:0/3749382840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d14d0 0x55d5005caa80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c000c00 tx=0x7ffa9c0038b0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffab0c84740 1 -- 172.21.15.94:0/3749382840 shutdown_connections 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffab0c84740 1 --2- 172.21.15.94:0/3749382840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d1e80 0x55d5005d9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffab0c84740 1 --2- 172.21.15.94:0/3749382840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d14d0 0x55d5005caa80 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffab0c84740 1 --2- 172.21.15.94:0/3749382840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 0x55d5005e67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffab0c84740 1 -- 172.21.15.94:0/3749382840 >> 172.21.15.94:0/3749382840 conn(0x55d5005d7bf0 msgr2=0x55d5005ca370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffab0c84740 1 -- 172.21.15.94:0/3749382840 shutdown_connections 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.247+0000 7ffab0c84740 1 -- 172.21.15.94:0/3749382840 wait complete. 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffab0c84740 1 -- start start 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffab0c84740 1 Processor -- start 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 0x55d5005e3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d14d0 0x55d5005de3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffab0c84740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005de900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffab0c84740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81ada30 con 0x55d5005d1e80 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffab0c84740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81aeea0 con 0x55d5005cbba0 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffab0c84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81ae100 con 0x55d5005d14d0 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffab0c84740 10 client.0 _ll_register_callbacks cb 0x7ffaa0018460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffaa7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005de900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffaa7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005de900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44856/0 (socket says 172.21.15.94:44856) 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffaa7fff640 1 -- 172.21.15.94:0/1253943094 learned_addr learned my addr 172.21.15.94:0/1253943094 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 0x55d5005e3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffaa6ffd640 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d14d0 0x55d5005de3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffaa7fff640 1 -- 172.21.15.94:0/1253943094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d14d0 msgr2=0x55d5005de3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffaa7fff640 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d14d0 0x55d5005de3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffaa7fff640 1 -- 172.21.15.94:0/1253943094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 msgr2=0x55d5005e3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.248+0000 7ffaa7fff640 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 0x55d5005e3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffaa7fff640 1 -- 172.21.15.94:0/1253943094 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5005df130 con 0x55d5005d1e80 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 0x55d5005e3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffaa7fff640 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005de900 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0016e20 tx=0x7ffaa0016e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca43e5e9b64a51f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffa86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffa86ffd640 1 -- 172.21.15.94:0/1253943094 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0038060 con 0x55d5005d1e80 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffa86ffd640 1 -- 172.21.15.94:0/1253943094 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ffaa00165d0 con 0x55d5005d1e80 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffab0c84740 1 -- 172.21.15.94:0/1253943094 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa9c09c570 con 0x55d5005d1e80 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffa86ffd640 1 -- 172.21.15.94:0/1253943094 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffaa0028020 con 0x55d5005d1e80 2024-04-21T08:46:00.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffab0c84740 10 client.17289 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffab0c84740 1 -- 172.21.15.94:0/1253943094 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d500651010 con 0x55d5005d1e80 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffa86ffd640 1 -- 172.21.15.94:0/1253943094 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9c09c570 con 0x55d5005d1e80 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffa86ffd640 1 -- 172.21.15.94:0/1253943094 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ffaa0033430 con 0x55d5005d1e80 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffab0c84740 10 client.17289 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffab0c84740 10 client.17289 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffab0c84740 20 client.17289 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffab0c84740 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55d5005daa00 0x55d5006537a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffab0c84740 4 client.17289 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.249+0000 7ffab0c84740 1 -- 172.21.15.94:0/1253943094 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d500543c10 con 0x55d5005daa00 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55d5005daa00 0x55d5006537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffaa77fe640 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55d5005daa00 0x55d5006537a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5c113bdd6edbd24 server_cookie=c6154cce2878f938 in_seq=0 out_seq=0 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffa86ffd640 10 client.17289 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffa86ffd640 1 -- 172.21.15.94:0/1253943094 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d500543c10 con 0x55d5005daa00 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffa86ffd640 10 client.17289 handle_command_reply: tid=0 2024-04-21T08:46:00.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 client.17289 shutdown 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 -- 172.21.15.94:0/1253943094 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 msgr2=0x55d5005de900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005de900 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0016e20 tx=0x7ffaa0016e50 comp rx=0 tx=0).stop 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 -- 172.21.15.94:0/1253943094 shutdown_connections 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5005d14d0 0x55d5005de3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55d5005daa00 0x55d5006537a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d5005d1e80 0x55d5005de900 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 --2- 172.21.15.94:0/1253943094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5005cbba0 0x55d5005e3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffa86ffd640 0 client.17289 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 -- 172.21.15.94:0/1253943094 >> 172.21.15.94:0/1253943094 conn(0x55d5005d7bf0 msgr2=0x55d5005d59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 -- 172.21.15.94:0/1253943094 shutdown_connections 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 1 -- 172.21.15.94:0/1253943094 wait complete. 2024-04-21T08:46:00.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.250+0000 7ffab0c84740 20 client.17289 trim_cache size 0 max 16384 2024-04-21T08:46:00.258 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:46:00.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 -- 172.21.15.94:0/4143414444 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffa900a3ec0 msgr2=0x7ffa900a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffa900a3ec0 0x7ffa900a6380 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000a510 tx=0x7ffaa000d040 comp rx=0 tx=0).stop 2024-04-21T08:46:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 -- 172.21.15.94:0/4143414444 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffaa8107260 msgr2=0x7ffaa8198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffaa8107260 0x7ffaa8198ca0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c00c970 tx=0x7ffa9c00ce30 comp rx=0 tx=0).stop 2024-04-21T08:46:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 -- 172.21.15.94:0/4143414444 shutdown_connections 2024-04-21T08:46:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffa900a3ec0 0x7ffa900a6380 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaa819d020 0x7ffaa819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffaa8107260 0x7ffaa8198ca0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 --2- 172.21.15.94:0/4143414444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaa8102190 0x7ffaa8198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 -- 172.21.15.94:0/4143414444 >> 172.21.15.94:0/4143414444 conn(0x7ffaa80793a0 msgr2=0x7ffaa80ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 -- 172.21.15.94:0/4143414444 shutdown_connections 2024-04-21T08:46:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.251+0000 7ffaae08c640 1 -- 172.21.15.94:0/4143414444 wait complete. 2024-04-21T08:46:00.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:45:59 smithi173 ceph-mon[31428]: pgmap v183: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:00.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:46:00.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 -- 172.21.15.94:0/1629815909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb17406b1f0 msgr2=0x7fb174106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 --2- 172.21.15.94:0/1629815909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb17406b1f0 0x7fb174106b90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fb160009900 tx=0x7fb16002bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 -- 172.21.15.94:0/1629815909 shutdown_connections 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 --2- 172.21.15.94:0/1629815909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb174107270 0x7fb17410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 --2- 172.21.15.94:0/1629815909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb17406b1f0 0x7fb174106b90 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 --2- 172.21.15.94:0/1629815909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb17406a840 0x7fb17406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 -- 172.21.15.94:0/1629815909 >> 172.21.15.94:0/1629815909 conn(0x7fb174077eb0 msgr2=0x7fb1740782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 -- 172.21.15.94:0/1629815909 shutdown_connections 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 -- 172.21.15.94:0/1629815909 wait complete. 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 Processor -- start 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 -- start start 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb17406a840 0x7fb174198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb174107270 0x7fb174198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.362+0000 7fb17ab8f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb17419bfa0 0x7fb17419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17ab8f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb174112210 con 0x7fb17406a840 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17ab8f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb174112090 con 0x7fb174107270 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17ab8f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb174112390 con 0x7fb17419bfa0 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17892b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb17406a840 0x7fb174198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17892b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb17406a840 0x7fb174198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44868/0 (socket says 172.21.15.94:44868) 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17892b640 1 -- 172.21.15.94:0/3231756778 learned_addr learned my addr 172.21.15.94:0/3231756778 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17912c640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb17419bfa0 0x7fb17419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb173fff640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb174107270 0x7fb174198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17892b640 1 -- 172.21.15.94:0/3231756778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb17419bfa0 msgr2=0x7fb17419c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17892b640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb17419bfa0 0x7fb17419c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17892b640 1 -- 172.21.15.94:0/3231756778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb174107270 msgr2=0x7fb174198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17892b640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb174107270 0x7fb174198cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17892b640 1 -- 172.21.15.94:0/3231756778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17419ca90 con 0x7fb17406a840 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17912c640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb17419bfa0 0x7fb17419c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:00.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17892b640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb17406a840 0x7fb174198790 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fb16800b4a0 tx=0x7fb16800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb171ffb640 1 -- 172.21.15.94:0/3231756778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168014070 con 0x7fb17406a840 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb171ffb640 1 -- 172.21.15.94:0/3231756778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb168010ac0 con 0x7fb17406a840 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb17ab8f640 1 -- 172.21.15.94:0/3231756778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17419cd80 con 0x7fb17406a840 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.363+0000 7fb171ffb640 1 -- 172.21.15.94:0/3231756778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168017c30 con 0x7fb17406a840 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.364+0000 7fb17ab8f640 1 -- 172.21.15.94:0/3231756778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1741a5620 con 0x7fb17406a840 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.365+0000 7fb171ffb640 1 -- 172.21.15.94:0/3231756778 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fb168017070 con 0x7fb17406a840 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb171ffb640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb1640a3d20 0x7fb1640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17d780740 1 Processor -- start 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17d780740 1 -- start start 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb173fff640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb1640a3d20 0x7fb1640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb171ffb640 1 -- 172.21.15.94:0/3231756778 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb1680d0260 con 0x7fb17406a840 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17d780740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e693084d0 0x558e693088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17d780740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69308e80 0x558e69311430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17d780740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e69313e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17d780740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x558e6926d1f0 con 0x558e69311a00 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17d780740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558e6920a4b0 con 0x558e693084d0 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17d780740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558e692691b0 con 0x558e69308e80 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17912c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e69313e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17912c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e69313e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44874/0 (socket says 172.21.15.94:44874) 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17912c640 1 -- 172.21.15.94:0/3396815520 learned_addr learned my addr 172.21.15.94:0/3396815520 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb17892b640 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e693084d0 0x558e693088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.366+0000 7fb173fff640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb1640a3d20 0x7fb1640a61e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb160006010 tx=0x7fb160000bc0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb173fff640 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69308e80 0x558e69311430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb17912c640 1 -- 172.21.15.94:0/3396815520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69308e80 msgr2=0x558e69311430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb17912c640 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69308e80 0x558e69311430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb17912c640 1 -- 172.21.15.94:0/3396815520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e693084d0 msgr2=0x558e693088b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb17912c640 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e693084d0 0x558e693088b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb17912c640 1 -- 172.21.15.94:0/3396815520 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e691fc1b0 con 0x558e69311a00 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb17892b640 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e693084d0 0x558e693088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb173fff640 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69308e80 0x558e69311430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb17912c640 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e69313e50 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb16c008e60 tx=0x7fb16c00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b02bf9eb71100 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb15effd640 1 -- 172.21.15.94:0/3396815520 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c00f830 con 0x558e69311a00 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb15effd640 1 -- 172.21.15.94:0/3396815520 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb16c018480 con 0x558e69311a00 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.367+0000 7fb15effd640 1 -- 172.21.15.94:0/3396815520 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c00e5d0 con 0x558e69311a00 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 -- 172.21.15.94:0/3396815520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 msgr2=0x558e69313e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e69313e50 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb16c008e60 tx=0x7fb16c00d710 comp rx=0 tx=0).stop 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 -- 172.21.15.94:0/3396815520 shutdown_connections 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69308e80 0x558e69311430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e69313e50 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 --2- 172.21.15.94:0/3396815520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e693084d0 0x558e693088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 -- 172.21.15.94:0/3396815520 >> 172.21.15.94:0/3396815520 conn(0x558e69302ba0 msgr2=0x558e69302f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 -- 172.21.15.94:0/3396815520 shutdown_connections 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 -- 172.21.15.94:0/3396815520 wait complete. 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 -- start start 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 Processor -- start 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 0x558e693112c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69308e80 0x558e6930e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69311a00 0x558e6930ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x558e69314970 con 0x558e69308e80 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558e69226500 con 0x558e69311a00 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558e69314af0 con 0x558e693084d0 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb17d780740 10 client.0 _ll_register_callbacks cb 0x558e691f4b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.368+0000 7fb173fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69308e80 0x558e6930e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb173fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69308e80 0x558e6930e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44880/0 (socket says 172.21.15.94:44880) 2024-04-21T08:46:00.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb173fff640 1 -- 172.21.15.94:0/2490809098 learned_addr learned my addr 172.21.15.94:0/2490809098 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb17892b640 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 0x558e693112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb173fff640 1 -- 172.21.15.94:0/2490809098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 msgr2=0x558e693112c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb173fff640 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 0x558e693112c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb173fff640 1 -- 172.21.15.94:0/2490809098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69311a00 msgr2=0x558e6930ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb173fff640 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69311a00 0x558e6930ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb173fff640 1 -- 172.21.15.94:0/2490809098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e68f7f3f0 con 0x558e69308e80 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb173fff640 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69308e80 0x558e6930e650 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb160036040 tx=0x7fb160009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf90cb2197d04195 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb153fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb153fff640 1 -- 172.21.15.94:0/2490809098 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16004f020 con 0x558e69308e80 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb153fff640 1 -- 172.21.15.94:0/2490809098 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb16003a040 con 0x558e69308e80 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb17d780740 1 -- 172.21.15.94:0/2490809098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb16c018480 con 0x558e69308e80 2024-04-21T08:46:00.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb153fff640 1 -- 172.21.15.94:0/2490809098 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb16005a050 con 0x558e69308e80 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb17d780740 10 client.17301 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb17d780740 1 -- 172.21.15.94:0/2490809098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558e690497d0 con 0x558e69308e80 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.369+0000 7fb153fff640 1 -- 172.21.15.94:0/2490809098 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb16c018480 con 0x558e69308e80 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.370+0000 7fb153fff640 1 -- 172.21.15.94:0/2490809098 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb160003710 con 0x558e69308e80 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.370+0000 7fb17d780740 10 client.17301 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.370+0000 7fb17d780740 10 client.17301 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.370+0000 7fb17d780740 20 client.17301 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.370+0000 7fb17d780740 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558e69321be0 0x558e693240a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.370+0000 7fb17d780740 4 client.17301 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.370+0000 7fb17d780740 1 -- 172.21.15.94:0/2490809098 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e6920c930 con 0x558e69321be0 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.370+0000 7fb17892b640 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558e69321be0 0x558e693240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.370+0000 7fb153fff640 10 client.17301 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:00.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.370+0000 7fb17892b640 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558e69321be0 0x558e693240a0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2830cbad9655552d server_cookie=be22b881ba76abcd in_seq=0 out_seq=0 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb153fff640 1 -- 172.21.15.94:0/2490809098 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558e6920c930 con 0x558e69321be0 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb153fff640 10 client.17301 handle_command_reply: tid=0 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 client.17301 shutdown 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 -- 172.21.15.94:0/2490809098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69308e80 msgr2=0x558e6930e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69308e80 0x558e6930e650 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb160036040 tx=0x7fb160009260 comp rx=0 tx=0).stop 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 -- 172.21.15.94:0/2490809098 shutdown_connections 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558e69321be0 0x558e693240a0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69311a00 0x558e6930ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69308e80 0x558e6930e650 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 --2- 172.21.15.94:0/2490809098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 0x558e693112c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb153fff640 0 client.17301 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 -- 172.21.15.94:0/2490809098 >> 172.21.15.94:0/2490809098 conn(0x558e69302ba0 msgr2=0x558e693048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 -- 172.21.15.94:0/2490809098 shutdown_connections 2024-04-21T08:46:00.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 1 -- 172.21.15.94:0/2490809098 wait complete. 2024-04-21T08:46:00.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.371+0000 7fb17d780740 20 client.17301 trim_cache size 0 max 16384 2024-04-21T08:46:00.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.384+0000 7fb17d780740 1 Processor -- start 2024-04-21T08:46:00.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17d780740 1 -- start start 2024-04-21T08:46:00.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17d780740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e693084d0 0x558e6931d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17d780740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69308e80 0x558e69310d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17d780740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69311a00 0x558e693112c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17d780740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741b1d50 con 0x558e693084d0 2024-04-21T08:46:00.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17d780740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741af620 con 0x558e69308e80 2024-04-21T08:46:00.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17d780740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741ad470 con 0x558e69311a00 2024-04-21T08:46:00.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17892b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e693084d0 0x558e6931d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17892b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e693084d0 0x558e6931d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44892/0 (socket says 172.21.15.94:44892) 2024-04-21T08:46:00.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17892b640 1 -- 172.21.15.94:0/3669945507 learned_addr learned my addr 172.21.15.94:0/3669945507 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17912c640 1 --2- 172.21.15.94:0/3669945507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69311a00 0x558e693112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17892b640 1 -- 172.21.15.94:0/3669945507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69311a00 msgr2=0x558e693112c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17892b640 1 --2- 172.21.15.94:0/3669945507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69311a00 0x558e693112c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17892b640 1 -- 172.21.15.94:0/3669945507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69308e80 msgr2=0x558e69310d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17892b640 1 --2- 172.21.15.94:0/3669945507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69308e80 0x558e69310d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17892b640 1 -- 172.21.15.94:0/3669945507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e693142b0 con 0x558e693084d0 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb17892b640 1 --2- 172.21.15.94:0/3669945507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e693084d0 0x558e6931d830 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fb168098390 tx=0x7fb1680cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f46ce982644553 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb15e7fc640 1 -- 172.21.15.94:0/3669945507 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16801e040 con 0x558e693084d0 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb15e7fc640 1 -- 172.21.15.94:0/3669945507 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1680a02e0 con 0x558e693084d0 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.385+0000 7fb15e7fc640 1 -- 172.21.15.94:0/3669945507 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1680985a0 con 0x558e693084d0 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 -- 172.21.15.94:0/3669945507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e693084d0 msgr2=0x558e6931d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 --2- 172.21.15.94:0/3669945507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e693084d0 0x558e6931d830 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fb168098390 tx=0x7fb1680cf040 comp rx=0 tx=0).stop 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 -- 172.21.15.94:0/3669945507 shutdown_connections 2024-04-21T08:46:00.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 --2- 172.21.15.94:0/3669945507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e69311a00 0x558e693112c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 --2- 172.21.15.94:0/3669945507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69308e80 0x558e69310d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 --2- 172.21.15.94:0/3669945507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e693084d0 0x558e6931d830 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 -- 172.21.15.94:0/3669945507 >> 172.21.15.94:0/3669945507 conn(0x558e69302ba0 msgr2=0x558e69301c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 -- 172.21.15.94:0/3669945507 shutdown_connections 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 -- 172.21.15.94:0/3669945507 wait complete. 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 -- start start 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.386+0000 7fb17d780740 1 Processor -- start 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17d780740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 0x558e6931a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17d780740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69308e80 0x558e6930b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17d780740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e6930b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17d780740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741adae0 con 0x558e69311a00 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17d780740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741aef50 con 0x558e69308e80 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17d780740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741ae1b0 con 0x558e693084d0 2024-04-21T08:46:00.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17d780740 10 client.0 _ll_register_callbacks cb 0x7fb16002c7e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17912c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e6930b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17912c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e6930b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44906/0 (socket says 172.21.15.94:44906) 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17912c640 1 -- 172.21.15.94:0/2122000756 learned_addr learned my addr 172.21.15.94:0/2122000756 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17892b640 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 0x558e6931a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb173fff640 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69308e80 0x558e6930b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17912c640 1 -- 172.21.15.94:0/2122000756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 msgr2=0x558e6931a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17912c640 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 0x558e6931a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17912c640 1 -- 172.21.15.94:0/2122000756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69308e80 msgr2=0x558e6930b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17912c640 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69308e80 0x558e6930b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17912c640 1 -- 172.21.15.94:0/2122000756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e6930bdd0 con 0x558e69311a00 2024-04-21T08:46:00.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb173fff640 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69308e80 0x558e6930b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17892b640 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 0x558e6931a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17912c640 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e6930b5a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009010 tx=0x7fb16c004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc83593854bbf60 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb153fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb153fff640 1 -- 172.21.15.94:0/2122000756 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c00f830 con 0x558e69311a00 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb153fff640 1 -- 172.21.15.94:0/2122000756 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fb16c00f9d0 con 0x558e69311a00 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.387+0000 7fb17d780740 1 -- 172.21.15.94:0/2122000756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1680a02e0 con 0x558e69311a00 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb153fff640 1 -- 172.21.15.94:0/2122000756 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb16c00fbd0 con 0x558e69311a00 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb17d780740 10 client.17307 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb17d780740 1 -- 172.21.15.94:0/2122000756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558e69388240 con 0x558e69311a00 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb153fff640 1 -- 172.21.15.94:0/2122000756 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1680a02e0 con 0x558e69311a00 2024-04-21T08:46:00.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb153fff640 1 -- 172.21.15.94:0/2122000756 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb16c018510 con 0x558e69311a00 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb17d780740 10 client.17307 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb17d780740 10 client.17307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb17d780740 20 client.17307 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb17d780740 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558e69388700 0x558e6938aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb17d780740 4 client.17307 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb17d780740 1 -- 172.21.15.94:0/2122000756 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558e6927ac10 con 0x558e69388700 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.388+0000 7fb17892b640 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558e69388700 0x558e6938aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb153fff640 10 client.17307 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17892b640 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558e69388700 0x558e6938aaf0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cd151ded1681cdb server_cookie=76dd525d1d21280d in_seq=0 out_seq=0 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb153fff640 1 -- 172.21.15.94:0/2122000756 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558e6927ac10 con 0x558e69388700 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb153fff640 10 client.17307 handle_command_reply: tid=0 2024-04-21T08:46:00.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 client.17307 shutdown 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 -- 172.21.15.94:0/2122000756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 msgr2=0x558e6930b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e6930b5a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009010 tx=0x7fb16c004f70 comp rx=0 tx=0).stop 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 -- 172.21.15.94:0/2122000756 shutdown_connections 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558e69388700 0x558e6938aaf0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558e69311a00 0x558e6930b5a0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558e69308e80 0x558e6930b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb153fff640 0 client.17307 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 --2- 172.21.15.94:0/2122000756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e693084d0 0x558e6931a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 -- 172.21.15.94:0/2122000756 >> 172.21.15.94:0/2122000756 conn(0x558e69302ba0 msgr2=0x558e69309260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 -- 172.21.15.94:0/2122000756 shutdown_connections 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 1 -- 172.21.15.94:0/2122000756 wait complete. 2024-04-21T08:46:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.389+0000 7fb17d780740 20 client.17307 trim_cache size 0 max 16384 2024-04-21T08:46:00.398 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:46:00.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 -- 172.21.15.94:0/3231756778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb1640a3d20 msgr2=0x7fb1640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb1640a3d20 0x7fb1640a61e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb160006010 tx=0x7fb160000bc0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 -- 172.21.15.94:0/3231756778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb17406a840 msgr2=0x7fb174198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb17406a840 0x7fb174198790 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fb16800b4a0 tx=0x7fb16800b960 comp rx=0 tx=0).stop 2024-04-21T08:46:00.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 -- 172.21.15.94:0/3231756778 shutdown_connections 2024-04-21T08:46:00.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fb1640a3d20 0x7fb1640a61e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb17419bfa0 0x7fb17419c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb174107270 0x7fb174198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 --2- 172.21.15.94:0/3231756778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb17406a840 0x7fb174198790 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 -- 172.21.15.94:0/3231756778 >> 172.21.15.94:0/3231756778 conn(0x7fb174077eb0 msgr2=0x7fb174100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 -- 172.21.15.94:0/3231756778 shutdown_connections 2024-04-21T08:46:00.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:00.390+0000 7fb17ab8f640 1 -- 172.21.15.94:0/3231756778 wait complete. 2024-04-21T08:46:01.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3307479677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:01.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3307479677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:01.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3307479677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:01.401 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:46:01.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:46:01.402 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:46:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.508+0000 7f0fe2752640 1 -- 172.21.15.94:0/2139838425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0fdc1072b0 msgr2=0x7f0fdc10eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.508+0000 7f0fe2752640 1 --2- 172.21.15.94:0/2139838425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0fdc1072b0 0x7f0fdc10eec0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009ca0 tx=0x7f0fd402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.508+0000 7f0fe2752640 1 -- 172.21.15.94:0/2139838425 shutdown_connections 2024-04-21T08:46:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.508+0000 7f0fe2752640 1 --2- 172.21.15.94:0/2139838425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fdc1014a0 0x7f0fdc106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.508+0000 7f0fe2752640 1 --2- 172.21.15.94:0/2139838425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0fdc1072b0 0x7f0fdc10eec0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.508+0000 7f0fe2752640 1 --2- 172.21.15.94:0/2139838425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fdc100af0 0x7f0fdc100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.508+0000 7f0fe2752640 1 -- 172.21.15.94:0/2139838425 >> 172.21.15.94:0/2139838425 conn(0x7f0fdc0760e0 msgr2=0x7f0fdc0764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.508+0000 7f0fe2752640 1 -- 172.21.15.94:0/2139838425 shutdown_connections 2024-04-21T08:46:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.508+0000 7f0fe2752640 1 -- 172.21.15.94:0/2139838425 wait complete. 2024-04-21T08:46:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fe2752640 1 Processor -- start 2024-04-21T08:46:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fe2752640 1 -- start start 2024-04-21T08:46:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fe2752640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0fdc100af0 0x7f0fdc198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fe2752640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fdc1072b0 0x7f0fdc198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fe2752640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fdc19cff0 0x7f0fdc19d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fe2752640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc112250 con 0x7f0fdc100af0 2024-04-21T08:46:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fe2752640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc1120d0 con 0x7f0fdc19cff0 2024-04-21T08:46:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fe2752640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc1123d0 con 0x7f0fdc1072b0 2024-04-21T08:46:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fdbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0fdc100af0 0x7f0fdc198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fdbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0fdc100af0 0x7f0fdc198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44922/0 (socket says 172.21.15.94:44922) 2024-04-21T08:46:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fdbfff640 1 -- 172.21.15.94:0/549281010 learned_addr learned my addr 172.21.15.94:0/549281010 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fdb7fe640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fdc1072b0 0x7f0fdc198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.509+0000 7f0fe0cef640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fdc19cff0 0x7f0fdc19d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fdbfff640 1 -- 172.21.15.94:0/549281010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fdc1072b0 msgr2=0x7f0fdc198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fdc1072b0 0x7f0fdc198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fdbfff640 1 -- 172.21.15.94:0/549281010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fdc19cff0 msgr2=0x7f0fdc19d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fdc19cff0 0x7f0fdc19d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fdbfff640 1 -- 172.21.15.94:0/549281010 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fdc19dae0 con 0x7f0fdc100af0 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0fdc100af0 0x7f0fdc198730 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000c920 tx=0x7f0fd000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fd97fa640 1 -- 172.21.15.94:0/549281010 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0007c70 con 0x7f0fdc100af0 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fe2752640 1 -- 172.21.15.94:0/549281010 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fdc19ddd0 con 0x7f0fdc100af0 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fd97fa640 1 -- 172.21.15.94:0/549281010 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0007e10 con 0x7f0fdc100af0 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fd97fa640 1 -- 172.21.15.94:0/549281010 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0018820 con 0x7f0fdc100af0 2024-04-21T08:46:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.510+0000 7f0fe2752640 1 -- 172.21.15.94:0/549281010 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fdc1a5610 con 0x7f0fdc100af0 2024-04-21T08:46:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.512+0000 7f0fd97fa640 1 -- 172.21.15.94:0/549281010 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f0fd000f450 con 0x7f0fdc100af0 2024-04-21T08:46:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.512+0000 7f0fe5422740 1 Processor -- start 2024-04-21T08:46:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.512+0000 7f0fe5422740 1 -- start start 2024-04-21T08:46:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.512+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dbfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dcb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fe5422740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x558f59d23f00 con 0x558f59dc0290 2024-04-21T08:46:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fe5422740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558f59cc1590 con 0x558f59dc8e10 2024-04-21T08:46:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fe5422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558f59d1fec0 con 0x558f59dbf8e0 2024-04-21T08:46:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fdbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fdbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44926/0 (socket says 172.21.15.94:44926) 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fdbfff640 1 -- 172.21.15.94:0/1020158593 learned_addr learned my addr 172.21.15.94:0/1020158593 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fd97fa640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f0fc40a3d20 0x7f0fc40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fe0cef640 1 --2- 172.21.15.94:0/1020158593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dbfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fdb7fe640 1 --2- 172.21.15.94:0/1020158593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dcb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fe0cef640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f0fc40a3d20 0x7f0fc40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fdbfff640 1 -- 172.21.15.94:0/1020158593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 msgr2=0x558f59dbfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/1020158593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dbfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fdbfff640 1 -- 172.21.15.94:0/1020158593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 msgr2=0x558f59dcb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/1020158593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dcb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fd97fa640 1 -- 172.21.15.94:0/549281010 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f0fd00d0c90 con 0x7f0fdc100af0 2024-04-21T08:46:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fdbfff640 1 -- 172.21.15.94:0/1020158593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f59afe7e0 con 0x558f59dc0290 2024-04-21T08:46:01.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/1020158593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc8840 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0099220 tx=0x7f0fd000fe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50b06d3d9b1f0b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fe0cef640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f0fc40a3d20 0x7f0fc40a61e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4004af0 tx=0x7f0fd4005c50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fc27fc640 1 -- 172.21.15.94:0/1020158593 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd009cf50 con 0x558f59dc0290 2024-04-21T08:46:01.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fc27fc640 1 -- 172.21.15.94:0/1020158593 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd004ce50 con 0x558f59dc0290 2024-04-21T08:46:01.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.513+0000 7f0fc27fc640 1 -- 172.21.15.94:0/1020158593 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd00ce280 con 0x558f59dc0290 2024-04-21T08:46:01.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.514+0000 7f0fe5422740 1 -- 172.21.15.94:0/1020158593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 msgr2=0x558f59dc8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.514+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1020158593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc8840 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0099220 tx=0x7f0fd000fe60 comp rx=0 tx=0).stop 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.514+0000 7f0fe5422740 1 -- 172.21.15.94:0/1020158593 shutdown_connections 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.514+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1020158593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dcb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.514+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1020158593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc8840 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.514+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1020158593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dbfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.514+0000 7f0fe5422740 1 -- 172.21.15.94:0/1020158593 >> 172.21.15.94:0/1020158593 conn(0x558f59db9fb0 msgr2=0x558f59dba390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.514+0000 7f0fe5422740 1 -- 172.21.15.94:0/1020158593 shutdown_connections 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.514+0000 7f0fe5422740 1 -- 172.21.15.94:0/1020158593 wait complete. 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.514+0000 7f0fe5422740 1 -- start start 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fe5422740 1 Processor -- start 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dc5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dc6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fe5422740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x558f59dcbd20 con 0x558f59dc0290 2024-04-21T08:46:01.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fe5422740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558f59cdd580 con 0x558f59dc8e10 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fe5422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558f59dcbea0 con 0x558f59dbf8e0 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fe5422740 10 client.0 _ll_register_callbacks cb 0x558f59ce3650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fdb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fdb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44936/0 (socket says 172.21.15.94:44936) 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fdb7fe640 1 -- 172.21.15.94:0/138640505 learned_addr learned my addr 172.21.15.94:0/138640505 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fe0cef640 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dc6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dc5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fdb7fe640 1 -- 172.21.15.94:0/138640505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 msgr2=0x558f59dc5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fdb7fe640 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dc5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fdb7fe640 1 -- 172.21.15.94:0/138640505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 msgr2=0x558f59dc6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fdb7fe640 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dc6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fdb7fe640 1 -- 172.21.15.94:0/138640505 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f59b5ef70 con 0x558f59dc0290 2024-04-21T08:46:01.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fdb7fe640 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc62e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00b920 tx=0x7f0fcc00bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63fa668388d89b4f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.515+0000 7f0fbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/138640505 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc003ea0 con 0x558f59dc0290 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/138640505 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f0fcc005220 con 0x558f59dc0290 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fe5422740 1 -- 172.21.15.94:0/138640505 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd004ce50 con 0x558f59dc0290 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/138640505 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0fcc005420 con 0x558f59dc0290 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fe5422740 10 client.17319 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fe5422740 1 -- 172.21.15.94:0/138640505 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558f599f6170 con 0x558f59dc0290 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/138640505 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd004ce50 con 0x558f59dc0290 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/138640505 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f0fcc019030 con 0x558f59dc0290 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fe5422740 10 client.17319 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fe5422740 10 client.17319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fe5422740 20 client.17319 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fe5422740 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558f59dd9340 0x558f59ddb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fe5422740 4 client.17319 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fe5422740 1 -- 172.21.15.94:0/138640505 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f59d31920 con 0x558f59dd9340 2024-04-21T08:46:01.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.516+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558f59dd9340 0x558f59ddb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.517+0000 7f0fbb7fe640 10 client.17319 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:01.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.517+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558f59dd9340 0x558f59ddb770 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1228a2380ac5b94 server_cookie=c1080ba46a4b9fe5 in_seq=0 out_seq=0 2024-04-21T08:46:01.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.517+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/138640505 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f59d31920 con 0x558f59dd9340 2024-04-21T08:46:01.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.517+0000 7f0fbb7fe640 10 client.17319 handle_command_reply: tid=0 2024-04-21T08:46:01.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.517+0000 7f0fe5422740 1 client.17319 shutdown 2024-04-21T08:46:01.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 1 -- 172.21.15.94:0/138640505 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 msgr2=0x558f59dc62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc62e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00b920 tx=0x7f0fcc00bde0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 1 -- 172.21.15.94:0/138640505 shutdown_connections 2024-04-21T08:46:01.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558f59dd9340 0x558f59ddb770 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dc6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc0290 0x558f59dc62e0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fbb7fe640 0 client.17319 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:01.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 1 --2- 172.21.15.94:0/138640505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dc5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 1 -- 172.21.15.94:0/138640505 >> 172.21.15.94:0/138640505 conn(0x558f59db9fb0 msgr2=0x558f59dbbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 1 -- 172.21.15.94:0/138640505 shutdown_connections 2024-04-21T08:46:01.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 1 -- 172.21.15.94:0/138640505 wait complete. 2024-04-21T08:46:01.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.518+0000 7f0fe5422740 20 client.17319 trim_cache size 0 max 16384 2024-04-21T08:46:01.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fe5422740 1 Processor -- start 2024-04-21T08:46:01.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fe5422740 1 -- start start 2024-04-21T08:46:01.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dbf8e0 0x558f59dc84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dc0290 0x558f59dd4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dc6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fe5422740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc1b1d00 con 0x558f59dbf8e0 2024-04-21T08:46:01.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fe5422740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc1af5d0 con 0x558f59dc8e10 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fe5422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc1ad420 con 0x558f59dc0290 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fdbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dbf8e0 0x558f59dc84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fdbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dbf8e0 0x558f59dc84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44950/0 (socket says 172.21.15.94:44950) 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.535+0000 7f0fdbfff640 1 -- 172.21.15.94:0/1967471224 learned_addr learned my addr 172.21.15.94:0/1967471224 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fdbfff640 1 -- 172.21.15.94:0/1967471224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dc0290 msgr2=0x558f59dd4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/1967471224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dc0290 0x558f59dd4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fdbfff640 1 -- 172.21.15.94:0/1967471224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 msgr2=0x558f59dc6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/1967471224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dc6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fdbfff640 1 -- 172.21.15.94:0/1967471224 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f59dc42f0 con 0x558f59dbf8e0 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/1967471224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dbf8e0 0x558f59dc84d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0099220 tx=0x7f0fd00d0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b631e56a339b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fc1ffb640 1 -- 172.21.15.94:0/1967471224 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd009cf50 con 0x558f59dbf8e0 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fc1ffb640 1 -- 172.21.15.94:0/1967471224 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd009c850 con 0x558f59dbf8e0 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fc1ffb640 1 -- 172.21.15.94:0/1967471224 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000f8d0 con 0x558f59dbf8e0 2024-04-21T08:46:01.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fe5422740 1 -- 172.21.15.94:0/1967471224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dbf8e0 msgr2=0x558f59dc84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1967471224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dbf8e0 0x558f59dc84d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0099220 tx=0x7f0fd00d0040 comp rx=0 tx=0).stop 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fe5422740 1 -- 172.21.15.94:0/1967471224 shutdown_connections 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1967471224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dc0290 0x558f59dd4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1967471224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc8e10 0x558f59dc6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1967471224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dbf8e0 0x558f59dc84d0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fe5422740 1 -- 172.21.15.94:0/1967471224 >> 172.21.15.94:0/1967471224 conn(0x558f59db9fb0 msgr2=0x558f59dc5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fe5422740 1 -- 172.21.15.94:0/1967471224 shutdown_connections 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.536+0000 7f0fe5422740 1 -- 172.21.15.94:0/1967471224 wait complete. 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe5422740 1 -- start start 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe5422740 1 Processor -- start 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dd12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc0290 0x558f59dce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe5422740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc8e10 0x558f59dce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe5422740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc1ada90 con 0x558f59dc8e10 2024-04-21T08:46:01.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe5422740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc1aef00 con 0x558f59dc0290 2024-04-21T08:46:01.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe5422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc1ae160 con 0x558f59dbf8e0 2024-04-21T08:46:01.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe5422740 10 client.0 _ll_register_callbacks cb 0x7f0fcc004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:01.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe0cef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc8e10 0x558f59dce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe0cef640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc8e10 0x558f59dce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44960/0 (socket says 172.21.15.94:44960) 2024-04-21T08:46:01.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe0cef640 1 -- 172.21.15.94:0/1832148071 learned_addr learned my addr 172.21.15.94:0/1832148071 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fdb7fe640 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc0290 0x558f59dce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dd12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe0cef640 1 -- 172.21.15.94:0/1832148071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 msgr2=0x558f59dd12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe0cef640 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dd12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe0cef640 1 -- 172.21.15.94:0/1832148071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc0290 msgr2=0x558f59dce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe0cef640 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc0290 0x558f59dce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fe0cef640 1 -- 172.21.15.94:0/1832148071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f59dcebf0 con 0x558f59dc8e10 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.537+0000 7f0fdb7fe640 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc0290 0x558f59dce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fe0cef640 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc8e10 0x558f59dce600 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4036000 tx=0x7f0fd40323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f56b951101c0f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/1832148071 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4053060 con 0x558f59dc8e10 2024-04-21T08:46:01.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/1832148071 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f0fd4004040 con 0x558f59dc8e10 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fe5422740 1 -- 172.21.15.94:0/1832148071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd009c850 con 0x558f59dc8e10 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/1832148071 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0fd4039040 con 0x558f59dc8e10 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fe5422740 10 client.17325 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fe5422740 1 -- 172.21.15.94:0/1832148071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558f59e40740 con 0x558f59dc8e10 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/1832148071 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd009c850 con 0x558f59dc8e10 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/1832148071 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f0fd404d070 con 0x558f59dc8e10 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fe5422740 10 client.17325 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fe5422740 10 client.17325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fe5422740 20 client.17325 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558f59e40e60 0x558f59e43280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fe5422740 4 client.17325 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.538+0000 7f0fe5422740 1 -- 172.21.15.94:0/1832148071 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558f59d2b790 con 0x558f59e40e60 2024-04-21T08:46:01.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.539+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558f59e40e60 0x558f59e43280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.539+0000 7f0fdbfff640 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558f59e40e60 0x558f59e43280 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c582038c9f6da34 server_cookie=cc9bfacff672527e in_seq=0 out_seq=0 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.539+0000 7f0fbb7fe640 10 client.17325 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fbb7fe640 1 -- 172.21.15.94:0/1832148071 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558f59d2b790 con 0x558f59e40e60 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fbb7fe640 10 client.17325 handle_command_reply: tid=0 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 client.17325 shutdown 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 -- 172.21.15.94:0/1832148071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc8e10 msgr2=0x558f59dce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc8e10 0x558f59dce600 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4036000 tx=0x7f0fd40323c0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 -- 172.21.15.94:0/1832148071 shutdown_connections 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558f59e40e60 0x558f59e43280 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558f59dc8e10 0x558f59dce600 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fbb7fe640 0 client.17325 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558f59dc0290 0x558f59dce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 --2- 172.21.15.94:0/1832148071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f59dbf8e0 0x558f59dd12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 -- 172.21.15.94:0/1832148071 >> 172.21.15.94:0/1832148071 conn(0x558f59db9fb0 msgr2=0x558f59dbbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 -- 172.21.15.94:0/1832148071 shutdown_connections 2024-04-21T08:46:01.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 1 -- 172.21.15.94:0/1832148071 wait complete. 2024-04-21T08:46:01.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.540+0000 7f0fe5422740 20 client.17325 trim_cache size 0 max 16384 2024-04-21T08:46:01.548 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:46:01.548 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:46:01.548 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "8da8ae19-5c2b-44fe-b517-f13704d3fd90", 2024-04-21T08:46:01.548 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:46:01.548 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:46:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 -- 172.21.15.94:0/549281010 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f0fc40a3d20 msgr2=0x7f0fc40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f0fc40a3d20 0x7f0fc40a61e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4004af0 tx=0x7f0fd4005c50 comp rx=0 tx=0).stop 2024-04-21T08:46:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 -- 172.21.15.94:0/549281010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0fdc100af0 msgr2=0x7f0fdc198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0fdc100af0 0x7f0fdc198730 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000c920 tx=0x7f0fd000cde0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 -- 172.21.15.94:0/549281010 shutdown_connections 2024-04-21T08:46:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f0fc40a3d20 0x7f0fc40a61e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fdc1072b0 0x7f0fdc198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fdc19cff0 0x7f0fdc19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 --2- 172.21.15.94:0/549281010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0fdc100af0 0x7f0fdc198730 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 -- 172.21.15.94:0/549281010 >> 172.21.15.94:0/549281010 conn(0x7f0fdc0760e0 msgr2=0x7f0fdc100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 -- 172.21.15.94:0/549281010 shutdown_connections 2024-04-21T08:46:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.541+0000 7f0fe2752640 1 -- 172.21.15.94:0/549281010 wait complete. 2024-04-21T08:46:01.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:01.552 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:01.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.652+0000 7f5731206640 1 -- 172.21.15.94:0/1352778317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f572c101a40 msgr2=0x7f572c10dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.652+0000 7f5731206640 1 --2- 172.21.15.94:0/1352778317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f572c101a40 0x7f572c10dc80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f5724009ca0 tx=0x7f572402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.652+0000 7f5731206640 1 -- 172.21.15.94:0/1352778317 shutdown_connections 2024-04-21T08:46:01.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.652+0000 7f5731206640 1 --2- 172.21.15.94:0/1352778317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c1010a0 0x7f572c101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.652+0000 7f5731206640 1 --2- 172.21.15.94:0/1352778317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f572c101a40 0x7f572c10dc80 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.652+0000 7f5731206640 1 --2- 172.21.15.94:0/1352778317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f572c105dd0 0x7f572c1061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.652+0000 7f5731206640 1 -- 172.21.15.94:0/1352778317 >> 172.21.15.94:0/1352778317 conn(0x7f572c077c40 msgr2=0x7f572c078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.652+0000 7f5731206640 1 -- 172.21.15.94:0/1352778317 shutdown_connections 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.652+0000 7f5731206640 1 -- 172.21.15.94:0/1352778317 wait complete. 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.652+0000 7f5731206640 1 Processor -- start 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f5731206640 1 -- start start 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f5731206640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f572c1010a0 0x7f572c199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f5731206640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f572c101a40 0x7f572c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f5731206640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c105dd0 0x7f572c1a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f5731206640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f572c111230 con 0x7f572c1010a0 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f5731206640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f572c1110b0 con 0x7f572c101a40 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f5731206640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f572c1113b0 con 0x7f572c105dd0 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f572ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f572c1010a0 0x7f572c199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f572ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f572c1010a0 0x7f572c199690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44966/0 (socket says 172.21.15.94:44966) 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f572ad9d640 1 -- 172.21.15.94:0/3739112560 learned_addr learned my addr 172.21.15.94:0/3739112560 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f572a59c640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f572c101a40 0x7f572c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f572b59e640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c105dd0 0x7f572c1a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f572ad9d640 1 -- 172.21.15.94:0/3739112560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c105dd0 msgr2=0x7f572c1a4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f572ad9d640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c105dd0 0x7f572c1a4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f572ad9d640 1 -- 172.21.15.94:0/3739112560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f572c101a40 msgr2=0x7f572c199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f572ad9d640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f572c101a40 0x7f572c199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.653+0000 7f572ad9d640 1 -- 172.21.15.94:0/3739112560 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c1a4c50 con 0x7f572c1010a0 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.654+0000 7f572ad9d640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f572c1010a0 0x7f572c199690 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f571c00b4a0 tx=0x7f571c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.654+0000 7f5723fff640 1 -- 172.21.15.94:0/3739112560 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c015020 con 0x7f572c1010a0 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.654+0000 7f5723fff640 1 -- 172.21.15.94:0/3739112560 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f571c011ce0 con 0x7f572c1010a0 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.654+0000 7f5731206640 1 -- 172.21.15.94:0/3739112560 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f572c1a4e80 con 0x7f572c1010a0 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.654+0000 7f5723fff640 1 -- 172.21.15.94:0/3739112560 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c010dc0 con 0x7f572c1010a0 2024-04-21T08:46:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.654+0000 7f5731206640 1 -- 172.21.15.94:0/3739112560 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f572c1a5340 con 0x7f572c1010a0 2024-04-21T08:46:01.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.656+0000 7f5723fff640 1 -- 172.21.15.94:0/3739112560 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f571c018020 con 0x7f572c1010a0 2024-04-21T08:46:01.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.656+0000 7f5733df3740 1 Processor -- start 2024-04-21T08:46:01.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.656+0000 7f5733df3740 1 -- start start 2024-04-21T08:46:01.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.656+0000 7f5733df3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3230 0x55b5a59c3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.656+0000 7f5733df3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3b50 0x55b5a59d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.656+0000 7f5733df3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 0x55b5a59d3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.656+0000 7f5733df3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b5a59281e0 con 0x55b5a59c3b50 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.656+0000 7f5733df3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b5a58c5390 con 0x55b5a59c3230 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.656+0000 7f5733df3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b5a59241a0 con 0x55b5a59d0e90 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.656+0000 7f5723fff640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57180a3d20 0x7f57180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3b50 0x55b5a59d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572b59e640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57180a3d20 0x7f57180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3b50 0x55b5a59d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44982/0 (socket says 172.21.15.94:44982) 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572ad9d640 1 -- 172.21.15.94:0/738468920 learned_addr learned my addr 172.21.15.94:0/738468920 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572a59c640 1 --2- 172.21.15.94:0/738468920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 0x55b5a59d3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f5723fff640 1 -- 172.21.15.94:0/3739112560 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f571c0d0310 con 0x7f572c1010a0 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572b59e640 1 --2- 172.21.15.94:0/738468920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3230 0x55b5a59c3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572ad9d640 1 -- 172.21.15.94:0/738468920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 msgr2=0x55b5a59d3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572ad9d640 1 --2- 172.21.15.94:0/738468920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 0x55b5a59d3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572ad9d640 1 -- 172.21.15.94:0/738468920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3230 msgr2=0x55b5a59c3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572ad9d640 1 --2- 172.21.15.94:0/738468920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3230 0x55b5a59c3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572ad9d640 1 -- 172.21.15.94:0/738468920 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a57047d0 con 0x55b5a59c3b50 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572a59c640 1 --2- 172.21.15.94:0/738468920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 0x55b5a59d3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572b59e640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57180a3d20 0x7f57180a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f57240098d0 tx=0x7f5724032cf0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f572ad9d640 1 --2- 172.21.15.94:0/738468920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3b50 0x55b5a59d0950 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f571c0989b0 tx=0x7f571c06ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6e37d0926e3169 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f57217fa640 1 -- 172.21.15.94:0/738468920 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c096070 con 0x55b5a59c3b50 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f57217fa640 1 -- 172.21.15.94:0/738468920 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f571c0749e0 con 0x55b5a59c3b50 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.657+0000 7f57217fa640 1 -- 172.21.15.94:0/738468920 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c09bfe0 con 0x55b5a59c3b50 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.658+0000 7f5733df3740 1 -- 172.21.15.94:0/738468920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3b50 msgr2=0x55b5a59d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.658+0000 7f5733df3740 1 --2- 172.21.15.94:0/738468920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3b50 0x55b5a59d0950 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f571c0989b0 tx=0x7f571c06ead0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.658+0000 7f5733df3740 1 -- 172.21.15.94:0/738468920 shutdown_connections 2024-04-21T08:46:01.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.658+0000 7f5733df3740 1 --2- 172.21.15.94:0/738468920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 0x55b5a59d3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.658+0000 7f5733df3740 1 --2- 172.21.15.94:0/738468920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3b50 0x55b5a59d0950 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.658+0000 7f5733df3740 1 --2- 172.21.15.94:0/738468920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3230 0x55b5a59c3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.658+0000 7f5733df3740 1 -- 172.21.15.94:0/738468920 >> 172.21.15.94:0/738468920 conn(0x55b5a59bdcb0 msgr2=0x55b5a59be090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.658+0000 7f5733df3740 1 -- 172.21.15.94:0/738468920 shutdown_connections 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.658+0000 7f5733df3740 1 -- 172.21.15.94:0/738468920 wait complete. 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.658+0000 7f5733df3740 1 -- start start 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f5733df3740 1 Processor -- start 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f5733df3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3230 0x55b5a59cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f5733df3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3b50 0x55b5a59ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f5733df3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59d0e90 0x55b5a59ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f5733df3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b5a59d3c70 con 0x55b5a59d0e90 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f5733df3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b5a58e12f0 con 0x55b5a59c3b50 2024-04-21T08:46:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f5733df3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b5a59d3df0 con 0x55b5a59c3230 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f5733df3740 10 client.0 _ll_register_callbacks cb 0x55b5a58ee580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572b59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59d0e90 0x55b5a59ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572b59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59d0e90 0x55b5a59ce870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44984/0 (socket says 172.21.15.94:44984) 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572b59e640 1 -- 172.21.15.94:0/4066342717 learned_addr learned my addr 172.21.15.94:0/4066342717 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572ad9d640 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3230 0x55b5a59cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572a59c640 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3b50 0x55b5a59ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572b59e640 1 -- 172.21.15.94:0/4066342717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3230 msgr2=0x55b5a59cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572b59e640 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3230 0x55b5a59cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572b59e640 1 -- 172.21.15.94:0/4066342717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3b50 msgr2=0x55b5a59ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572b59e640 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3b50 0x55b5a59ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572b59e640 1 -- 172.21.15.94:0/4066342717 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a563a3f0 con 0x55b5a59d0e90 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f572b59e640 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59d0e90 0x55b5a59ce870 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f5724009ca0 tx=0x7f5724009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780737906386c4a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.659+0000 7f57127fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:01.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f57127fc640 1 -- 172.21.15.94:0/4066342717 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5724052060 con 0x55b5a59d0e90 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f57127fc640 1 -- 172.21.15.94:0/4066342717 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5724004510 con 0x55b5a59d0e90 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f5733df3740 1 -- 172.21.15.94:0/4066342717 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f571c0749e0 con 0x55b5a59d0e90 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f57127fc640 1 -- 172.21.15.94:0/4066342717 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5724059050 con 0x55b5a59d0e90 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f5733df3740 10 client.17337 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f5733df3740 1 -- 172.21.15.94:0/4066342717 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b5a55fa0f0 con 0x55b5a59d0e90 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f57127fc640 1 -- 172.21.15.94:0/4066342717 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f571c0749e0 con 0x55b5a59d0e90 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f57127fc640 1 -- 172.21.15.94:0/4066342717 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f572403f030 con 0x55b5a59d0e90 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f5733df3740 10 client.17337 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f5733df3740 10 client.17337 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f5733df3740 20 client.17337 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f5733df3740 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b5a59e0ed0 0x55b5a59e32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f5733df3740 4 client.17337 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f5733df3740 1 -- 172.21.15.94:0/4066342717 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5a5935c00 con 0x55b5a59e0ed0 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.660+0000 7f572ad9d640 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b5a59e0ed0 0x55b5a59e32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.661+0000 7f57127fc640 10 client.17337 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.661+0000 7f572ad9d640 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b5a59e0ed0 0x55b5a59e32f0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffcde26bcc648fbf server_cookie=459c51f89d232955 in_seq=0 out_seq=0 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.661+0000 7f57127fc640 1 -- 172.21.15.94:0/4066342717 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5a5935c00 con 0x55b5a59e0ed0 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.661+0000 7f57127fc640 10 client.17337 handle_command_reply: tid=0 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.661+0000 7f5733df3740 1 client.17337 shutdown 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 1 -- 172.21.15.94:0/4066342717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59d0e90 msgr2=0x55b5a59ce870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59d0e90 0x55b5a59ce870 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f5724009ca0 tx=0x7f5724009800 comp rx=0 tx=0).stop 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 1 -- 172.21.15.94:0/4066342717 shutdown_connections 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b5a59e0ed0 0x55b5a59e32f0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59d0e90 0x55b5a59ce870 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f57127fc640 0 client.17337 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3b50 0x55b5a59ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 1 --2- 172.21.15.94:0/4066342717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3230 0x55b5a59cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 1 -- 172.21.15.94:0/4066342717 >> 172.21.15.94:0/4066342717 conn(0x55b5a59bdcb0 msgr2=0x55b5a59bfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 1 -- 172.21.15.94:0/4066342717 shutdown_connections 2024-04-21T08:46:01.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 1 -- 172.21.15.94:0/4066342717 wait complete. 2024-04-21T08:46:01.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.662+0000 7f5733df3740 20 client.17337 trim_cache size 0 max 16384 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f5733df3740 1 Processor -- start 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f5733df3740 1 -- start start 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f5733df3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f5733df3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3b50 0x55b5a59dcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f5733df3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 0x55b5a59e8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f5733df3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f572c1afc40 con 0x55b5a59c3230 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f5733df3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f572c1ad510 con 0x55b5a59c3b50 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f5733df3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f572c1ab360 con 0x55b5a59d0e90 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f572ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f572ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44996/0 (socket says 172.21.15.94:44996) 2024-04-21T08:46:01.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f572ad9d640 1 -- 172.21.15.94:0/233823451 learned_addr learned my addr 172.21.15.94:0/233823451 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f572b59e640 1 --2- 172.21.15.94:0/233823451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 0x55b5a59e8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f572ad9d640 1 -- 172.21.15.94:0/233823451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 msgr2=0x55b5a59e8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f572ad9d640 1 --2- 172.21.15.94:0/233823451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 0x55b5a59e8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f572ad9d640 1 -- 172.21.15.94:0/233823451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3b50 msgr2=0x55b5a59dcbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f572ad9d640 1 --2- 172.21.15.94:0/233823451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3b50 0x55b5a59dcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f572ad9d640 1 -- 172.21.15.94:0/233823451 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a59c2670 con 0x55b5a59c3230 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.679+0000 7f572b59e640 1 --2- 172.21.15.94:0/233823451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 0x55b5a59e8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f572ad9d640 1 --2- 172.21.15.94:0/233823451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59ccd30 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f571c004810 tx=0x7f571c09a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96c388ada3af91a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5720ff9640 1 -- 172.21.15.94:0/233823451 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c0d0120 con 0x55b5a59c3230 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5720ff9640 1 -- 172.21.15.94:0/233823451 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f571c09bfe0 con 0x55b5a59c3230 2024-04-21T08:46:01.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5720ff9640 1 -- 172.21.15.94:0/233823451 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c010410 con 0x55b5a59c3230 2024-04-21T08:46:01.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5733df3740 1 -- 172.21.15.94:0/233823451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 msgr2=0x55b5a59ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5733df3740 1 --2- 172.21.15.94:0/233823451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59ccd30 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f571c004810 tx=0x7f571c09a660 comp rx=0 tx=0).stop 2024-04-21T08:46:01.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5733df3740 1 -- 172.21.15.94:0/233823451 shutdown_connections 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5733df3740 1 --2- 172.21.15.94:0/233823451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59d0e90 0x55b5a59e8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5733df3740 1 --2- 172.21.15.94:0/233823451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59c3b50 0x55b5a59dcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5733df3740 1 --2- 172.21.15.94:0/233823451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59ccd30 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5733df3740 1 -- 172.21.15.94:0/233823451 >> 172.21.15.94:0/233823451 conn(0x55b5a59bdcb0 msgr2=0x55b5a59bf990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5733df3740 1 -- 172.21.15.94:0/233823451 shutdown_connections 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.680+0000 7f5733df3740 1 -- 172.21.15.94:0/233823451 wait complete. 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f5733df3740 1 -- start start 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f5733df3740 1 Processor -- start 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f5733df3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59d5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f5733df3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3b50 0x55b5a59d5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f5733df3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59d0e90 0x55b5a5a46430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f5733df3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f572c1ab9d0 con 0x55b5a59c3230 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f5733df3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f572c1ace40 con 0x55b5a59d0e90 2024-04-21T08:46:01.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f5733df3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f572c1ac0a0 con 0x55b5a59c3b50 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f5733df3740 10 client.0 _ll_register_callbacks cb 0x7f5724009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59d5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59d5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45000/0 (socket says 172.21.15.94:45000) 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572ad9d640 1 -- 172.21.15.94:0/492120932 learned_addr learned my addr 172.21.15.94:0/492120932 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572a59c640 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3b50 0x55b5a59d5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572b59e640 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59d0e90 0x55b5a5a46430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572ad9d640 1 -- 172.21.15.94:0/492120932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3b50 msgr2=0x55b5a59d5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572ad9d640 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3b50 0x55b5a59d5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572ad9d640 1 -- 172.21.15.94:0/492120932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59d0e90 msgr2=0x55b5a5a46430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572ad9d640 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59d0e90 0x55b5a5a46430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572ad9d640 1 -- 172.21.15.94:0/492120932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a5a46970 con 0x55b5a59c3230 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572a59c640 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3b50 0x55b5a59d5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:01.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572b59e640 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59d0e90 0x55b5a5a46430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f572ad9d640 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59d5140 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f571c09a710 tx=0x7f571c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea392b64f5c8fce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.681+0000 7f57127fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f57127fc640 1 -- 172.21.15.94:0/492120932 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c0b9070 con 0x55b5a59c3230 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f57127fc640 1 -- 172.21.15.94:0/492120932 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f571c098650 con 0x55b5a59c3230 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f5733df3740 1 -- 172.21.15.94:0/492120932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f571c09bfe0 con 0x55b5a59c3230 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f57127fc640 1 -- 172.21.15.94:0/492120932 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f571c09a8d0 con 0x55b5a59c3230 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f5733df3740 10 client.17343 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f5733df3740 1 -- 172.21.15.94:0/492120932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b5a5a47af0 con 0x55b5a59c3230 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f57127fc640 1 -- 172.21.15.94:0/492120932 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f571c09bfe0 con 0x55b5a59c3230 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f57127fc640 1 -- 172.21.15.94:0/492120932 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f571c09ab10 con 0x55b5a59c3230 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f5733df3740 10 client.17343 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f5733df3740 10 client.17343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f5733df3740 20 client.17343 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f5733df3740 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b5a5a47de0 0x55b5a5a4a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f5733df3740 4 client.17343 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f5733df3740 1 -- 172.21.15.94:0/492120932 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b5a592fa70 con 0x55b5a5a47de0 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.682+0000 7f572a59c640 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b5a5a47de0 0x55b5a5a4a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f572a59c640 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b5a5a47de0 0x55b5a5a4a1d0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66a3cea67e1aa128 server_cookie=c7406c49900c469e in_seq=0 out_seq=0 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f57127fc640 10 client.17343 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f57127fc640 1 -- 172.21.15.94:0/492120932 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b5a592fa70 con 0x55b5a5a47de0 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f57127fc640 10 client.17343 handle_command_reply: tid=0 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 client.17343 shutdown 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 -- 172.21.15.94:0/492120932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 msgr2=0x55b5a59d5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59d5140 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f571c09a710 tx=0x7f571c0046c0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 -- 172.21.15.94:0/492120932 shutdown_connections 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b5a5a47de0 0x55b5a5a4a1d0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a59c3b50 0x55b5a59d5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b5a59d0e90 0x55b5a5a46430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f57127fc640 0 client.17343 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:01.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 --2- 172.21.15.94:0/492120932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5a59c3230 0x55b5a59d5140 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 -- 172.21.15.94:0/492120932 >> 172.21.15.94:0/492120932 conn(0x55b5a59bdcb0 msgr2=0x55b5a59bf100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 -- 172.21.15.94:0/492120932 shutdown_connections 2024-04-21T08:46:01.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 1 -- 172.21.15.94:0/492120932 wait complete. 2024-04-21T08:46:01.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.683+0000 7f5733df3740 20 client.17343 trim_cache size 0 max 16384 2024-04-21T08:46:01.691 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:46:01.691 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:46:01.691 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:46:01.692 INFO:teuthology.orchestra.run.smithi094.stdout: "8da8ae19-5c2b-44fe-b517-f13704d3fd90": { 2024-04-21T08:46:01.692 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:46:01.692 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "8da8ae19-5c2b-44fe-b517-f13704d3fd90", 2024-04-21T08:46:01.692 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:46:01.692 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:46:01.692 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:46:01.692 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 -- 172.21.15.94:0/3739112560 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57180a3d20 msgr2=0x7f57180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57180a3d20 0x7f57180a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f57240098d0 tx=0x7f5724032cf0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 -- 172.21.15.94:0/3739112560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f572c1010a0 msgr2=0x7f572c199690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f572c1010a0 0x7f572c199690 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f571c00b4a0 tx=0x7f571c00b8a0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 -- 172.21.15.94:0/3739112560 shutdown_connections 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57180a3d20 0x7f57180a61e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f572c105dd0 0x7f572c1a4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f572c101a40 0x7f572c199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 --2- 172.21.15.94:0/3739112560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f572c1010a0 0x7f572c199690 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 -- 172.21.15.94:0/3739112560 >> 172.21.15.94:0/3739112560 conn(0x7f572c077c40 msgr2=0x7f572c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 -- 172.21.15.94:0/3739112560 shutdown_connections 2024-04-21T08:46:01.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:01.684+0000 7f5731206640 1 -- 172.21.15.94:0/3739112560 wait complete. 2024-04-21T08:46:01.694 INFO:tasks.cephfs.filesystem:scrub status for tag:8da8ae19-5c2b-44fe-b517-f13704d3fd90 - {'path': '/', 'tag': '8da8ae19-5c2b-44fe-b517-f13704d3fd90', 'options': 'recursive,force'} 2024-04-21T08:46:02.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:01 smithi094 ceph-mon[27988]: pgmap v184: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:46:02.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:01 smithi111 ceph-mon[31493]: pgmap v184: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:46:02.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:01 smithi173 ceph-mon[31428]: pgmap v184: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:46:03.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:02 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:46:03.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:02 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:03.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:02 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:46:03.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:02 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:03.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:02 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:46:03.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:02 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:03.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:02 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:46:03.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:02 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:03.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:02 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:46:03.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:02 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:03.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:02 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:46:03.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:02 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:04.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:03 smithi111 ceph-mon[31493]: pgmap v185: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:04.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:03 smithi094 ceph-mon[27988]: pgmap v185: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:04.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:03 smithi173 ceph-mon[31428]: pgmap v185: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:06.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:05 smithi111 ceph-mon[31493]: pgmap v186: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-21T08:46:06.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:05 smithi094 ceph-mon[27988]: pgmap v186: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-21T08:46:06.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:05 smithi173 ceph-mon[31428]: pgmap v186: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-21T08:46:07.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:06 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:46:07.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:06 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:46:07.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:06 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:46:08.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:07 smithi111 ceph-mon[31493]: pgmap v187: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:46:08.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:07 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:08.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:07 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:08.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:07 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:08.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:07 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:46:08.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:07 smithi094 ceph-mon[27988]: pgmap v187: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:46:08.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:07 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:08.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:07 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:08.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:07 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:08.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:07 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:46:08.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:07 smithi173 ceph-mon[31428]: pgmap v187: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:46:08.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:07 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:08.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:07 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:08.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:07 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:08.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:07 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:46:10.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:09 smithi111 ceph-mon[31493]: pgmap v188: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:46:10.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:09 smithi094 ceph-mon[27988]: pgmap v188: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:46:10.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:09 smithi173 ceph-mon[31428]: pgmap v188: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:46:12.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:11 smithi111 ceph-mon[31493]: pgmap v189: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-21T08:46:12.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:11 smithi094 ceph-mon[27988]: pgmap v189: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-21T08:46:12.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:11 smithi173 ceph-mon[31428]: pgmap v189: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-21T08:46:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:13 smithi111 ceph-mon[31493]: pgmap v190: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:46:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:13 smithi094 ceph-mon[27988]: pgmap v190: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:46:14.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:13 smithi173 ceph-mon[31428]: pgmap v190: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-21T08:46:16.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:15 smithi111 ceph-mon[31493]: pgmap v191: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-21T08:46:16.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:15 smithi094 ceph-mon[27988]: pgmap v191: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-21T08:46:16.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:15 smithi173 ceph-mon[31428]: pgmap v191: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-21T08:46:17.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:16 smithi111 ceph-mon[31493]: pgmap v192: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:46:17.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:16 smithi094 ceph-mon[27988]: pgmap v192: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:46:17.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:16 smithi173 ceph-mon[31428]: pgmap v192: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-21T08:46:19.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:19 smithi111 ceph-mon[31493]: pgmap v193: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:46:19.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:19 smithi094 ceph-mon[27988]: pgmap v193: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:46:19.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:19 smithi173 ceph-mon[31428]: pgmap v193: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:46:21.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:21 smithi111 ceph-mon[31493]: pgmap v194: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:46:21.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:21 smithi094 ceph-mon[27988]: pgmap v194: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:46:21.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:21 smithi173 ceph-mon[31428]: pgmap v194: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:46:23.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:23 smithi111 ceph-mon[31493]: pgmap v195: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:23.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:23 smithi094 ceph-mon[27988]: pgmap v195: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:23.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:23 smithi173 ceph-mon[31428]: pgmap v195: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:25.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:25 smithi111 ceph-mon[31493]: pgmap v196: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:46:25.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:25 smithi094 ceph-mon[27988]: pgmap v196: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:46:25.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:25 smithi173 ceph-mon[31428]: pgmap v196: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:46:27.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:27 smithi111 ceph-mon[31493]: pgmap v197: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:46:27.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:27 smithi094 ceph-mon[27988]: pgmap v197: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:46:27.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:27 smithi173 ceph-mon[31428]: pgmap v197: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:46:29.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:29 smithi111 ceph-mon[31493]: pgmap v198: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:29.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:29 smithi094 ceph-mon[27988]: pgmap v198: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:29.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:29 smithi173 ceph-mon[31428]: pgmap v198: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:31.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:31 smithi111 ceph-mon[31493]: pgmap v199: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:46:31.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:31 smithi094 ceph-mon[27988]: pgmap v199: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:46:31.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:31 smithi173 ceph-mon[31428]: pgmap v199: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:46:31.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:31.696 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:31.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.814+0000 7fd472300640 1 -- 172.21.15.94:0/261730879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd46c1031c0 msgr2=0x7fd46c1035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.814+0000 7fd472300640 1 --2- 172.21.15.94:0/261730879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd46c1031c0 0x7fd46c1035a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd454009900 tx=0x7fd45402bd60 comp rx=0 tx=0).stop 2024-04-21T08:46:31.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.814+0000 7fd472300640 1 -- 172.21.15.94:0/261730879 shutdown_connections 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.814+0000 7fd472300640 1 --2- 172.21.15.94:0/261730879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd46c103ae0 0x7fd46c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.814+0000 7fd472300640 1 --2- 172.21.15.94:0/261730879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c10c500 0x7fd46c10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.814+0000 7fd472300640 1 --2- 172.21.15.94:0/261730879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd46c1031c0 0x7fd46c1035a0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.814+0000 7fd472300640 1 -- 172.21.15.94:0/261730879 >> 172.21.15.94:0/261730879 conn(0x7fd46c078000 msgr2=0x7fd46c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.814+0000 7fd472300640 1 -- 172.21.15.94:0/261730879 shutdown_connections 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.814+0000 7fd472300640 1 -- 172.21.15.94:0/261730879 wait complete. 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd472300640 1 Processor -- start 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd472300640 1 -- start start 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd472300640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd46c1031c0 0x7fd46c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd472300640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd46c103ae0 0x7fd46c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd472300640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c10c500 0x7fd46c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd472300640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c112190 con 0x7fd46c1031c0 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd472300640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c112010 con 0x7fd46c10c500 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd472300640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c112310 con 0x7fd46c103ae0 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd46bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd46c1031c0 0x7fd46c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd46bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd46c1031c0 0x7fd46c198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51790/0 (socket says 172.21.15.94:51790) 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd46bfff640 1 -- 172.21.15.94:0/3434107749 learned_addr learned my addr 172.21.15.94:0/3434107749 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd46c103ae0 0x7fd46c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd47089d640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c10c500 0x7fd46c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd46bfff640 1 -- 172.21.15.94:0/3434107749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd46c103ae0 msgr2=0x7fd46c198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd46bfff640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd46c103ae0 0x7fd46c198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd46bfff640 1 -- 172.21.15.94:0/3434107749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c10c500 msgr2=0x7fd46c19cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd46bfff640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c10c500 0x7fd46c19cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd46bfff640 1 -- 172.21.15.94:0/3434107749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd46c19d680 con 0x7fd46c1031c0 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.815+0000 7fd47089d640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c10c500 0x7fd46c19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.816+0000 7fd46bfff640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd46c1031c0 0x7fd46c198690 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd454005940 tx=0x7fd45402d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:31.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.816+0000 7fd4697fa640 1 -- 172.21.15.94:0/3434107749 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454043020 con 0x7fd46c1031c0 2024-04-21T08:46:31.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.816+0000 7fd4697fa640 1 -- 172.21.15.94:0/3434107749 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd45402dda0 con 0x7fd46c1031c0 2024-04-21T08:46:31.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.816+0000 7fd472300640 1 -- 172.21.15.94:0/3434107749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd46c19d910 con 0x7fd46c1031c0 2024-04-21T08:46:31.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.816+0000 7fd4697fa640 1 -- 172.21.15.94:0/3434107749 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454003bf0 con 0x7fd46c1031c0 2024-04-21T08:46:31.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.816+0000 7fd472300640 1 -- 172.21.15.94:0/3434107749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd46c1a5200 con 0x7fd46c1031c0 2024-04-21T08:46:31.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd4697fa640 1 -- 172.21.15.94:0/3434107749 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd454046020 con 0x7fd46c1031c0 2024-04-21T08:46:31.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd474fcd740 1 Processor -- start 2024-04-21T08:46:31.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd474fcd740 1 -- start start 2024-04-21T08:46:31.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d04f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d058540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d05ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd474fcd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d79cfb41e0 con 0x55d79d04f580 2024-04-21T08:46:31.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd474fcd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d79cf51390 con 0x55d79d058a80 2024-04-21T08:46:31.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd474fcd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d79cfb01a0 con 0x55d79d04fea0 2024-04-21T08:46:31.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd47089d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d04f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd47089d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d04f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51804/0 (socket says 172.21.15.94:51804) 2024-04-21T08:46:31.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd47089d640 1 -- 172.21.15.94:0/3725774841 learned_addr learned my addr 172.21.15.94:0/3725774841 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:31.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.819+0000 7fd4697fa640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd4580a3d20 0x7fd4580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd46bfff640 1 --2- 172.21.15.94:0/3725774841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d058540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/3725774841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d05ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd47089d640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd4580a3d20 0x7fd4580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd47089d640 1 -- 172.21.15.94:0/3725774841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 msgr2=0x55d79d058540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd47089d640 1 --2- 172.21.15.94:0/3725774841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d058540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd47089d640 1 -- 172.21.15.94:0/3725774841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 msgr2=0x55d79d05ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd47089d640 1 --2- 172.21.15.94:0/3725774841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d05ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd47089d640 1 -- 172.21.15.94:0/3725774841 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d79cd907d0 con 0x55d79d04f580 2024-04-21T08:46:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd4697fa640 1 -- 172.21.15.94:0/3434107749 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd4540f2f50 con 0x7fd46c1031c0 2024-04-21T08:46:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/3725774841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d05ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd47089d640 1 --2- 172.21.15.94:0/3725774841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d04f960 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fd4640149a0 tx=0x7fd464014e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca0cdbfcfc7882d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd4627fc640 1 -- 172.21.15.94:0/3725774841 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd464021070 con 0x55d79d04f580 2024-04-21T08:46:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd4627fc640 1 -- 172.21.15.94:0/3725774841 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd464013460 con 0x55d79d04f580 2024-04-21T08:46:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd4627fc640 1 -- 172.21.15.94:0/3725774841 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46400e690 con 0x55d79d04f580 2024-04-21T08:46:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.820+0000 7fd47089d640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd4580a3d20 0x7fd4580a61e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd46401e000 tx=0x7fd464019000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.821+0000 7fd474fcd740 1 -- 172.21.15.94:0/3725774841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 msgr2=0x55d79d04f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.821+0000 7fd474fcd740 1 --2- 172.21.15.94:0/3725774841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d04f960 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fd4640149a0 tx=0x7fd464014e60 comp rx=0 tx=0).stop 2024-04-21T08:46:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.821+0000 7fd474fcd740 1 -- 172.21.15.94:0/3725774841 shutdown_connections 2024-04-21T08:46:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.821+0000 7fd474fcd740 1 --2- 172.21.15.94:0/3725774841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d058540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.821+0000 7fd474fcd740 1 --2- 172.21.15.94:0/3725774841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d05ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.821+0000 7fd474fcd740 1 --2- 172.21.15.94:0/3725774841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d04f960 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.821+0000 7fd474fcd740 1 -- 172.21.15.94:0/3725774841 >> 172.21.15.94:0/3725774841 conn(0x55d79d049cb0 msgr2=0x55d79d04a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.821+0000 7fd474fcd740 1 -- 172.21.15.94:0/3725774841 shutdown_connections 2024-04-21T08:46:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.821+0000 7fd474fcd740 1 -- 172.21.15.94:0/3725774841 wait complete. 2024-04-21T08:46:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.821+0000 7fd474fcd740 1 -- start start 2024-04-21T08:46:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd474fcd740 1 Processor -- start 2024-04-21T08:46:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d055a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d055fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d0564e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd474fcd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d79d05b870 con 0x55d79d04f580 2024-04-21T08:46:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd474fcd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d79cf6d2f0 con 0x55d79d058a80 2024-04-21T08:46:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd474fcd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d79d05b9f0 con 0x55d79d04fea0 2024-04-21T08:46:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd474fcd740 10 client.0 _ll_register_callbacks cb 0x55d79cf7a580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd46bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d055a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd46bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d055a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51812/0 (socket says 172.21.15.94:51812) 2024-04-21T08:46:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd46bfff640 1 -- 172.21.15.94:0/3285571319 learned_addr learned my addr 172.21.15.94:0/3285571319 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd47089d640 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d0564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d055fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd46bfff640 1 -- 172.21.15.94:0/3285571319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 msgr2=0x55d79d055fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd46bfff640 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d055fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd46bfff640 1 -- 172.21.15.94:0/3285571319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 msgr2=0x55d79d0564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd46bfff640 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d0564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd46bfff640 1 -- 172.21.15.94:0/3285571319 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d79ccc63f0 con 0x55d79d04f580 2024-04-21T08:46:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd46bfff640 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d055a60 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fd4540f0430 tx=0x7fd45402bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54deecba9bbe013 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.822+0000 7fd44b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd44b7fe640 1 -- 172.21.15.94:0/3285571319 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4540c2fd0 con 0x55d79d04f580 2024-04-21T08:46:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd44b7fe640 1 -- 172.21.15.94:0/3285571319 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd4540f5110 con 0x55d79d04f580 2024-04-21T08:46:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd474fcd740 1 -- 172.21.15.94:0/3285571319 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd464013460 con 0x55d79d04f580 2024-04-21T08:46:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd44b7fe640 1 -- 172.21.15.94:0/3285571319 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd4540c7890 con 0x55d79d04f580 2024-04-21T08:46:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd474fcd740 10 client.17355 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd474fcd740 1 -- 172.21.15.94:0/3285571319 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d79cc860f0 con 0x55d79d04f580 2024-04-21T08:46:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd44b7fe640 1 -- 172.21.15.94:0/3285571319 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd464013460 con 0x55d79d04f580 2024-04-21T08:46:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd44b7fe640 1 -- 172.21.15.94:0/3285571319 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd4540df0d0 con 0x55d79d04f580 2024-04-21T08:46:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd474fcd740 10 client.17355 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd474fcd740 10 client.17355 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd474fcd740 20 client.17355 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd474fcd740 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d79d068980 0x55d79d06ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd474fcd740 4 client.17355 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd474fcd740 1 -- 172.21.15.94:0/3285571319 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d79cfc1c00 con 0x55d79d068980 2024-04-21T08:46:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.823+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d79d068980 0x55d79d06ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.824+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d79d068980 0x55d79d06ade0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4131726125d3fc82 server_cookie=fb513b31b6c98661 in_seq=0 out_seq=0 2024-04-21T08:46:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.824+0000 7fd44b7fe640 10 client.17355 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.824+0000 7fd44b7fe640 1 -- 172.21.15.94:0/3285571319 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d79cfc1c00 con 0x55d79d068980 2024-04-21T08:46:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd44b7fe640 10 client.17355 handle_command_reply: tid=0 2024-04-21T08:46:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 client.17355 shutdown 2024-04-21T08:46:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 -- 172.21.15.94:0/3285571319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 msgr2=0x55d79d055a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d055a60 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fd4540f0430 tx=0x7fd45402bd60 comp rx=0 tx=0).stop 2024-04-21T08:46:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 -- 172.21.15.94:0/3285571319 shutdown_connections 2024-04-21T08:46:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d79d068980 0x55d79d06ade0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d055fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd44b7fe640 0 client.17355 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d0564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 --2- 172.21.15.94:0/3285571319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d055a60 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 -- 172.21.15.94:0/3285571319 >> 172.21.15.94:0/3285571319 conn(0x55d79d049cb0 msgr2=0x55d79d04ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 -- 172.21.15.94:0/3285571319 shutdown_connections 2024-04-21T08:46:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 1 -- 172.21.15.94:0/3285571319 wait complete. 2024-04-21T08:46:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.825+0000 7fd474fcd740 20 client.17355 trim_cache size 0 max 16384 2024-04-21T08:46:31.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.842+0000 7fd474fcd740 1 Processor -- start 2024-04-21T08:46:31.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.842+0000 7fd474fcd740 1 -- start start 2024-04-21T08:46:31.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d058190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d064760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d048bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd474fcd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c1b1c40 con 0x55d79d04f580 2024-04-21T08:46:31.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd474fcd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c1af510 con 0x55d79d058a80 2024-04-21T08:46:31.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd474fcd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c1ad360 con 0x55d79d04fea0 2024-04-21T08:46:31.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd46bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d058190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd46bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d058190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51814/0 (socket says 172.21.15.94:51814) 2024-04-21T08:46:31.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd46bfff640 1 -- 172.21.15.94:0/1714804440 learned_addr learned my addr 172.21.15.94:0/1714804440 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:31.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd46bfff640 1 -- 172.21.15.94:0/1714804440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 msgr2=0x55d79d064760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:31.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd46bfff640 1 --2- 172.21.15.94:0/1714804440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d064760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd46bfff640 1 -- 172.21.15.94:0/1714804440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 msgr2=0x55d79d048bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:31.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd46bfff640 1 --2- 172.21.15.94:0/1714804440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d048bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd46bfff640 1 -- 172.21.15.94:0/1714804440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d79d049ae0 con 0x55d79d04f580 2024-04-21T08:46:31.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd46bfff640 1 --2- 172.21.15.94:0/1714804440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d058190 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd4540be920 tx=0x7fd4540beda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58de429b6c577d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:31.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd461ffb640 1 -- 172.21.15.94:0/1714804440 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454087070 con 0x55d79d04f580 2024-04-21T08:46:31.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd461ffb640 1 -- 172.21.15.94:0/1714804440 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4540bcf20 con 0x55d79d04f580 2024-04-21T08:46:31.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.843+0000 7fd461ffb640 1 -- 172.21.15.94:0/1714804440 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4540bef50 con 0x55d79d04f580 2024-04-21T08:46:31.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 -- 172.21.15.94:0/1714804440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 msgr2=0x55d79d058190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 --2- 172.21.15.94:0/1714804440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d058190 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd4540be920 tx=0x7fd4540beda0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 -- 172.21.15.94:0/1714804440 shutdown_connections 2024-04-21T08:46:31.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 --2- 172.21.15.94:0/1714804440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04fea0 0x55d79d064760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 --2- 172.21.15.94:0/1714804440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d048bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 --2- 172.21.15.94:0/1714804440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04f580 0x55d79d058190 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 -- 172.21.15.94:0/1714804440 >> 172.21.15.94:0/1714804440 conn(0x55d79d049cb0 msgr2=0x55d79d0484e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 -- 172.21.15.94:0/1714804440 shutdown_connections 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 -- 172.21.15.94:0/1714804440 wait complete. 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 -- start start 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.844+0000 7fd474fcd740 1 Processor -- start 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04f580 0x55d79d052800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04fea0 0x55d79d052d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd474fcd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d053280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd474fcd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c1ad9d0 con 0x55d79d04fea0 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd474fcd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c1aee40 con 0x55d79d058a80 2024-04-21T08:46:31.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd474fcd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c1ae0a0 con 0x55d79d04f580 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd474fcd740 10 client.0 _ll_register_callbacks cb 0x7fd4540057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd46b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04fea0 0x55d79d052d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd46b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04fea0 0x55d79d052d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51828/0 (socket says 172.21.15.94:51828) 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd46b7fe640 1 -- 172.21.15.94:0/1012648557 learned_addr learned my addr 172.21.15.94:0/1012648557 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd46b7fe640 1 -- 172.21.15.94:0/1012648557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04f580 msgr2=0x55d79d052800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04f580 0x55d79d052800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd46b7fe640 1 -- 172.21.15.94:0/1012648557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 msgr2=0x55d79d053280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d053280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd46b7fe640 1 -- 172.21.15.94:0/1012648557 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d79d0ce020 con 0x55d79d04fea0 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04fea0 0x55d79d052d40 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x55d79d04ae40 tx=0x7fd45c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa1fa1c15cf3eb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd44b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd44b7fe640 1 -- 172.21.15.94:0/1012648557 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c01e070 con 0x55d79d04fea0 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd44b7fe640 1 -- 172.21.15.94:0/1012648557 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd45c004020 con 0x55d79d04fea0 2024-04-21T08:46:31.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.845+0000 7fd474fcd740 1 -- 172.21.15.94:0/1012648557 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4540bcf20 con 0x55d79d04fea0 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd44b7fe640 1 -- 172.21.15.94:0/1012648557 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd45c029050 con 0x55d79d04fea0 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd474fcd740 10 client.17361 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd474fcd740 1 -- 172.21.15.94:0/1012648557 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d79d0cf360 con 0x55d79d04fea0 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd44b7fe640 1 -- 172.21.15.94:0/1012648557 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4540bcf20 con 0x55d79d04fea0 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd44b7fe640 1 -- 172.21.15.94:0/1012648557 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd45c003820 con 0x55d79d04fea0 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd474fcd740 10 client.17361 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd474fcd740 10 client.17361 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd474fcd740 20 client.17361 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd474fcd740 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d79d0cfb10 0x55d79d0d1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd474fcd740 4 client.17361 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd474fcd740 1 -- 172.21.15.94:0/1012648557 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d79cfbba70 con 0x55d79d0cfb10 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.846+0000 7fd46bfff640 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d79d0cfb10 0x55d79d0d1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd46bfff640 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d79d0cfb10 0x55d79d0d1f30 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7276ccb5e150282 server_cookie=7e3689f4e45e0987 in_seq=0 out_seq=0 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd44b7fe640 10 client.17361 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:31.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd44b7fe640 1 -- 172.21.15.94:0/1012648557 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d79cfbba70 con 0x55d79d0cfb10 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd44b7fe640 10 client.17361 handle_command_reply: tid=0 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 client.17361 shutdown 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 -- 172.21.15.94:0/1012648557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04fea0 msgr2=0x55d79d052d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04fea0 0x55d79d052d40 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x55d79d04ae40 tx=0x7fd45c011b00 comp rx=0 tx=0).stop 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 -- 172.21.15.94:0/1012648557 shutdown_connections 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d79d0cfb10 0x55d79d0d1f30 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d79d058a80 0x55d79d053280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d79d04fea0 0x55d79d052d40 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd44b7fe640 0 client.17361 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 --2- 172.21.15.94:0/1012648557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d79d04f580 0x55d79d052800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 -- 172.21.15.94:0/1012648557 >> 172.21.15.94:0/1012648557 conn(0x55d79d049cb0 msgr2=0x55d79d04b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 -- 172.21.15.94:0/1012648557 shutdown_connections 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 1 -- 172.21.15.94:0/1012648557 wait complete. 2024-04-21T08:46:31.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.847+0000 7fd474fcd740 20 client.17361 trim_cache size 0 max 16384 2024-04-21T08:46:31.861 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:46:31.861 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:46:31.861 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:46:31.861 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:46:31.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 -- 172.21.15.94:0/3434107749 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd4580a3d20 msgr2=0x7fd4580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd4580a3d20 0x7fd4580a61e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd46401e000 tx=0x7fd464019000 comp rx=0 tx=0).stop 2024-04-21T08:46:31.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 -- 172.21.15.94:0/3434107749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd46c1031c0 msgr2=0x7fd46c198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd46c1031c0 0x7fd46c198690 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd454005940 tx=0x7fd45402d9a0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 -- 172.21.15.94:0/3434107749 shutdown_connections 2024-04-21T08:46:31.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd4580a3d20 0x7fd4580a61e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd46c103ae0 0x7fd46c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c10c500 0x7fd46c19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 --2- 172.21.15.94:0/3434107749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd46c1031c0 0x7fd46c198690 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 -- 172.21.15.94:0/3434107749 >> 172.21.15.94:0/3434107749 conn(0x7fd46c078000 msgr2=0x7fd46c0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:31.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 -- 172.21.15.94:0/3434107749 shutdown_connections 2024-04-21T08:46:31.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.848+0000 7fd472300640 1 -- 172.21.15.94:0/3434107749 wait complete. 2024-04-21T08:46:31.863 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:46:31.864 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:46:31.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.963+0000 7f8c24621640 1 -- 172.21.15.94:0/2812645597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c1c107270 msgr2=0x7f8c1c10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.963+0000 7f8c24621640 1 --2- 172.21.15.94:0/2812645597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c1c107270 0x7f8c1c10ee40 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009ca0 tx=0x7f8c1402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.963+0000 7f8c24621640 1 -- 172.21.15.94:0/2812645597 shutdown_connections 2024-04-21T08:46:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.963+0000 7f8c24621640 1 --2- 172.21.15.94:0/2812645597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c1c102b50 0x7f8c1c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.963+0000 7f8c24621640 1 --2- 172.21.15.94:0/2812645597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c1c107270 0x7f8c1c10ee40 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.963+0000 7f8c24621640 1 --2- 172.21.15.94:0/2812645597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1c1021a0 0x7f8c1c102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.963+0000 7f8c24621640 1 -- 172.21.15.94:0/2812645597 >> 172.21.15.94:0/2812645597 conn(0x7f8c1c0793a0 msgr2=0x7f8c1c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.963+0000 7f8c24621640 1 -- 172.21.15.94:0/2812645597 shutdown_connections 2024-04-21T08:46:31.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.963+0000 7f8c24621640 1 -- 172.21.15.94:0/2812645597 wait complete. 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c24621640 1 Processor -- start 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c24621640 1 -- start start 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c24621640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1c1021a0 0x7f8c1c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c24621640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c1c107270 0x7f8c1c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c24621640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c1c19cf30 0x7f8c1c19d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c24621640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1c1121c0 con 0x7f8c1c107270 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c24621640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1c112040 con 0x7f8c1c1021a0 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c24621640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1c112340 con 0x7f8c1c19cf30 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c21bbc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c1c107270 0x7f8c1c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c21bbc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c1c107270 0x7f8c1c198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51850/0 (socket says 172.21.15.94:51850) 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c21bbc640 1 -- 172.21.15.94:0/2212801382 learned_addr learned my addr 172.21.15.94:0/2212801382 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c223bd640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1c1021a0 0x7f8c1c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c22bbe640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c1c19cf30 0x7f8c1c19d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c21bbc640 1 -- 172.21.15.94:0/2212801382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c1c19cf30 msgr2=0x7f8c1c19d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c21bbc640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c1c19cf30 0x7f8c1c19d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c21bbc640 1 -- 172.21.15.94:0/2212801382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1c1021a0 msgr2=0x7f8c1c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c21bbc640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1c1021a0 0x7f8c1c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c21bbc640 1 -- 172.21.15.94:0/2212801382 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1c19da20 con 0x7f8c1c107270 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c223bd640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1c1021a0 0x7f8c1c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c22bbe640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c1c19cf30 0x7f8c1c19d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.964+0000 7f8c21bbc640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c1c107270 0x7f8c1c198c00 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800e9c0 tx=0x7f8c1800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.965+0000 7f8c137fe640 1 -- 172.21.15.94:0/2212801382 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800cd80 con 0x7f8c1c107270 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.965+0000 7f8c137fe640 1 -- 172.21.15.94:0/2212801382 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c180054d0 con 0x7f8c1c107270 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.965+0000 7f8c24621640 1 -- 172.21.15.94:0/2212801382 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c1c19dd10 con 0x7f8c1c107270 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.965+0000 7f8c137fe640 1 -- 172.21.15.94:0/2212801382 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c18010430 con 0x7f8c1c107270 2024-04-21T08:46:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.965+0000 7f8c24621640 1 -- 172.21.15.94:0/2212801382 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c1c1a5550 con 0x7f8c1c107270 2024-04-21T08:46:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.966+0000 7f8c24621640 1 -- 172.21.15.94:0/2212801382 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c1c069230 con 0x7f8c1c107270 2024-04-21T08:46:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.967+0000 7f8c137fe640 1 -- 172.21.15.94:0/2212801382 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8c18004fb0 con 0x7f8c1c107270 2024-04-21T08:46:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.967+0000 7f8c137fe640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8c080a3da0 0x7f8c080a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:31.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.967+0000 7f8c223bd640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8c080a3da0 0x7f8c080a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:31.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.967+0000 7f8c137fe640 1 -- 172.21.15.94:0/2212801382 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f8c180d1240 con 0x7f8c1c107270 2024-04-21T08:46:31.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.968+0000 7f8c223bd640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8c080a3da0 0x7f8c080a6260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c009b40 tx=0x7f8c0c005e50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:31.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:31.971+0000 7f8c137fe640 1 -- 172.21.15.94:0/2212801382 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c18053ff0 con 0x7f8c1c107270 2024-04-21T08:46:32.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.190+0000 7f8c24621640 1 -- 172.21.15.94:0/2212801382 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c1c1999c0 con 0x7f8c1c107270 2024-04-21T08:46:32.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.191+0000 7f8c137fe640 1 -- 172.21.15.94:0/2212801382 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f8c1c1999c0 con 0x7f8c1c107270 2024-04-21T08:46:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:46:32.191 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:46:32.191 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:46:32.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 -- 172.21.15.94:0/2212801382 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8c080a3da0 msgr2=0x7f8c080a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8c080a3da0 0x7f8c080a6260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c009b40 tx=0x7f8c0c005e50 comp rx=0 tx=0).stop 2024-04-21T08:46:32.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 -- 172.21.15.94:0/2212801382 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c1c107270 msgr2=0x7f8c1c198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c1c107270 0x7f8c1c198c00 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800e9c0 tx=0x7f8c1800ee80 comp rx=0 tx=0).stop 2024-04-21T08:46:32.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 -- 172.21.15.94:0/2212801382 shutdown_connections 2024-04-21T08:46:32.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8c080a3da0 0x7f8c080a6260 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c1c19cf30 0x7f8c1c19d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c1c107270 0x7f8c1c198c00 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 --2- 172.21.15.94:0/2212801382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1c1021a0 0x7f8c1c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 -- 172.21.15.94:0/2212801382 >> 172.21.15.94:0/2212801382 conn(0x7f8c1c0793a0 msgr2=0x7f8c1c0ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 -- 172.21.15.94:0/2212801382 shutdown_connections 2024-04-21T08:46:32.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.193+0000 7f8c24621640 1 -- 172.21.15.94:0/2212801382 wait complete. 2024-04-21T08:46:32.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:46:32.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.305+0000 7fadcdef2640 1 -- 172.21.15.94:0/4236587186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc806b1f0 msgr2=0x7fadc8106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.305+0000 7fadcdef2640 1 --2- 172.21.15.94:0/4236587186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc806b1f0 0x7fadc8106b90 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009900 tx=0x7fadbc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:32.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.305+0000 7fadcdef2640 1 -- 172.21.15.94:0/4236587186 shutdown_connections 2024-04-21T08:46:32.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.305+0000 7fadcdef2640 1 --2- 172.21.15.94:0/4236587186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadc8107270 0x7fadc810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.305+0000 7fadcdef2640 1 --2- 172.21.15.94:0/4236587186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc806b1f0 0x7fadc8106b90 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.305+0000 7fadcdef2640 1 --2- 172.21.15.94:0/4236587186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadc806a840 0x7fadc806ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.305+0000 7fadcdef2640 1 -- 172.21.15.94:0/4236587186 >> 172.21.15.94:0/4236587186 conn(0x7fadc8077eb0 msgr2=0x7fadc80782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.305+0000 7fadcdef2640 1 -- 172.21.15.94:0/4236587186 shutdown_connections 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.305+0000 7fadcdef2640 1 -- 172.21.15.94:0/4236587186 wait complete. 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.305+0000 7fadcdef2640 1 Processor -- start 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadcdef2640 1 -- start start 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadcdef2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadc806a840 0x7fadc819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadcdef2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadc8107270 0x7fadc819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadcdef2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc8198600 0x7fadc8198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadcdef2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8112280 con 0x7fadc8198600 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadcdef2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8112100 con 0x7fadc806a840 2024-04-21T08:46:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadcdef2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8112400 con 0x7fadc8107270 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc8198600 0x7fadc8198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc8198600 0x7fadc8198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51864/0 (socket says 172.21.15.94:51864) 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc7fff640 1 -- 172.21.15.94:0/3521951762 learned_addr learned my addr 172.21.15.94:0/3521951762 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadc8107270 0x7fadc819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc77fe640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadc806a840 0x7fadc819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc7fff640 1 -- 172.21.15.94:0/3521951762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadc8107270 msgr2=0x7fadc819e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc7fff640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadc8107270 0x7fadc819e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc7fff640 1 -- 172.21.15.94:0/3521951762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadc806a840 msgr2=0x7fadc819e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc7fff640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadc806a840 0x7fadc819e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc7fff640 1 -- 172.21.15.94:0/3521951762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc8199120 con 0x7fadc8198600 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.306+0000 7fadc77fe640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadc806a840 0x7fadc819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.307+0000 7fadc7fff640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc8198600 0x7fadc8198ab0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fadc000b970 tx=0x7fadc000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.307+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3521951762 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0019070 con 0x7fadc8198600 2024-04-21T08:46:32.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.307+0000 7fadcdef2640 1 -- 172.21.15.94:0/3521951762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc8199410 con 0x7fadc8198600 2024-04-21T08:46:32.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.307+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3521951762 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadc0014440 con 0x7fadc8198600 2024-04-21T08:46:32.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.307+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3521951762 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc00075d0 con 0x7fadc8198600 2024-04-21T08:46:32.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.307+0000 7fadcdef2640 1 -- 172.21.15.94:0/3521951762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc81a55c0 con 0x7fadc8198600 2024-04-21T08:46:32.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadd0bc3740 1 Processor -- start 2024-04-21T08:46:32.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3521951762 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fadc001c070 con 0x7fadc8198600 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadd0bc3740 1 -- start start 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efb4d0 0x563751efb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efbe80 0x563751f04430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751f04a00 0x563751f06e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563751e601f0 con 0x563751efbe80 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563751dfd4b0 con 0x563751efb4d0 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563751e5c1b0 con 0x563751f04a00 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efbe80 0x563751f04430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efbe80 0x563751f04430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51880/0 (socket says 172.21.15.94:51880) 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc77fe640 1 -- 172.21.15.94:0/2330191468 learned_addr learned my addr 172.21.15.94:0/2330191468 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc4ff9640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadb00a3d20 0x7fadb00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc77fe640 1 -- 172.21.15.94:0/2330191468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751f04a00 msgr2=0x563751f06e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc77fe640 1 --2- 172.21.15.94:0/2330191468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751f04a00 0x563751f06e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc77fe640 1 -- 172.21.15.94:0/2330191468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efb4d0 msgr2=0x563751efb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc77fe640 1 --2- 172.21.15.94:0/2330191468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efb4d0 0x563751efb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc77fe640 1 -- 172.21.15.94:0/2330191468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563751def1b0 con 0x563751efbe80 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc7fff640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadb00a3d20 0x7fadb00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc77fe640 1 --2- 172.21.15.94:0/2330191468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efbe80 0x563751f04430 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fadb800bb70 tx=0x7fadb800d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01a7f1af6b7573a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.310+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3521951762 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fadc00d0060 con 0x7fadc8198600 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadadffb640 1 -- 172.21.15.94:0/2330191468 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8017620 con 0x563751efbe80 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadadffb640 1 -- 172.21.15.94:0/2330191468 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadb8017c70 con 0x563751efbe80 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadadffb640 1 -- 172.21.15.94:0/2330191468 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb800fb90 con 0x563751efbe80 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadc7fff640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadb00a3d20 0x7fadb00a61e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fadc00cd590 tx=0x7fadc007a040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadd0bc3740 1 -- 172.21.15.94:0/2330191468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efbe80 msgr2=0x563751f04430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/2330191468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efbe80 0x563751f04430 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fadb800bb70 tx=0x7fadb800d710 comp rx=0 tx=0).stop 2024-04-21T08:46:32.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadd0bc3740 1 -- 172.21.15.94:0/2330191468 shutdown_connections 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/2330191468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751f04a00 0x563751f06e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/2330191468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efbe80 0x563751f04430 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/2330191468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efb4d0 0x563751efb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadd0bc3740 1 -- 172.21.15.94:0/2330191468 >> 172.21.15.94:0/2330191468 conn(0x563751ef5ba0 msgr2=0x563751ef5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadd0bc3740 1 -- 172.21.15.94:0/2330191468 shutdown_connections 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.311+0000 7fadd0bc3740 1 -- 172.21.15.94:0/2330191468 wait complete. 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadd0bc3740 1 -- start start 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadd0bc3740 1 Processor -- start 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f04320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751efbe80 0x563751f016b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751f04a00 0x563751f01c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563751f07970 con 0x563751efb4d0 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563751e19500 con 0x563751f04a00 2024-04-21T08:46:32.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563751f07af0 con 0x563751efbe80 2024-04-21T08:46:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadd0bc3740 10 client.0 _ll_register_callbacks cb 0x563751de7b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadc6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f04320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadc6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f04320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51882/0 (socket says 172.21.15.94:51882) 2024-04-21T08:46:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadc6ffd640 1 -- 172.21.15.94:0/855137087 learned_addr learned my addr 172.21.15.94:0/855137087 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751f04a00 0x563751f01c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadc77fe640 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751efbe80 0x563751f016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadc6ffd640 1 -- 172.21.15.94:0/855137087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751efbe80 msgr2=0x563751f016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751efbe80 0x563751f016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadc6ffd640 1 -- 172.21.15.94:0/855137087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751f04a00 msgr2=0x563751f01c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751f04a00 0x563751f01c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.312+0000 7fadc6ffd640 1 -- 172.21.15.94:0/855137087 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563751b723f0 con 0x563751efb4d0 2024-04-21T08:46:32.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f04320 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fadbc005ec0 tx=0x7fadbc005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0c85a80e07b1e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fada6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fada6ffd640 1 -- 172.21.15.94:0/855137087 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc047020 con 0x563751efb4d0 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fada6ffd640 1 -- 172.21.15.94:0/855137087 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fadbc005250 con 0x563751efb4d0 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadd0bc3740 1 -- 172.21.15.94:0/855137087 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadb8017c70 con 0x563751efb4d0 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fada6ffd640 1 -- 172.21.15.94:0/855137087 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fadbc052050 con 0x563751efb4d0 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadd0bc3740 10 client.17379 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadd0bc3740 1 -- 172.21.15.94:0/855137087 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563751c3c7d0 con 0x563751efb4d0 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fada6ffd640 1 -- 172.21.15.94:0/855137087 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadb8017c70 con 0x563751efb4d0 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fada6ffd640 1 -- 172.21.15.94:0/855137087 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fadbc040040 con 0x563751efb4d0 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadd0bc3740 10 client.17379 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadd0bc3740 10 client.17379 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadd0bc3740 20 client.17379 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563751f14c10 0x563751f17020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadd0bc3740 4 client.17379 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:32.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadd0bc3740 1 -- 172.21.15.94:0/855137087 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563751dff930 con 0x563751f14c10 2024-04-21T08:46:32.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.313+0000 7fadc77fe640 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563751f14c10 0x563751f17020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.314+0000 7fada6ffd640 10 client.17379 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:46:32.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.314+0000 7fadc77fe640 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563751f14c10 0x563751f17020 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=55789fe6eeb2cc0b server_cookie=249d94d238b5238f in_seq=0 out_seq=0 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fada6ffd640 1 -- 172.21.15.94:0/855137087 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563751dff930 con 0x563751f14c10 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fada6ffd640 10 client.17379 handle_command_reply: tid=0 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 client.17379 shutdown 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 -- 172.21.15.94:0/855137087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 msgr2=0x563751f04320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f04320 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fadbc005ec0 tx=0x7fadbc005b50 comp rx=0 tx=0).stop 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 -- 172.21.15.94:0/855137087 shutdown_connections 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563751f14c10 0x563751f17020 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751efbe80 0x563751f016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fada6ffd640 0 client.17379 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751f04a00 0x563751f01c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/855137087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f04320 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 -- 172.21.15.94:0/855137087 >> 172.21.15.94:0/855137087 conn(0x563751ef5ba0 msgr2=0x563751ef78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 -- 172.21.15.94:0/855137087 shutdown_connections 2024-04-21T08:46:32.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 1 -- 172.21.15.94:0/855137087 wait complete. 2024-04-21T08:46:32.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.315+0000 7fadd0bc3740 20 client.17379 trim_cache size 0 max 16384 2024-04-21T08:46:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadd0bc3740 1 Processor -- start 2024-04-21T08:46:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadd0bc3740 1 -- start start 2024-04-21T08:46:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f10830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efbe80 0x563751f00810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751f04a00 0x563751ef7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fadc81b1c80 con 0x563751efb4d0 2024-04-21T08:46:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fadc81af550 con 0x563751efbe80 2024-04-21T08:46:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fadc81ad3a0 con 0x563751f04a00 2024-04-21T08:46:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadc6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f10830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadc6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f10830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51892/0 (socket says 172.21.15.94:51892) 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadc6ffd640 1 -- 172.21.15.94:0/1357953807 learned_addr learned my addr 172.21.15.94:0/1357953807 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadc6ffd640 1 -- 172.21.15.94:0/1357953807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751f04a00 msgr2=0x563751ef7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/1357953807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751f04a00 0x563751ef7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadc6ffd640 1 -- 172.21.15.94:0/1357953807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efbe80 msgr2=0x563751f00810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/1357953807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efbe80 0x563751f00810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.328+0000 7fadc6ffd640 1 -- 172.21.15.94:0/1357953807 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563751f01fc0 con 0x563751efb4d0 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/1357953807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f10830 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009900 tx=0x7fadbc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1ae647ed9e2aca server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadad7fa640 1 -- 172.21.15.94:0/1357953807 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc047040 con 0x563751efb4d0 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadad7fa640 1 -- 172.21.15.94:0/1357953807 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadbc005700 con 0x563751efb4d0 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadad7fa640 1 -- 172.21.15.94:0/1357953807 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc004920 con 0x563751efb4d0 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadd0bc3740 1 -- 172.21.15.94:0/1357953807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 msgr2=0x563751f10830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/1357953807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f10830 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009900 tx=0x7fadbc005fd0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadd0bc3740 1 -- 172.21.15.94:0/1357953807 shutdown_connections 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/1357953807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751f04a00 0x563751ef7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/1357953807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efbe80 0x563751f00810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/1357953807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751efb4d0 0x563751f10830 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadd0bc3740 1 -- 172.21.15.94:0/1357953807 >> 172.21.15.94:0/1357953807 conn(0x563751ef5ba0 msgr2=0x563751ef6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadd0bc3740 1 -- 172.21.15.94:0/1357953807 shutdown_connections 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.329+0000 7fadd0bc3740 1 -- 172.21.15.94:0/1357953807 wait complete. 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadd0bc3740 1 -- start start 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadd0bc3740 1 Processor -- start 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751efb4d0 0x563751f0cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efbe80 0x563751f0d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadd0bc3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751f04a00 0x563751efdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fadc81ada10 con 0x563751f04a00 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fadc81aee80 con 0x563751efbe80 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadd0bc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fadc81ae0e0 con 0x563751efb4d0 2024-04-21T08:46:32.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadd0bc3740 10 client.0 _ll_register_callbacks cb 0x7fadbc02c820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadc6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751f04a00 0x563751efdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadc6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751f04a00 0x563751efdd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51902/0 (socket says 172.21.15.94:51902) 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadc6ffd640 1 -- 172.21.15.94:0/709904495 learned_addr learned my addr 172.21.15.94:0/709904495 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751efb4d0 0x563751f0cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadc77fe640 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efbe80 0x563751f0d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadc6ffd640 1 -- 172.21.15.94:0/709904495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751efb4d0 msgr2=0x563751f0cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751efb4d0 0x563751f0cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadc6ffd640 1 -- 172.21.15.94:0/709904495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efbe80 msgr2=0x563751f0d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efbe80 0x563751f0d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.330+0000 7fadc6ffd640 1 -- 172.21.15.94:0/709904495 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563751efe580 con 0x563751f04a00 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadc77fe640 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efbe80 0x563751f0d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadc6ffd640 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751f04a00 0x563751efdd50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fadbc004230 tx=0x7fadbc003ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c191770663f7d0c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fada6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fada6ffd640 1 -- 172.21.15.94:0/709904495 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc047070 con 0x563751f04a00 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadd0bc3740 1 -- 172.21.15.94:0/709904495 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadbc005700 con 0x563751f04a00 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fada6ffd640 1 -- 172.21.15.94:0/709904495 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fadbc0046c0 con 0x563751f04a00 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fada6ffd640 1 -- 172.21.15.94:0/709904495 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fadbc003f50 con 0x563751f04a00 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadd0bc3740 10 client.17385 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadd0bc3740 1 -- 172.21.15.94:0/709904495 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563751f7ac50 con 0x563751f04a00 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fada6ffd640 1 -- 172.21.15.94:0/709904495 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadbc005700 con 0x563751f04a00 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fada6ffd640 1 -- 172.21.15.94:0/709904495 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fadbc063860 con 0x563751f04a00 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadd0bc3740 10 client.17385 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadd0bc3740 10 client.17385 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadd0bc3740 20 client.17385 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563751f14c10 0x563751f7d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadd0bc3740 4 client.17385 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.331+0000 7fadd0bc3740 1 -- 172.21.15.94:0/709904495 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563751e6dc10 con 0x563751f14c10 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadc77fe640 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563751f14c10 0x563751f7d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadc77fe640 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563751f14c10 0x563751f7d2e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ed4364ca122918a server_cookie=810a535d95a6be3 in_seq=0 out_seq=0 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fada6ffd640 10 client.17385 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fada6ffd640 1 -- 172.21.15.94:0/709904495 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563751e6dc10 con 0x563751f14c10 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fada6ffd640 10 client.17385 handle_command_reply: tid=0 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadd0bc3740 1 client.17385 shutdown 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadd0bc3740 1 -- 172.21.15.94:0/709904495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751f04a00 msgr2=0x563751efdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751f04a00 0x563751efdd50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fadbc004230 tx=0x7fadbc003ab0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadd0bc3740 1 -- 172.21.15.94:0/709904495 shutdown_connections 2024-04-21T08:46:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563751f14c10 0x563751f7d2e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563751f04a00 0x563751efdd50 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fada6ffd640 0 client.17385 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:46:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563751efbe80 0x563751f0d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadd0bc3740 1 --2- 172.21.15.94:0/709904495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563751efb4d0 0x563751f0cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadd0bc3740 1 -- 172.21.15.94:0/709904495 >> 172.21.15.94:0/709904495 conn(0x563751ef5ba0 msgr2=0x563751ef6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.332+0000 7fadd0bc3740 1 -- 172.21.15.94:0/709904495 shutdown_connections 2024-04-21T08:46:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadd0bc3740 1 -- 172.21.15.94:0/709904495 wait complete. 2024-04-21T08:46:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadd0bc3740 20 client.17385 trim_cache size 0 max 16384 2024-04-21T08:46:32.341 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:46:32.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 -- 172.21.15.94:0/3521951762 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadb00a3d20 msgr2=0x7fadb00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadb00a3d20 0x7fadb00a61e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fadc00cd590 tx=0x7fadc007a040 comp rx=0 tx=0).stop 2024-04-21T08:46:32.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 -- 172.21.15.94:0/3521951762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc8198600 msgr2=0x7fadc8198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc8198600 0x7fadc8198ab0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fadc000b970 tx=0x7fadc000be30 comp rx=0 tx=0).stop 2024-04-21T08:46:32.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 -- 172.21.15.94:0/3521951762 shutdown_connections 2024-04-21T08:46:32.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fadb00a3d20 0x7fadb00a61e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadc8107270 0x7fadc819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc8198600 0x7fadc8198ab0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 --2- 172.21.15.94:0/3521951762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadc806a840 0x7fadc819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 -- 172.21.15.94:0/3521951762 >> 172.21.15.94:0/3521951762 conn(0x7fadc8077eb0 msgr2=0x7fadc80fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 -- 172.21.15.94:0/3521951762 shutdown_connections 2024-04-21T08:46:32.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.333+0000 7fadcdef2640 1 -- 172.21.15.94:0/3521951762 wait complete. 2024-04-21T08:46:32.344 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:46:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 -- 172.21.15.94:0/3574786428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd314102b40 msgr2=0x7fd314106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 --2- 172.21.15.94:0/3574786428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd314102b40 0x7fd314106b80 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fd308009900 tx=0x7fd30802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 -- 172.21.15.94:0/3574786428 shutdown_connections 2024-04-21T08:46:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 --2- 172.21.15.94:0/3574786428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd314107260 0x7fd31410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 --2- 172.21.15.94:0/3574786428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd314102b40 0x7fd314106b80 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 --2- 172.21.15.94:0/3574786428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd314102190 0x7fd314102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 -- 172.21.15.94:0/3574786428 >> 172.21.15.94:0/3574786428 conn(0x7fd3140793a0 msgr2=0x7fd3140797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 -- 172.21.15.94:0/3574786428 shutdown_connections 2024-04-21T08:46:32.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 -- 172.21.15.94:0/3574786428 wait complete. 2024-04-21T08:46:32.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 Processor -- start 2024-04-21T08:46:32.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.445+0000 7fd319315640 1 -- start start 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd319315640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd314102190 0x7fd314198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd319315640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd314107260 0x7fd314198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd319315640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd31419cfc0 0x7fd31419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd319315640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141121a0 con 0x7fd314102190 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd319315640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd314112020 con 0x7fd314107260 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd319315640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd314112320 con 0x7fd31419cfc0 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd312ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd314102190 0x7fd314198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd312ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd314102190 0x7fd314198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51920/0 (socket says 172.21.15.94:51920) 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd312ffd640 1 -- 172.21.15.94:0/897937424 learned_addr learned my addr 172.21.15.94:0/897937424 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd3137fe640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd31419cfc0 0x7fd31419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd3127fc640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd314107260 0x7fd314198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd312ffd640 1 -- 172.21.15.94:0/897937424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd31419cfc0 msgr2=0x7fd31419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd312ffd640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd31419cfc0 0x7fd31419d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd312ffd640 1 -- 172.21.15.94:0/897937424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd314107260 msgr2=0x7fd314198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd312ffd640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd314107260 0x7fd314198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd312ffd640 1 -- 172.21.15.94:0/897937424 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31419dab0 con 0x7fd314102190 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd312ffd640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd314102190 0x7fd314198700 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fd30400cce0 tx=0x7fd30400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd2fbfff640 1 -- 172.21.15.94:0/897937424 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd304004b20 con 0x7fd314102190 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd2fbfff640 1 -- 172.21.15.94:0/897937424 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd304004cc0 con 0x7fd314102190 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd2fbfff640 1 -- 172.21.15.94:0/897937424 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30400f7c0 con 0x7fd314102190 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd319315640 1 -- 172.21.15.94:0/897937424 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31419dda0 con 0x7fd314102190 2024-04-21T08:46:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.446+0000 7fd319315640 1 -- 172.21.15.94:0/897937424 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3141a5540 con 0x7fd314102190 2024-04-21T08:46:32.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.448+0000 7fd2fbfff640 1 -- 172.21.15.94:0/897937424 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd304005560 con 0x7fd314102190 2024-04-21T08:46:32.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd31bfe9740 1 Processor -- start 2024-04-21T08:46:32.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd31bfe9740 1 -- start start 2024-04-21T08:46:32.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c1d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c26430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c28e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55fa42b821f0 con 0x55fa42c26a00 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fa42b1f4b0 con 0x55fa42c1d4d0 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fa42b7e1b0 con 0x55fa42c1de80 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd3127fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c28e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd2fbfff640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd2fc0a3d20 0x7fd2fc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd3127fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c28e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51930/0 (socket says 172.21.15.94:51930) 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd3127fc640 1 -- 172.21.15.94:0/2328737749 learned_addr learned my addr 172.21.15.94:0/2328737749 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd312ffd640 1 --2- 172.21.15.94:0/2328737749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c26430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd3137fe640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd2fc0a3d20 0x7fd2fc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd3137fe640 1 --2- 172.21.15.94:0/2328737749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c1d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd3127fc640 1 -- 172.21.15.94:0/2328737749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 msgr2=0x55fa42c26430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd3127fc640 1 --2- 172.21.15.94:0/2328737749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c26430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd3127fc640 1 -- 172.21.15.94:0/2328737749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 msgr2=0x55fa42c1d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd3127fc640 1 --2- 172.21.15.94:0/2328737749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c1d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd3127fc640 1 -- 172.21.15.94:0/2328737749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa42b111b0 con 0x55fa42c26a00 2024-04-21T08:46:32.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd2fbfff640 1 -- 172.21.15.94:0/897937424 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd3040d0d50 con 0x7fd314102190 2024-04-21T08:46:32.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.449+0000 7fd312ffd640 1 --2- 172.21.15.94:0/2328737749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c26430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:32.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd3137fe640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd2fc0a3d20 0x7fd2fc0a61e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd30c005fd0 tx=0x7fd30c005e40 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd3127fc640 1 --2- 172.21.15.94:0/2328737749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c28e50 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fd308036040 tx=0x7fd30802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c451b0dc8d3c3c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd2f97fa640 1 -- 172.21.15.94:0/2328737749 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd308043020 con 0x55fa42c26a00 2024-04-21T08:46:32.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd2f97fa640 1 -- 172.21.15.94:0/2328737749 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd30802c7e0 con 0x55fa42c26a00 2024-04-21T08:46:32.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd2f97fa640 1 -- 172.21.15.94:0/2328737749 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3080055f0 con 0x55fa42c26a00 2024-04-21T08:46:32.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2328737749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 msgr2=0x55fa42c28e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2328737749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c28e50 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fd308036040 tx=0x7fd30802c650 comp rx=0 tx=0).stop 2024-04-21T08:46:32.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2328737749 shutdown_connections 2024-04-21T08:46:32.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2328737749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c26430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2328737749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c28e50 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2328737749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c1d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2328737749 >> 172.21.15.94:0/2328737749 conn(0x55fa42c17ba0 msgr2=0x55fa42c17f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2328737749 shutdown_connections 2024-04-21T08:46:32.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.450+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2328737749 wait complete. 2024-04-21T08:46:32.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd31bfe9740 1 -- start start 2024-04-21T08:46:32.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd31bfe9740 1 Processor -- start 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c262d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c23660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c23bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55fa42c29910 con 0x55fa42c26a00 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fa42b3b500 con 0x55fa42c1d4d0 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fa42c29a90 con 0x55fa42c1de80 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd31bfe9740 10 client.0 _ll_register_callbacks cb 0x55fa42b09b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd3137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c23bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd3137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c23bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51938/0 (socket says 172.21.15.94:51938) 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd3137fe640 1 -- 172.21.15.94:0/3355243295 learned_addr learned my addr 172.21.15.94:0/3355243295 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd312ffd640 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c262d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd3127fc640 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c23660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd3137fe640 1 -- 172.21.15.94:0/3355243295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 msgr2=0x55fa42c23660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd3137fe640 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c23660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd3137fe640 1 -- 172.21.15.94:0/3355243295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 msgr2=0x55fa42c262d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd3137fe640 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c262d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.451+0000 7fd3137fe640 1 -- 172.21.15.94:0/3355243295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa428943f0 con 0x55fa42c26a00 2024-04-21T08:46:32.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd3137fe640 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c23bc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fd30c010e30 tx=0x7fd30c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e542c60ef0aaaaa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd2f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:32.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/3355243295 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c021070 con 0x55fa42c26a00 2024-04-21T08:46:32.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/3355243295 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd30c01d410 con 0x55fa42c26a00 2024-04-21T08:46:32.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd31bfe9740 1 -- 172.21.15.94:0/3355243295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd30802c7e0 con 0x55fa42c26a00 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/3355243295 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd30c02d050 con 0x55fa42c26a00 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd31bfe9740 10 client.17397 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd31bfe9740 1 -- 172.21.15.94:0/3355243295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fa4295e7d0 con 0x55fa42c26a00 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/3355243295 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd30802c7e0 con 0x55fa42c26a00 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/3355243295 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd30c0254f0 con 0x55fa42c26a00 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd31bfe9740 10 client.17397 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd31bfe9740 10 client.17397 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd31bfe9740 20 client.17397 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55fa42c36b70 0x55fa42c38fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd31bfe9740 4 client.17397 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.452+0000 7fd31bfe9740 1 -- 172.21.15.94:0/3355243295 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa42b21930 con 0x55fa42c36b70 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.453+0000 7fd312ffd640 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55fa42c36b70 0x55fa42c38fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.453+0000 7fd312ffd640 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55fa42c36b70 0x55fa42c38fc0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=10fe743e1f09a701 server_cookie=582d1a68118e3b1b in_seq=0 out_seq=0 2024-04-21T08:46:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.453+0000 7fd2f6ff5640 10 client.17397 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/3355243295 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa42b21930 con 0x55fa42c36b70 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd2f6ff5640 10 client.17397 handle_command_reply: tid=0 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 client.17397 shutdown 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 -- 172.21.15.94:0/3355243295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 msgr2=0x55fa42c23bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c23bc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fd30c010e30 tx=0x7fd30c0046c0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 -- 172.21.15.94:0/3355243295 shutdown_connections 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c23660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55fa42c36b70 0x55fa42c38fc0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c23bc0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd2f6ff5640 0 client.17397 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/3355243295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c262d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 -- 172.21.15.94:0/3355243295 >> 172.21.15.94:0/3355243295 conn(0x55fa42c17ba0 msgr2=0x55fa42c198f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 -- 172.21.15.94:0/3355243295 shutdown_connections 2024-04-21T08:46:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 1 -- 172.21.15.94:0/3355243295 wait complete. 2024-04-21T08:46:32.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.454+0000 7fd31bfe9740 20 client.17397 trim_cache size 0 max 16384 2024-04-21T08:46:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd31bfe9740 1 Processor -- start 2024-04-21T08:46:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd31bfe9740 1 -- start start 2024-04-21T08:46:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c327d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c19880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c25d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141b1ce0 con 0x55fa42c26a00 2024-04-21T08:46:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141af5b0 con 0x55fa42c1d4d0 2024-04-21T08:46:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141ad400 con 0x55fa42c1de80 2024-04-21T08:46:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd3137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c25d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd3137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c25d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51950/0 (socket says 172.21.15.94:51950) 2024-04-21T08:46:32.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.467+0000 7fd3137fe640 1 -- 172.21.15.94:0/2429545683 learned_addr learned my addr 172.21.15.94:0/2429545683 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd3127fc640 1 --2- 172.21.15.94:0/2429545683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c19880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd312ffd640 1 --2- 172.21.15.94:0/2429545683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd3137fe640 1 -- 172.21.15.94:0/2429545683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 msgr2=0x55fa42c19880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd3137fe640 1 --2- 172.21.15.94:0/2429545683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c19880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd3137fe640 1 -- 172.21.15.94:0/2429545683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 msgr2=0x55fa42c327d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd3137fe640 1 --2- 172.21.15.94:0/2429545683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c327d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd3137fe640 1 -- 172.21.15.94:0/2429545683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa42c22610 con 0x55fa42c26a00 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd312ffd640 1 --2- 172.21.15.94:0/2429545683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c327d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd3137fe640 1 --2- 172.21.15.94:0/2429545683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c25d90 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fd30c025a80 tx=0x7fd30c005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8030d9458d4fada0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd2f8ff9640 1 -- 172.21.15.94:0/2429545683 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c021040 con 0x55fa42c26a00 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd2f8ff9640 1 -- 172.21.15.94:0/2429545683 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd30c036070 con 0x55fa42c26a00 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd2f8ff9640 1 -- 172.21.15.94:0/2429545683 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c025d30 con 0x55fa42c26a00 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2429545683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 msgr2=0x55fa42c25d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2429545683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c25d90 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fd30c025a80 tx=0x7fd30c005cc0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2429545683 shutdown_connections 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2429545683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1de80 0x55fa42c19880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2429545683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c25d90 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2429545683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1d4d0 0x55fa42c327d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.468+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2429545683 >> 172.21.15.94:0/2429545683 conn(0x55fa42c17ba0 msgr2=0x55fa42c17f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2429545683 shutdown_connections 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2429545683 wait complete. 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 1 -- start start 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 1 Processor -- start 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1d4d0 0x55fa42c2a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1de80 0x55fa42c2aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c2b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141ada70 con 0x55fa42c26a00 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141aeee0 con 0x55fa42c1de80 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141ae140 con 0x55fa42c1d4d0 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd31bfe9740 10 client.0 _ll_register_callbacks cb 0x7fd30c010ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:32.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd3137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c2b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd3137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c2b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51966/0 (socket says 172.21.15.94:51966) 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd3137fe640 1 -- 172.21.15.94:0/2795013985 learned_addr learned my addr 172.21.15.94:0/2795013985 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd312ffd640 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1d4d0 0x55fa42c2a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd3127fc640 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1de80 0x55fa42c2aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd3137fe640 1 -- 172.21.15.94:0/2795013985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1d4d0 msgr2=0x55fa42c2a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd3137fe640 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1d4d0 0x55fa42c2a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd3137fe640 1 -- 172.21.15.94:0/2795013985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1de80 msgr2=0x55fa42c2aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd3137fe640 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1de80 0x55fa42c2aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd3137fe640 1 -- 172.21.15.94:0/2795013985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa42c9c420 con 0x55fa42c26a00 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.469+0000 7fd3127fc640 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1de80 0x55fa42c2aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd3137fe640 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c2b450 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fd30c033040 tx=0x7fd30c025ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef977f4bc7f7da8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd2f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/2795013985 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c03e020 con 0x55fa42c26a00 2024-04-21T08:46:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/2795013985 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fd30c0255b0 con 0x55fa42c26a00 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2795013985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd30c036070 con 0x55fa42c26a00 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/2795013985 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd30c049050 con 0x55fa42c26a00 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd31bfe9740 10 client.17403 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2795013985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fa42c9d630 con 0x55fa42c26a00 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/2795013985 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd30c036070 con 0x55fa42c26a00 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/2795013985 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd30c003a80 con 0x55fa42c26a00 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd31bfe9740 10 client.17403 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd31bfe9740 10 client.17403 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd31bfe9740 20 client.17403 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55fa42c9dc30 0x55fa42ca0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd31bfe9740 4 client.17403 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.470+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2795013985 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fa42b8fc10 con 0x55fa42c9dc30 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd312ffd640 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55fa42c9dc30 0x55fa42ca0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd2f6ff5640 10 client.17403 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd312ffd640 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55fa42c9dc30 0x55fa42ca0050 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2bc647749de1f560 server_cookie=ea0d658382d564ba in_seq=0 out_seq=0 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd2f6ff5640 1 -- 172.21.15.94:0/2795013985 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fa42b8fc10 con 0x55fa42c9dc30 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd2f6ff5640 10 client.17403 handle_command_reply: tid=0 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 client.17403 shutdown 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2795013985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 msgr2=0x55fa42c2b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c2b450 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fd30c033040 tx=0x7fd30c025ab0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2795013985 shutdown_connections 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55fa42c9dc30 0x55fa42ca0050 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fa42c26a00 0x55fa42c2b450 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa42c1de80 0x55fa42c2aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd2f6ff5640 0 client.17403 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 --2- 172.21.15.94:0/2795013985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa42c1d4d0 0x55fa42c2a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2795013985 >> 172.21.15.94:0/2795013985 conn(0x55fa42c17ba0 msgr2=0x55fa42c16c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2795013985 shutdown_connections 2024-04-21T08:46:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 1 -- 172.21.15.94:0/2795013985 wait complete. 2024-04-21T08:46:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.471+0000 7fd31bfe9740 20 client.17403 trim_cache size 0 max 16384 2024-04-21T08:46:32.483 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:46:32.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 -- 172.21.15.94:0/897937424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd2fc0a3d20 msgr2=0x7fd2fc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd2fc0a3d20 0x7fd2fc0a61e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd30c005fd0 tx=0x7fd30c005e40 comp rx=0 tx=0).stop 2024-04-21T08:46:32.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 -- 172.21.15.94:0/897937424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd314102190 msgr2=0x7fd314198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd314102190 0x7fd314198700 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fd30400cce0 tx=0x7fd30400a620 comp rx=0 tx=0).stop 2024-04-21T08:46:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 -- 172.21.15.94:0/897937424 shutdown_connections 2024-04-21T08:46:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd2fc0a3d20 0x7fd2fc0a61e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd31419cfc0 0x7fd31419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd314107260 0x7fd314198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 --2- 172.21.15.94:0/897937424 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd314102190 0x7fd314198700 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 -- 172.21.15.94:0/897937424 >> 172.21.15.94:0/897937424 conn(0x7fd3140793a0 msgr2=0x7fd3140fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 -- 172.21.15.94:0/897937424 shutdown_connections 2024-04-21T08:46:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.472+0000 7fd319315640 1 -- 172.21.15.94:0/897937424 wait complete. 2024-04-21T08:46:32.485 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:46:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.586+0000 7f22d633d640 1 -- 172.21.15.94:0/835780117 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22d01072b0 msgr2=0x7f22d010ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.586+0000 7f22d633d640 1 --2- 172.21.15.94:0/835780117 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22d01072b0 0x7f22d010ee80 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009ca0 tx=0x7f22c802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.586+0000 7f22d633d640 1 -- 172.21.15.94:0/835780117 shutdown_connections 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.586+0000 7f22d633d640 1 --2- 172.21.15.94:0/835780117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22d0102b90 0x7f22d0106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.586+0000 7f22d633d640 1 --2- 172.21.15.94:0/835780117 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22d01072b0 0x7f22d010ee80 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.586+0000 7f22d633d640 1 --2- 172.21.15.94:0/835780117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d01021e0 0x7f22d01025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.586+0000 7f22d633d640 1 -- 172.21.15.94:0/835780117 >> 172.21.15.94:0/835780117 conn(0x7f22d00fda60 msgr2=0x7f22d00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.586+0000 7f22d633d640 1 -- 172.21.15.94:0/835780117 shutdown_connections 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.586+0000 7f22d633d640 1 -- 172.21.15.94:0/835780117 wait complete. 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22d633d640 1 Processor -- start 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22d633d640 1 -- start start 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22d633d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22d01021e0 0x7f22d019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22d633d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d01072b0 0x7f22d019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22d633d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22d01985e0 0x7f22d0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22d633d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f22d0112220 con 0x7f22d01021e0 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22d633d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01120a0 con 0x7f22d01072b0 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22d633d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01123a0 con 0x7f22d01985e0 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22cffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22d01021e0 0x7f22d019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22cffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22d01021e0 0x7f22d019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51972/0 (socket says 172.21.15.94:51972) 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22cffff640 1 -- 172.21.15.94:0/24724115 learned_addr learned my addr 172.21.15.94:0/24724115 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22d48da640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22d01985e0 0x7f22d0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.587+0000 7f22cf7fe640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d01072b0 0x7f22d019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22cffff640 1 -- 172.21.15.94:0/24724115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22d01985e0 msgr2=0x7f22d0198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22cffff640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22d01985e0 0x7f22d0198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22cffff640 1 -- 172.21.15.94:0/24724115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d01072b0 msgr2=0x7f22d019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22cffff640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d01072b0 0x7f22d019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22cffff640 1 -- 172.21.15.94:0/24724115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d0199100 con 0x7f22d01021e0 2024-04-21T08:46:32.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22cf7fe640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d01072b0 0x7f22d019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:32.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22cffff640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22d01021e0 0x7f22d019e3e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f22c000c890 tx=0x7f22c000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22cd7fa640 1 -- 172.21.15.94:0/24724115 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0004220 con 0x7f22d01021e0 2024-04-21T08:46:32.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22cd7fa640 1 -- 172.21.15.94:0/24724115 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22c00043c0 con 0x7f22d01021e0 2024-04-21T08:46:32.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22d633d640 1 -- 172.21.15.94:0/24724115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d01993f0 con 0x7f22d01021e0 2024-04-21T08:46:32.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22cd7fa640 1 -- 172.21.15.94:0/24724115 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0004a00 con 0x7f22d01021e0 2024-04-21T08:46:32.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.588+0000 7f22d633d640 1 -- 172.21.15.94:0/24724115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d01a55f0 con 0x7f22d01021e0 2024-04-21T08:46:32.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.590+0000 7f22cd7fa640 1 -- 172.21.15.94:0/24724115 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f22c00079b0 con 0x7f22d01021e0 2024-04-21T08:46:32.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.590+0000 7f22d900a740 1 Processor -- start 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.590+0000 7f22d900a740 1 -- start start 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cd7fa640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22b80a3d20 0x7f22b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22d900a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e698b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22d900a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e72430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22d900a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 0x5582c9e74e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22d900a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5582c9dce1f0 con 0x5582c9e69e80 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22d900a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5582c9d6b4b0 con 0x5582c9e72a00 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22d900a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5582c9dca1b0 con 0x5582c9e694d0 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22d48da640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22b80a3d20 0x7f22b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cd7fa640 1 -- 172.21.15.94:0/24724115 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f22c00d0930 con 0x7f22d01021e0 2024-04-21T08:46:32.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e72430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e72430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51976/0 (socket says 172.21.15.94:51976) 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cf7fe640 1 -- 172.21.15.94:0/919556230 learned_addr learned my addr 172.21.15.94:0/919556230 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cffff640 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cf7fe640 1 -- 172.21.15.94:0/919556230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 msgr2=0x5582c9e698b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22d48da640 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 0x5582c9e74e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cf7fe640 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e698b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cf7fe640 1 -- 172.21.15.94:0/919556230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 msgr2=0x5582c9e74e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cf7fe640 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 0x5582c9e74e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cf7fe640 1 -- 172.21.15.94:0/919556230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582c9d5d1b0 con 0x5582c9e69e80 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cffff640 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22d48da640 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 0x5582c9e74e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22d48da640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22b80a3d20 0x7f22b80a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f22c8004af0 tx=0x7f22c8032380 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22cf7fe640 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e72430 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009b30 tx=0x7f22c400e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa48f6d28f84433 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22b67fc640 1 -- 172.21.15.94:0/919556230 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4016070 con 0x5582c9e69e80 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22b67fc640 1 -- 172.21.15.94:0/919556230 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22c400ed20 con 0x5582c9e69e80 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.591+0000 7f22b67fc640 1 -- 172.21.15.94:0/919556230 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4010730 con 0x5582c9e69e80 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.592+0000 7f22d900a740 1 -- 172.21.15.94:0/919556230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 msgr2=0x5582c9e72430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.592+0000 7f22d900a740 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e72430 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009b30 tx=0x7f22c400e2f0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.592+0000 7f22d900a740 1 -- 172.21.15.94:0/919556230 shutdown_connections 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.592+0000 7f22d900a740 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 0x5582c9e74e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.592+0000 7f22d900a740 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e72430 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.592+0000 7f22d900a740 1 --2- 172.21.15.94:0/919556230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.592+0000 7f22d900a740 1 -- 172.21.15.94:0/919556230 >> 172.21.15.94:0/919556230 conn(0x5582c9e63ba0 msgr2=0x5582c9e63f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.592+0000 7f22d900a740 1 -- 172.21.15.94:0/919556230 shutdown_connections 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.592+0000 7f22d900a740 1 -- 172.21.15.94:0/919556230 wait complete. 2024-04-21T08:46:32.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.592+0000 7f22d900a740 1 -- start start 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d900a740 1 Processor -- start 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d900a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e694d0 0x5582c9e72290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d900a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e69e80 0x5582c9e6f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d900a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e6fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d900a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5582c9e75970 con 0x5582c9e72a00 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d900a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5582c9d87500 con 0x5582c9e694d0 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d900a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5582c9e75af0 con 0x5582c9e69e80 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d900a740 10 client.0 _ll_register_callbacks cb 0x5582c9d55b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d48da640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e6fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d48da640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e6fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51988/0 (socket says 172.21.15.94:51988) 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d48da640 1 -- 172.21.15.94:0/2483270265 learned_addr learned my addr 172.21.15.94:0/2483270265 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22cffff640 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e694d0 0x5582c9e72290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22cf7fe640 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e69e80 0x5582c9e6f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d48da640 1 -- 172.21.15.94:0/2483270265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e69e80 msgr2=0x5582c9e6f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d48da640 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e69e80 0x5582c9e6f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d48da640 1 -- 172.21.15.94:0/2483270265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e694d0 msgr2=0x5582c9e72290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d48da640 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e694d0 0x5582c9e72290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d48da640 1 -- 172.21.15.94:0/2483270265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582c9ae03f0 con 0x5582c9e72a00 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.593+0000 7f22d48da640 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e6fb60 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009ca0 tx=0x7f22c802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d07e323ed858106 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22af7fe640 1 -- 172.21.15.94:0/2483270265 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8050060 con 0x5582c9e72a00 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22af7fe640 1 -- 172.21.15.94:0/2483270265 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f22c8003cc0 con 0x5582c9e72a00 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22d900a740 1 -- 172.21.15.94:0/2483270265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22c400ed20 con 0x5582c9e72a00 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22af7fe640 1 -- 172.21.15.94:0/2483270265 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f22c8057050 con 0x5582c9e72a00 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22d900a740 10 client.17415 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22d900a740 1 -- 172.21.15.94:0/2483270265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5582c9baa7d0 con 0x5582c9e72a00 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22af7fe640 1 -- 172.21.15.94:0/2483270265 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22c400ed20 con 0x5582c9e72a00 2024-04-21T08:46:32.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22af7fe640 1 -- 172.21.15.94:0/2483270265 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f22c803f040 con 0x5582c9e72a00 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22d900a740 10 client.17415 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22d900a740 10 client.17415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22d900a740 20 client.17415 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22d900a740 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5582c9e82b60 0x5582c9e84fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22d900a740 4 client.17415 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.594+0000 7f22d900a740 1 -- 172.21.15.94:0/2483270265 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582c9d6d930 con 0x5582c9e82b60 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.595+0000 7f22cffff640 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5582c9e82b60 0x5582c9e84fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.595+0000 7f22cffff640 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5582c9e82b60 0x5582c9e84fe0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b00b34d89f4ad1d9 server_cookie=3037c0acf7b6edc4 in_seq=0 out_seq=0 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.595+0000 7f22af7fe640 10 client.17415 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22af7fe640 1 -- 172.21.15.94:0/2483270265 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582c9d6d930 con 0x5582c9e82b60 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22af7fe640 10 client.17415 handle_command_reply: tid=0 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 client.17415 shutdown 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 -- 172.21.15.94:0/2483270265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 msgr2=0x5582c9e6fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e6fb60 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009ca0 tx=0x7f22c802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 -- 172.21.15.94:0/2483270265 shutdown_connections 2024-04-21T08:46:32.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5582c9e82b60 0x5582c9e84fe0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e69e80 0x5582c9e6f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e6fb60 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 --2- 172.21.15.94:0/2483270265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e694d0 0x5582c9e72290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 -- 172.21.15.94:0/2483270265 >> 172.21.15.94:0/2483270265 conn(0x5582c9e63ba0 msgr2=0x5582c9e658f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22af7fe640 0 client.17415 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:32.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 -- 172.21.15.94:0/2483270265 shutdown_connections 2024-04-21T08:46:32.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 1 -- 172.21.15.94:0/2483270265 wait complete. 2024-04-21T08:46:32.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.596+0000 7f22d900a740 20 client.17415 trim_cache size 0 max 16384 2024-04-21T08:46:32.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d900a740 1 Processor -- start 2024-04-21T08:46:32.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d900a740 1 -- start start 2024-04-21T08:46:32.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d900a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e7e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d900a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e69e80 0x5582c9e6e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d900a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e7b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d900a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01b1c80 con 0x5582c9e72a00 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d900a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01af550 con 0x5582c9e69e80 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d900a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01ad3a0 con 0x5582c9e694d0 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d48da640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e7b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d48da640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e7b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51996/0 (socket says 172.21.15.94:51996) 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.609+0000 7f22d48da640 1 -- 172.21.15.94:0/2944225860 learned_addr learned my addr 172.21.15.94:0/2944225860 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22d48da640 1 -- 172.21.15.94:0/2944225860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 msgr2=0x5582c9e7e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22d48da640 1 --2- 172.21.15.94:0/2944225860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e7e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22d48da640 1 -- 172.21.15.94:0/2944225860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e69e80 msgr2=0x5582c9e6e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22d48da640 1 --2- 172.21.15.94:0/2944225860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e69e80 0x5582c9e6e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22d48da640 1 -- 172.21.15.94:0/2944225860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582c9e64520 con 0x5582c9e72a00 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22d48da640 1 --2- 172.21.15.94:0/2944225860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e7b120 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f22c8006010 tx=0x7f22c802c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece83c4ad1ff93d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22b5ffb640 1 -- 172.21.15.94:0/2944225860 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8050070 con 0x5582c9e72a00 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22b5ffb640 1 -- 172.21.15.94:0/2944225860 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22c8034bf0 con 0x5582c9e72a00 2024-04-21T08:46:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22b5ffb640 1 -- 172.21.15.94:0/2944225860 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8039040 con 0x5582c9e72a00 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22d900a740 1 -- 172.21.15.94:0/2944225860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 msgr2=0x5582c9e7b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.610+0000 7f22d900a740 1 --2- 172.21.15.94:0/2944225860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e7b120 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f22c8006010 tx=0x7f22c802c790 comp rx=0 tx=0).stop 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 -- 172.21.15.94:0/2944225860 shutdown_connections 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 --2- 172.21.15.94:0/2944225860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e72a00 0x5582c9e7b120 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 --2- 172.21.15.94:0/2944225860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e69e80 0x5582c9e6e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 --2- 172.21.15.94:0/2944225860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e7e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 -- 172.21.15.94:0/2944225860 >> 172.21.15.94:0/2944225860 conn(0x5582c9e63ba0 msgr2=0x5582c9e64ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 -- 172.21.15.94:0/2944225860 shutdown_connections 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 -- 172.21.15.94:0/2944225860 wait complete. 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 -- start start 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 Processor -- start 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e716e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e71c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 0x5582c9e72160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01ada10 con 0x5582c9e69e80 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01aee80 con 0x5582c9e72a00 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01ae0e0 con 0x5582c9e694d0 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22d900a740 10 client.0 _ll_register_callbacks cb 0x7f22c80324c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22cf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e71c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22cf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e71c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52002/0 (socket says 172.21.15.94:52002) 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.611+0000 7f22cf7fe640 1 -- 172.21.15.94:0/1909603635 learned_addr learned my addr 172.21.15.94:0/1909603635 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22cffff640 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e716e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22d48da640 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 0x5582c9e72160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22cf7fe640 1 -- 172.21.15.94:0/1909603635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 msgr2=0x5582c9e716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22cf7fe640 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e716e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22cf7fe640 1 -- 172.21.15.94:0/1909603635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 msgr2=0x5582c9e72160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22cf7fe640 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 0x5582c9e72160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22cf7fe640 1 -- 172.21.15.94:0/1909603635 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582c9eec150 con 0x5582c9e69e80 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22cf7fe640 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e71c20 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009a80 tx=0x7f22c40094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1f08fb02947bd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22af7fe640 1 -- 172.21.15.94:0/1909603635 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4016040 con 0x5582c9e69e80 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22af7fe640 1 -- 172.21.15.94:0/1909603635 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f22c4004620 con 0x5582c9e69e80 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22d900a740 1 -- 172.21.15.94:0/1909603635 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22c8034bf0 con 0x5582c9e69e80 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22af7fe640 1 -- 172.21.15.94:0/1909603635 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f22c4004820 con 0x5582c9e69e80 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22d900a740 10 client.17421 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22d900a740 1 -- 172.21.15.94:0/1909603635 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5582c9eed3d0 con 0x5582c9e69e80 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22af7fe640 1 -- 172.21.15.94:0/1909603635 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22c8034bf0 con 0x5582c9e69e80 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22af7fe640 1 -- 172.21.15.94:0/1909603635 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f22c400f040 con 0x5582c9e69e80 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22d900a740 10 client.17421 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22d900a740 10 client.17421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22d900a740 20 client.17421 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22d900a740 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5582c9eed9e0 0x5582c9eefe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22d900a740 4 client.17421 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.612+0000 7f22d900a740 1 -- 172.21.15.94:0/1909603635 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5582c9ddbc10 con 0x5582c9eed9e0 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22cffff640 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5582c9eed9e0 0x5582c9eefe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22cffff640 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5582c9eed9e0 0x5582c9eefe00 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=618057bbdbfffaf4 server_cookie=a0a9330d060a978 in_seq=0 out_seq=0 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22af7fe640 10 client.17421 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22af7fe640 1 -- 172.21.15.94:0/1909603635 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5582c9ddbc10 con 0x5582c9eed9e0 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22af7fe640 10 client.17421 handle_command_reply: tid=0 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22d900a740 1 client.17421 shutdown 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22d900a740 1 -- 172.21.15.94:0/1909603635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 msgr2=0x5582c9e71c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22d900a740 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e71c20 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009a80 tx=0x7f22c40094d0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22d900a740 1 -- 172.21.15.94:0/1909603635 shutdown_connections 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22d900a740 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5582c9eed9e0 0x5582c9eefe00 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22d900a740 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5582c9e72a00 0x5582c9e72160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22af7fe640 0 client.17421 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:32.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22d900a740 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5582c9e69e80 0x5582c9e71c20 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22d900a740 1 --2- 172.21.15.94:0/1909603635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582c9e694d0 0x5582c9e716e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.613+0000 7f22d900a740 1 -- 172.21.15.94:0/1909603635 >> 172.21.15.94:0/1909603635 conn(0x5582c9e63ba0 msgr2=0x5582c9e6ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d900a740 1 -- 172.21.15.94:0/1909603635 shutdown_connections 2024-04-21T08:46:32.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d900a740 1 -- 172.21.15.94:0/1909603635 wait complete. 2024-04-21T08:46:32.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d900a740 20 client.17421 trim_cache size 0 max 16384 2024-04-21T08:46:32.621 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:46:32.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 -- 172.21.15.94:0/24724115 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22b80a3d20 msgr2=0x7f22b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22b80a3d20 0x7f22b80a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f22c8004af0 tx=0x7f22c8032380 comp rx=0 tx=0).stop 2024-04-21T08:46:32.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 -- 172.21.15.94:0/24724115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22d01021e0 msgr2=0x7f22d019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22d01021e0 0x7f22d019e3e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f22c000c890 tx=0x7f22c000cd50 comp rx=0 tx=0).stop 2024-04-21T08:46:32.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 -- 172.21.15.94:0/24724115 shutdown_connections 2024-04-21T08:46:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22b80a3d20 0x7f22b80a61e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22d01985e0 0x7f22d0198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d01072b0 0x7f22d019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 --2- 172.21.15.94:0/24724115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22d01021e0 0x7f22d019e3e0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 -- 172.21.15.94:0/24724115 >> 172.21.15.94:0/24724115 conn(0x7f22d00fda60 msgr2=0x7f22d00ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 -- 172.21.15.94:0/24724115 shutdown_connections 2024-04-21T08:46:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:32.614+0000 7f22d633d640 1 -- 172.21.15.94:0/24724115 wait complete. 2024-04-21T08:46:33.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:33 smithi111 ceph-mon[31493]: pgmap v200: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:33.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:33 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2212801382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:33.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:33 smithi094 ceph-mon[27988]: pgmap v200: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:33.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:33 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2212801382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:33.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:33 smithi173 ceph-mon[31428]: pgmap v200: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:33.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:33 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2212801382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:33.626 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:46:33.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:46:33.627 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:46:33.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 -- 172.21.15.94:0/1719758434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff98006aa90 msgr2=0x7ff98006aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 --2- 172.21.15.94:0/1719758434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff98006aa90 0x7ff98006aef0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff970009900 tx=0x7ff97002bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:33.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 -- 172.21.15.94:0/1719758434 shutdown_connections 2024-04-21T08:46:33.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 --2- 172.21.15.94:0/1719758434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff98010e810 0x7ff980110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 --2- 172.21.15.94:0/1719758434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff98006aa90 0x7ff98006aef0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 --2- 172.21.15.94:0/1719758434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff980102e20 0x7ff98006a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 -- 172.21.15.94:0/1719758434 >> 172.21.15.94:0/1719758434 conn(0x7ff980076c50 msgr2=0x7ff980077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 -- 172.21.15.94:0/1719758434 shutdown_connections 2024-04-21T08:46:33.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 -- 172.21.15.94:0/1719758434 wait complete. 2024-04-21T08:46:33.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 Processor -- start 2024-04-21T08:46:33.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.743+0000 7ff98734b640 1 -- start start 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff98734b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff98006aa90 0x7ff9801a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff98734b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff980102e20 0x7ff9801a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff98734b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff98010e810 0x7ff98019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff98734b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801143c0 con 0x7ff980102e20 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff98734b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff980114240 con 0x7ff98010e810 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff98734b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff980114540 con 0x7ff98006aa90 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9848e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff980102e20 0x7ff9801a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9848e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff980102e20 0x7ff9801a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38890/0 (socket says 172.21.15.94:38890) 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9848e6640 1 -- 172.21.15.94:0/1947659846 learned_addr learned my addr 172.21.15.94:0/1947659846 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9858e8640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff98010e810 0x7ff98019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9850e7640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff98006aa90 0x7ff9801a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9848e6640 1 -- 172.21.15.94:0/1947659846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff98006aa90 msgr2=0x7ff9801a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9848e6640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff98006aa90 0x7ff9801a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9848e6640 1 -- 172.21.15.94:0/1947659846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff98010e810 msgr2=0x7ff98019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9848e6640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff98010e810 0x7ff98019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9848e6640 1 -- 172.21.15.94:0/1947659846 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98019b020 con 0x7ff980102e20 2024-04-21T08:46:33.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9848e6640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff980102e20 0x7ff9801a0be0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff970005940 tx=0x7ff97002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9767fc640 1 -- 172.21.15.94:0/1947659846 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970043020 con 0x7ff980102e20 2024-04-21T08:46:33.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9767fc640 1 -- 172.21.15.94:0/1947659846 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff97002cbe0 con 0x7ff980102e20 2024-04-21T08:46:33.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff98734b640 1 -- 172.21.15.94:0/1947659846 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98019b2b0 con 0x7ff980102e20 2024-04-21T08:46:33.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff9767fc640 1 -- 172.21.15.94:0/1947659846 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970003bf0 con 0x7ff980102e20 2024-04-21T08:46:33.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.744+0000 7ff98734b640 1 -- 172.21.15.94:0/1947659846 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9801a7620 con 0x7ff980102e20 2024-04-21T08:46:33.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.747+0000 7ff9767fc640 1 -- 172.21.15.94:0/1947659846 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff970046020 con 0x7ff980102e20 2024-04-21T08:46:33.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff98a011740 1 Processor -- start 2024-04-21T08:46:33.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff98a011740 1 -- start start 2024-04-21T08:46:33.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff98a011740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 0x55b4bfb79cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff98a011740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb82840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff98a011740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb85260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff98a011740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b4bfaddf00 con 0x55b4bfb82e10 2024-04-21T08:46:33.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff98a011740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b4bfa7b590 con 0x55b4bfb7a290 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff98a011740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b4bfad9ec0 con 0x55b4bfb798e0 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9848e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb85260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9848e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb85260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38894/0 (socket says 172.21.15.94:38894) 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9848e6640 1 -- 172.21.15.94:0/958635005 learned_addr learned my addr 172.21.15.94:0/958635005 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9767fc640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff96c0a3d20 0x7ff96c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9850e7640 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb82840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9858e8640 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 0x55b4bfb79cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9848e6640 1 -- 172.21.15.94:0/958635005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 msgr2=0x55b4bfb79cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9848e6640 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 0x55b4bfb79cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9848e6640 1 -- 172.21.15.94:0/958635005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 msgr2=0x55b4bfb82840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9848e6640 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb82840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9848e6640 1 -- 172.21.15.94:0/958635005 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4bf8b87e0 con 0x55b4bfb82e10 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9850e7640 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb82840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:33.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9858e8640 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 0x55b4bfb79cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9858e8640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff96c0a3d20 0x7ff96c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.748+0000 7ff9767fc640 1 -- 172.21.15.94:0/1947659846 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff9700f2dc0 con 0x7ff980102e20 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.749+0000 7ff9848e6640 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb85260 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ff9700f01b0 tx=0x7ff97004f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f02ca73248b441 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.749+0000 7ff96b7fe640 1 -- 172.21.15.94:0/958635005 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9700be5d0 con 0x55b4bfb82e10 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.749+0000 7ff96b7fe640 1 -- 172.21.15.94:0/958635005 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff97003c350 con 0x55b4bfb82e10 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.749+0000 7ff96b7fe640 1 -- 172.21.15.94:0/958635005 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97003eaa0 con 0x55b4bfb82e10 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.749+0000 7ff9858e8640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff96c0a3d20 0x7ff96c0a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff9780058c0 tx=0x7ff9780046c0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.749+0000 7ff98a011740 1 -- 172.21.15.94:0/958635005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 msgr2=0x55b4bfb85260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.749+0000 7ff98a011740 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb85260 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ff9700f01b0 tx=0x7ff97004f000 comp rx=0 tx=0).stop 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 -- 172.21.15.94:0/958635005 shutdown_connections 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb85260 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb82840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 --2- 172.21.15.94:0/958635005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 0x55b4bfb79cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 -- 172.21.15.94:0/958635005 >> 172.21.15.94:0/958635005 conn(0x55b4bfb73fb0 msgr2=0x55b4bfb74390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 -- 172.21.15.94:0/958635005 shutdown_connections 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 -- 172.21.15.94:0/958635005 wait complete. 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 -- start start 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 Processor -- start 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 0x55b4bfb7fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb80200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb80740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55b4bfb85b40 con 0x55b4bfb82e10 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b4bfa97580 con 0x55b4bfb7a290 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b4bfb85cc0 con 0x55b4bfb798e0 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.750+0000 7ff98a011740 10 client.0 _ll_register_callbacks cb 0x55b4bfa9d650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff9858e8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb80740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff9858e8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb80740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38904/0 (socket says 172.21.15.94:38904) 2024-04-21T08:46:33.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff9858e8640 1 -- 172.21.15.94:0/3288280722 learned_addr learned my addr 172.21.15.94:0/3288280722 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff9850e7640 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 0x55b4bfb7fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff9858e8640 1 -- 172.21.15.94:0/3288280722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 msgr2=0x55b4bfb7fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff9858e8640 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 0x55b4bfb7fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff9858e8640 1 -- 172.21.15.94:0/3288280722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 msgr2=0x55b4bfb80200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff9858e8640 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb80200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff9858e8640 1 -- 172.21.15.94:0/3288280722 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4bf918f70 con 0x55b4bfb82e10 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff9858e8640 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb80740 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7ff978012690 tx=0x7ff978005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6a1219d3ee3ff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff968ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff968ff9640 1 -- 172.21.15.94:0/3288280722 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97801b960 con 0x55b4bfb82e10 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff968ff9640 1 -- 172.21.15.94:0/3288280722 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff97801bb00 con 0x55b4bfb82e10 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff98a011740 1 -- 172.21.15.94:0/3288280722 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff97003c350 con 0x55b4bfb82e10 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff968ff9640 1 -- 172.21.15.94:0/3288280722 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff97802f050 con 0x55b4bfb82e10 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff98a011740 10 client.17433 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:33.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff98a011740 1 -- 172.21.15.94:0/3288280722 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b4bf7b0170 con 0x55b4bfb82e10 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.751+0000 7ff968ff9640 1 -- 172.21.15.94:0/3288280722 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff97003c350 con 0x55b4bfb82e10 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.752+0000 7ff968ff9640 1 -- 172.21.15.94:0/3288280722 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff978018030 con 0x55b4bfb82e10 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.752+0000 7ff98a011740 10 client.17433 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.752+0000 7ff98a011740 10 client.17433 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.752+0000 7ff98a011740 20 client.17433 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.752+0000 7ff98a011740 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b4bfb92f90 0x55b4bfb953e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.752+0000 7ff98a011740 4 client.17433 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.752+0000 7ff98a011740 1 -- 172.21.15.94:0/3288280722 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4bfaeb920 con 0x55b4bfb92f90 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.752+0000 7ff9850e7640 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b4bfb92f90 0x55b4bfb953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff968ff9640 10 client.17433 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.752+0000 7ff9850e7640 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b4bfb92f90 0x55b4bfb953e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bc24940d1a6c419 server_cookie=5e514a82dca1fa36 in_seq=0 out_seq=0 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff968ff9640 1 -- 172.21.15.94:0/3288280722 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b4bfaeb920 con 0x55b4bfb92f90 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff968ff9640 10 client.17433 handle_command_reply: tid=0 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff98a011740 1 client.17433 shutdown 2024-04-21T08:46:33.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff98a011740 1 -- 172.21.15.94:0/3288280722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 msgr2=0x55b4bfb80740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff98a011740 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb80740 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7ff978012690 tx=0x7ff978005850 comp rx=0 tx=0).stop 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff98a011740 1 -- 172.21.15.94:0/3288280722 shutdown_connections 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff98a011740 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b4bfb92f90 0x55b4bfb953e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff98a011740 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb82e10 0x55b4bfb80740 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff98a011740 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb80200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff968ff9640 0 client.17433 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff98a011740 1 --2- 172.21.15.94:0/3288280722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb798e0 0x55b4bfb7fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.753+0000 7ff98a011740 1 -- 172.21.15.94:0/3288280722 >> 172.21.15.94:0/3288280722 conn(0x55b4bfb73fb0 msgr2=0x55b4bfb75d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.754+0000 7ff98a011740 1 -- 172.21.15.94:0/3288280722 shutdown_connections 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.754+0000 7ff98a011740 1 -- 172.21.15.94:0/3288280722 wait complete. 2024-04-21T08:46:33.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.754+0000 7ff98a011740 20 client.17433 trim_cache size 0 max 16384 2024-04-21T08:46:33.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff98a011740 1 Processor -- start 2024-04-21T08:46:33.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff98a011740 1 -- start start 2024-04-21T08:46:33.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff98a011740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb798e0 0x55b4bfb823f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff98a011740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb7a290 0x55b4bfb8e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff98a011740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb82e10 0x55b4bfb80390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff98a011740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801b3f30 con 0x55b4bfb7a290 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff98a011740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801b1800 con 0x55b4bfb798e0 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff98a011740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801af650 con 0x55b4bfb82e10 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff9848e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb7a290 0x55b4bfb8e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff9848e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb7a290 0x55b4bfb8e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38908/0 (socket says 172.21.15.94:38908) 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff9848e6640 1 -- 172.21.15.94:0/317205936 learned_addr learned my addr 172.21.15.94:0/317205936 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff9858e8640 1 --2- 172.21.15.94:0/317205936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb82e10 0x55b4bfb80390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff9848e6640 1 -- 172.21.15.94:0/317205936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb82e10 msgr2=0x55b4bfb80390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff9848e6640 1 --2- 172.21.15.94:0/317205936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb82e10 0x55b4bfb80390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff9848e6640 1 -- 172.21.15.94:0/317205936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb798e0 msgr2=0x55b4bfb823f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff9848e6640 1 --2- 172.21.15.94:0/317205936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb798e0 0x55b4bfb823f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff9848e6640 1 -- 172.21.15.94:0/317205936 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4bfb73de0 con 0x55b4bfb7a290 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff9848e6640 1 --2- 172.21.15.94:0/317205936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb7a290 0x55b4bfb8e8d0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ff970000c00 tx=0x7ff97002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ab6bf7c3ce90ad server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.772+0000 7ff96affd640 1 -- 172.21.15.94:0/317205936 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9700be3f0 con 0x55b4bfb7a290 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff96affd640 1 -- 172.21.15.94:0/317205936 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff97003c350 con 0x55b4bfb7a290 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff96affd640 1 -- 172.21.15.94:0/317205936 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9700bea90 con 0x55b4bfb7a290 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff98a011740 1 -- 172.21.15.94:0/317205936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb7a290 msgr2=0x55b4bfb8e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff98a011740 1 --2- 172.21.15.94:0/317205936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb7a290 0x55b4bfb8e8d0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ff970000c00 tx=0x7ff97002bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff98a011740 1 -- 172.21.15.94:0/317205936 shutdown_connections 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff98a011740 1 --2- 172.21.15.94:0/317205936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb82e10 0x55b4bfb80390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff98a011740 1 --2- 172.21.15.94:0/317205936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb7a290 0x55b4bfb8e8d0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff98a011740 1 --2- 172.21.15.94:0/317205936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb798e0 0x55b4bfb823f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff98a011740 1 -- 172.21.15.94:0/317205936 >> 172.21.15.94:0/317205936 conn(0x55b4bfb73fb0 msgr2=0x55b4bfb7fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff98a011740 1 -- 172.21.15.94:0/317205936 shutdown_connections 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.773+0000 7ff98a011740 1 -- 172.21.15.94:0/317205936 wait complete. 2024-04-21T08:46:33.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff98a011740 1 -- start start 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff98a011740 1 Processor -- start 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff98a011740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb798e0 0x55b4bfb8ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff98a011740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb8b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff98a011740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb82e10 0x55b4bfb87db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff98a011740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801afcc0 con 0x55b4bfb798e0 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff98a011740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801b1130 con 0x55b4bfb7a290 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff98a011740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801b0390 con 0x55b4bfb82e10 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff98a011740 10 client.0 _ll_register_callbacks cb 0x7ff97800c880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff9850e7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb798e0 0x55b4bfb8ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff9850e7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb798e0 0x55b4bfb8ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38924/0 (socket says 172.21.15.94:38924) 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff9850e7640 1 -- 172.21.15.94:0/617374214 learned_addr learned my addr 172.21.15.94:0/617374214 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:33.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff9858e8640 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb82e10 0x55b4bfb87db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff9850e7640 1 -- 172.21.15.94:0/617374214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb82e10 msgr2=0x55b4bfb87db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff9850e7640 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb82e10 0x55b4bfb87db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff9850e7640 1 -- 172.21.15.94:0/617374214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 msgr2=0x55b4bfb8b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff9850e7640 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb8b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.774+0000 7ff9850e7640 1 -- 172.21.15.94:0/617374214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4bfb88370 con 0x55b4bfb798e0 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff9850e7640 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb798e0 0x55b4bfb8ade0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009370 tx=0x7ff97c013a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e54d8f3ef8d5c5c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff968ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff968ff9640 1 -- 172.21.15.94:0/617374214 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c003970 con 0x55b4bfb798e0 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff968ff9640 1 -- 172.21.15.94:0/617374214 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7ff97c003b10 con 0x55b4bfb798e0 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff98a011740 1 -- 172.21.15.94:0/617374214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff97003c350 con 0x55b4bfb798e0 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff968ff9640 1 -- 172.21.15.94:0/617374214 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff97c02c050 con 0x55b4bfb798e0 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff98a011740 10 client.17439 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff98a011740 1 -- 172.21.15.94:0/617374214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b4bfbf9e50 con 0x55b4bfb798e0 2024-04-21T08:46:33.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff968ff9640 1 -- 172.21.15.94:0/617374214 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff97003c350 con 0x55b4bfb798e0 2024-04-21T08:46:33.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff968ff9640 1 -- 172.21.15.94:0/617374214 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff97c011980 con 0x55b4bfb798e0 2024-04-21T08:46:33.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff98a011740 10 client.17439 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:33.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff98a011740 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:33.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff98a011740 20 client.17439 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:33.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff98a011740 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b4bfbfa5d0 0x55b4bfbfc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff98a011740 4 client.17439 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:46:33.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.775+0000 7ff98a011740 1 -- 172.21.15.94:0/617374214 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b4bfae5790 con 0x55b4bfbfa5d0 2024-04-21T08:46:33.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.776+0000 7ff9848e6640 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b4bfbfa5d0 0x55b4bfbfc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.776+0000 7ff9848e6640 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b4bfbfa5d0 0x55b4bfbfc9f0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e68b1aa1d2e5e1a server_cookie=bd50d63116fc0e31 in_seq=0 out_seq=0 2024-04-21T08:46:33.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.776+0000 7ff968ff9640 10 client.17439 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:33.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff968ff9640 1 -- 172.21.15.94:0/617374214 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b4bfae5790 con 0x55b4bfbfa5d0 2024-04-21T08:46:33.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff968ff9640 10 client.17439 handle_command_reply: tid=0 2024-04-21T08:46:33.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 client.17439 shutdown 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 -- 172.21.15.94:0/617374214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb798e0 msgr2=0x55b4bfb8ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb798e0 0x55b4bfb8ade0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009370 tx=0x7ff97c013a90 comp rx=0 tx=0).stop 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 -- 172.21.15.94:0/617374214 shutdown_connections 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55b4bfbfa5d0 0x55b4bfbfc9f0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b4bfb82e10 0x55b4bfb87db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff968ff9640 0 client.17439 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4bfb7a290 0x55b4bfb8b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 --2- 172.21.15.94:0/617374214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b4bfb798e0 0x55b4bfb8ade0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 -- 172.21.15.94:0/617374214 >> 172.21.15.94:0/617374214 conn(0x55b4bfb73fb0 msgr2=0x55b4bfb80510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 -- 172.21.15.94:0/617374214 shutdown_connections 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 1 -- 172.21.15.94:0/617374214 wait complete. 2024-04-21T08:46:33.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.777+0000 7ff98a011740 20 client.17439 trim_cache size 0 max 16384 2024-04-21T08:46:33.784 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:46:33.784 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:46:33.784 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "f568425a-56ac-46ae-8a87-432dee7e731e", 2024-04-21T08:46:33.784 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:46:33.784 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 -- 172.21.15.94:0/1947659846 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff96c0a3d20 msgr2=0x7ff96c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff96c0a3d20 0x7ff96c0a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff9780058c0 tx=0x7ff9780046c0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 -- 172.21.15.94:0/1947659846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff980102e20 msgr2=0x7ff9801a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff980102e20 0x7ff9801a0be0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff970005940 tx=0x7ff97002c7e0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 -- 172.21.15.94:0/1947659846 shutdown_connections 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff96c0a3d20 0x7ff96c0a61e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff98010e810 0x7ff98019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff980102e20 0x7ff9801a0be0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 --2- 172.21.15.94:0/1947659846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff98006aa90 0x7ff9801a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 -- 172.21.15.94:0/1947659846 >> 172.21.15.94:0/1947659846 conn(0x7ff980076c50 msgr2=0x7ff98010f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 -- 172.21.15.94:0/1947659846 shutdown_connections 2024-04-21T08:46:33.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.778+0000 7ff98734b640 1 -- 172.21.15.94:0/1947659846 wait complete. 2024-04-21T08:46:33.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:33.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:33.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.889+0000 7f5b35c2e640 1 -- 172.21.15.94:0/3393650565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b30106fe0 msgr2=0x7f5b301073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.889+0000 7f5b35c2e640 1 --2- 172.21.15.94:0/3393650565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b30106fe0 0x7f5b301073c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f5b28009ca0 tx=0x7f5b2802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.889+0000 7f5b35c2e640 1 -- 172.21.15.94:0/3393650565 shutdown_connections 2024-04-21T08:46:33.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.889+0000 7f5b35c2e640 1 --2- 172.21.15.94:0/3393650565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b30101130 0x7f5b30101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.889+0000 7f5b35c2e640 1 --2- 172.21.15.94:0/3393650565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b30101ad0 0x7f5b3010dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.889+0000 7f5b35c2e640 1 --2- 172.21.15.94:0/3393650565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b30106fe0 0x7f5b301073c0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.889+0000 7f5b35c2e640 1 -- 172.21.15.94:0/3393650565 >> 172.21.15.94:0/3393650565 conn(0x7f5b30077e30 msgr2=0x7f5b30078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.889+0000 7f5b35c2e640 1 -- 172.21.15.94:0/3393650565 shutdown_connections 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.889+0000 7f5b35c2e640 1 -- 172.21.15.94:0/3393650565 wait complete. 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.889+0000 7f5b35c2e640 1 Processor -- start 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b35c2e640 1 -- start start 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b35c2e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b30101130 0x7f5b30198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b35c2e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b30101ad0 0x7f5b30198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b35c2e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b30106fe0 0x7f5b3019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b35c2e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5b301110c0 con 0x7f5b30101ad0 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b35c2e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5b30110f40 con 0x7f5b30106fe0 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b35c2e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b30111240 con 0x7f5b30101130 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b30101ad0 0x7f5b30198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b30101ad0 0x7f5b30198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38954/0 (socket says 172.21.15.94:38954) 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2effd640 1 -- 172.21.15.94:0/108554493 learned_addr learned my addr 172.21.15.94:0/108554493 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2f7fe640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b30101130 0x7f5b30198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b30106fe0 0x7f5b3019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2effd640 1 -- 172.21.15.94:0/108554493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b30101130 msgr2=0x7f5b30198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2effd640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b30101130 0x7f5b30198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2effd640 1 -- 172.21.15.94:0/108554493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b30106fe0 msgr2=0x7f5b3019cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2effd640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b30106fe0 0x7f5b3019cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2effd640 1 -- 172.21.15.94:0/108554493 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3019d6f0 con 0x7f5b30101ad0 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b30106fe0 0x7f5b3019cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2effd640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b30101ad0 0x7f5b30198c40 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800e9f0 tx=0x7f5b1800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2cff9640 1 -- 172.21.15.94:0/108554493 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1800cd80 con 0x7f5b30101ad0 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2cff9640 1 -- 172.21.15.94:0/108554493 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b180054d0 con 0x7f5b30101ad0 2024-04-21T08:46:33.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b35c2e640 1 -- 172.21.15.94:0/108554493 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b3019d9e0 con 0x7f5b30101ad0 2024-04-21T08:46:33.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.890+0000 7f5b2cff9640 1 -- 172.21.15.94:0/108554493 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b18010690 con 0x7f5b30101ad0 2024-04-21T08:46:33.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.891+0000 7f5b35c2e640 1 -- 172.21.15.94:0/108554493 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b301a5270 con 0x7f5b30101ad0 2024-04-21T08:46:33.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.892+0000 7f5b2cff9640 1 -- 172.21.15.94:0/108554493 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5b18004e90 con 0x7f5b30101ad0 2024-04-21T08:46:33.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b38817740 1 Processor -- start 2024-04-21T08:46:33.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b38817740 1 -- start start 2024-04-21T08:46:33.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b38817740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df5a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b38817740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 0x555b0df67950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b38817740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df67e90 0x555b0df6a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b38817740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x555b0debf1e0 con 0x555b0df5a2b0 2024-04-21T08:46:33.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b38817740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555b0de5c390 con 0x555b0df5abd0 2024-04-21T08:46:33.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b38817740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555b0debb1a0 con 0x555b0df67e90 2024-04-21T08:46:33.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df5a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b2cff9640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b1c0a3d20 0x7f5b1c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df5a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38958/0 (socket says 172.21.15.94:38958) 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b2ffff640 1 -- 172.21.15.94:0/2614492975 learned_addr learned my addr 172.21.15.94:0/2614492975 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b1c0a3d20 0x7f5b1c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b2f7fe640 1 --2- 172.21.15.94:0/2614492975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 0x555b0df67950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b2cff9640 1 -- 172.21.15.94:0/108554493 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5b180d1120 con 0x7f5b30101ad0 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.893+0000 7f5b2effd640 1 --2- 172.21.15.94:0/2614492975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df67e90 0x555b0df6a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b2ffff640 1 -- 172.21.15.94:0/2614492975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df67e90 msgr2=0x555b0df6a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/2614492975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df67e90 0x555b0df6a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b2ffff640 1 -- 172.21.15.94:0/2614492975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 msgr2=0x555b0df67950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/2614492975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 0x555b0df67950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b2ffff640 1 -- 172.21.15.94:0/2614492975 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b0dc9b7d0 con 0x555b0df5a2b0 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/2614492975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df5a690 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f5b20005fd0 tx=0x7f5b20015c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed149e4994f92311 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b1c0a3d20 0x7f5b1c0a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b20013770 tx=0x7f5b2001c000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b25ffb640 1 -- 172.21.15.94:0/2614492975 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20022070 con 0x555b0df5a2b0 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b25ffb640 1 -- 172.21.15.94:0/2614492975 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b20009970 con 0x555b0df5a2b0 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b25ffb640 1 -- 172.21.15.94:0/2614492975 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2001e690 con 0x555b0df5a2b0 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b38817740 1 -- 172.21.15.94:0/2614492975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 msgr2=0x555b0df5a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.894+0000 7f5b38817740 1 --2- 172.21.15.94:0/2614492975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df5a690 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f5b20005fd0 tx=0x7f5b20015c60 comp rx=0 tx=0).stop 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 -- 172.21.15.94:0/2614492975 shutdown_connections 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 --2- 172.21.15.94:0/2614492975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df67e90 0x555b0df6a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 --2- 172.21.15.94:0/2614492975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 0x555b0df67950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 --2- 172.21.15.94:0/2614492975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df5a690 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 -- 172.21.15.94:0/2614492975 >> 172.21.15.94:0/2614492975 conn(0x555b0df54cb0 msgr2=0x555b0df55090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 -- 172.21.15.94:0/2614492975 shutdown_connections 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 -- 172.21.15.94:0/2614492975 wait complete. 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 -- start start 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 Processor -- start 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df64f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5abd0 0x555b0df65440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df67e90 0x555b0df65980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x555b0df6abf0 con 0x555b0df5a2b0 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555b0de782f0 con 0x555b0df67e90 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555b0df6ad70 con 0x555b0df5abd0 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b38817740 10 client.0 _ll_register_callbacks cb 0x555b0de85580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b2f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df64f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b2f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df64f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38964/0 (socket says 172.21.15.94:38964) 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.895+0000 7f5b2f7fe640 1 -- 172.21.15.94:0/3043942369 learned_addr learned my addr 172.21.15.94:0/3043942369 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df67e90 0x555b0df65980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b2effd640 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5abd0 0x555b0df65440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b2f7fe640 1 -- 172.21.15.94:0/3043942369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5abd0 msgr2=0x555b0df65440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b2f7fe640 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5abd0 0x555b0df65440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b2f7fe640 1 -- 172.21.15.94:0/3043942369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df67e90 msgr2=0x555b0df65980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b2f7fe640 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df67e90 0x555b0df65980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b2f7fe640 1 -- 172.21.15.94:0/3043942369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b0dbd13f0 con 0x555b0df5a2b0 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df67e90 0x555b0df65980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b2f7fe640 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df64f00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f5b28005e30 tx=0x7f5b28009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d06988c83f8c002 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b0effd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b0effd640 1 -- 172.21.15.94:0/3043942369 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b28044070 con 0x555b0df5a2b0 2024-04-21T08:46:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b0effd640 1 -- 172.21.15.94:0/3043942369 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5b28032db0 con 0x555b0df5a2b0 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b38817740 1 -- 172.21.15.94:0/3043942369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b20009970 con 0x555b0df5a2b0 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b0effd640 1 -- 172.21.15.94:0/3043942369 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5b28050050 con 0x555b0df5a2b0 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b38817740 10 client.17451 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b38817740 1 -- 172.21.15.94:0/3043942369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555b0db910f0 con 0x555b0df5a2b0 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.896+0000 7f5b0effd640 1 -- 172.21.15.94:0/3043942369 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b20009970 con 0x555b0df5a2b0 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.897+0000 7f5b0effd640 1 -- 172.21.15.94:0/3043942369 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5b28004310 con 0x555b0df5a2b0 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.897+0000 7f5b38817740 10 client.17451 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.897+0000 7f5b38817740 10 client.17451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.897+0000 7f5b38817740 20 client.17451 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.897+0000 7f5b38817740 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b0df77e20 0x555b0df7a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.897+0000 7f5b38817740 4 client.17451 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.897+0000 7f5b38817740 1 -- 172.21.15.94:0/3043942369 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b0deccc00 con 0x555b0df77e20 2024-04-21T08:46:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.897+0000 7f5b2effd640 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b0df77e20 0x555b0df7a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.897+0000 7f5b0effd640 10 client.17451 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.897+0000 7f5b2effd640 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b0df77e20 0x555b0df7a250 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=837a886c353d05a0 server_cookie=8bc4f39e2f9df61 in_seq=0 out_seq=0 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b0effd640 1 -- 172.21.15.94:0/3043942369 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b0deccc00 con 0x555b0df77e20 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b0effd640 10 client.17451 handle_command_reply: tid=0 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 client.17451 shutdown 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 -- 172.21.15.94:0/3043942369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 msgr2=0x555b0df64f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df64f00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f5b28005e30 tx=0x7f5b28009890 comp rx=0 tx=0).stop 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 -- 172.21.15.94:0/3043942369 shutdown_connections 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b0df77e20 0x555b0df7a250 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5abd0 0x555b0df65440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b0effd640 0 client.17451 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df67e90 0x555b0df65980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 --2- 172.21.15.94:0/3043942369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df5a2b0 0x555b0df64f00 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 -- 172.21.15.94:0/3043942369 >> 172.21.15.94:0/3043942369 conn(0x555b0df54cb0 msgr2=0x555b0df56a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 -- 172.21.15.94:0/3043942369 shutdown_connections 2024-04-21T08:46:33.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 1 -- 172.21.15.94:0/3043942369 wait complete. 2024-04-21T08:46:33.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.898+0000 7f5b38817740 20 client.17451 trim_cache size 0 max 16384 2024-04-21T08:46:33.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.915+0000 7f5b38817740 1 Processor -- start 2024-04-21T08:46:33.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.915+0000 7f5b38817740 1 -- start start 2024-04-21T08:46:33.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b38817740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5a2b0 0x555b0df67630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b38817740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5abd0 0x555b0df73af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b38817740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df64060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b38817740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5b301afab0 con 0x555b0df67e90 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b38817740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5b301ad380 con 0x555b0df5a2b0 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b38817740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b301ab1d0 con 0x555b0df5abd0 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df64060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df64060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38972/0 (socket says 172.21.15.94:38972) 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b2ffff640 1 -- 172.21.15.94:0/2243289652 learned_addr learned my addr 172.21.15.94:0/2243289652 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b2ffff640 1 -- 172.21.15.94:0/2243289652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5abd0 msgr2=0x555b0df73af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/2243289652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5abd0 0x555b0df73af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b2ffff640 1 -- 172.21.15.94:0/2243289652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5a2b0 msgr2=0x555b0df67630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/2243289652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5a2b0 0x555b0df67630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b2ffff640 1 -- 172.21.15.94:0/2243289652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b0df5aa00 con 0x555b0df67e90 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/2243289652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df64060 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f5b200058c0 tx=0x7f5b20009400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ae28841916accd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b257fa640 1 -- 172.21.15.94:0/2243289652 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20022070 con 0x555b0df67e90 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b257fa640 1 -- 172.21.15.94:0/2243289652 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b20013e10 con 0x555b0df67e90 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.916+0000 7f5b257fa640 1 -- 172.21.15.94:0/2243289652 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b200152a0 con 0x555b0df67e90 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 -- 172.21.15.94:0/2243289652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 msgr2=0x555b0df64060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 --2- 172.21.15.94:0/2243289652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df64060 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f5b200058c0 tx=0x7f5b20009400 comp rx=0 tx=0).stop 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 -- 172.21.15.94:0/2243289652 shutdown_connections 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 --2- 172.21.15.94:0/2243289652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5abd0 0x555b0df73af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 --2- 172.21.15.94:0/2243289652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df64060 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 --2- 172.21.15.94:0/2243289652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5a2b0 0x555b0df67630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 -- 172.21.15.94:0/2243289652 >> 172.21.15.94:0/2243289652 conn(0x555b0df54cb0 msgr2=0x555b0df56090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 -- 172.21.15.94:0/2243289652 shutdown_connections 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 -- 172.21.15.94:0/2243289652 wait complete. 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 -- start start 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.917+0000 7f5b38817740 1 Processor -- start 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b38817740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5a2b0 0x555b0df6ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b38817740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 0x555b0df6bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b38817740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df6c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b38817740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5b301ab840 con 0x555b0df67e90 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b38817740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5b301accb0 con 0x555b0df5abd0 2024-04-21T08:46:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b38817740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b301abf10 con 0x555b0df5a2b0 2024-04-21T08:46:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b38817740 10 client.0 _ll_register_callbacks cb 0x7f5b18005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df6c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df6c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38976/0 (socket says 172.21.15.94:38976) 2024-04-21T08:46:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2ffff640 1 -- 172.21.15.94:0/1594747195 learned_addr learned my addr 172.21.15.94:0/1594747195 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:46:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2effd640 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 0x555b0df6bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2f7fe640 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5a2b0 0x555b0df6ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2ffff640 1 -- 172.21.15.94:0/1594747195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5a2b0 msgr2=0x555b0df6ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5a2b0 0x555b0df6ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2ffff640 1 -- 172.21.15.94:0/1594747195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 msgr2=0x555b0df6bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 0x555b0df6bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2ffff640 1 -- 172.21.15.94:0/1594747195 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b0dfdd390 con 0x555b0df67e90 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2f7fe640 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5a2b0 0x555b0df6ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2effd640 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 0x555b0df6bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b2ffff640 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df6c4a0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f5b20010f80 tx=0x7f5b20005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d279c5672753d20 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b0effd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b0effd640 1 -- 172.21.15.94:0/1594747195 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20036070 con 0x555b0df67e90 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b0effd640 1 -- 172.21.15.94:0/1594747195 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5b200106e0 con 0x555b0df67e90 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.918+0000 7f5b38817740 1 -- 172.21.15.94:0/1594747195 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b20013e10 con 0x555b0df67e90 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b0effd640 1 -- 172.21.15.94:0/1594747195 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5b20022050 con 0x555b0df67e90 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b38817740 10 client.17457 fetch_fsmap learned FSMap version 24 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b38817740 1 -- 172.21.15.94:0/1594747195 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555b0dfde680 con 0x555b0df67e90 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b0effd640 1 -- 172.21.15.94:0/1594747195 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b20013e10 con 0x555b0df67e90 2024-04-21T08:46:33.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b0effd640 1 -- 172.21.15.94:0/1594747195 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5b2001d070 con 0x555b0df67e90 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b38817740 10 client.17457 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b38817740 10 client.17457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b38817740 20 client.17457 populate_metadata read hostname 'smithi094' 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b38817740 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b0dfdec30 0x555b0dfe1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b38817740 4 client.17457 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b38817740 1 -- 172.21.15.94:0/1594747195 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555b0dec6a70 con 0x555b0dfdec30 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.919+0000 7f5b2f7fe640 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b0dfdec30 0x555b0dfe1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b2f7fe640 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b0dfdec30 0x555b0dfe1050 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7459662ee123562d server_cookie=645bb06fedced1c in_seq=0 out_seq=0 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b0effd640 10 client.17457 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b0effd640 1 -- 172.21.15.94:0/1594747195 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555b0dec6a70 con 0x555b0dfdec30 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b0effd640 10 client.17457 handle_command_reply: tid=0 2024-04-21T08:46:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 client.17457 shutdown 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 -- 172.21.15.94:0/1594747195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 msgr2=0x555b0df6c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df6c4a0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f5b20010f80 tx=0x7f5b20005b00 comp rx=0 tx=0).stop 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 -- 172.21.15.94:0/1594747195 shutdown_connections 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x555b0dfdec30 0x555b0dfe1050 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x555b0df67e90 0x555b0df6c4a0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555b0df5abd0 0x555b0df6bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b0effd640 0 client.17457 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 --2- 172.21.15.94:0/1594747195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b0df5a2b0 0x555b0df6ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 -- 172.21.15.94:0/1594747195 >> 172.21.15.94:0/1594747195 conn(0x555b0df54cb0 msgr2=0x555b0df534e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 -- 172.21.15.94:0/1594747195 shutdown_connections 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 1 -- 172.21.15.94:0/1594747195 wait complete. 2024-04-21T08:46:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.920+0000 7f5b38817740 20 client.17457 trim_cache size 0 max 16384 2024-04-21T08:46:33.934 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:46:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:46:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:46:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "f568425a-56ac-46ae-8a87-432dee7e731e": { 2024-04-21T08:46:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:46:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "f568425a-56ac-46ae-8a87-432dee7e731e", 2024-04-21T08:46:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:46:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:46:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:46:33.934 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 -- 172.21.15.94:0/108554493 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b1c0a3d20 msgr2=0x7f5b1c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b1c0a3d20 0x7f5b1c0a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b20013770 tx=0x7f5b2001c000 comp rx=0 tx=0).stop 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 -- 172.21.15.94:0/108554493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b30101ad0 msgr2=0x7f5b30198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b30101ad0 0x7f5b30198c40 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800e9f0 tx=0x7f5b1800edf0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 -- 172.21.15.94:0/108554493 shutdown_connections 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5b1c0a3d20 0x7f5b1c0a61e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b30106fe0 0x7f5b3019cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b30101ad0 0x7f5b30198c40 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 --2- 172.21.15.94:0/108554493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b30101130 0x7f5b30198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 -- 172.21.15.94:0/108554493 >> 172.21.15.94:0/108554493 conn(0x7f5b30077e30 msgr2=0x7f5b3006a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 -- 172.21.15.94:0/108554493 shutdown_connections 2024-04-21T08:46:33.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:46:33.921+0000 7f5b35c2e640 1 -- 172.21.15.94:0/108554493 wait complete. 2024-04-21T08:46:33.937 INFO:tasks.cephfs.filesystem:scrub status for tag:f568425a-56ac-46ae-8a87-432dee7e731e - {'path': '/', 'tag': 'f568425a-56ac-46ae-8a87-432dee7e731e', 'options': 'recursive,force'} 2024-04-21T08:46:35.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:35 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:46:35.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:35 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:35.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:35 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:46:35.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:35 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:35.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:35 smithi111 ceph-mon[31493]: pgmap v201: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:46:35.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:35 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:46:35.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:35 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:35.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:35 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:46:35.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:35 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:35.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:35 smithi094 ceph-mon[27988]: pgmap v201: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:46:35.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:35 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:46:35.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:35 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:35.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:35 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:46:35.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:35 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:35.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:35 smithi173 ceph-mon[31428]: pgmap v201: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:46:37.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:37 smithi111 ceph-mon[31493]: pgmap v202: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:46:37.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:37 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:46:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:37 smithi094 ceph-mon[27988]: pgmap v202: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:46:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:37 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:46:37.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:37 smithi173 ceph-mon[31428]: pgmap v202: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:46:37.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:37 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:46:39.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:39 smithi111 ceph-mon[31493]: pgmap v203: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:46:39.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:39 smithi094 ceph-mon[27988]: pgmap v203: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:46:39.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:39 smithi173 ceph-mon[31428]: pgmap v203: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:46:41.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:41 smithi111 ceph-mon[31493]: pgmap v204: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:46:41.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:41 smithi094 ceph-mon[27988]: pgmap v204: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:46:41.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:41 smithi173 ceph-mon[31428]: pgmap v204: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:46:43.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:43 smithi111 ceph-mon[31493]: pgmap v205: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:46:43.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:43 smithi094 ceph-mon[27988]: pgmap v205: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:46:43.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:43 smithi173 ceph-mon[31428]: pgmap v205: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:46:45.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:45 smithi111 ceph-mon[31493]: pgmap v206: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:46:45.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:45 smithi094 ceph-mon[27988]: pgmap v206: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:46:45.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:45 smithi173 ceph-mon[31428]: pgmap v206: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:46:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:47 smithi111 ceph-mon[31493]: pgmap v207: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:46:47.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:47 smithi094 ceph-mon[27988]: pgmap v207: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:46:47.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:47 smithi173 ceph-mon[31428]: pgmap v207: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:46:49.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:49 smithi111 ceph-mon[31493]: pgmap v208: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:49.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:49 smithi094 ceph-mon[27988]: pgmap v208: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:49.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:49 smithi173 ceph-mon[31428]: pgmap v208: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:51.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:51 smithi111 ceph-mon[31493]: pgmap v209: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:46:51.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:51 smithi094 ceph-mon[27988]: pgmap v209: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:46:51.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:51 smithi173 ceph-mon[31428]: pgmap v209: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:46:53.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:53 smithi111 ceph-mon[31493]: pgmap v210: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:53.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:53 smithi094 ceph-mon[27988]: pgmap v210: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:53.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:53 smithi173 ceph-mon[31428]: pgmap v210: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:55 smithi111 ceph-mon[31493]: pgmap v211: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:46:55.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:55 smithi094 ceph-mon[27988]: pgmap v211: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:46:55.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:55 smithi173 ceph-mon[31428]: pgmap v211: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:46:57.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:57 smithi111 ceph-mon[31493]: pgmap v212: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:46:57.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:57 smithi094 ceph-mon[27988]: pgmap v212: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:46:57.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:57 smithi173 ceph-mon[31428]: pgmap v212: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:46:59.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:46:59 smithi111 ceph-mon[31493]: pgmap v213: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:59.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:46:59 smithi094 ceph-mon[27988]: pgmap v213: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:46:59.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:46:59 smithi173 ceph-mon[31428]: pgmap v213: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:01.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:01 smithi111 ceph-mon[31493]: pgmap v214: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:47:01.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:01 smithi094 ceph-mon[27988]: pgmap v214: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:47:01.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:01 smithi173 ceph-mon[31428]: pgmap v214: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:47:03.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:03 smithi111 ceph-mon[31493]: pgmap v215: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:03.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:03 smithi094 ceph-mon[27988]: pgmap v215: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:03.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:03 smithi173 ceph-mon[31428]: pgmap v215: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:03.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:47:03.938 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:47:04.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.052+0000 7f4b435b6640 1 -- 172.21.15.94:0/1621859061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b3c074380 msgr2=0x7f4b3c074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.052+0000 7f4b435b6640 1 --2- 172.21.15.94:0/1621859061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b3c074380 0x7f4b3c074760 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f4b38009900 tx=0x7f4b3802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.052+0000 7f4b435b6640 1 -- 172.21.15.94:0/1621859061 shutdown_connections 2024-04-21T08:47:04.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.052+0000 7f4b435b6640 1 --2- 172.21.15.94:0/1621859061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b3c074ca0 0x7f4b3c10e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.052+0000 7f4b435b6640 1 --2- 172.21.15.94:0/1621859061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b3c10e770 0x7f4b3c110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.052+0000 7f4b435b6640 1 --2- 172.21.15.94:0/1621859061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b3c074380 0x7f4b3c074760 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.052+0000 7f4b435b6640 1 -- 172.21.15.94:0/1621859061 >> 172.21.15.94:0/1621859061 conn(0x7f4b3c0fda40 msgr2=0x7f4b3c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.052+0000 7f4b435b6640 1 -- 172.21.15.94:0/1621859061 shutdown_connections 2024-04-21T08:47:04.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.052+0000 7f4b435b6640 1 -- 172.21.15.94:0/1621859061 wait complete. 2024-04-21T08:47:04.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b435b6640 1 Processor -- start 2024-04-21T08:47:04.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b435b6640 1 -- start start 2024-04-21T08:47:04.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b435b6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b3c074380 0x7f4b3c1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b435b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b3c074ca0 0x7f4b3c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b435b6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b3c10e770 0x7f4b3c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b435b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c114400 con 0x7f4b3c074ca0 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b435b6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c114280 con 0x7f4b3c10e770 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b435b6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c114580 con 0x7f4b3c074380 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b40b51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b3c074ca0 0x7f4b3c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b40b51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b3c074ca0 0x7f4b3c1a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49040/0 (socket says 172.21.15.94:49040) 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b40b51640 1 -- 172.21.15.94:0/4170678508 learned_addr learned my addr 172.21.15.94:0/4170678508 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.053+0000 7f4b41352640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b3c074380 0x7f4b3c1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b41b53640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b3c10e770 0x7f4b3c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b40b51640 1 -- 172.21.15.94:0/4170678508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b3c074380 msgr2=0x7f4b3c1a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b40b51640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b3c074380 0x7f4b3c1a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b40b51640 1 -- 172.21.15.94:0/4170678508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b3c10e770 msgr2=0x7f4b3c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b40b51640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b3c10e770 0x7f4b3c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b40b51640 1 -- 172.21.15.94:0/4170678508 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3c19b040 con 0x7f4b3c074ca0 2024-04-21T08:47:04.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b41b53640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b3c10e770 0x7f4b3c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:04.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b40b51640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b3c074ca0 0x7f4b3c1a0bb0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c00c920 tx=0x7f4b2c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b327fc640 1 -- 172.21.15.94:0/4170678508 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c007c70 con 0x7f4b3c074ca0 2024-04-21T08:47:04.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b327fc640 1 -- 172.21.15.94:0/4170678508 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b2c007e10 con 0x7f4b3c074ca0 2024-04-21T08:47:04.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b435b6640 1 -- 172.21.15.94:0/4170678508 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b3c19b330 con 0x7f4b3c074ca0 2024-04-21T08:47:04.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b327fc640 1 -- 172.21.15.94:0/4170678508 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c018820 con 0x7f4b3c074ca0 2024-04-21T08:47:04.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.054+0000 7f4b435b6640 1 -- 172.21.15.94:0/4170678508 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b3c1a75f0 con 0x7f4b3c074ca0 2024-04-21T08:47:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.056+0000 7f4b327fc640 1 -- 172.21.15.94:0/4170678508 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4b2c00f450 con 0x7f4b3c074ca0 2024-04-21T08:47:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.057+0000 7f4b461a9740 1 Processor -- start 2024-04-21T08:47:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.057+0000 7f4b461a9740 1 -- start start 2024-04-21T08:47:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.057+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18580 0x556bd9a18960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.057+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18ea0 0x556bd9a21540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.057+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 0x556bd9a23e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.057+0000 7f4b461a9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556bd997d1e0 con 0x556bd9a18580 2024-04-21T08:47:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.057+0000 7f4b461a9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556bd991a390 con 0x556bd9a18ea0 2024-04-21T08:47:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.057+0000 7f4b461a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556bd99791a0 con 0x556bd9a21a80 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.057+0000 7f4b327fc640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4b280a3d20 0x7f4b280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.057+0000 7f4b41b53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18580 0x556bd9a18960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b40b51640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 0x556bd9a23e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41b53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18580 0x556bd9a18960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49044/0 (socket says 172.21.15.94:49044) 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41b53640 1 -- 172.21.15.94:0/3733926554 learned_addr learned my addr 172.21.15.94:0/3733926554 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41b53640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4b280a3d20 0x7f4b280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41352640 1 --2- 172.21.15.94:0/3733926554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18ea0 0x556bd9a21540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b327fc640 1 -- 172.21.15.94:0/4170678508 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4b2c0d0d70 con 0x7f4b3c074ca0 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41b53640 1 -- 172.21.15.94:0/3733926554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 msgr2=0x556bd9a23e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41b53640 1 --2- 172.21.15.94:0/3733926554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 0x556bd9a23e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41b53640 1 -- 172.21.15.94:0/3733926554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18ea0 msgr2=0x556bd9a21540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41b53640 1 --2- 172.21.15.94:0/3733926554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18ea0 0x556bd9a21540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41b53640 1 -- 172.21.15.94:0/3733926554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bd97597d0 con 0x556bd9a18580 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41b53640 1 --2- 172.21.15.94:0/3733926554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18580 0x556bd9a18960 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400ed70 tx=0x7f4b3400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b168a939d8bc6a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b41b53640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4b280a3d20 0x7f4b280a61e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400cc40 tx=0x7f4b34019000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b277fe640 1 -- 172.21.15.94:0/3733926554 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34018b70 con 0x556bd9a18580 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b277fe640 1 -- 172.21.15.94:0/3733926554 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b34018d10 con 0x556bd9a18580 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.058+0000 7f4b277fe640 1 -- 172.21.15.94:0/3733926554 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b340233a0 con 0x556bd9a18580 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 -- 172.21.15.94:0/3733926554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18580 msgr2=0x556bd9a18960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 --2- 172.21.15.94:0/3733926554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18580 0x556bd9a18960 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400ed70 tx=0x7f4b3400c740 comp rx=0 tx=0).stop 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 -- 172.21.15.94:0/3733926554 shutdown_connections 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 --2- 172.21.15.94:0/3733926554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 0x556bd9a23e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 --2- 172.21.15.94:0/3733926554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18ea0 0x556bd9a21540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 --2- 172.21.15.94:0/3733926554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18580 0x556bd9a18960 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 -- 172.21.15.94:0/3733926554 >> 172.21.15.94:0/3733926554 conn(0x556bd9a12cb0 msgr2=0x556bd9a13090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 -- 172.21.15.94:0/3733926554 shutdown_connections 2024-04-21T08:47:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 -- 172.21.15.94:0/3733926554 wait complete. 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 -- start start 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.059+0000 7f4b461a9740 1 Processor -- start 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a18580 0x556bd9a1eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18ea0 0x556bd9a1f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a21a80 0x556bd9a1f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b461a9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556bd9a24810 con 0x556bd9a21a80 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b461a9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556bd99362f0 con 0x556bd9a18ea0 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b461a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556bd9a24990 con 0x556bd9a18580 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b461a9740 10 client.0 _ll_register_callbacks cb 0x556bd9943580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b41b53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a21a80 0x556bd9a1f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b41b53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a21a80 0x556bd9a1f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49046/0 (socket says 172.21.15.94:49046) 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b41b53640 1 -- 172.21.15.94:0/4005784678 learned_addr learned my addr 172.21.15.94:0/4005784678 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b41352640 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a18580 0x556bd9a1eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b40b51640 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18ea0 0x556bd9a1f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b41b53640 1 -- 172.21.15.94:0/4005784678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a18580 msgr2=0x556bd9a1eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b41b53640 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a18580 0x556bd9a1eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b41b53640 1 -- 172.21.15.94:0/4005784678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18ea0 msgr2=0x556bd9a1f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b41b53640 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18ea0 0x556bd9a1f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b41b53640 1 -- 172.21.15.94:0/4005784678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bd968f3f0 con 0x556bd9a21a80 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.060+0000 7f4b41352640 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a18580 0x556bd9a1eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b41b53640 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a21a80 0x556bd9a1f5c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4b34004a80 tx=0x7f4b34018ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13193f54f0b522d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b24ff9640 1 -- 172.21.15.94:0/4005784678 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34028070 con 0x556bd9a21a80 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b24ff9640 1 -- 172.21.15.94:0/4005784678 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4b3402d460 con 0x556bd9a21a80 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b461a9740 1 -- 172.21.15.94:0/4005784678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b34018d10 con 0x556bd9a21a80 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b24ff9640 1 -- 172.21.15.94:0/4005784678 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4b34017040 con 0x556bd9a21a80 2024-04-21T08:47:04.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b461a9740 10 client.17469 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b461a9740 1 -- 172.21.15.94:0/4005784678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556bd964f0f0 con 0x556bd9a21a80 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b24ff9640 1 -- 172.21.15.94:0/4005784678 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b34018d10 con 0x556bd9a21a80 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b24ff9640 1 -- 172.21.15.94:0/4005784678 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4b34024590 con 0x556bd9a21a80 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b461a9740 10 client.17469 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b461a9740 10 client.17469 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b461a9740 20 client.17469 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b461a9740 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bd9a31aa0 0x556bd9a33e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b461a9740 4 client.17469 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.061+0000 7f4b461a9740 1 -- 172.21.15.94:0/4005784678 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bd998ac00 con 0x556bd9a31aa0 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.062+0000 7f4b41352640 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bd9a31aa0 0x556bd9a33e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.062+0000 7f4b41352640 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bd9a31aa0 0x556bd9a33e90 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80b1b98925530c27 server_cookie=31de5fc9b54c8332 in_seq=0 out_seq=0 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.062+0000 7f4b24ff9640 10 client.17469 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b24ff9640 1 -- 172.21.15.94:0/4005784678 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556bd998ac00 con 0x556bd9a31aa0 2024-04-21T08:47:04.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b24ff9640 10 client.17469 handle_command_reply: tid=0 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 client.17469 shutdown 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 -- 172.21.15.94:0/4005784678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a21a80 msgr2=0x556bd9a1f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a21a80 0x556bd9a1f5c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4b34004a80 tx=0x7f4b34018ec0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 -- 172.21.15.94:0/4005784678 shutdown_connections 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bd9a31aa0 0x556bd9a33e90 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a21a80 0x556bd9a1f5c0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18ea0 0x556bd9a1f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b24ff9640 0 client.17469 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 --2- 172.21.15.94:0/4005784678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a18580 0x556bd9a1eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 -- 172.21.15.94:0/4005784678 >> 172.21.15.94:0/4005784678 conn(0x556bd9a12cb0 msgr2=0x556bd9a14a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 -- 172.21.15.94:0/4005784678 shutdown_connections 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 1 -- 172.21.15.94:0/4005784678 wait complete. 2024-04-21T08:47:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.063+0000 7f4b461a9740 20 client.17469 trim_cache size 0 max 16384 2024-04-21T08:47:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.080+0000 7f4b461a9740 1 Processor -- start 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.080+0000 7f4b461a9740 1 -- start start 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18580 0x556bd9a21270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a2d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 0x556bd9a11bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b461a9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c1b3ee0 con 0x556bd9a18ea0 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b461a9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c1b17b0 con 0x556bd9a18580 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b461a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c1af600 con 0x556bd9a21a80 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b40b51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a2d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b40b51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a2d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49060/0 (socket says 172.21.15.94:49060) 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b40b51640 1 -- 172.21.15.94:0/2518674023 learned_addr learned my addr 172.21.15.94:0/2518674023 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b40b51640 1 -- 172.21.15.94:0/2518674023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 msgr2=0x556bd9a11bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b40b51640 1 --2- 172.21.15.94:0/2518674023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 0x556bd9a11bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b40b51640 1 -- 172.21.15.94:0/2518674023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18580 msgr2=0x556bd9a21270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b41352640 1 --2- 172.21.15.94:0/2518674023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18580 0x556bd9a21270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b40b51640 1 --2- 172.21.15.94:0/2518674023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18580 0x556bd9a21270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b40b51640 1 -- 172.21.15.94:0/2518674023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bd9a18cd0 con 0x556bd9a18ea0 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b41352640 1 --2- 172.21.15.94:0/2518674023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18580 0x556bd9a21270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b40b51640 1 --2- 172.21.15.94:0/2518674023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a2d6d0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c09c860 tx=0x7f4b2c087040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef393abd6fb7e2f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b26ffd640 1 -- 172.21.15.94:0/2518674023 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c09c8d0 con 0x556bd9a18ea0 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b26ffd640 1 -- 172.21.15.94:0/2518674023 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b2c0a0c80 con 0x556bd9a18ea0 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.081+0000 7f4b26ffd640 1 -- 172.21.15.94:0/2518674023 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c00f860 con 0x556bd9a18ea0 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 -- 172.21.15.94:0/2518674023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 msgr2=0x556bd9a2d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 --2- 172.21.15.94:0/2518674023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a2d6d0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c09c860 tx=0x7f4b2c087040 comp rx=0 tx=0).stop 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 -- 172.21.15.94:0/2518674023 shutdown_connections 2024-04-21T08:47:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 --2- 172.21.15.94:0/2518674023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 0x556bd9a11bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 --2- 172.21.15.94:0/2518674023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a2d6d0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 --2- 172.21.15.94:0/2518674023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18580 0x556bd9a21270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 -- 172.21.15.94:0/2518674023 >> 172.21.15.94:0/2518674023 conn(0x556bd9a12cb0 msgr2=0x556bd9a114e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 -- 172.21.15.94:0/2518674023 shutdown_connections 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 -- 172.21.15.94:0/2518674023 wait complete. 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 -- start start 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.082+0000 7f4b461a9740 1 Processor -- start 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18580 0x556bd9a25830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a25d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b461a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 0x556bd9a262b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b461a9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c1afc70 con 0x556bd9a18ea0 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b461a9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c1b10e0 con 0x556bd9a18580 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b461a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c1b0340 con 0x556bd9a21a80 2024-04-21T08:47:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b461a9740 10 client.0 _ll_register_callbacks cb 0x7f4b2c00cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b40b51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a25d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b40b51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a25d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49066/0 (socket says 172.21.15.94:49066) 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b40b51640 1 -- 172.21.15.94:0/256717926 learned_addr learned my addr 172.21.15.94:0/256717926 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b40b51640 1 -- 172.21.15.94:0/256717926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 msgr2=0x556bd9a262b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b40b51640 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 0x556bd9a262b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b40b51640 1 -- 172.21.15.94:0/256717926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18580 msgr2=0x556bd9a25830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b40b51640 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18580 0x556bd9a25830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b40b51640 1 -- 172.21.15.94:0/256717926 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bd9a96fb0 con 0x556bd9a18ea0 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b40b51640 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a25d70 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c00fea0 tx=0x7f4b2c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d85451f5ae6d231 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b24ff9640 1 -- 172.21.15.94:0/256717926 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c051070 con 0x556bd9a18ea0 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b24ff9640 1 -- 172.21.15.94:0/256717926 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f4b2c0870b0 con 0x556bd9a18ea0 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.083+0000 7f4b461a9740 1 -- 172.21.15.94:0/256717926 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b2c0a0c80 con 0x556bd9a18ea0 2024-04-21T08:47:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b24ff9640 1 -- 172.21.15.94:0/256717926 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4b2c0ce500 con 0x556bd9a18ea0 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b461a9740 10 client.17475 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b461a9740 1 -- 172.21.15.94:0/256717926 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556bd9a98240 con 0x556bd9a18ea0 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b24ff9640 1 -- 172.21.15.94:0/256717926 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b2c0a0c80 con 0x556bd9a18ea0 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b24ff9640 1 -- 172.21.15.94:0/256717926 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4b2c098620 con 0x556bd9a18ea0 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b461a9740 10 client.17475 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b461a9740 10 client.17475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b461a9740 20 client.17475 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b461a9740 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bd9a98920 0x556bd9a9ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b461a9740 4 client.17475 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b461a9740 1 -- 172.21.15.94:0/256717926 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556bd9984a70 con 0x556bd9a98920 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.084+0000 7f4b41352640 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bd9a98920 0x556bd9a9ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b24ff9640 10 client.17475 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b41352640 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bd9a98920 0x556bd9a9ad60 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3fe251d947798d8 server_cookie=7d4b17bf24dda49f in_seq=0 out_seq=0 2024-04-21T08:47:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b24ff9640 1 -- 172.21.15.94:0/256717926 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556bd9984a70 con 0x556bd9a98920 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b24ff9640 10 client.17475 handle_command_reply: tid=0 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 client.17475 shutdown 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 -- 172.21.15.94:0/256717926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 msgr2=0x556bd9a25d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a25d70 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c00fea0 tx=0x7f4b2c00f740 comp rx=0 tx=0).stop 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 -- 172.21.15.94:0/256717926 shutdown_connections 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bd9a98920 0x556bd9a9ad60 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bd9a21a80 0x556bd9a262b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bd9a18ea0 0x556bd9a25d70 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b24ff9640 0 client.17475 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 --2- 172.21.15.94:0/256717926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bd9a18580 0x556bd9a25830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 -- 172.21.15.94:0/256717926 >> 172.21.15.94:0/256717926 conn(0x556bd9a12cb0 msgr2=0x556bd9a14840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 -- 172.21.15.94:0/256717926 shutdown_connections 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 1 -- 172.21.15.94:0/256717926 wait complete. 2024-04-21T08:47:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.085+0000 7f4b461a9740 20 client.17475 trim_cache size 0 max 16384 2024-04-21T08:47:04.092 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:47:04.092 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:47:04.092 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:47:04.092 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 -- 172.21.15.94:0/4170678508 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4b280a3d20 msgr2=0x7f4b280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4b280a3d20 0x7f4b280a61e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400cc40 tx=0x7f4b34019000 comp rx=0 tx=0).stop 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 -- 172.21.15.94:0/4170678508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b3c074ca0 msgr2=0x7f4b3c1a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b3c074ca0 0x7f4b3c1a0bb0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c00c920 tx=0x7f4b2c00cde0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 -- 172.21.15.94:0/4170678508 shutdown_connections 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4b280a3d20 0x7f4b280a61e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b3c10e770 0x7f4b3c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b3c074ca0 0x7f4b3c1a0bb0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 --2- 172.21.15.94:0/4170678508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b3c074380 0x7f4b3c1a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 -- 172.21.15.94:0/4170678508 >> 172.21.15.94:0/4170678508 conn(0x7f4b3c0fda40 msgr2=0x7f4b3c0ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 -- 172.21.15.94:0/4170678508 shutdown_connections 2024-04-21T08:47:04.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.086+0000 7f4b435b6640 1 -- 172.21.15.94:0/4170678508 wait complete. 2024-04-21T08:47:04.098 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:47:04.098 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 -- 172.21.15.94:0/4115819315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44600690e0 msgr2=0x7f4460069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 --2- 172.21.15.94:0/4115819315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44600690e0 0x7f4460069560 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f444c009900 tx=0x7f444c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 -- 172.21.15.94:0/4115819315 shutdown_connections 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 --2- 172.21.15.94:0/4115819315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4460069aa0 0x7f446010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 --2- 172.21.15.94:0/4115819315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44600690e0 0x7f4460069560 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 --2- 172.21.15.94:0/4115819315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4460106f50 0x7f4460107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 -- 172.21.15.94:0/4115819315 >> 172.21.15.94:0/4115819315 conn(0x7f4460077e30 msgr2=0x7f4460078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 -- 172.21.15.94:0/4115819315 shutdown_connections 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 -- 172.21.15.94:0/4115819315 wait complete. 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 Processor -- start 2024-04-21T08:47:04.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 -- start start 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44600690e0 0x7f4460198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4460069aa0 0x7f4460198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4460106f50 0x7f446019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4460112150 con 0x7f4460106f50 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4460111fd0 con 0x7f4460069aa0 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f4466ad2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f44601122d0 con 0x7f44600690e0 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f446506f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4460106f50 0x7f446019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.198+0000 7f446506f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4460106f50 0x7f446019cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49070/0 (socket says 172.21.15.94:49070) 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f446506f640 1 -- 172.21.15.94:0/634489961 learned_addr learned my addr 172.21.15.94:0/634489961 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f445ffff640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4460069aa0 0x7f4460198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f446486e640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44600690e0 0x7f4460198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f446506f640 1 -- 172.21.15.94:0/634489961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44600690e0 msgr2=0x7f4460198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f446506f640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44600690e0 0x7f4460198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f446506f640 1 -- 172.21.15.94:0/634489961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4460069aa0 msgr2=0x7f4460198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f446506f640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4460069aa0 0x7f4460198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f446506f640 1 -- 172.21.15.94:0/634489961 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446019d5d0 con 0x7f4460106f50 2024-04-21T08:47:04.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f446506f640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4460106f50 0x7f446019cea0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f445800cce0 tx=0x7f445800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f445dffb640 1 -- 172.21.15.94:0/634489961 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445800f450 con 0x7f4460106f50 2024-04-21T08:47:04.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f4466ad2640 1 -- 172.21.15.94:0/634489961 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f446019d8c0 con 0x7f4460106f50 2024-04-21T08:47:04.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f445dffb640 1 -- 172.21.15.94:0/634489961 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f445800faa0 con 0x7f4460106f50 2024-04-21T08:47:04.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f445dffb640 1 -- 172.21.15.94:0/634489961 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4458018b70 con 0x7f4460106f50 2024-04-21T08:47:04.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.199+0000 7f4466ad2640 1 -- 172.21.15.94:0/634489961 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44601a5150 con 0x7f4460106f50 2024-04-21T08:47:04.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.200+0000 7f4466ad2640 1 -- 172.21.15.94:0/634489961 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4460068de0 con 0x7f4460106f50 2024-04-21T08:47:04.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.201+0000 7f445dffb640 1 -- 172.21.15.94:0/634489961 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4458018d10 con 0x7f4460106f50 2024-04-21T08:47:04.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.202+0000 7f445dffb640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44500a3d20 0x7f44500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.202+0000 7f446486e640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44500a3d20 0x7f44500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.202+0000 7f445dffb640 1 -- 172.21.15.94:0/634489961 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f44580d1040 con 0x7f4460106f50 2024-04-21T08:47:04.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.202+0000 7f446486e640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44500a3d20 0x7f44500a61e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4454005fd0 tx=0x7f4454005850 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.207+0000 7f445dffb640 1 -- 172.21.15.94:0/634489961 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4458014030 con 0x7f4460106f50 2024-04-21T08:47:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.425+0000 7f4466ad2640 1 -- 172.21.15.94:0/634489961 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f446010a500 con 0x7f4460106f50 2024-04-21T08:47:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.425+0000 7f445dffb640 1 -- 172.21.15.94:0/634489961 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f445809d300 con 0x7f4460106f50 2024-04-21T08:47:04.426 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:47:04.426 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:47:04.426 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:47:04.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 -- 172.21.15.94:0/634489961 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44500a3d20 msgr2=0x7f44500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44500a3d20 0x7f44500a61e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4454005fd0 tx=0x7f4454005850 comp rx=0 tx=0).stop 2024-04-21T08:47:04.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 -- 172.21.15.94:0/634489961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4460106f50 msgr2=0x7f446019cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4460106f50 0x7f446019cea0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f445800cce0 tx=0x7f445800a620 comp rx=0 tx=0).stop 2024-04-21T08:47:04.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 -- 172.21.15.94:0/634489961 shutdown_connections 2024-04-21T08:47:04.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44500a3d20 0x7f44500a61e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4460106f50 0x7f446019cea0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4460069aa0 0x7f4460198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 --2- 172.21.15.94:0/634489961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44600690e0 0x7f4460198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 -- 172.21.15.94:0/634489961 >> 172.21.15.94:0/634489961 conn(0x7f4460077e30 msgr2=0x7f44600fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 -- 172.21.15.94:0/634489961 shutdown_connections 2024-04-21T08:47:04.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.428+0000 7f4466ad2640 1 -- 172.21.15.94:0/634489961 wait complete. 2024-04-21T08:47:04.439 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.539+0000 7f5e7586d640 1 -- 172.21.15.94:0/3063186749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e701010a0 msgr2=0x7f5e70101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.539+0000 7f5e7586d640 1 --2- 172.21.15.94:0/3063186749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e701010a0 0x7f5e70101480 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009900 tx=0x7f5e6002bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.540+0000 7f5e7586d640 1 -- 172.21.15.94:0/3063186749 shutdown_connections 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.540+0000 7f5e7586d640 1 --2- 172.21.15.94:0/3063186749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e70106170 0x7f5e7010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.540+0000 7f5e7586d640 1 --2- 172.21.15.94:0/3063186749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e70101a50 0x7f5e70105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.540+0000 7f5e7586d640 1 --2- 172.21.15.94:0/3063186749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e701010a0 0x7f5e70101480 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.540+0000 7f5e7586d640 1 -- 172.21.15.94:0/3063186749 >> 172.21.15.94:0/3063186749 conn(0x7f5e700771d0 msgr2=0x7f5e700775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.540+0000 7f5e7586d640 1 -- 172.21.15.94:0/3063186749 shutdown_connections 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.540+0000 7f5e7586d640 1 -- 172.21.15.94:0/3063186749 wait complete. 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.540+0000 7f5e7586d640 1 Processor -- start 2024-04-21T08:47:04.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.540+0000 7f5e7586d640 1 -- start start 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7586d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e701010a0 0x7f5e7019e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7586d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e70106170 0x7f5e7019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7586d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e70198610 0x7f5e70198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7586d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5e70112250 con 0x7f5e70106170 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7586d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701120d0 con 0x7f5e701010a0 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7586d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701123d0 con 0x7f5e70198610 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e70106170 0x7f5e7019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e70106170 0x7f5e7019e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49082/0 (socket says 172.21.15.94:49082) 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6e7fc640 1 -- 172.21.15.94:0/531052485 learned_addr learned my addr 172.21.15.94:0/531052485 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6effd640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e701010a0 0x7f5e7019e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6f7fe640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e70198610 0x7f5e70198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6e7fc640 1 -- 172.21.15.94:0/531052485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e70198610 msgr2=0x7f5e70198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6e7fc640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e70198610 0x7f5e70198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6e7fc640 1 -- 172.21.15.94:0/531052485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e701010a0 msgr2=0x7f5e7019e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6e7fc640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e701010a0 0x7f5e7019e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6e7fc640 1 -- 172.21.15.94:0/531052485 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e70199130 con 0x7f5e70106170 2024-04-21T08:47:04.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6effd640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e701010a0 0x7f5e7019e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:04.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e6e7fc640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e70106170 0x7f5e7019e9a0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f5e6400b450 tx=0x7f5e6400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7486b640 1 -- 172.21.15.94:0/531052485 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e64015020 con 0x7f5e70106170 2024-04-21T08:47:04.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7486b640 1 -- 172.21.15.94:0/531052485 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e64011d40 con 0x7f5e70106170 2024-04-21T08:47:04.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7586d640 1 -- 172.21.15.94:0/531052485 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e70199420 con 0x7f5e70106170 2024-04-21T08:47:04.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7486b640 1 -- 172.21.15.94:0/531052485 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6400f340 con 0x7f5e70106170 2024-04-21T08:47:04.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.541+0000 7f5e7586d640 1 -- 172.21.15.94:0/531052485 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e701a5670 con 0x7f5e70106170 2024-04-21T08:47:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.543+0000 7f5e7486b640 1 -- 172.21.15.94:0/531052485 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5e64018020 con 0x7f5e70106170 2024-04-21T08:47:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e78466740 1 Processor -- start 2024-04-21T08:47:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e78466740 1 -- start start 2024-04-21T08:47:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e7486b640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e580a3d20 0x7f5e580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e78466740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67330 0x563584e67710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e78466740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e70820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e78466740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e70e40 0x563584e712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e78466740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563584dcc1f0 con 0x563584e67330 2024-04-21T08:47:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e78466740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563584d694b0 con 0x563584e67ce0 2024-04-21T08:47:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e78466740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563584dc81b0 con 0x563584e70e40 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e6effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67330 0x563584e67710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e6f7fe640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e580a3d20 0x7f5e580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e6effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67330 0x563584e67710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49086/0 (socket says 172.21.15.94:49086) 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e6effd640 1 -- 172.21.15.94:0/761006626 learned_addr learned my addr 172.21.15.94:0/761006626 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e7486b640 1 -- 172.21.15.94:0/531052485 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5e640d0620 con 0x7f5e70106170 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e6e7fc640 1 --2- 172.21.15.94:0/761006626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e70820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e6effd640 1 -- 172.21.15.94:0/761006626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e70e40 msgr2=0x563584e712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e6effd640 1 --2- 172.21.15.94:0/761006626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e70e40 0x563584e712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e6effd640 1 -- 172.21.15.94:0/761006626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 msgr2=0x563584e70820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e6effd640 1 --2- 172.21.15.94:0/761006626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e70820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.544+0000 7f5e6effd640 1 -- 172.21.15.94:0/761006626 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563584d5b1b0 con 0x563584e67330 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e6f7fe640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e580a3d20 0x7f5e580a61e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5e68005fd0 tx=0x7f5e680097a0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e6effd640 1 --2- 172.21.15.94:0/761006626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67330 0x563584e67710 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f5e60000c00 tx=0x7f5e60005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5323818733b7739 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e55ffb640 1 -- 172.21.15.94:0/761006626 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60044020 con 0x563584e67330 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e55ffb640 1 -- 172.21.15.94:0/761006626 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e6002cc90 con 0x563584e67330 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e55ffb640 1 -- 172.21.15.94:0/761006626 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e600046c0 con 0x563584e67330 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e78466740 1 -- 172.21.15.94:0/761006626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67330 msgr2=0x563584e67710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e78466740 1 --2- 172.21.15.94:0/761006626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67330 0x563584e67710 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f5e60000c00 tx=0x7f5e60005e80 comp rx=0 tx=0).stop 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e78466740 1 -- 172.21.15.94:0/761006626 shutdown_connections 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e78466740 1 --2- 172.21.15.94:0/761006626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e70e40 0x563584e712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e78466740 1 --2- 172.21.15.94:0/761006626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e70820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e78466740 1 --2- 172.21.15.94:0/761006626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67330 0x563584e67710 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.545+0000 7f5e78466740 1 -- 172.21.15.94:0/761006626 >> 172.21.15.94:0/761006626 conn(0x563584e61ba0 msgr2=0x563584e61f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 1 -- 172.21.15.94:0/761006626 shutdown_connections 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 1 -- 172.21.15.94:0/761006626 wait complete. 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 1 -- start start 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 1 Processor -- start 2024-04-21T08:47:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 0x563584e77280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e74570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e74b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563584e77d60 con 0x563584e70e40 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563584d85500 con 0x563584e67ce0 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563584e77ee0 con 0x563584e67330 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e78466740 10 client.0 _ll_register_callbacks cb 0x563584d53b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e6f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e74b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e6f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e74b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49100/0 (socket says 172.21.15.94:49100) 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.546+0000 7f5e6f7fe640 1 -- 172.21.15.94:0/2738539719 learned_addr learned my addr 172.21.15.94:0/2738539719 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e6e7fc640 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e74570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e6effd640 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 0x563584e77280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e6f7fe640 1 -- 172.21.15.94:0/2738539719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 msgr2=0x563584e77280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e6f7fe640 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 0x563584e77280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e6f7fe640 1 -- 172.21.15.94:0/2738539719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 msgr2=0x563584e74570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e6f7fe640 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e74570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e6f7fe640 1 -- 172.21.15.94:0/2738539719 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563584ade3f0 con 0x563584e70e40 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e6effd640 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 0x563584e77280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e6e7fc640 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e74570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e6f7fe640 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e74b10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009400 tx=0x7f5e68004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda057af7d3b8aa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e4effd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e4effd640 1 -- 172.21.15.94:0/2738539719 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e68023070 con 0x563584e70e40 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e78466740 1 -- 172.21.15.94:0/2738539719 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e6002cc90 con 0x563584e70e40 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e4effd640 1 -- 172.21.15.94:0/2738539719 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5e6800e920 con 0x563584e70e40 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e4effd640 1 -- 172.21.15.94:0/2738539719 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e6802f050 con 0x563584e70e40 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e78466740 10 client.17493 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e78466740 1 -- 172.21.15.94:0/2738539719 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563584ba87d0 con 0x563584e70e40 2024-04-21T08:47:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e4effd640 1 -- 172.21.15.94:0/2738539719 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e6002cc90 con 0x563584e70e40 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.547+0000 7f5e4effd640 1 -- 172.21.15.94:0/2738539719 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5e680273b0 con 0x563584e70e40 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.548+0000 7f5e78466740 10 client.17493 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.548+0000 7f5e78466740 10 client.17493 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.548+0000 7f5e78466740 20 client.17493 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.548+0000 7f5e78466740 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563584e84ff0 0x563584e87450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.548+0000 7f5e78466740 4 client.17493 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.548+0000 7f5e78466740 1 -- 172.21.15.94:0/2738539719 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563584d6b930 con 0x563584e84ff0 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.548+0000 7f5e6effd640 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563584e84ff0 0x563584e87450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.548+0000 7f5e4effd640 10 client.17493 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.548+0000 7f5e6effd640 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563584e84ff0 0x563584e87450 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce76ed1809b73535 server_cookie=64ce615bbf7ba698 in_seq=0 out_seq=0 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e4effd640 1 -- 172.21.15.94:0/2738539719 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563584d6b930 con 0x563584e84ff0 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e4effd640 10 client.17493 handle_command_reply: tid=0 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 client.17493 shutdown 2024-04-21T08:47:04.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 -- 172.21.15.94:0/2738539719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 msgr2=0x563584e74b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e74b10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009400 tx=0x7f5e68004ab0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 -- 172.21.15.94:0/2738539719 shutdown_connections 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563584e84ff0 0x563584e87450 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e74b10 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e74570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e4effd640 0 client.17493 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 --2- 172.21.15.94:0/2738539719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 0x563584e77280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 -- 172.21.15.94:0/2738539719 >> 172.21.15.94:0/2738539719 conn(0x563584e61ba0 msgr2=0x563584e638f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 -- 172.21.15.94:0/2738539719 shutdown_connections 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 1 -- 172.21.15.94:0/2738539719 wait complete. 2024-04-21T08:47:04.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.549+0000 7f5e78466740 20 client.17493 trim_cache size 0 max 16384 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.562+0000 7f5e78466740 1 Processor -- start 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.562+0000 7f5e78466740 1 -- start start 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e78466740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67330 0x563584e80c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e78466740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67ce0 0x563584e60a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e78466740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e70e40 0x563584e76db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e78466740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701afb70 con 0x563584e67ce0 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e78466740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701ad440 con 0x563584e67330 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e78466740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701ab290 con 0x563584e70e40 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e6e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67ce0 0x563584e60a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e6e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67ce0 0x563584e60a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49108/0 (socket says 172.21.15.94:49108) 2024-04-21T08:47:04.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e6e7fc640 1 -- 172.21.15.94:0/922673874 learned_addr learned my addr 172.21.15.94:0/922673874 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e6e7fc640 1 -- 172.21.15.94:0/922673874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e70e40 msgr2=0x563584e76db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e6e7fc640 1 --2- 172.21.15.94:0/922673874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e70e40 0x563584e76db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e6e7fc640 1 -- 172.21.15.94:0/922673874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67330 msgr2=0x563584e80c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e6e7fc640 1 --2- 172.21.15.94:0/922673874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67330 0x563584e80c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e6e7fc640 1 -- 172.21.15.94:0/922673874 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563584e66560 con 0x563584e67ce0 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e6e7fc640 1 --2- 172.21.15.94:0/922673874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67ce0 0x563584e60a80 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5e640cdb00 tx=0x7f5e6409bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3556eda6e6bde149 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e557fa640 1 -- 172.21.15.94:0/922673874 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e64096e60 con 0x563584e67ce0 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e557fa640 1 -- 172.21.15.94:0/922673874 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e6409c2f0 con 0x563584e67ce0 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.563+0000 7f5e557fa640 1 -- 172.21.15.94:0/922673874 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e64010500 con 0x563584e67ce0 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 -- 172.21.15.94:0/922673874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67ce0 msgr2=0x563584e60a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 --2- 172.21.15.94:0/922673874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67ce0 0x563584e60a80 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5e640cdb00 tx=0x7f5e6409bf60 comp rx=0 tx=0).stop 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 -- 172.21.15.94:0/922673874 shutdown_connections 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 --2- 172.21.15.94:0/922673874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e70e40 0x563584e76db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 --2- 172.21.15.94:0/922673874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e67ce0 0x563584e60a80 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 --2- 172.21.15.94:0/922673874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67330 0x563584e80c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 -- 172.21.15.94:0/922673874 >> 172.21.15.94:0/922673874 conn(0x563584e61ba0 msgr2=0x563584e60370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 -- 172.21.15.94:0/922673874 shutdown_connections 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 -- 172.21.15.94:0/922673874 wait complete. 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 -- start start 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 Processor -- start 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 0x563584e7d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e7d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e7a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701ab900 con 0x563584e70e40 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.564+0000 7f5e78466740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701acd70 con 0x563584e67ce0 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e78466740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701abfd0 con 0x563584e67330 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e78466740 10 client.0 _ll_register_callbacks cb 0x7f5e68009510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e7a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e7a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49118/0 (socket says 172.21.15.94:49118) 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6f7fe640 1 -- 172.21.15.94:0/2290684056 learned_addr learned my addr 172.21.15.94:0/2290684056 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6e7fc640 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e7d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6effd640 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 0x563584e7d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6f7fe640 1 -- 172.21.15.94:0/2290684056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 msgr2=0x563584e7d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6f7fe640 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 0x563584e7d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6f7fe640 1 -- 172.21.15.94:0/2290684056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 msgr2=0x563584e7d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6f7fe640 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e7d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6f7fe640 1 -- 172.21.15.94:0/2290684056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563584e7a750 con 0x563584e70e40 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e6f7fe640 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e7a140 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009270 tx=0x7f5e68011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3a8969333fd44d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e4effd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e4effd640 1 -- 172.21.15.94:0/2290684056 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e68038060 con 0x563584e70e40 2024-04-21T08:47:04.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e4effd640 1 -- 172.21.15.94:0/2290684056 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f5e6800e720 con 0x563584e70e40 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e78466740 1 -- 172.21.15.94:0/2290684056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e6409c2f0 con 0x563584e70e40 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e4effd640 1 -- 172.21.15.94:0/2290684056 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e6803f050 con 0x563584e70e40 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e78466740 10 client.17499 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.565+0000 7f5e78466740 1 -- 172.21.15.94:0/2290684056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563584eebef0 con 0x563584e70e40 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e4effd640 1 -- 172.21.15.94:0/2290684056 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e6409c2f0 con 0x563584e70e40 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e4effd640 1 -- 172.21.15.94:0/2290684056 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5e68028720 con 0x563584e70e40 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e78466740 10 client.17499 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e78466740 10 client.17499 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e78466740 20 client.17499 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e78466740 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563584eec640 0x563584eeea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e78466740 4 client.17499 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e78466740 1 -- 172.21.15.94:0/2290684056 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563584dd9c10 con 0x563584eec640 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e6effd640 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563584eec640 0x563584eeea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e4effd640 10 client.17499 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:47:04.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.566+0000 7f5e6effd640 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563584eec640 0x563584eeea60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bab702f91f6f5556 server_cookie=91a6efca86f957a in_seq=0 out_seq=0 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e4effd640 1 -- 172.21.15.94:0/2290684056 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563584dd9c10 con 0x563584eec640 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e4effd640 10 client.17499 handle_command_reply: tid=0 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 client.17499 shutdown 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 -- 172.21.15.94:0/2290684056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 msgr2=0x563584e7a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e7a140 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009270 tx=0x7f5e68011d30 comp rx=0 tx=0).stop 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 -- 172.21.15.94:0/2290684056 shutdown_connections 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x563584eec640 0x563584eeea60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563584e70e40 0x563584e7a140 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563584e67ce0 0x563584e7d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 --2- 172.21.15.94:0/2290684056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563584e67330 0x563584e7d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e4effd640 0 client.17499 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 -- 172.21.15.94:0/2290684056 >> 172.21.15.94:0/2290684056 conn(0x563584e61ba0 msgr2=0x563584e60c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 -- 172.21.15.94:0/2290684056 shutdown_connections 2024-04-21T08:47:04.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 1 -- 172.21.15.94:0/2290684056 wait complete. 2024-04-21T08:47:04.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e78466740 20 client.17499 trim_cache size 0 max 16384 2024-04-21T08:47:04.574 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:47:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e7586d640 1 -- 172.21.15.94:0/531052485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e580a3d20 msgr2=0x7f5e580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e7586d640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e580a3d20 0x7f5e580a61e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5e68005fd0 tx=0x7f5e680097a0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e7586d640 1 -- 172.21.15.94:0/531052485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e70106170 msgr2=0x7f5e7019e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e7586d640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e70106170 0x7f5e7019e9a0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f5e6400b450 tx=0x7f5e6400b910 comp rx=0 tx=0).stop 2024-04-21T08:47:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e7586d640 1 -- 172.21.15.94:0/531052485 shutdown_connections 2024-04-21T08:47:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e7586d640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e580a3d20 0x7f5e580a61e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e7586d640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e70198610 0x7f5e70198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e7586d640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e70106170 0x7f5e7019e9a0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e7586d640 1 --2- 172.21.15.94:0/531052485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e701010a0 0x7f5e7019e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.567+0000 7f5e7586d640 1 -- 172.21.15.94:0/531052485 >> 172.21.15.94:0/531052485 conn(0x7f5e700771d0 msgr2=0x7f5e700feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.568+0000 7f5e7586d640 1 -- 172.21.15.94:0/531052485 shutdown_connections 2024-04-21T08:47:04.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.568+0000 7f5e7586d640 1 -- 172.21.15.94:0/531052485 wait complete. 2024-04-21T08:47:04.578 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:47:04.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 -- 172.21.15.94:0/4072500474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc50c101090 msgr2=0x7fc50c101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 --2- 172.21.15.94:0/4072500474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc50c101090 0x7fc50c101470 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fc504009ca0 tx=0x7fc50402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 -- 172.21.15.94:0/4072500474 shutdown_connections 2024-04-21T08:47:04.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 --2- 172.21.15.94:0/4072500474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc50c106160 0x7fc50c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 --2- 172.21.15.94:0/4072500474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc50c101a40 0x7fc50c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 --2- 172.21.15.94:0/4072500474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc50c101090 0x7fc50c101470 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 -- 172.21.15.94:0/4072500474 >> 172.21.15.94:0/4072500474 conn(0x7fc50c077f50 msgr2=0x7fc50c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 -- 172.21.15.94:0/4072500474 shutdown_connections 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 -- 172.21.15.94:0/4072500474 wait complete. 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 Processor -- start 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.678+0000 7fc5136a0640 1 -- start start 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5136a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc50c101090 0x7fc50c1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5136a0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc50c106160 0x7fc50c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5136a0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc50c19be60 0x7fc50c19c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5136a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1110a0 con 0x7fc50c101090 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5136a0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c110f20 con 0x7fc50c19be60 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5136a0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c111220 con 0x7fc50c106160 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc51143c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc50c101090 0x7fc50c1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc51143c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc50c101090 0x7fc50c1975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49138/0 (socket says 172.21.15.94:49138) 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc51143c640 1 -- 172.21.15.94:0/3236168273 learned_addr learned my addr 172.21.15.94:0/3236168273 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc511c3d640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc50c19be60 0x7fc50c19c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc51143c640 1 -- 172.21.15.94:0/3236168273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc50c106160 msgr2=0x7fc50c197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc51143c640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc50c106160 0x7fc50c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc51143c640 1 -- 172.21.15.94:0/3236168273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc50c19be60 msgr2=0x7fc50c19c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc51143c640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc50c19be60 0x7fc50c19c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc51143c640 1 -- 172.21.15.94:0/3236168273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50c19c950 con 0x7fc50c101090 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc511c3d640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc50c19be60 0x7fc50c19c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc51143c640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc50c101090 0x7fc50c1975a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc5040098d0 tx=0x7fc504035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5027fc640 1 -- 172.21.15.94:0/3236168273 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504039040 con 0x7fc50c101090 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5027fc640 1 -- 172.21.15.94:0/3236168273 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc504044070 con 0x7fc50c101090 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5136a0640 1 -- 172.21.15.94:0/3236168273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc50c19cbe0 con 0x7fc50c101090 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5027fc640 1 -- 172.21.15.94:0/3236168273 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040042a0 con 0x7fc50c101090 2024-04-21T08:47:04.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.679+0000 7fc5136a0640 1 -- 172.21.15.94:0/3236168273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc50c1a43a0 con 0x7fc50c101090 2024-04-21T08:47:04.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.681+0000 7fc5027fc640 1 -- 172.21.15.94:0/3236168273 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fc504004440 con 0x7fc50c101090 2024-04-21T08:47:04.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc516368740 1 Processor -- start 2024-04-21T08:47:04.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc516368740 1 -- start start 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc516368740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d5636e0 0x55736d563ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc516368740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d564090 0x55736d56c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc516368740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d56f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc516368740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55736d4c61f0 con 0x55736d564090 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc516368740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55736d4634b0 con 0x55736d56cc10 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc516368740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55736d4c21b0 con 0x55736d5636e0 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc5027fc640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4f80a3d20 0x7fc4f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc51143c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d564090 0x55736d56c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc511c3d640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4f80a3d20 0x7fc4f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc51143c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d564090 0x55736d56c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49152/0 (socket says 172.21.15.94:49152) 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc51143c640 1 -- 172.21.15.94:0/1813416213 learned_addr learned my addr 172.21.15.94:0/1813416213 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc5027fc640 1 -- 172.21.15.94:0/3236168273 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc5040f3070 con 0x7fc50c101090 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc511c3d640 1 --2- 172.21.15.94:0/1813416213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d5636e0 0x55736d563ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc510c3b640 1 --2- 172.21.15.94:0/1813416213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d56f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc51143c640 1 -- 172.21.15.94:0/1813416213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d5636e0 msgr2=0x55736d563ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc51143c640 1 --2- 172.21.15.94:0/1813416213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d5636e0 0x55736d563ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc51143c640 1 -- 172.21.15.94:0/1813416213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 msgr2=0x55736d56f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.682+0000 7fc51143c640 1 --2- 172.21.15.94:0/1813416213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d56f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.683+0000 7fc51143c640 1 -- 172.21.15.94:0/1813416213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55736d4551b0 con 0x55736d564090 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.683+0000 7fc511c3d640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4f80a3d20 0x7fc4f80a61e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc50800de50 tx=0x7fc508014840 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.683+0000 7fc51143c640 1 --2- 172.21.15.94:0/1813416213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d564090 0x55736d56c640 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fc5040becd0 tx=0x7fc504090780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d366a60a22a68639 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.683+0000 7fc4f77fe640 1 -- 172.21.15.94:0/1813416213 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040beb10 con 0x55736d564090 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.683+0000 7fc4f77fe640 1 -- 172.21.15.94:0/1813416213 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5040a8280 con 0x55736d564090 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.683+0000 7fc4f77fe640 1 -- 172.21.15.94:0/1813416213 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040be6a0 con 0x55736d564090 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.683+0000 7fc516368740 1 -- 172.21.15.94:0/1813416213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d564090 msgr2=0x55736d56c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.683+0000 7fc516368740 1 --2- 172.21.15.94:0/1813416213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d564090 0x55736d56c640 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fc5040becd0 tx=0x7fc504090780 comp rx=0 tx=0).stop 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 -- 172.21.15.94:0/1813416213 shutdown_connections 2024-04-21T08:47:04.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 --2- 172.21.15.94:0/1813416213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d56f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 --2- 172.21.15.94:0/1813416213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d564090 0x55736d56c640 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 --2- 172.21.15.94:0/1813416213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d5636e0 0x55736d563ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 -- 172.21.15.94:0/1813416213 >> 172.21.15.94:0/1813416213 conn(0x55736d55ddb0 msgr2=0x55736d55e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 -- 172.21.15.94:0/1813416213 shutdown_connections 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 -- 172.21.15.94:0/1813416213 wait complete. 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 -- start start 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 Processor -- start 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d56c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d569950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d569fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55736d56f940 con 0x55736d5636e0 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55736d47f500 con 0x55736d56cc10 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55736d56fac0 con 0x55736d564090 2024-04-21T08:47:04.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc516368740 10 client.0 _ll_register_callbacks cb 0x55736d44db80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc51143c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d56c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc51143c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d56c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49158/0 (socket says 172.21.15.94:49158) 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.684+0000 7fc51143c640 1 -- 172.21.15.94:0/186700663 learned_addr learned my addr 172.21.15.94:0/186700663 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc510c3b640 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d569950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc51143c640 1 -- 172.21.15.94:0/186700663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 msgr2=0x55736d569950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc51143c640 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d569950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc51143c640 1 -- 172.21.15.94:0/186700663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 msgr2=0x55736d569fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc51143c640 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d569fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc51143c640 1 -- 172.21.15.94:0/186700663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55736d1d83f0 con 0x55736d5636e0 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc51143c640 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d56c550 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fc5040f05a0 tx=0x7fc504004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83e482a40f1b807 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc4f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/186700663 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040be880 con 0x55736d5636e0 2024-04-21T08:47:04.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/186700663 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc50403e3a0 con 0x55736d5636e0 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc516368740 1 -- 172.21.15.94:0/186700663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5040a8280 con 0x55736d5636e0 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/186700663 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc5040dd050 con 0x55736d5636e0 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc516368740 10 client.17511 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc516368740 1 -- 172.21.15.94:0/186700663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55736d2a27d0 con 0x55736d5636e0 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/186700663 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5040a8280 con 0x55736d5636e0 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/186700663 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc50403c030 con 0x55736d5636e0 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc516368740 10 client.17511 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.685+0000 7fc516368740 10 client.17511 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.686+0000 7fc516368740 20 client.17511 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.686+0000 7fc516368740 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55736d57ca60 0x55736d57ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.686+0000 7fc516368740 4 client.17511 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.686+0000 7fc516368740 1 -- 172.21.15.94:0/186700663 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55736d465930 con 0x55736d57ca60 2024-04-21T08:47:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.686+0000 7fc510c3b640 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55736d57ca60 0x55736d57ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.686+0000 7fc4f4ff9640 10 client.17511 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.686+0000 7fc510c3b640 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55736d57ca60 0x55736d57ee50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd40c8fc9b970393 server_cookie=cb5c04785ee6556b in_seq=0 out_seq=0 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/186700663 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55736d465930 con 0x55736d57ca60 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc4f4ff9640 10 client.17511 handle_command_reply: tid=0 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 client.17511 shutdown 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 -- 172.21.15.94:0/186700663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 msgr2=0x55736d56c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d56c550 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fc5040f05a0 tx=0x7fc504004960 comp rx=0 tx=0).stop 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 -- 172.21.15.94:0/186700663 shutdown_connections 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d569950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55736d57ca60 0x55736d57ee50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d569fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 --2- 172.21.15.94:0/186700663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d56c550 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc4f4ff9640 0 client.17511 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:47:04.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 -- 172.21.15.94:0/186700663 >> 172.21.15.94:0/186700663 conn(0x55736d55ddb0 msgr2=0x55736d55fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 -- 172.21.15.94:0/186700663 shutdown_connections 2024-04-21T08:47:04.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 1 -- 172.21.15.94:0/186700663 wait complete. 2024-04-21T08:47:04.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.687+0000 7fc516368740 20 client.17511 trim_cache size 0 max 16384 2024-04-21T08:47:04.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.700+0000 7fc516368740 1 Processor -- start 2024-04-21T08:47:04.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.700+0000 7fc516368740 1 -- start start 2024-04-21T08:47:04.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.700+0000 7fc516368740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d5636e0 0x55736d569100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc516368740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d56c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc516368740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d56cc10 0x55736d584700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc516368740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1af9e0 con 0x55736d56cc10 2024-04-21T08:47:04.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc516368740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1ad2b0 con 0x55736d5636e0 2024-04-21T08:47:04.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc516368740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1ab100 con 0x55736d564090 2024-04-21T08:47:04.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc511c3d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d56cc10 0x55736d584700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc511c3d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d56cc10 0x55736d584700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49172/0 (socket says 172.21.15.94:49172) 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc511c3d640 1 -- 172.21.15.94:0/3907761863 learned_addr learned my addr 172.21.15.94:0/3907761863 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc511c3d640 1 -- 172.21.15.94:0/3907761863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 msgr2=0x55736d56c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc511c3d640 1 --2- 172.21.15.94:0/3907761863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d56c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc511c3d640 1 -- 172.21.15.94:0/3907761863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d5636e0 msgr2=0x55736d569100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc511c3d640 1 --2- 172.21.15.94:0/3907761863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d5636e0 0x55736d569100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc511c3d640 1 -- 172.21.15.94:0/3907761863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55736d567ea0 con 0x55736d56cc10 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc511c3d640 1 --2- 172.21.15.94:0/3907761863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d56cc10 0x55736d584700 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc508006000 tx=0x7fc508014f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a42316036686d0c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc4f6ffd640 1 -- 172.21.15.94:0/3907761863 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5080142d0 con 0x55736d56cc10 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc4f6ffd640 1 -- 172.21.15.94:0/3907761863 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc508011e10 con 0x55736d56cc10 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.701+0000 7fc4f6ffd640 1 -- 172.21.15.94:0/3907761863 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50801f780 con 0x55736d56cc10 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 -- 172.21.15.94:0/3907761863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d56cc10 msgr2=0x55736d584700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 --2- 172.21.15.94:0/3907761863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d56cc10 0x55736d584700 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc508006000 tx=0x7fc508014f90 comp rx=0 tx=0).stop 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 -- 172.21.15.94:0/3907761863 shutdown_connections 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 --2- 172.21.15.94:0/3907761863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d56c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 --2- 172.21.15.94:0/3907761863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d56cc10 0x55736d584700 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 --2- 172.21.15.94:0/3907761863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d5636e0 0x55736d569100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 -- 172.21.15.94:0/3907761863 >> 172.21.15.94:0/3907761863 conn(0x55736d55ddb0 msgr2=0x55736d564d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 -- 172.21.15.94:0/3907761863 shutdown_connections 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 -- 172.21.15.94:0/3907761863 wait complete. 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 -- start start 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.702+0000 7fc516368740 1 Processor -- start 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc516368740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d570800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc516368740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d570d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc516368740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d571280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc516368740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1ab770 con 0x55736d5636e0 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc516368740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1acbe0 con 0x55736d56cc10 2024-04-21T08:47:04.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc516368740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1abe40 con 0x55736d564090 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc516368740 10 client.0 _ll_register_callbacks cb 0x7fc5040bf480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc51143c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d570800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc51143c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d570800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49178/0 (socket says 172.21.15.94:49178) 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc51143c640 1 -- 172.21.15.94:0/615817420 learned_addr learned my addr 172.21.15.94:0/615817420 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc511c3d640 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d571280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc510c3b640 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d570d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc51143c640 1 -- 172.21.15.94:0/615817420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 msgr2=0x55736d570d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc51143c640 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d570d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc51143c640 1 -- 172.21.15.94:0/615817420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 msgr2=0x55736d571280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc51143c640 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d571280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc51143c640 1 -- 172.21.15.94:0/615817420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55736d5e2170 con 0x55736d5636e0 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc511c3d640 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d571280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:04.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc51143c640 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d570800 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc504000f80 tx=0x7fc5040be760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb12c3a92d901f2c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc4f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/615817420 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040be880 con 0x55736d5636e0 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/615817420 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fc5040bd040 con 0x55736d5636e0 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc516368740 1 -- 172.21.15.94:0/615817420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc508011e10 con 0x55736d5636e0 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/615817420 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc5040beb10 con 0x55736d5636e0 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc516368740 10 client.17517 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc516368740 1 -- 172.21.15.94:0/615817420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55736d5e2170 con 0x55736d5636e0 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.703+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/615817420 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc508011e10 con 0x55736d5636e0 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/615817420 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc5040c7520 con 0x55736d5636e0 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc516368740 10 client.17517 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc516368740 10 client.17517 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc516368740 20 client.17517 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc516368740 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55736d5e36f0 0x55736d5e5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc516368740 4 client.17517 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc516368740 1 -- 172.21.15.94:0/615817420 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55736d4d3c10 con 0x55736d5e36f0 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc510c3b640 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55736d5e36f0 0x55736d5e5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc510c3b640 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55736d5e36f0 0x55736d5e5b10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=61d8dbcb412a164a server_cookie=aa4d5d5f51275656 in_seq=0 out_seq=0 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc4f4ff9640 10 client.17517 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc4f4ff9640 1 -- 172.21.15.94:0/615817420 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55736d4d3c10 con 0x55736d5e36f0 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc4f4ff9640 10 client.17517 handle_command_reply: tid=0 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.704+0000 7fc516368740 1 client.17517 shutdown 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 1 -- 172.21.15.94:0/615817420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 msgr2=0x55736d570800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d570800 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc504000f80 tx=0x7fc5040be760 comp rx=0 tx=0).stop 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 1 -- 172.21.15.94:0/615817420 shutdown_connections 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55736d564090 0x55736d570d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55736d5e36f0 0x55736d5e5b10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55736d56cc10 0x55736d571280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 1 --2- 172.21.15.94:0/615817420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55736d5636e0 0x55736d570800 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc4f4ff9640 0 client.17517 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:47:04.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 1 -- 172.21.15.94:0/615817420 >> 172.21.15.94:0/615817420 conn(0x55736d55ddb0 msgr2=0x55736d55ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 1 -- 172.21.15.94:0/615817420 shutdown_connections 2024-04-21T08:47:04.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 1 -- 172.21.15.94:0/615817420 wait complete. 2024-04-21T08:47:04.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc516368740 20 client.17517 trim_cache size 0 max 16384 2024-04-21T08:47:04.712 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:47:04.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 -- 172.21.15.94:0/3236168273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4f80a3d20 msgr2=0x7fc4f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4f80a3d20 0x7fc4f80a61e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc50800de50 tx=0x7fc508014840 comp rx=0 tx=0).stop 2024-04-21T08:47:04.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 -- 172.21.15.94:0/3236168273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc50c101090 msgr2=0x7fc50c1975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc50c101090 0x7fc50c1975a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc5040098d0 tx=0x7fc504035d60 comp rx=0 tx=0).stop 2024-04-21T08:47:04.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 -- 172.21.15.94:0/3236168273 shutdown_connections 2024-04-21T08:47:04.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc4f80a3d20 0x7fc4f80a61e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc50c106160 0x7fc50c197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc50c19be60 0x7fc50c19c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 --2- 172.21.15.94:0/3236168273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc50c101090 0x7fc50c1975a0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 -- 172.21.15.94:0/3236168273 >> 172.21.15.94:0/3236168273 conn(0x7fc50c077f50 msgr2=0x7fc50c0fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 -- 172.21.15.94:0/3236168273 shutdown_connections 2024-04-21T08:47:04.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.705+0000 7fc5136a0640 1 -- 172.21.15.94:0/3236168273 wait complete. 2024-04-21T08:47:04.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.817+0000 7f3c58625640 1 -- 172.21.15.94:0/3546994530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c501011a0 msgr2=0x7f3c50101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.817+0000 7f3c58625640 1 --2- 172.21.15.94:0/3546994530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c501011a0 0x7f3c50101600 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009930 tx=0x7f3c4c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.817+0000 7f3c58625640 1 -- 172.21.15.94:0/3546994530 shutdown_connections 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.817+0000 7f3c58625640 1 --2- 172.21.15.94:0/3546994530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c50101b40 0x7f3c5010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.817+0000 7f3c58625640 1 --2- 172.21.15.94:0/3546994530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c501011a0 0x7f3c50101600 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.817+0000 7f3c58625640 1 --2- 172.21.15.94:0/3546994530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c50107030 0x7f3c50107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.817+0000 7f3c58625640 1 -- 172.21.15.94:0/3546994530 >> 172.21.15.94:0/3546994530 conn(0x7f3c50076c50 msgr2=0x7f3c50077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.817+0000 7f3c58625640 1 -- 172.21.15.94:0/3546994530 shutdown_connections 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.817+0000 7f3c58625640 1 -- 172.21.15.94:0/3546994530 wait complete. 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c58625640 1 Processor -- start 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c58625640 1 -- start start 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c58625640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c501011a0 0x7f3c5019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c58625640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c50101b40 0x7f3c5019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c58625640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c50107030 0x7f3c501985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c58625640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3c501110c0 con 0x7f3c50101b40 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c58625640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3c50110f40 con 0x7f3c501011a0 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c58625640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c50111240 con 0x7f3c50107030 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c55bc0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c50101b40 0x7f3c5019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c55bc0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c50101b40 0x7f3c5019e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49192/0 (socket says 172.21.15.94:49192) 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c55bc0640 1 -- 172.21.15.94:0/3478553882 learned_addr learned my addr 172.21.15.94:0/3478553882 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c563c1640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c501011a0 0x7f3c5019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c55bc0640 1 -- 172.21.15.94:0/3478553882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c50107030 msgr2=0x7f3c501985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c50107030 0x7f3c501985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c55bc0640 1 -- 172.21.15.94:0/3478553882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c501011a0 msgr2=0x7f3c5019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c501011a0 0x7f3c5019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c55bc0640 1 -- 172.21.15.94:0/3478553882 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c50198d50 con 0x7f3c50101b40 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c563c1640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c501011a0 0x7f3c5019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c50101b40 0x7f3c5019e960 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009230 tx=0x7f3c4c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c477fe640 1 -- 172.21.15.94:0/3478553882 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c043020 con 0x7f3c50101b40 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c58625640 1 -- 172.21.15.94:0/3478553882 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c50198fe0 con 0x7f3c50101b40 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c477fe640 1 -- 172.21.15.94:0/3478553882 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c4c02cb70 con 0x7f3c50101b40 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c477fe640 1 -- 172.21.15.94:0/3478553882 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c003b50 con 0x7f3c50101b40 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.818+0000 7f3c58625640 1 -- 172.21.15.94:0/3478553882 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c501a52e0 con 0x7f3c50101b40 2024-04-21T08:47:04.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c477fe640 1 -- 172.21.15.94:0/3478553882 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f3c4c046020 con 0x7f3c50101b40 2024-04-21T08:47:04.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c5b212740 1 Processor -- start 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c5b212740 1 -- start start 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805d64d0 0x564e805d68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d6e80 0x564e805df430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805dfa00 0x564e805e1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c5b212740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564e8053b1f0 con 0x564e805d6e80 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c5b212740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564e804d84b0 con 0x564e805dfa00 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c5b212740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564e805371b0 con 0x564e805d64d0 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c563c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d6e80 0x564e805df430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.821+0000 7f3c477fe640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3c3c0a3d20 0x7f3c3c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c563c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d6e80 0x564e805df430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49194/0 (socket says 172.21.15.94:49194) 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c563c1640 1 -- 172.21.15.94:0/3924398835 learned_addr learned my addr 172.21.15.94:0/3924398835 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/3924398835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805dfa00 0x564e805e1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c56bc2640 1 --2- 172.21.15.94:0/3924398835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805d64d0 0x564e805d68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c56bc2640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3c3c0a3d20 0x7f3c3c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c477fe640 1 -- 172.21.15.94:0/3478553882 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3c4c0f2dc0 con 0x7f3c50101b40 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c563c1640 1 -- 172.21.15.94:0/3924398835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805d64d0 msgr2=0x564e805d68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c563c1640 1 --2- 172.21.15.94:0/3924398835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805d64d0 0x564e805d68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c563c1640 1 -- 172.21.15.94:0/3924398835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805dfa00 msgr2=0x564e805e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c563c1640 1 --2- 172.21.15.94:0/3924398835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805dfa00 0x564e805e1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c563c1640 1 -- 172.21.15.94:0/3924398835 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e804ca1b0 con 0x564e805d6e80 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c563c1640 1 --2- 172.21.15.94:0/3924398835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d6e80 0x564e805df430 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f3c40005fd0 tx=0x7f3c4000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed147337fee50a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c56bc2640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3c3c0a3d20 0x7f3c3c0a61e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009fd0 tx=0x7f3c48005bd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c44bfe640 1 -- 172.21.15.94:0/3924398835 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40016070 con 0x564e805d6e80 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c44bfe640 1 -- 172.21.15.94:0/3924398835 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c400054d0 con 0x564e805d6e80 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.822+0000 7f3c44bfe640 1 -- 172.21.15.94:0/3924398835 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40011470 con 0x564e805d6e80 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 -- 172.21.15.94:0/3924398835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d6e80 msgr2=0x564e805df430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 --2- 172.21.15.94:0/3924398835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d6e80 0x564e805df430 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f3c40005fd0 tx=0x7f3c4000e6c0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 -- 172.21.15.94:0/3924398835 shutdown_connections 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 --2- 172.21.15.94:0/3924398835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805dfa00 0x564e805e1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 --2- 172.21.15.94:0/3924398835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d6e80 0x564e805df430 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 --2- 172.21.15.94:0/3924398835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805d64d0 0x564e805d68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 -- 172.21.15.94:0/3924398835 >> 172.21.15.94:0/3924398835 conn(0x564e805d0ba0 msgr2=0x564e805d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 -- 172.21.15.94:0/3924398835 shutdown_connections 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 -- 172.21.15.94:0/3924398835 wait complete. 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 -- start start 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.823+0000 7f3c5b212740 1 Processor -- start 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805df260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805dc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 0x564e805dcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c5b212740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564e805e2970 con 0x564e805d64d0 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c5b212740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564e804f4500 con 0x564e805d6e80 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c5b212740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564e805e2af0 con 0x564e805dfa00 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c5b212740 10 client.0 _ll_register_callbacks cb 0x564e804c2b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c563c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805df260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c563c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805df260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49204/0 (socket says 172.21.15.94:49204) 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c563c1640 1 -- 172.21.15.94:0/2307952589 learned_addr learned my addr 172.21.15.94:0/2307952589 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c563c1640 1 -- 172.21.15.94:0/2307952589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 msgr2=0x564e805dcb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c563c1640 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 0x564e805dcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805dc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c563c1640 1 -- 172.21.15.94:0/2307952589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 msgr2=0x564e805dc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c563c1640 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805dc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c563c1640 1 -- 172.21.15.94:0/2307952589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e8024d3f0 con 0x564e805d64d0 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c563c1640 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805df260 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f3c40017000 tx=0x7f3c40005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02d47fd98f049ae server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:04.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c39ffb640 1 -- 172.21.15.94:0/2307952589 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40028060 con 0x564e805d64d0 2024-04-21T08:47:04.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c39ffb640 1 -- 172.21.15.94:0/2307952589 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3c400097e0 con 0x564e805d64d0 2024-04-21T08:47:04.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c5b212740 1 -- 172.21.15.94:0/2307952589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c400054d0 con 0x564e805d64d0 2024-04-21T08:47:04.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c39ffb640 1 -- 172.21.15.94:0/2307952589 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3c4002f050 con 0x564e805d64d0 2024-04-21T08:47:04.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c5b212740 10 client.17529 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:04.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.824+0000 7f3c5b212740 1 -- 172.21.15.94:0/2307952589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564e803177d0 con 0x564e805d64d0 2024-04-21T08:47:04.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.825+0000 7f3c39ffb640 1 -- 172.21.15.94:0/2307952589 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c400054d0 con 0x564e805d64d0 2024-04-21T08:47:04.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.825+0000 7f3c39ffb640 1 -- 172.21.15.94:0/2307952589 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3c4000f040 con 0x564e805d64d0 2024-04-21T08:47:04.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.825+0000 7f3c5b212740 10 client.17529 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:04.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.825+0000 7f3c5b212740 10 client.17529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:04.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.825+0000 7f3c5b212740 20 client.17529 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:04.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.825+0000 7f3c5b212740 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564e805efb70 0x564e805f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.825+0000 7f3c5b212740 4 client.17529 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:04.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.825+0000 7f3c5b212740 1 -- 172.21.15.94:0/2307952589 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e804da930 con 0x564e805efb70 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.825+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564e805efb70 0x564e805f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.826+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564e805efb70 0x564e805f1f60 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5973c9ea1c00ce5 server_cookie=2f797d10e421a1c in_seq=0 out_seq=0 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.826+0000 7f3c39ffb640 10 client.17529 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.826+0000 7f3c39ffb640 1 -- 172.21.15.94:0/2307952589 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e804da930 con 0x564e805efb70 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.826+0000 7f3c39ffb640 10 client.17529 handle_command_reply: tid=0 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.826+0000 7f3c5b212740 1 client.17529 shutdown 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.826+0000 7f3c5b212740 1 -- 172.21.15.94:0/2307952589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 msgr2=0x564e805df260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.826+0000 7f3c5b212740 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805df260 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f3c40017000 tx=0x7f3c40005390 comp rx=0 tx=0).stop 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.827+0000 7f3c5b212740 1 -- 172.21.15.94:0/2307952589 shutdown_connections 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.827+0000 7f3c5b212740 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564e805efb70 0x564e805f1f60 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.827+0000 7f3c5b212740 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 0x564e805dcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.827+0000 7f3c5b212740 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805dc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.827+0000 7f3c5b212740 1 --2- 172.21.15.94:0/2307952589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805df260 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.827+0000 7f3c39ffb640 0 client.17529 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:04.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.827+0000 7f3c5b212740 1 -- 172.21.15.94:0/2307952589 >> 172.21.15.94:0/2307952589 conn(0x564e805d0ba0 msgr2=0x564e805d28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.827+0000 7f3c5b212740 1 -- 172.21.15.94:0/2307952589 shutdown_connections 2024-04-21T08:47:04.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.827+0000 7f3c5b212740 1 -- 172.21.15.94:0/2307952589 wait complete. 2024-04-21T08:47:04.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.827+0000 7f3c5b212740 20 client.17529 trim_cache size 0 max 16384 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c5b212740 1 Processor -- start 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c5b212740 1 -- start start 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805eb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805ded20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 0x564e805df260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c5b212740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3c501b1ca0 con 0x564e805d64d0 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c5b212740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3c501af570 con 0x564e805d6e80 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c5b212740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c501ad3c0 con 0x564e805dfa00 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c563c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805eb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c563c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805eb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49220/0 (socket says 172.21.15.94:49220) 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c563c1640 1 -- 172.21.15.94:0/2385844575 learned_addr learned my addr 172.21.15.94:0/2385844575 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/2385844575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805ded20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c563c1640 1 -- 172.21.15.94:0/2385844575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 msgr2=0x564e805df260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c563c1640 1 --2- 172.21.15.94:0/2385844575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 0x564e805df260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c563c1640 1 -- 172.21.15.94:0/2385844575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 msgr2=0x564e805ded20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c563c1640 1 --2- 172.21.15.94:0/2385844575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805ded20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.840+0000 7f3c563c1640 1 -- 172.21.15.94:0/2385844575 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e805e1df0 con 0x564e805d64d0 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c563c1640 1 --2- 172.21.15.94:0/2385844575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805eb890 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f3c400094b0 tx=0x7f3c4000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14c18e730c02ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c3bfff640 1 -- 172.21.15.94:0/2385844575 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40028030 con 0x564e805d64d0 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c3bfff640 1 -- 172.21.15.94:0/2385844575 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c40009d60 con 0x564e805d64d0 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c3bfff640 1 -- 172.21.15.94:0/2385844575 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40016040 con 0x564e805d64d0 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c5b212740 1 -- 172.21.15.94:0/2385844575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 msgr2=0x564e805eb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c5b212740 1 --2- 172.21.15.94:0/2385844575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805eb890 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f3c400094b0 tx=0x7f3c4000ecf0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c5b212740 1 -- 172.21.15.94:0/2385844575 shutdown_connections 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c5b212740 1 --2- 172.21.15.94:0/2385844575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 0x564e805df260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c5b212740 1 --2- 172.21.15.94:0/2385844575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805ded20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c5b212740 1 --2- 172.21.15.94:0/2385844575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805eb890 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c5b212740 1 -- 172.21.15.94:0/2385844575 >> 172.21.15.94:0/2385844575 conn(0x564e805d0ba0 msgr2=0x564e805db620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c5b212740 1 -- 172.21.15.94:0/2385844575 shutdown_connections 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.841+0000 7f3c5b212740 1 -- 172.21.15.94:0/2385844575 wait complete. 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c5b212740 1 -- start start 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c5b212740 1 Processor -- start 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805e8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805e33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c5b212740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 0x564e805e38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c5b212740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3c501ada30 con 0x564e805d64d0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c5b212740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3c501aeea0 con 0x564e805d6e80 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c5b212740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c501ae100 con 0x564e805dfa00 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c5b212740 10 client.0 _ll_register_callbacks cb 0x7f3c40009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c563c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c563c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805e8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49226/0 (socket says 172.21.15.94:49226) 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c563c1640 1 -- 172.21.15.94:0/1145319930 learned_addr learned my addr 172.21.15.94:0/1145319930 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805e33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c563c1640 1 -- 172.21.15.94:0/1145319930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 msgr2=0x564e805e38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c563c1640 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 0x564e805e38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c563c1640 1 -- 172.21.15.94:0/1145319930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 msgr2=0x564e805e33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c563c1640 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805e33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c563c1640 1 -- 172.21.15.94:0/1145319930 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e805e3fd0 con 0x564e805d64d0 2024-04-21T08:47:04.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c563c1640 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805e8030 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f3c400054e0 tx=0x7f3c40009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af45ce2362bd2cf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.842+0000 7f3c39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:04.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c39ffb640 1 -- 172.21.15.94:0/1145319930 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40016070 con 0x564e805d64d0 2024-04-21T08:47:04.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c39ffb640 1 -- 172.21.15.94:0/1145319930 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f3c400048d0 con 0x564e805d64d0 2024-04-21T08:47:04.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c5b212740 1 -- 172.21.15.94:0/1145319930 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c40009d60 con 0x564e805d64d0 2024-04-21T08:47:04.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c39ffb640 1 -- 172.21.15.94:0/1145319930 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3c40044050 con 0x564e805d64d0 2024-04-21T08:47:04.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c5b212740 10 client.17535 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:04.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c5b212740 1 -- 172.21.15.94:0/1145319930 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564e80655d80 con 0x564e805d64d0 2024-04-21T08:47:04.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c39ffb640 1 -- 172.21.15.94:0/1145319930 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c40009d60 con 0x564e805d64d0 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c39ffb640 1 -- 172.21.15.94:0/1145319930 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3c4003a030 con 0x564e805d64d0 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c5b212740 10 client.17535 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c5b212740 10 client.17535 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c5b212740 20 client.17535 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c5b212740 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564e806563e0 0x564e806587d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c5b212740 4 client.17535 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:47:04.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c5b212740 1 -- 172.21.15.94:0/1145319930 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e80548c10 con 0x564e806563e0 2024-04-21T08:47:04.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.843+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564e806563e0 0x564e806587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c39ffb640 10 client.17535 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:04.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c55bc0640 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564e806563e0 0x564e806587d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b434acb40b9cce11 server_cookie=b6e0e669f2eda16b in_seq=0 out_seq=0 2024-04-21T08:47:04.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c39ffb640 1 -- 172.21.15.94:0/1145319930 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e80548c10 con 0x564e806563e0 2024-04-21T08:47:04.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c39ffb640 10 client.17535 handle_command_reply: tid=0 2024-04-21T08:47:04.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 client.17535 shutdown 2024-04-21T08:47:04.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 -- 172.21.15.94:0/1145319930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 msgr2=0x564e805e8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805e8030 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f3c400054e0 tx=0x7f3c40009f80 comp rx=0 tx=0).stop 2024-04-21T08:47:04.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 -- 172.21.15.94:0/1145319930 shutdown_connections 2024-04-21T08:47:04.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564e806563e0 0x564e806587d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e805dfa00 0x564e805e38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e805d6e80 0x564e805e33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 --2- 172.21.15.94:0/1145319930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564e805d64d0 0x564e805e8030 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c39ffb640 0 client.17535 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:04.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 -- 172.21.15.94:0/1145319930 >> 172.21.15.94:0/1145319930 conn(0x564e805d0ba0 msgr2=0x564e805dbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 -- 172.21.15.94:0/1145319930 shutdown_connections 2024-04-21T08:47:04.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 1 -- 172.21.15.94:0/1145319930 wait complete. 2024-04-21T08:47:04.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c5b212740 20 client.17535 trim_cache size 0 max 16384 2024-04-21T08:47:04.859 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:47:04.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c58625640 1 -- 172.21.15.94:0/3478553882 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3c3c0a3d20 msgr2=0x7f3c3c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c58625640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3c3c0a3d20 0x7f3c3c0a61e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009fd0 tx=0x7f3c48005bd0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c58625640 1 -- 172.21.15.94:0/3478553882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c50101b40 msgr2=0x7f3c5019e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.844+0000 7f3c58625640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c50101b40 0x7f3c5019e960 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009230 tx=0x7f3c4c0045e0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.845+0000 7f3c58625640 1 -- 172.21.15.94:0/3478553882 shutdown_connections 2024-04-21T08:47:04.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.845+0000 7f3c58625640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3c3c0a3d20 0x7f3c3c0a61e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.845+0000 7f3c58625640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c50107030 0x7f3c501985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.845+0000 7f3c58625640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c50101b40 0x7f3c5019e960 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.845+0000 7f3c58625640 1 --2- 172.21.15.94:0/3478553882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c501011a0 0x7f3c5019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.845+0000 7f3c58625640 1 -- 172.21.15.94:0/3478553882 >> 172.21.15.94:0/3478553882 conn(0x7f3c50076c50 msgr2=0x7f3c5006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.845+0000 7f3c58625640 1 -- 172.21.15.94:0/3478553882 shutdown_connections 2024-04-21T08:47:04.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:04.845+0000 7f3c58625640 1 -- 172.21.15.94:0/3478553882 wait complete. 2024-04-21T08:47:05.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:05 smithi111 ceph-mon[31493]: pgmap v216: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-21T08:47:05.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:05 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/634489961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:05 smithi094 ceph-mon[27988]: pgmap v216: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-21T08:47:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:05 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/634489961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:05.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:05 smithi173 ceph-mon[31428]: pgmap v216: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-21T08:47:05.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:05 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/634489961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:05.864 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:47:05.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:47:05.864 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 -- 172.21.15.94:0/1029716658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe21406b1f0 msgr2=0x7fe214106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 --2- 172.21.15.94:0/1029716658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe21406b1f0 0x7fe214106b90 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe204009900 tx=0x7fe20402bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 -- 172.21.15.94:0/1029716658 shutdown_connections 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 --2- 172.21.15.94:0/1029716658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214107270 0x7fe21410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 --2- 172.21.15.94:0/1029716658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe21406b1f0 0x7fe214106b90 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 --2- 172.21.15.94:0/1029716658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe21406a840 0x7fe21406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 -- 172.21.15.94:0/1029716658 >> 172.21.15.94:0/1029716658 conn(0x7fe214077eb0 msgr2=0x7fe2140782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 -- 172.21.15.94:0/1029716658 shutdown_connections 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 -- 172.21.15.94:0/1029716658 wait complete. 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 Processor -- start 2024-04-21T08:47:05.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.984+0000 7fe21b9a4640 1 -- start start 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe21b9a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe21406a840 0x7fe2141a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe21b9a4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214107270 0x7fe2141a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe21b9a4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe21419a6f0 0x7fe21419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe21b9a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe214112190 con 0x7fe21406a840 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe21b9a4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe214112010 con 0x7fe214107270 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe21b9a4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe214112310 con 0x7fe21419a6f0 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219740640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe21406a840 0x7fe2141a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219740640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe21406a840 0x7fe2141a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49236/0 (socket says 172.21.15.94:49236) 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219740640 1 -- 172.21.15.94:0/2871690167 learned_addr learned my addr 172.21.15.94:0/2871690167 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219f41640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe21419a6f0 0x7fe21419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219740640 1 -- 172.21.15.94:0/2871690167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe21419a6f0 msgr2=0x7fe21419ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219740640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe21419a6f0 0x7fe21419ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe218f3f640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214107270 0x7fe2141a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219740640 1 -- 172.21.15.94:0/2871690167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214107270 msgr2=0x7fe2141a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219740640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214107270 0x7fe2141a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219740640 1 -- 172.21.15.94:0/2871690167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe21419b2f0 con 0x7fe21406a840 2024-04-21T08:47:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219f41640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe21419a6f0 0x7fe21419ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe219740640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe21406a840 0x7fe2141a0630 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fe21000d950 tx=0x7fe21000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe20a7fc640 1 -- 172.21.15.94:0/2871690167 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210017030 con 0x7fe21406a840 2024-04-21T08:47:05.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe20a7fc640 1 -- 172.21.15.94:0/2871690167 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe210004020 con 0x7fe21406a840 2024-04-21T08:47:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe21b9a4640 1 -- 172.21.15.94:0/2871690167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2141a73d0 con 0x7fe21406a840 2024-04-21T08:47:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.985+0000 7fe20a7fc640 1 -- 172.21.15.94:0/2871690167 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2100047a0 con 0x7fe21406a840 2024-04-21T08:47:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.986+0000 7fe21b9a4640 1 -- 172.21.15.94:0/2871690167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2141a7830 con 0x7fe21406a840 2024-04-21T08:47:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.988+0000 7fe20a7fc640 1 -- 172.21.15.94:0/2871690167 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fe210009a30 con 0x7fe21406a840 2024-04-21T08:47:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.988+0000 7fe21e599740 1 Processor -- start 2024-04-21T08:47:05.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.988+0000 7fe21e599740 1 -- start start 2024-04-21T08:47:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.988+0000 7fe21e599740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc528cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.988+0000 7fe21e599740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 0x556bdc531840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.988+0000 7fe21e599740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc534260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.988+0000 7fe21e599740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556bdc48cf00 con 0x556bdc5288e0 2024-04-21T08:47:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.988+0000 7fe21e599740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556bdc42a590 con 0x556bdc531e10 2024-04-21T08:47:05.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.988+0000 7fe21e599740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556bdc488ec0 con 0x556bdc529290 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219f41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc528cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219f41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc528cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49246/0 (socket says 172.21.15.94:49246) 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219f41640 1 -- 172.21.15.94:0/3969142034 learned_addr learned my addr 172.21.15.94:0/3969142034 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe20a7fc640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2000a3da0 0x7fe2000a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219740640 1 --2- 172.21.15.94:0/3969142034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 0x556bdc531840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe218f3f640 1 --2- 172.21.15.94:0/3969142034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc534260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219f41640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2000a3da0 0x7fe2000a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219f41640 1 -- 172.21.15.94:0/3969142034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 msgr2=0x556bdc531840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219f41640 1 --2- 172.21.15.94:0/3969142034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 0x556bdc531840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219f41640 1 -- 172.21.15.94:0/3969142034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 msgr2=0x556bdc534260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219f41640 1 --2- 172.21.15.94:0/3969142034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc534260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219f41640 1 -- 172.21.15.94:0/3969142034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bdc2677e0 con 0x556bdc5288e0 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe20a7fc640 1 -- 172.21.15.94:0/2871690167 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe2100d06c0 con 0x7fe21406a840 2024-04-21T08:47:05.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe219f41640 1 --2- 172.21.15.94:0/3969142034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc528cc0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fe20c013910 tx=0x7fe20c013dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6790e27f1635f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe1ff7fe640 1 -- 172.21.15.94:0/3969142034 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c004b20 con 0x556bdc5288e0 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe1ff7fe640 1 -- 172.21.15.94:0/3969142034 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe20c004cc0 con 0x556bdc5288e0 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.989+0000 7fe1ff7fe640 1 -- 172.21.15.94:0/3969142034 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c005740 con 0x556bdc5288e0 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe21e599740 1 -- 172.21.15.94:0/3969142034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 msgr2=0x556bdc528cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe219f41640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2000a3da0 0x7fe2000a6260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe20c020000 tx=0x7fe20c01f000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe21e599740 1 --2- 172.21.15.94:0/3969142034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc528cc0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fe20c013910 tx=0x7fe20c013dd0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe21e599740 1 -- 172.21.15.94:0/3969142034 shutdown_connections 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe21e599740 1 --2- 172.21.15.94:0/3969142034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 0x556bdc531840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe21e599740 1 --2- 172.21.15.94:0/3969142034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc534260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe21e599740 1 --2- 172.21.15.94:0/3969142034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc528cc0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe21e599740 1 -- 172.21.15.94:0/3969142034 >> 172.21.15.94:0/3969142034 conn(0x556bdc522fb0 msgr2=0x556bdc523390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe21e599740 1 -- 172.21.15.94:0/3969142034 shutdown_connections 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe21e599740 1 -- 172.21.15.94:0/3969142034 wait complete. 2024-04-21T08:47:05.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.990+0000 7fe21e599740 1 -- start start 2024-04-21T08:47:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe21e599740 1 Processor -- start 2024-04-21T08:47:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe21e599740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc52ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe21e599740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc529290 0x556bdc52f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe21e599740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc531e10 0x556bdc52f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe21e599740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556bdc534d80 con 0x556bdc5288e0 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe21e599740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556bdc446580 con 0x556bdc529290 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe21e599740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556bdc534f00 con 0x556bdc531e10 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe21e599740 10 client.0 _ll_register_callbacks cb 0x556bdc44c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe219740640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc52ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe219740640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc52ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49260/0 (socket says 172.21.15.94:49260) 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe219740640 1 -- 172.21.15.94:0/1179357213 learned_addr learned my addr 172.21.15.94:0/1179357213 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe218f3f640 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc529290 0x556bdc52f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe219f41640 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc531e10 0x556bdc52f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe219740640 1 -- 172.21.15.94:0/1179357213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc531e10 msgr2=0x556bdc52f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe219740640 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc531e10 0x556bdc52f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe219740640 1 -- 172.21.15.94:0/1179357213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc529290 msgr2=0x556bdc52f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe219740640 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc529290 0x556bdc52f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe219740640 1 -- 172.21.15.94:0/1179357213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bdc2c7f70 con 0x556bdc5288e0 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe219f41640 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc531e10 0x556bdc52f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.991+0000 7fe218f3f640 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc529290 0x556bdc52f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe219740640 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc52ee20 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe210098c50 tx=0x7fe21006ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42be57bbb1eb99c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe1fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1179357213 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210066d90 con 0x556bdc5288e0 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1179357213 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe210009d20 con 0x556bdc5288e0 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe21e599740 1 -- 172.21.15.94:0/1179357213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe20c004cc0 con 0x556bdc5288e0 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1179357213 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe21009c980 con 0x556bdc5288e0 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe21e599740 10 client.17547 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe21e599740 1 -- 172.21.15.94:0/1179357213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556bdc15f170 con 0x556bdc5288e0 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1179357213 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe20c004cc0 con 0x556bdc5288e0 2024-04-21T08:47:06.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1179357213 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe2100cdcb0 con 0x556bdc5288e0 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe21e599740 10 client.17547 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe21e599740 10 client.17547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe21e599740 20 client.17547 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe21e599740 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bdc5420b0 0x556bdc5444e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe21e599740 4 client.17547 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.992+0000 7fe21e599740 1 -- 172.21.15.94:0/1179357213 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bdc49a920 con 0x556bdc5420b0 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.993+0000 7fe218f3f640 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bdc5420b0 0x556bdc5444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.993+0000 7fe1fcff9640 10 client.17547 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.993+0000 7fe218f3f640 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bdc5420b0 0x556bdc5444e0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebc3ee496996c51 server_cookie=d1cf0e121a10aee7 in_seq=0 out_seq=0 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1179357213 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556bdc49a920 con 0x556bdc5420b0 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe1fcff9640 10 client.17547 handle_command_reply: tid=0 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 client.17547 shutdown 2024-04-21T08:47:06.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 -- 172.21.15.94:0/1179357213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 msgr2=0x556bdc52ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc52ee20 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe210098c50 tx=0x7fe21006ac30 comp rx=0 tx=0).stop 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 -- 172.21.15.94:0/1179357213 shutdown_connections 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bdc5420b0 0x556bdc5444e0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc531e10 0x556bdc52f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc529290 0x556bdc52f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 --2- 172.21.15.94:0/1179357213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc52ee20 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe1fcff9640 0 client.17547 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 -- 172.21.15.94:0/1179357213 >> 172.21.15.94:0/1179357213 conn(0x556bdc522fb0 msgr2=0x556bdc524d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 -- 172.21.15.94:0/1179357213 shutdown_connections 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.994+0000 7fe21e599740 1 -- 172.21.15.94:0/1179357213 wait complete. 2024-04-21T08:47:06.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:05.995+0000 7fe21e599740 20 client.17547 trim_cache size 0 max 16384 2024-04-21T08:47:06.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe21e599740 1 Processor -- start 2024-04-21T08:47:06.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe21e599740 1 -- start start 2024-04-21T08:47:06.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe21e599740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc5288e0 0x556bdc531550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe21e599740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc529290 0x556bdc53dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe21e599740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc52dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe21e599740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe2141b3f50 con 0x556bdc529290 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe21e599740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe2141b1820 con 0x556bdc531e10 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe21e599740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2141af670 con 0x556bdc5288e0 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe218f3f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc529290 0x556bdc53dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe218f3f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc529290 0x556bdc53dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49274/0 (socket says 172.21.15.94:49274) 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.012+0000 7fe218f3f640 1 -- 172.21.15.94:0/2911661867 learned_addr learned my addr 172.21.15.94:0/2911661867 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe219740640 1 --2- 172.21.15.94:0/2911661867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc5288e0 0x556bdc531550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe218f3f640 1 -- 172.21.15.94:0/2911661867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc5288e0 msgr2=0x556bdc531550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe218f3f640 1 --2- 172.21.15.94:0/2911661867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc5288e0 0x556bdc531550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe218f3f640 1 -- 172.21.15.94:0/2911661867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 msgr2=0x556bdc52dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe218f3f640 1 --2- 172.21.15.94:0/2911661867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc52dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe218f3f640 1 -- 172.21.15.94:0/2911661867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bdc52fd80 con 0x556bdc529290 2024-04-21T08:47:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe219740640 1 --2- 172.21.15.94:0/2911661867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc5288e0 0x556bdc531550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe218f3f640 1 --2- 172.21.15.94:0/2911661867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc529290 0x556bdc53dc40 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe204000c00 tx=0x7fe2040095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9381ca600e4a949 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe1feffd640 1 -- 172.21.15.94:0/2911661867 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20404c020 con 0x556bdc529290 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe1feffd640 1 -- 172.21.15.94:0/2911661867 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe20402c7e0 con 0x556bdc529290 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.013+0000 7fe1feffd640 1 -- 172.21.15.94:0/2911661867 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2040055e0 con 0x556bdc529290 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 -- 172.21.15.94:0/2911661867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc529290 msgr2=0x556bdc53dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 --2- 172.21.15.94:0/2911661867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc529290 0x556bdc53dc40 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe204000c00 tx=0x7fe2040095f0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 -- 172.21.15.94:0/2911661867 shutdown_connections 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 --2- 172.21.15.94:0/2911661867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc52dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 --2- 172.21.15.94:0/2911661867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc529290 0x556bdc53dc40 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 --2- 172.21.15.94:0/2911661867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc5288e0 0x556bdc531550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 -- 172.21.15.94:0/2911661867 >> 172.21.15.94:0/2911661867 conn(0x556bdc522fb0 msgr2=0x556bdc52f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 -- 172.21.15.94:0/2911661867 shutdown_connections 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 -- 172.21.15.94:0/2911661867 wait complete. 2024-04-21T08:47:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 -- start start 2024-04-21T08:47:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 Processor -- start 2024-04-21T08:47:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc536a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 0x556bdc536fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc537510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe2141afce0 con 0x556bdc5288e0 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe2141b1150 con 0x556bdc531e10 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2141b03b0 con 0x556bdc529290 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.014+0000 7fe21e599740 10 client.0 _ll_register_callbacks cb 0x7fe210098580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe219740640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc536a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe219740640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc536a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49284/0 (socket says 172.21.15.94:49284) 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe219740640 1 -- 172.21.15.94:0/1200556056 learned_addr learned my addr 172.21.15.94:0/1200556056 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe219f41640 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc537510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe219740640 1 -- 172.21.15.94:0/1200556056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 msgr2=0x556bdc536fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe218f3f640 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 0x556bdc536fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe219740640 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 0x556bdc536fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe219740640 1 -- 172.21.15.94:0/1200556056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 msgr2=0x556bdc537510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe219740640 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc537510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe219740640 1 -- 172.21.15.94:0/1200556056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bdc5a8600 con 0x556bdc5288e0 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe218f3f640 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 0x556bdc536fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe219740640 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc536a90 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fe210005fd0 tx=0x7fe21009c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc388339f39c341 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe1fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1200556056 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2100a4c60 con 0x556bdc5288e0 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1200556056 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe2100d0280 con 0x556bdc5288e0 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe21e599740 1 -- 172.21.15.94:0/1200556056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe20402c7e0 con 0x556bdc5288e0 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1200556056 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe21009c280 con 0x556bdc5288e0 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe21e599740 10 client.17553 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:06.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe21e599740 1 -- 172.21.15.94:0/1200556056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556bdc5a97b0 con 0x556bdc5288e0 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.015+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1200556056 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe20402c7e0 con 0x556bdc5288e0 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.016+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1200556056 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe2100a0160 con 0x556bdc5288e0 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.016+0000 7fe21e599740 10 client.17553 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.016+0000 7fe21e599740 10 client.17553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.016+0000 7fe21e599740 20 client.17553 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.016+0000 7fe21e599740 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bdc5a9d70 0x556bdc5ac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.016+0000 7fe21e599740 4 client.17553 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.016+0000 7fe21e599740 1 -- 172.21.15.94:0/1200556056 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556bdc494790 con 0x556bdc5a9d70 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.016+0000 7fe218f3f640 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bdc5a9d70 0x556bdc5ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.016+0000 7fe218f3f640 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bdc5a9d70 0x556bdc5ac190 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c771f9f0f9dbefbc server_cookie=cc661632d2e1e5e5 in_seq=0 out_seq=0 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.016+0000 7fe1fcff9640 10 client.17553 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe1fcff9640 1 -- 172.21.15.94:0/1200556056 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556bdc494790 con 0x556bdc5a9d70 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe1fcff9640 10 client.17553 handle_command_reply: tid=0 2024-04-21T08:47:06.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 client.17553 shutdown 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 -- 172.21.15.94:0/1200556056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 msgr2=0x556bdc536a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc536a90 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fe210005fd0 tx=0x7fe21009c050 comp rx=0 tx=0).stop 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 -- 172.21.15.94:0/1200556056 shutdown_connections 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x556bdc5a9d70 0x556bdc5ac190 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bdc529290 0x556bdc536fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556bdc531e10 0x556bdc537510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 --2- 172.21.15.94:0/1200556056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556bdc5288e0 0x556bdc536a90 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe1fcff9640 0 client.17553 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 -- 172.21.15.94:0/1200556056 >> 172.21.15.94:0/1200556056 conn(0x556bdc522fb0 msgr2=0x556bdc521780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 -- 172.21.15.94:0/1200556056 shutdown_connections 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 1 -- 172.21.15.94:0/1200556056 wait complete. 2024-04-21T08:47:06.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.018+0000 7fe21e599740 20 client.17553 trim_cache size 0 max 16384 2024-04-21T08:47:06.025 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:47:06.025 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:47:06.025 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "71e00ee3-e9ff-46a2-a895-1fec175fdad4", 2024-04-21T08:47:06.025 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:47:06.025 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 -- 172.21.15.94:0/2871690167 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2000a3da0 msgr2=0x7fe2000a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2000a3da0 0x7fe2000a6260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe20c020000 tx=0x7fe20c01f000 comp rx=0 tx=0).stop 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 -- 172.21.15.94:0/2871690167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe21406a840 msgr2=0x7fe2141a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe21406a840 0x7fe2141a0630 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fe21000d950 tx=0x7fe21000de10 comp rx=0 tx=0).stop 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 -- 172.21.15.94:0/2871690167 shutdown_connections 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2000a3da0 0x7fe2000a6260 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe21419a6f0 0x7fe21419ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214107270 0x7fe2141a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 --2- 172.21.15.94:0/2871690167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe21406a840 0x7fe2141a0630 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 -- 172.21.15.94:0/2871690167 >> 172.21.15.94:0/2871690167 conn(0x7fe214077eb0 msgr2=0x7fe21410d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 -- 172.21.15.94:0/2871690167 shutdown_connections 2024-04-21T08:47:06.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.019+0000 7fe21b9a4640 1 -- 172.21.15.94:0/2871690167 wait complete. 2024-04-21T08:47:06.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:47:06.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:47:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.130+0000 7fcbb4511640 1 -- 172.21.15.94:0/645654474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbac106f30 msgr2=0x7fcbac107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.130+0000 7fcbb4511640 1 --2- 172.21.15.94:0/645654474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbac106f30 0x7fcbac107310 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009930 tx=0x7fcba802bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 -- 172.21.15.94:0/645654474 shutdown_connections 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 --2- 172.21.15.94:0/645654474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbac100450 0x7fcbac1008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 --2- 172.21.15.94:0/645654474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac100e10 0x7fcbac10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 --2- 172.21.15.94:0/645654474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbac106f30 0x7fcbac107310 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 -- 172.21.15.94:0/645654474 >> 172.21.15.94:0/645654474 conn(0x7fcbac077410 msgr2=0x7fcbac0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 -- 172.21.15.94:0/645654474 shutdown_connections 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 -- 172.21.15.94:0/645654474 wait complete. 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 Processor -- start 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 -- start start 2024-04-21T08:47:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbac100450 0x7fcbac1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac100e10 0x7fcbac1a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbac106f30 0x7fcbac19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac112270 con 0x7fcbac100450 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1120f0 con 0x7fcbac100e10 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb4511640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1123f0 con 0x7fcbac106f30 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb22ad640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbac100450 0x7fcbac1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb22ad640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbac100450 0x7fcbac1a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49304/0 (socket says 172.21.15.94:49304) 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb22ad640 1 -- 172.21.15.94:0/1259754151 learned_addr learned my addr 172.21.15.94:0/1259754151 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.131+0000 7fcbb2aae640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbac106f30 0x7fcbac19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac100e10 0x7fcbac1a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcbb22ad640 1 -- 172.21.15.94:0/1259754151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbac106f30 msgr2=0x7fcbac19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbac106f30 0x7fcbac19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcbb22ad640 1 -- 172.21.15.94:0/1259754151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac100e10 msgr2=0x7fcbac1a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac100e10 0x7fcbac1a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcbb22ad640 1 -- 172.21.15.94:0/1259754151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbac19afe0 con 0x7fcbac100450 2024-04-21T08:47:06.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbac100450 0x7fcbac1a06a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009230 tx=0x7fcba80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcba37fe640 1 -- 172.21.15.94:0/1259754151 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8043020 con 0x7fcbac100450 2024-04-21T08:47:06.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcba37fe640 1 -- 172.21.15.94:0/1259754151 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcba802cb70 con 0x7fcbac100450 2024-04-21T08:47:06.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcbb4511640 1 -- 172.21.15.94:0/1259754151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbac19b270 con 0x7fcbac100450 2024-04-21T08:47:06.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcba37fe640 1 -- 172.21.15.94:0/1259754151 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8003b50 con 0x7fcbac100450 2024-04-21T08:47:06.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.132+0000 7fcbb4511640 1 -- 172.21.15.94:0/1259754151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbac1a74f0 con 0x7fcbac100450 2024-04-21T08:47:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.134+0000 7fcba37fe640 1 -- 172.21.15.94:0/1259754151 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fcba8046020 con 0x7fcbac100450 2024-04-21T08:47:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb71d6740 1 Processor -- start 2024-04-21T08:47:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb71d6740 1 -- start start 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339677580 0x559339677960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339680540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339680a80 0x559339682e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcba37fe640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb980a3d20 0x7fcb980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5593395dc1e0 con 0x559339677ea0 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559339579390 con 0x559339677580 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5593395d81a0 con 0x559339680a80 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb22ad640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339680540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb2aae640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb980a3d20 0x7fcb980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb22ad640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339680540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49312/0 (socket says 172.21.15.94:49312) 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb22ad640 1 -- 172.21.15.94:0/917547188 learned_addr learned my addr 172.21.15.94:0/917547188 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/917547188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339680a80 0x559339682e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb2aae640 1 --2- 172.21.15.94:0/917547188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339677580 0x559339677960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcba37fe640 1 -- 172.21.15.94:0/1259754151 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fcba80f2c50 con 0x7fcbac100450 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb22ad640 1 -- 172.21.15.94:0/917547188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339680a80 msgr2=0x559339682e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/917547188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339680a80 0x559339682e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb22ad640 1 -- 172.21.15.94:0/917547188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339677580 msgr2=0x559339677960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.135+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/917547188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339677580 0x559339677960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.136+0000 7fcbb22ad640 1 -- 172.21.15.94:0/917547188 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593393b87d0 con 0x559339677ea0 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.136+0000 7fcbb2aae640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb980a3d20 0x7fcb980a61e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcba4005fd0 tx=0x7fcba4014b70 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.136+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/917547188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339680540 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fcba8006010 tx=0x7fcba802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22dedcb99f5f6130 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.136+0000 7fcba0bfe640 1 -- 172.21.15.94:0/917547188 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba80a7e60 con 0x559339677ea0 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.136+0000 7fcba0bfe640 1 -- 172.21.15.94:0/917547188 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcba80bf020 con 0x559339677ea0 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.136+0000 7fcba0bfe640 1 -- 172.21.15.94:0/917547188 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba80baf90 con 0x559339677ea0 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 -- 172.21.15.94:0/917547188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 msgr2=0x559339680540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/917547188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339680540 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fcba8006010 tx=0x7fcba802bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 -- 172.21.15.94:0/917547188 shutdown_connections 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/917547188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339680a80 0x559339682e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/917547188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339680540 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/917547188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339677580 0x559339677960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 -- 172.21.15.94:0/917547188 >> 172.21.15.94:0/917547188 conn(0x559339671cb0 msgr2=0x559339672090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 -- 172.21.15.94:0/917547188 shutdown_connections 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 -- 172.21.15.94:0/917547188 wait complete. 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 -- start start 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 Processor -- start 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677580 0x55933967da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677ea0 0x55933967df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x55933967e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x559339683630 con 0x559339677580 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5593395952f0 con 0x559339680a80 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5593396837b0 con 0x559339677ea0 2024-04-21T08:47:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb71d6740 10 client.0 _ll_register_callbacks cb 0x5593395a2580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb22ad640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677580 0x55933967da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb22ad640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677580 0x55933967da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49326/0 (socket says 172.21.15.94:49326) 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.137+0000 7fcbb22ad640 1 -- 172.21.15.94:0/180935789 learned_addr learned my addr 172.21.15.94:0/180935789 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677ea0 0x55933967df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb2aae640 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x55933967e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb22ad640 1 -- 172.21.15.94:0/180935789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677ea0 msgr2=0x55933967df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677ea0 0x55933967df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb22ad640 1 -- 172.21.15.94:0/180935789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 msgr2=0x55933967e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x55933967e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb22ad640 1 -- 172.21.15.94:0/180935789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593392ee3f0 con 0x559339677580 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb2aae640 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x55933967e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677580 0x55933967da40 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fcba80be8b0 tx=0x7fcba80c2d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a014e1047530c88c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcb95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcb95ffb640 1 -- 172.21.15.94:0/180935789 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba80a7e60 con 0x559339677580 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcb95ffb640 1 -- 172.21.15.94:0/180935789 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fcba80a90e0 con 0x559339677580 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb71d6740 1 -- 172.21.15.94:0/180935789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcba80bf020 con 0x559339677580 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcb95ffb640 1 -- 172.21.15.94:0/180935789 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcba80056a0 con 0x559339677580 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb71d6740 10 client.17565 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb71d6740 1 -- 172.21.15.94:0/180935789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5593392ae0f0 con 0x559339677580 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcb95ffb640 1 -- 172.21.15.94:0/180935789 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcba80bf020 con 0x559339677580 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcb95ffb640 1 -- 172.21.15.94:0/180935789 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fcba80c7100 con 0x559339677580 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb71d6740 10 client.17565 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.138+0000 7fcbb71d6740 10 client.17565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.139+0000 7fcbb71d6740 20 client.17565 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.139+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5593396905d0 0x5593396929c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.139+0000 7fcbb71d6740 4 client.17565 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.139+0000 7fcbb71d6740 1 -- 172.21.15.94:0/180935789 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593395e9c00 con 0x5593396905d0 2024-04-21T08:47:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.139+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5593396905d0 0x5593396929c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.139+0000 7fcb95ffb640 10 client.17565 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.139+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5593396905d0 0x5593396929c0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5874a5d342d2584 server_cookie=5356f05a1420e419 in_seq=0 out_seq=0 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcb95ffb640 1 -- 172.21.15.94:0/180935789 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5593395e9c00 con 0x5593396905d0 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcb95ffb640 10 client.17565 handle_command_reply: tid=0 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 client.17565 shutdown 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 -- 172.21.15.94:0/180935789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677580 msgr2=0x55933967da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677580 0x55933967da40 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fcba80be8b0 tx=0x7fcba80c2d40 comp rx=0 tx=0).stop 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 -- 172.21.15.94:0/180935789 shutdown_connections 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5593396905d0 0x5593396929c0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677ea0 0x55933967df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcb95ffb640 0 client.17565 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x55933967e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/180935789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677580 0x55933967da40 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 -- 172.21.15.94:0/180935789 >> 172.21.15.94:0/180935789 conn(0x559339671cb0 msgr2=0x559339673a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 -- 172.21.15.94:0/180935789 shutdown_connections 2024-04-21T08:47:06.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 1 -- 172.21.15.94:0/180935789 wait complete. 2024-04-21T08:47:06.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.140+0000 7fcbb71d6740 20 client.17565 trim_cache size 0 max 16384 2024-04-21T08:47:06.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb71d6740 1 Processor -- start 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb71d6740 1 -- start start 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677580 0x559339680170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x55933968c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x559339673710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1b3f50 con 0x559339677ea0 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1b1820 con 0x559339680a80 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1af670 con 0x559339677580 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb1aac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x55933968c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb1aac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x55933968c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49332/0 (socket says 172.21.15.94:49332) 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb1aac640 1 -- 172.21.15.94:0/1979451481 learned_addr learned my addr 172.21.15.94:0/1979451481 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/1979451481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677580 0x559339680170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb2aae640 1 --2- 172.21.15.94:0/1979451481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x559339673710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb1aac640 1 -- 172.21.15.94:0/1979451481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677580 msgr2=0x559339680170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/1979451481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677580 0x559339680170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb1aac640 1 -- 172.21.15.94:0/1979451481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 msgr2=0x559339673710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/1979451481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x559339673710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb1aac640 1 -- 172.21.15.94:0/1979451481 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559339692d30 con 0x559339677ea0 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb2aae640 1 --2- 172.21.15.94:0/1979451481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x559339673710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.158+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/1979451481 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x55933968c460 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c005d00 tx=0x7fcb9c00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1f2998d14eb62c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcb97fff640 1 -- 172.21.15.94:0/1979451481 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c004220 con 0x559339677ea0 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcb97fff640 1 -- 172.21.15.94:0/1979451481 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb9c0043c0 con 0x559339677ea0 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcb97fff640 1 -- 172.21.15.94:0/1979451481 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c020230 con 0x559339677ea0 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcbb71d6740 1 -- 172.21.15.94:0/1979451481 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 msgr2=0x55933968c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/1979451481 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x55933968c460 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c005d00 tx=0x7fcb9c00b410 comp rx=0 tx=0).stop 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcbb71d6740 1 -- 172.21.15.94:0/1979451481 shutdown_connections 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/1979451481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x559339673710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/1979451481 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x55933968c460 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/1979451481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677580 0x559339680170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcbb71d6740 1 -- 172.21.15.94:0/1979451481 >> 172.21.15.94:0/1979451481 conn(0x559339671cb0 msgr2=0x559339673000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcbb71d6740 1 -- 172.21.15.94:0/1979451481 shutdown_connections 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.159+0000 7fcbb71d6740 1 -- 172.21.15.94:0/1979451481 wait complete. 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb71d6740 1 -- start start 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb71d6740 1 Processor -- start 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677580 0x5593396844e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339684a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb71d6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x559339684f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1afce0 con 0x559339677ea0 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1b1150 con 0x559339680a80 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb71d6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1b03b0 con 0x559339677580 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb71d6740 10 client.0 _ll_register_callbacks cb 0x7fcb9c00b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb1aac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339684a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb1aac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339684a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49334/0 (socket says 172.21.15.94:49334) 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb1aac640 1 -- 172.21.15.94:0/3203972984 learned_addr learned my addr 172.21.15.94:0/3203972984 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677580 0x5593396844e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb2aae640 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x559339684f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb1aac640 1 -- 172.21.15.94:0/3203972984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677580 msgr2=0x5593396844e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677580 0x5593396844e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb1aac640 1 -- 172.21.15.94:0/3203972984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 msgr2=0x559339684f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x559339684f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb1aac640 1 -- 172.21.15.94:0/3203972984 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593396f5c10 con 0x559339677ea0 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcbb1aac640 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339684a20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c025000 tx=0x7fcb9c004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d6f8030f0e8b2e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.160+0000 7fcb95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcb95ffb640 1 -- 172.21.15.94:0/3203972984 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c004930 con 0x559339677ea0 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcb95ffb640 1 -- 172.21.15.94:0/3203972984 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fcb9c020230 con 0x559339677ea0 2024-04-21T08:47:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcbb71d6740 1 -- 172.21.15.94:0/3203972984 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb9c0043c0 con 0x559339677ea0 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcb95ffb640 1 -- 172.21.15.94:0/3203972984 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcb9c037050 con 0x559339677ea0 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcbb71d6740 10 client.17571 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcbb71d6740 1 -- 172.21.15.94:0/3203972984 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5593396f6e70 con 0x559339677ea0 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcb95ffb640 1 -- 172.21.15.94:0/3203972984 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb9c0043c0 con 0x559339677ea0 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcb95ffb640 1 -- 172.21.15.94:0/3203972984 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fcb9c0054c0 con 0x559339677ea0 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcbb71d6740 10 client.17571 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcbb71d6740 10 client.17571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcbb71d6740 20 client.17571 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5593396f7560 0x5593396f9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcbb71d6740 4 client.17571 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcbb71d6740 1 -- 172.21.15.94:0/3203972984 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5593395e3a70 con 0x5593396f7560 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.161+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5593396f7560 0x5593396f9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcb95ffb640 10 client.17571 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb22ad640 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5593396f7560 0x5593396f9980 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e917e8fa9d461c3 server_cookie=9eec591261e8a568 in_seq=0 out_seq=0 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcb95ffb640 1 -- 172.21.15.94:0/3203972984 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5593395e3a70 con 0x5593396f7560 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcb95ffb640 10 client.17571 handle_command_reply: tid=0 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 client.17571 shutdown 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 -- 172.21.15.94:0/3203972984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 msgr2=0x559339684a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339684a20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c025000 tx=0x7fcb9c004cc0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 -- 172.21.15.94:0/3203972984 shutdown_connections 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5593396f7560 0x5593396f9980 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559339680a80 0x559339684f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559339677ea0 0x559339684a20 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcb95ffb640 0 client.17571 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 --2- 172.21.15.94:0/3203972984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559339677580 0x5593396844e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 -- 172.21.15.94:0/3203972984 >> 172.21.15.94:0/3203972984 conn(0x559339671cb0 msgr2=0x55933967e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 -- 172.21.15.94:0/3203972984 shutdown_connections 2024-04-21T08:47:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 1 -- 172.21.15.94:0/3203972984 wait complete. 2024-04-21T08:47:06.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.162+0000 7fcbb71d6740 20 client.17571 trim_cache size 0 max 16384 2024-04-21T08:47:06.170 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:47:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:47:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:47:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "71e00ee3-e9ff-46a2-a895-1fec175fdad4": { 2024-04-21T08:47:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:47:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "71e00ee3-e9ff-46a2-a895-1fec175fdad4", 2024-04-21T08:47:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:47:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:47:06.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:47:06.171 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:47:06.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 -- 172.21.15.94:0/1259754151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb980a3d20 msgr2=0x7fcb980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb980a3d20 0x7fcb980a61e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcba4005fd0 tx=0x7fcba4014b70 comp rx=0 tx=0).stop 2024-04-21T08:47:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 -- 172.21.15.94:0/1259754151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbac100450 msgr2=0x7fcbac1a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbac100450 0x7fcbac1a06a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009230 tx=0x7fcba80045e0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 -- 172.21.15.94:0/1259754151 shutdown_connections 2024-04-21T08:47:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb980a3d20 0x7fcb980a61e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbac106f30 0x7fcbac19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac100e10 0x7fcbac1a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 --2- 172.21.15.94:0/1259754151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbac100450 0x7fcbac1a06a0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 -- 172.21.15.94:0/1259754151 >> 172.21.15.94:0/1259754151 conn(0x7fcbac077410 msgr2=0x7fcbac1052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 -- 172.21.15.94:0/1259754151 shutdown_connections 2024-04-21T08:47:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:06.163+0000 7fcbb4511640 1 -- 172.21.15.94:0/1259754151 wait complete. 2024-04-21T08:47:06.173 INFO:tasks.cephfs.filesystem:scrub status for tag:71e00ee3-e9ff-46a2-a895-1fec175fdad4 - {'path': '/', 'tag': '71e00ee3-e9ff-46a2-a895-1fec175fdad4', 'options': 'recursive,force'} 2024-04-21T08:47:07.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:07 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:47:07.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:07 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:07.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:07 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:47:07.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:07 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:07.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:07 smithi111 ceph-mon[31493]: pgmap v217: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:47:07.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:07 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:47:07.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:07 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:47:07.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:07 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:07.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:07 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:47:07.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:07 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:07.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:07 smithi094 ceph-mon[27988]: pgmap v217: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:47:07.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:07 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:47:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:07 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:47:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:07 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:07 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:47:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:07 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:07 smithi173 ceph-mon[31428]: pgmap v217: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:47:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:07 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:47:08.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:08 smithi111 ceph-mon[31493]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:47:08.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:08 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:08.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:08 smithi094 ceph-mon[27988]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:47:08.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:08 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:08.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:08 smithi173 ceph-mon[31428]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:47:08.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:08 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:09.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:09 smithi111 ceph-mon[31493]: pgmap v219: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:47:09.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:09 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:09.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:09 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:09.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:09 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:47:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:09 smithi094 ceph-mon[27988]: pgmap v219: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:47:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:09 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:09 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:09 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:47:09.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:09 smithi173 ceph-mon[31428]: pgmap v219: 129 pgs: 129 active+clean; 11 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:47:09.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:09 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:09.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:09 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:09.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:09 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:47:11.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:11 smithi111 ceph-mon[31493]: pgmap v220: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-04-21T08:47:11.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:11 smithi094 ceph-mon[27988]: pgmap v220: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-04-21T08:47:11.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:11 smithi173 ceph-mon[31428]: pgmap v220: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-04-21T08:47:13.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:12 smithi111 ceph-mon[31493]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:47:13.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:12 smithi094 ceph-mon[27988]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:47:13.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:12 smithi173 ceph-mon[31428]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:47:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:13 smithi111 ceph-mon[31493]: pgmap v222: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 32 KiB/s wr, 27 op/s 2024-04-21T08:47:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:13 smithi094 ceph-mon[27988]: pgmap v222: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 32 KiB/s wr, 27 op/s 2024-04-21T08:47:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:13 smithi173 ceph-mon[31428]: pgmap v222: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 32 KiB/s wr, 27 op/s 2024-04-21T08:47:16.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:15 smithi111 ceph-mon[31493]: pgmap v223: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:47:16.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:15 smithi094 ceph-mon[27988]: pgmap v223: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:47:16.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:15 smithi173 ceph-mon[31428]: pgmap v223: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:47:18.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:17 smithi111 ceph-mon[31493]: pgmap v224: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 230 B/s wr, 8 op/s 2024-04-21T08:47:18.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:17 smithi094 ceph-mon[27988]: pgmap v224: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 230 B/s wr, 8 op/s 2024-04-21T08:47:18.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:17 smithi173 ceph-mon[31428]: pgmap v224: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 230 B/s wr, 8 op/s 2024-04-21T08:47:20.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:19 smithi111 ceph-mon[31493]: pgmap v225: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-04-21T08:47:20.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:19 smithi094 ceph-mon[27988]: pgmap v225: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-04-21T08:47:20.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:19 smithi173 ceph-mon[31428]: pgmap v225: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-04-21T08:47:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:21 smithi111 ceph-mon[31493]: pgmap v226: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T08:47:22.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:21 smithi094 ceph-mon[27988]: pgmap v226: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T08:47:22.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:21 smithi173 ceph-mon[31428]: pgmap v226: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T08:47:24.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:23 smithi111 ceph-mon[31493]: pgmap v227: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 100 B/s wr, 5 op/s 2024-04-21T08:47:24.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:23 smithi094 ceph-mon[27988]: pgmap v227: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 100 B/s wr, 5 op/s 2024-04-21T08:47:24.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:23 smithi173 ceph-mon[31428]: pgmap v227: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 100 B/s wr, 5 op/s 2024-04-21T08:47:26.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:25 smithi111 ceph-mon[31493]: pgmap v228: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:47:26.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:25 smithi094 ceph-mon[27988]: pgmap v228: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:47:26.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:25 smithi173 ceph-mon[31428]: pgmap v228: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:47:28.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:27 smithi111 ceph-mon[31493]: pgmap v229: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T08:47:28.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:27 smithi094 ceph-mon[27988]: pgmap v229: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T08:47:28.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:27 smithi173 ceph-mon[31428]: pgmap v229: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T08:47:30.190 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:29 smithi111 ceph-mon[31493]: pgmap v230: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T08:47:30.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:29 smithi094 ceph-mon[27988]: pgmap v230: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T08:47:30.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:29 smithi173 ceph-mon[31428]: pgmap v230: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T08:47:32.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:31 smithi111 ceph-mon[31493]: pgmap v231: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:47:32.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:31 smithi094 ceph-mon[27988]: pgmap v231: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:47:32.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:31 smithi173 ceph-mon[31428]: pgmap v231: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:47:34.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:33 smithi111 ceph-mon[31493]: pgmap v232: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:34.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:33 smithi094 ceph-mon[27988]: pgmap v232: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:34.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:33 smithi173 ceph-mon[31428]: pgmap v232: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:36.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:47:36.174 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:47:36.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:35 smithi111 ceph-mon[31493]: pgmap v233: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:47:36.199 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:35 smithi094 ceph-mon[27988]: pgmap v233: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:47:36.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:35 smithi173 ceph-mon[31428]: pgmap v233: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:47:36.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/4070139634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f18101080 msgr2=0x7f7f18101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 --2- 172.21.15.94:0/4070139634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f18101080 0x7f7f18101460 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009ca0 tx=0x7f7f0802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/4070139634 shutdown_connections 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 --2- 172.21.15.94:0/4070139634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f18106150 0x7f7f1810dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 --2- 172.21.15.94:0/4070139634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f18101a30 0x7f7f18105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 --2- 172.21.15.94:0/4070139634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f18101080 0x7f7f18101460 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/4070139634 >> 172.21.15.94:0/4070139634 conn(0x7f7f18078000 msgr2=0x7f7f18078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/4070139634 shutdown_connections 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/4070139634 wait complete. 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 Processor -- start 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.285+0000 7f7f1d9bd640 1 -- start start 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f1d9bd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f18101080 0x7f7f18198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f1d9bd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f18106150 0x7f7f18198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f1d9bd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1819cfc0 0x7f7f1819d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f1d9bd640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181110c0 con 0x7f7f18101080 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f1d9bd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7f18110f40 con 0x7f7f1819cfc0 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f1d9bd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f18111240 con 0x7f7f18106150 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f16ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f18101080 0x7f7f18198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f16ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f18101080 0x7f7f18198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47916/0 (socket says 172.21.15.94:47916) 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f16ffd640 1 -- 172.21.15.94:0/3231798246 learned_addr learned my addr 172.21.15.94:0/3231798246 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f167fc640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f18106150 0x7f7f18198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f177fe640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1819cfc0 0x7f7f1819d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f16ffd640 1 -- 172.21.15.94:0/3231798246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f18106150 msgr2=0x7f7f18198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f18106150 0x7f7f18198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f16ffd640 1 -- 172.21.15.94:0/3231798246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1819cfc0 msgr2=0x7f7f1819d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1819cfc0 0x7f7f1819d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f16ffd640 1 -- 172.21.15.94:0/3231798246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1819dab0 con 0x7f7f18101080 2024-04-21T08:47:36.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f177fe640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1819cfc0 0x7f7f1819d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:36.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f18101080 0x7f7f18198700 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f7f08005e30 tx=0x7f7f08034780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f1c9bb640 1 -- 172.21.15.94:0/3231798246 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08039040 con 0x7f7f18101080 2024-04-21T08:47:36.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f1c9bb640 1 -- 172.21.15.94:0/3231798246 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f08044070 con 0x7f7f18101080 2024-04-21T08:47:36.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f1c9bb640 1 -- 172.21.15.94:0/3231798246 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08003e60 con 0x7f7f18101080 2024-04-21T08:47:36.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.286+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/3231798246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1819dd40 con 0x7f7f18101080 2024-04-21T08:47:36.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.287+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/3231798246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f181a54a0 con 0x7f7f18101080 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f1c9bb640 1 -- 172.21.15.94:0/3231798246 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f7f080324c0 con 0x7f7f18101080 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f205b3740 1 Processor -- start 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f205b3740 1 -- start start 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f1c9bb640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f000a3d20 0x7f7f000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586be580 0x564f586be960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586beea0 0x564f586c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586c7a80 0x564f586c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f205b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564f586231e0 con 0x564f586c7a80 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f205b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564f585c0390 con 0x564f586beea0 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f205b3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564f5861f1a0 con 0x564f586be580 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f177fe640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f000a3d20 0x7f7f000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586c7a80 0x564f586c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f1c9bb640 1 -- 172.21.15.94:0/3231798246 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7f7f080f2ef0 con 0x7f7f18101080 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f16ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586be580 0x564f586be960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586c7a80 0x564f586c9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47930/0 (socket says 172.21.15.94:47930) 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.289+0000 7f7f177fe640 1 -- 172.21.15.94:0/2068586847 learned_addr learned my addr 172.21.15.94:0/2068586847 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7f167fc640 1 --2- 172.21.15.94:0/2068586847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586beea0 0x564f586c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7f177fe640 1 -- 172.21.15.94:0/2068586847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586be580 msgr2=0x564f586be960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7f177fe640 1 --2- 172.21.15.94:0/2068586847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586be580 0x564f586be960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7f177fe640 1 -- 172.21.15.94:0/2068586847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586beea0 msgr2=0x564f586c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7f177fe640 1 --2- 172.21.15.94:0/2068586847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586beea0 0x564f586c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7f177fe640 1 -- 172.21.15.94:0/2068586847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f583ff7d0 con 0x564f586c7a80 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/2068586847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586be580 0x564f586be960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7f177fe640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f000a3d20 0x7f7f000a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7f10013800 tx=0x7f7f1000ba50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7f177fe640 1 --2- 172.21.15.94:0/2068586847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586c7a80 0x564f586c9e70 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f7f10005cd0 tx=0x7f7f1001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e865836d205b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7efdffb640 1 -- 172.21.15.94:0/2068586847 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10023070 con 0x564f586c7a80 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7efdffb640 1 -- 172.21.15.94:0/2068586847 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f1000ec10 con 0x564f586c7a80 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.290+0000 7f7efdffb640 1 -- 172.21.15.94:0/2068586847 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1001fb00 con 0x564f586c7a80 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 -- 172.21.15.94:0/2068586847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586c7a80 msgr2=0x564f586c9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 --2- 172.21.15.94:0/2068586847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586c7a80 0x564f586c9e70 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f7f10005cd0 tx=0x7f7f1001a000 comp rx=0 tx=0).stop 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 -- 172.21.15.94:0/2068586847 shutdown_connections 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 --2- 172.21.15.94:0/2068586847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586c7a80 0x564f586c9e70 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 --2- 172.21.15.94:0/2068586847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586beea0 0x564f586c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 --2- 172.21.15.94:0/2068586847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586be580 0x564f586be960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 -- 172.21.15.94:0/2068586847 >> 172.21.15.94:0/2068586847 conn(0x564f586b8cb0 msgr2=0x564f586b9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 -- 172.21.15.94:0/2068586847 shutdown_connections 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 -- 172.21.15.94:0/2068586847 wait complete. 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 -- start start 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 Processor -- start 2024-04-21T08:47:36.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586be580 0x564f586c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.291+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586beea0 0x564f586c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586c7a80 0x564f586c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f205b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564f586ca810 con 0x564f586beea0 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f205b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564f585dc2f0 con 0x564f586be580 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f205b3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564f586ca990 con 0x564f586c7a80 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f205b3740 10 client.0 _ll_register_callbacks cb 0x564f585e9580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f167fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586beea0 0x564f586c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f167fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586beea0 0x564f586c4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47940/0 (socket says 172.21.15.94:47940) 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f167fc640 1 -- 172.21.15.94:0/3231173176 learned_addr learned my addr 172.21.15.94:0/3231173176 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f177fe640 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586c7a80 0x564f586c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586be580 0x564f586c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f167fc640 1 -- 172.21.15.94:0/3231173176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586c7a80 msgr2=0x564f586c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f167fc640 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586c7a80 0x564f586c5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f167fc640 1 -- 172.21.15.94:0/3231173176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586be580 msgr2=0x564f586c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f167fc640 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586be580 0x564f586c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f167fc640 1 -- 172.21.15.94:0/3231173176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f583353f0 con 0x564f586beea0 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586be580 0x564f586c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f167fc640 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586beea0 0x564f586c4fc0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00be50 tx=0x7f7f0c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1330fb2f9cb23fa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7ef6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/3231173176 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c004230 con 0x564f586beea0 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/3231173176 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7f7f0c0043d0 con 0x564f586beea0 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f205b3740 1 -- 172.21.15.94:0/3231173176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f1000ec10 con 0x564f586beea0 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/3231173176 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7f0c024050 con 0x564f586beea0 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f205b3740 10 client.17583 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.292+0000 7f7f205b3740 1 -- 172.21.15.94:0/3231173176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564f582f50f0 con 0x564f586beea0 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.293+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/3231173176 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f1000ec10 con 0x564f586beea0 2024-04-21T08:47:36.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.293+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/3231173176 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7f0c01b740 con 0x564f586beea0 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.293+0000 7f7f205b3740 10 client.17583 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.293+0000 7f7f205b3740 10 client.17583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.293+0000 7f7f205b3740 20 client.17583 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.293+0000 7f7f205b3740 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564f586d7ab0 0x564f586d9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.293+0000 7f7f205b3740 4 client.17583 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.293+0000 7f7f205b3740 1 -- 172.21.15.94:0/3231173176 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f58630c00 con 0x564f586d7ab0 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.293+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564f586d7ab0 0x564f586d9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.294+0000 7f7ef6ffd640 10 client.17583 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.294+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564f586d7ab0 0x564f586d9f00 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5136eae81415f8f7 server_cookie=69019af6acc4a2b5 in_seq=0 out_seq=0 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.294+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/3231173176 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f58630c00 con 0x564f586d7ab0 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.294+0000 7f7ef6ffd640 10 client.17583 handle_command_reply: tid=0 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.294+0000 7f7f205b3740 1 client.17583 shutdown 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.294+0000 7f7f205b3740 1 -- 172.21.15.94:0/3231173176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586beea0 msgr2=0x564f586c4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.294+0000 7f7f205b3740 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586beea0 0x564f586c4fc0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00be50 tx=0x7f7f0c0117b0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.294+0000 7f7f205b3740 1 -- 172.21.15.94:0/3231173176 shutdown_connections 2024-04-21T08:47:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.294+0000 7f7f205b3740 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564f586d7ab0 0x564f586d9f00 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.295+0000 7f7f205b3740 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586c7a80 0x564f586c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.295+0000 7f7f205b3740 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586beea0 0x564f586c4fc0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.295+0000 7f7ef6ffd640 0 client.17583 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.295+0000 7f7f205b3740 1 --2- 172.21.15.94:0/3231173176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586be580 0x564f586c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.295+0000 7f7f205b3740 1 -- 172.21.15.94:0/3231173176 >> 172.21.15.94:0/3231173176 conn(0x564f586b8cb0 msgr2=0x564f586baa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.295+0000 7f7f205b3740 1 -- 172.21.15.94:0/3231173176 shutdown_connections 2024-04-21T08:47:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.295+0000 7f7f205b3740 1 -- 172.21.15.94:0/3231173176 wait complete. 2024-04-21T08:47:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.295+0000 7f7f205b3740 20 client.17583 trim_cache size 0 max 16384 2024-04-21T08:47:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f205b3740 1 Processor -- start 2024-04-21T08:47:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f205b3740 1 -- start start 2024-04-21T08:47:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f586c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586beea0 0x564f586d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 0x564f586d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f205b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181b1db0 con 0x564f586be580 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f205b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181af680 con 0x564f586c7a80 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f205b3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181ad4f0 con 0x564f586beea0 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f16ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f586c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f16ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f586c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47942/0 (socket says 172.21.15.94:47942) 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f16ffd640 1 -- 172.21.15.94:0/3071312970 learned_addr learned my addr 172.21.15.94:0/3071312970 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f167fc640 1 --2- 172.21.15.94:0/3071312970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586beea0 0x564f586d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f177fe640 1 --2- 172.21.15.94:0/3071312970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 0x564f586d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f16ffd640 1 -- 172.21.15.94:0/3071312970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586beea0 msgr2=0x564f586d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/3071312970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586beea0 0x564f586d3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f16ffd640 1 -- 172.21.15.94:0/3071312970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 msgr2=0x564f586d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/3071312970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 0x564f586d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f16ffd640 1 -- 172.21.15.94:0/3071312970 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f586c9e70 con 0x564f586be580 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f177fe640 1 --2- 172.21.15.94:0/3071312970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 0x564f586d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.312+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/3071312970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f586c71b0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7f08035fd0 tx=0x7f7f0802c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223727959d33981c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7efd7fa640 1 -- 172.21.15.94:0/3071312970 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08035df0 con 0x564f586be580 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7efd7fa640 1 -- 172.21.15.94:0/3071312970 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f080bf270 con 0x564f586be580 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7efd7fa640 1 -- 172.21.15.94:0/3071312970 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f080bcca0 con 0x564f586be580 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7f205b3740 1 -- 172.21.15.94:0/3071312970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 msgr2=0x564f586c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7f205b3740 1 --2- 172.21.15.94:0/3071312970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f586c71b0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7f08035fd0 tx=0x7f7f0802c910 comp rx=0 tx=0).stop 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7f205b3740 1 -- 172.21.15.94:0/3071312970 shutdown_connections 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7f205b3740 1 --2- 172.21.15.94:0/3071312970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586beea0 0x564f586d3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7f205b3740 1 --2- 172.21.15.94:0/3071312970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 0x564f586d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7f205b3740 1 --2- 172.21.15.94:0/3071312970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f586c71b0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7f205b3740 1 -- 172.21.15.94:0/3071312970 >> 172.21.15.94:0/3071312970 conn(0x564f586b8cb0 msgr2=0x564f586ba100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7f205b3740 1 -- 172.21.15.94:0/3071312970 shutdown_connections 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.313+0000 7f7f205b3740 1 -- 172.21.15.94:0/3071312970 wait complete. 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f205b3740 1 -- start start 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f205b3740 1 Processor -- start 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f58736000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586beea0 0x564f58736540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f205b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 0x564f58736a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f205b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181adb40 con 0x564f586be580 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f205b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181aefb0 con 0x564f586c7a80 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f205b3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181ae210 con 0x564f586beea0 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f205b3740 10 client.0 _ll_register_callbacks cb 0x7f7f0c00b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f16ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f58736000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f16ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f58736000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47946/0 (socket says 172.21.15.94:47946) 2024-04-21T08:47:36.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f16ffd640 1 -- 172.21.15.94:0/2081764566 learned_addr learned my addr 172.21.15.94:0/2081764566 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f167fc640 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586beea0 0x564f58736540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f177fe640 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 0x564f58736a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f16ffd640 1 -- 172.21.15.94:0/2081764566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586beea0 msgr2=0x564f58736540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586beea0 0x564f58736540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f16ffd640 1 -- 172.21.15.94:0/2081764566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 msgr2=0x564f58736a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 0x564f58736a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f16ffd640 1 -- 172.21.15.94:0/2081764566 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f587412a0 con 0x564f586be580 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7f16ffd640 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f58736000 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f7f080bafd0 tx=0x7f7f08035e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f859c351b9b498 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.314+0000 7f7ef6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/2081764566 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f080c2f20 con 0x564f586be580 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/2081764566 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7f7f080dda00 con 0x564f586be580 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7f205b3740 1 -- 172.21.15.94:0/2081764566 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f080bf270 con 0x564f586be580 2024-04-21T08:47:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/2081764566 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7f08088050 con 0x564f586be580 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7f205b3740 10 client.17589 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7f205b3740 1 -- 172.21.15.94:0/2081764566 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564f58742500 con 0x564f586be580 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/2081764566 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f080bf270 con 0x564f586be580 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/2081764566 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7f080d9070 con 0x564f586be580 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7f205b3740 10 client.17589 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7f205b3740 10 client.17589 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7f205b3740 20 client.17589 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7f205b3740 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564f58742b10 0x564f58744f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7f205b3740 4 client.17589 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7f205b3740 1 -- 172.21.15.94:0/2081764566 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f5862aa70 con 0x564f58742b10 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.315+0000 7f7f167fc640 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564f58742b10 0x564f58744f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7ef6ffd640 10 client.17589 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f167fc640 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564f58742b10 0x564f58744f30 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cb1971f4bab3d1a server_cookie=8dd28b5fbc30851f in_seq=0 out_seq=0 2024-04-21T08:47:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7ef6ffd640 1 -- 172.21.15.94:0/2081764566 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564f5862aa70 con 0x564f58742b10 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7ef6ffd640 10 client.17589 handle_command_reply: tid=0 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 client.17589 shutdown 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 -- 172.21.15.94:0/2081764566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 msgr2=0x564f58736000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f58736000 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f7f080bafd0 tx=0x7f7f08035e60 comp rx=0 tx=0).stop 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 -- 172.21.15.94:0/2081764566 shutdown_connections 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x564f58742b10 0x564f58744f30 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f586beea0 0x564f58736540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7ef6ffd640 0 client.17589 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564f586c7a80 0x564f58736a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 --2- 172.21.15.94:0/2081764566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564f586be580 0x564f58736000 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 -- 172.21.15.94:0/2081764566 >> 172.21.15.94:0/2081764566 conn(0x564f586b8cb0 msgr2=0x564f586ba970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 -- 172.21.15.94:0/2081764566 shutdown_connections 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 1 -- 172.21.15.94:0/2081764566 wait complete. 2024-04-21T08:47:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.316+0000 7f7f205b3740 20 client.17589 trim_cache size 0 max 16384 2024-04-21T08:47:36.324 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:47:36.324 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:47:36.324 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:47:36.324 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:47:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/3231798246 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f000a3d20 msgr2=0x7f7f000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f000a3d20 0x7f7f000a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7f10013800 tx=0x7f7f1000ba50 comp rx=0 tx=0).stop 2024-04-21T08:47:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/3231798246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f18101080 msgr2=0x7f7f18198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f18101080 0x7f7f18198700 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f7f08005e30 tx=0x7f7f08034780 comp rx=0 tx=0).stop 2024-04-21T08:47:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/3231798246 shutdown_connections 2024-04-21T08:47:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f7f000a3d20 0x7f7f000a61e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f18106150 0x7f7f18198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f1819cfc0 0x7f7f1819d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 --2- 172.21.15.94:0/3231798246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f18101080 0x7f7f18198700 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/3231798246 >> 172.21.15.94:0/3231798246 conn(0x7f7f18078000 msgr2=0x7f7f18069440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/3231798246 shutdown_connections 2024-04-21T08:47:36.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.317+0000 7f7f1d9bd640 1 -- 172.21.15.94:0/3231798246 wait complete. 2024-04-21T08:47:36.327 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:47:36.328 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:47:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.465+0000 7f34bd32c640 1 -- 172.21.15.94:0/2006667983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34b8105e30 msgr2=0x7f34b8106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.465+0000 7f34bd32c640 1 --2- 172.21.15.94:0/2006667983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34b8105e30 0x7f34b8106210 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009930 tx=0x7f34ac02bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.465+0000 7f34bd32c640 1 -- 172.21.15.94:0/2006667983 shutdown_connections 2024-04-21T08:47:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.465+0000 7f34bd32c640 1 --2- 172.21.15.94:0/2006667983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34b81011a0 0x7f34b8101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.465+0000 7f34bd32c640 1 --2- 172.21.15.94:0/2006667983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b8101b60 0x7f34b810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.465+0000 7f34bd32c640 1 --2- 172.21.15.94:0/2006667983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34b8105e30 0x7f34b8106210 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.465+0000 7f34bd32c640 1 -- 172.21.15.94:0/2006667983 >> 172.21.15.94:0/2006667983 conn(0x7f34b8077e30 msgr2=0x7f34b8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.465+0000 7f34bd32c640 1 -- 172.21.15.94:0/2006667983 shutdown_connections 2024-04-21T08:47:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.465+0000 7f34bd32c640 1 -- 172.21.15.94:0/2006667983 wait complete. 2024-04-21T08:47:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34bd32c640 1 Processor -- start 2024-04-21T08:47:36.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34bd32c640 1 -- start start 2024-04-21T08:47:36.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34bd32c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34b81011a0 0x7f34b819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34bd32c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b8101b60 0x7f34b819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34bd32c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34b8105e30 0x7f34b81985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34bd32c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f34b81110e0 con 0x7f34b8105e30 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34bd32c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f34b8110f60 con 0x7f34b8101b60 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34bd32c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f34b8111260 con 0x7f34b81011a0 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34b77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34b8105e30 0x7f34b81985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34b77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34b8105e30 0x7f34b81985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47970/0 (socket says 172.21.15.94:47970) 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34b77fe640 1 -- 172.21.15.94:0/595109039 learned_addr learned my addr 172.21.15.94:0/595109039 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.466+0000 7f34b6ffd640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34b81011a0 0x7f34b819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f34b67fc640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b8101b60 0x7f34b819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f34b77fe640 1 -- 172.21.15.94:0/595109039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34b81011a0 msgr2=0x7f34b819e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f34b77fe640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34b81011a0 0x7f34b819e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f34b77fe640 1 -- 172.21.15.94:0/595109039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b8101b60 msgr2=0x7f34b819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f34b77fe640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b8101b60 0x7f34b819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f34b77fe640 1 -- 172.21.15.94:0/595109039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b8198d20 con 0x7f34b8105e30 2024-04-21T08:47:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f34b77fe640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34b8105e30 0x7f34b81985a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009fd0 tx=0x7f34b000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f349ffff640 1 -- 172.21.15.94:0/595109039 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0017070 con 0x7f34b8105e30 2024-04-21T08:47:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f34bd32c640 1 -- 172.21.15.94:0/595109039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b8199010 con 0x7f34b8105e30 2024-04-21T08:47:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f349ffff640 1 -- 172.21.15.94:0/595109039 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34b0003e20 con 0x7f34b8105e30 2024-04-21T08:47:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f349ffff640 1 -- 172.21.15.94:0/595109039 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0010460 con 0x7f34b8105e30 2024-04-21T08:47:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.467+0000 7f34bd32c640 1 -- 172.21.15.94:0/595109039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b81a52b0 con 0x7f34b8105e30 2024-04-21T08:47:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.468+0000 7f34bd32c640 1 -- 172.21.15.94:0/595109039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b8102570 con 0x7f34b8105e30 2024-04-21T08:47:36.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.470+0000 7f349ffff640 1 -- 172.21.15.94:0/595109039 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f34b0003970 con 0x7f34b8105e30 2024-04-21T08:47:36.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.470+0000 7f349ffff640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f34a00a3d20 0x7f34a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.471+0000 7f34b6ffd640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f34a00a3d20 0x7f34a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.471+0000 7f349ffff640 1 -- 172.21.15.94:0/595109039 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7f34b00d09a0 con 0x7f34b8105e30 2024-04-21T08:47:36.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.471+0000 7f34b6ffd640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f34a00a3d20 0x7f34a00a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f34ac005ab0 tx=0x7f34ac005fd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.474+0000 7f349ffff640 1 -- 172.21.15.94:0/595109039 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34b009cc10 con 0x7f34b8105e30 2024-04-21T08:47:36.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.691+0000 7f34bd32c640 1 -- 172.21.15.94:0/595109039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34b8068de0 con 0x7f34b8105e30 2024-04-21T08:47:36.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.691+0000 7f349ffff640 1 -- 172.21.15.94:0/595109039 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f34b8068de0 con 0x7f34b8105e30 2024-04-21T08:47:36.692 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:47:36.692 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:47:36.692 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.693+0000 7f34bd32c640 1 -- 172.21.15.94:0/595109039 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f34a00a3d20 msgr2=0x7f34a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.693+0000 7f34bd32c640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f34a00a3d20 0x7f34a00a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f34ac005ab0 tx=0x7f34ac005fd0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.693+0000 7f34bd32c640 1 -- 172.21.15.94:0/595109039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34b8105e30 msgr2=0x7f34b81985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.693+0000 7f34bd32c640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34b8105e30 0x7f34b81985a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009fd0 tx=0x7f34b000b260 comp rx=0 tx=0).stop 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.693+0000 7f34bd32c640 1 -- 172.21.15.94:0/595109039 shutdown_connections 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.693+0000 7f34bd32c640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f34a00a3d20 0x7f34a00a61e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.693+0000 7f34bd32c640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34b8105e30 0x7f34b81985a0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.693+0000 7f34bd32c640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b8101b60 0x7f34b819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.693+0000 7f34bd32c640 1 --2- 172.21.15.94:0/595109039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34b81011a0 0x7f34b819e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.693+0000 7f34bd32c640 1 -- 172.21.15.94:0/595109039 >> 172.21.15.94:0/595109039 conn(0x7f34b8077e30 msgr2=0x7f34b80693e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.694+0000 7f34bd32c640 1 -- 172.21.15.94:0/595109039 shutdown_connections 2024-04-21T08:47:36.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.694+0000 7f34bd32c640 1 -- 172.21.15.94:0/595109039 wait complete. 2024-04-21T08:47:36.705 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:47:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.805+0000 7fcaf2462640 1 -- 172.21.15.94:0/23778450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaec10d330 msgr2=0x7fcaec1197f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.805+0000 7fcaf2462640 1 --2- 172.21.15.94:0/23778450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaec10d330 0x7fcaec1197f0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009cd0 tx=0x7fcae402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.805+0000 7fcaf2462640 1 -- 172.21.15.94:0/23778450 shutdown_connections 2024-04-21T08:47:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.805+0000 7fcaf2462640 1 --2- 172.21.15.94:0/23778450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcaec10c970 0x7fcaec10cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.805+0000 7fcaf2462640 1 --2- 172.21.15.94:0/23778450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaec10d330 0x7fcaec1197f0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.805+0000 7fcaf2462640 1 --2- 172.21.15.94:0/23778450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaec112930 0x7fcaec112d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.805+0000 7fcaf2462640 1 -- 172.21.15.94:0/23778450 >> 172.21.15.94:0/23778450 conn(0x7fcaec06cfc0 msgr2=0x7fcaec10af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.805+0000 7fcaf2462640 1 -- 172.21.15.94:0/23778450 shutdown_connections 2024-04-21T08:47:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.805+0000 7fcaf2462640 1 -- 172.21.15.94:0/23778450 wait complete. 2024-04-21T08:47:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaf2462640 1 Processor -- start 2024-04-21T08:47:36.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaf2462640 1 -- start start 2024-04-21T08:47:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaf2462640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaec10c970 0x7fcaec1a3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaf2462640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcaec10d330 0x7fcaec1a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaf2462640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaec112930 0x7fcaec1a7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaf2462640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec11cbf0 con 0x7fcaec10c970 2024-04-21T08:47:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaf2462640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec11ca70 con 0x7fcaec112930 2024-04-21T08:47:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaf2462640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec11cd70 con 0x7fcaec10d330 2024-04-21T08:47:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaec10c970 0x7fcaec1a3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaec10c970 0x7fcaec1a3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47982/0 (socket says 172.21.15.94:47982) 2024-04-21T08:47:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaebfff640 1 -- 172.21.15.94:0/3378207799 learned_addr learned my addr 172.21.15.94:0/3378207799 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.806+0000 7fcaf09ff640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaec112930 0x7fcaec1a7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcaeb7fe640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcaec10d330 0x7fcaec1a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcaebfff640 1 -- 172.21.15.94:0/3378207799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcaec10d330 msgr2=0x7fcaec1a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcaebfff640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcaec10d330 0x7fcaec1a36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcaebfff640 1 -- 172.21.15.94:0/3378207799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaec112930 msgr2=0x7fcaec1a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcaebfff640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaec112930 0x7fcaec1a7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcaebfff640 1 -- 172.21.15.94:0/3378207799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaec1a8150 con 0x7fcaec10c970 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcaebfff640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaec10c970 0x7fcaec1a3160 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00d950 tx=0x7fcadc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcae97fa640 1 -- 172.21.15.94:0/3378207799 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc017070 con 0x7fcaec10c970 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcaf2462640 1 -- 172.21.15.94:0/3378207799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaec1a8440 con 0x7fcaec10c970 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcae97fa640 1 -- 172.21.15.94:0/3378207799 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcadc009c30 con 0x7fcaec10c970 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcae97fa640 1 -- 172.21.15.94:0/3378207799 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc00f620 con 0x7fcaec10c970 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.807+0000 7fcaf2462640 1 -- 172.21.15.94:0/3378207799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaec1afcd0 con 0x7fcaec10c970 2024-04-21T08:47:36.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.809+0000 7fcae97fa640 1 -- 172.21.15.94:0/3378207799 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fcadc010460 con 0x7fcaec10c970 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.809+0000 7fcaf504c740 1 Processor -- start 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.809+0000 7fcaf504c740 1 -- start start 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.809+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5628988584d0 0x5628988588b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.809+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898861430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.809+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x562898863e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.809+0000 7fcaf504c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5628987bd1f0 con 0x562898858e80 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.809+0000 7fcaf504c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56289875a4b0 con 0x5628988584d0 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.809+0000 7fcaf504c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5628987b91b0 con 0x562898861a00 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898861430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898861430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47994/0 (socket says 172.21.15.94:47994) 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaebfff640 1 -- 172.21.15.94:0/3608760099 learned_addr learned my addr 172.21.15.94:0/3608760099 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcae97fa640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcad40a3d20 0x7fcad40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaf09ff640 1 --2- 172.21.15.94:0/3608760099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5628988584d0 0x5628988588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaeb7fe640 1 --2- 172.21.15.94:0/3608760099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x562898863e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaf09ff640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcad40a3d20 0x7fcad40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcae97fa640 1 -- 172.21.15.94:0/3378207799 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7fcadc0d07e0 con 0x7fcaec10c970 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaebfff640 1 -- 172.21.15.94:0/3608760099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 msgr2=0x562898863e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaebfff640 1 --2- 172.21.15.94:0/3608760099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x562898863e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaebfff640 1 -- 172.21.15.94:0/3608760099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5628988584d0 msgr2=0x5628988588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaebfff640 1 --2- 172.21.15.94:0/3608760099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5628988584d0 0x5628988588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaebfff640 1 -- 172.21.15.94:0/3608760099 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56289874c1b0 con 0x562898858e80 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaf09ff640 1 --2- 172.21.15.94:0/3608760099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5628988584d0 0x5628988588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaebfff640 1 --2- 172.21.15.94:0/3608760099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898861430 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fcadc09c050 tx=0x7fcadc010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d895eb59555049 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcad27fc640 1 -- 172.21.15.94:0/3608760099 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc020050 con 0x562898858e80 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcaf09ff640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcad40a3d20 0x7fcad40a61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcae4005cc0 tx=0x7fcae402c0f0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcad27fc640 1 -- 172.21.15.94:0/3608760099 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcadc09ca50 con 0x562898858e80 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.810+0000 7fcad27fc640 1 -- 172.21.15.94:0/3608760099 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc0cd340 con 0x562898858e80 2024-04-21T08:47:36.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.811+0000 7fcaf504c740 1 -- 172.21.15.94:0/3608760099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 msgr2=0x562898861430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.811+0000 7fcaf504c740 1 --2- 172.21.15.94:0/3608760099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898861430 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fcadc09c050 tx=0x7fcadc010cb0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.811+0000 7fcaf504c740 1 -- 172.21.15.94:0/3608760099 shutdown_connections 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.811+0000 7fcaf504c740 1 --2- 172.21.15.94:0/3608760099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x562898863e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.811+0000 7fcaf504c740 1 --2- 172.21.15.94:0/3608760099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898861430 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.811+0000 7fcaf504c740 1 --2- 172.21.15.94:0/3608760099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5628988584d0 0x5628988588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.811+0000 7fcaf504c740 1 -- 172.21.15.94:0/3608760099 >> 172.21.15.94:0/3608760099 conn(0x562898852ba0 msgr2=0x562898852f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.811+0000 7fcaf504c740 1 -- 172.21.15.94:0/3608760099 shutdown_connections 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.811+0000 7fcaf504c740 1 -- 172.21.15.94:0/3608760099 wait complete. 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.811+0000 7fcaf504c740 1 -- start start 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf504c740 1 Processor -- start 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x562898861320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562898858e80 0x56289885e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x56289885ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf504c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x562898864910 con 0x5628988584d0 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf504c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562898776500 con 0x562898858e80 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf504c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562898864a90 con 0x562898861a00 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf504c740 10 client.0 _ll_register_callbacks cb 0x562898744b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x562898861320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x562898861320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48000/0 (socket says 172.21.15.94:48000) 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaebfff640 1 -- 172.21.15.94:0/514944793 learned_addr learned my addr 172.21.15.94:0/514944793 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf09ff640 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x56289885ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaebfff640 1 -- 172.21.15.94:0/514944793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 msgr2=0x56289885ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaebfff640 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x56289885ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaebfff640 1 -- 172.21.15.94:0/514944793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562898858e80 msgr2=0x56289885e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaebfff640 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562898858e80 0x56289885e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaebfff640 1 -- 172.21.15.94:0/514944793 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628984cf3f0 con 0x5628988584d0 2024-04-21T08:47:36.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf09ff640 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x56289885ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaebfff640 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x562898861320 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fcadc0cdbb0 tx=0x7fcadc0d4080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3d068badfeeeb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcacb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcacb7fe640 1 -- 172.21.15.94:0/514944793 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc0b5060 con 0x5628988584d0 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcacb7fe640 1 -- 172.21.15.94:0/514944793 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7fcadc0ccfc0 con 0x5628988584d0 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.812+0000 7fcaf504c740 1 -- 172.21.15.94:0/514944793 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcadc09ca50 con 0x5628988584d0 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcacb7fe640 1 -- 172.21.15.94:0/514944793 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcadc07c020 con 0x5628988584d0 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcaf504c740 10 client.17607 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcaf504c740 1 -- 172.21.15.94:0/514944793 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5628985997d0 con 0x5628988584d0 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcacb7fe640 1 -- 172.21.15.94:0/514944793 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcadc09ca50 con 0x5628988584d0 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcacb7fe640 1 -- 172.21.15.94:0/514944793 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fcadc0cdd80 con 0x5628988584d0 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcaf504c740 10 client.17607 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcaf504c740 10 client.17607 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcaf504c740 20 client.17607 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:36.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcaf504c740 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562898871aa0 0x562898873e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcaf504c740 4 client.17607 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:36.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcaf504c740 1 -- 172.21.15.94:0/514944793 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56289875c930 con 0x562898871aa0 2024-04-21T08:47:36.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.813+0000 7fcaeb7fe640 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562898871aa0 0x562898873e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.814+0000 7fcaeb7fe640 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562898871aa0 0x562898873e90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b1d80a91f5de87d1 server_cookie=f7b5f3c0b6580904 in_seq=0 out_seq=0 2024-04-21T08:47:36.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.814+0000 7fcacb7fe640 10 client.17607 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:47:36.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.814+0000 7fcacb7fe640 1 -- 172.21.15.94:0/514944793 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56289875c930 con 0x562898871aa0 2024-04-21T08:47:36.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.814+0000 7fcacb7fe640 10 client.17607 handle_command_reply: tid=0 2024-04-21T08:47:36.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.814+0000 7fcaf504c740 1 client.17607 shutdown 2024-04-21T08:47:36.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 1 -- 172.21.15.94:0/514944793 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 msgr2=0x562898861320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x562898861320 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fcadc0cdbb0 tx=0x7fcadc0d4080 comp rx=0 tx=0).stop 2024-04-21T08:47:36.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 1 -- 172.21.15.94:0/514944793 shutdown_connections 2024-04-21T08:47:36.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562898871aa0 0x562898873e90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x56289885ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562898858e80 0x56289885e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 1 --2- 172.21.15.94:0/514944793 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x562898861320 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcacb7fe640 0 client.17607 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:47:36.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 1 -- 172.21.15.94:0/514944793 >> 172.21.15.94:0/514944793 conn(0x562898852ba0 msgr2=0x5628988548f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 1 -- 172.21.15.94:0/514944793 shutdown_connections 2024-04-21T08:47:36.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 1 -- 172.21.15.94:0/514944793 wait complete. 2024-04-21T08:47:36.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.815+0000 7fcaf504c740 20 client.17607 trim_cache size 0 max 16384 2024-04-21T08:47:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaf504c740 1 Processor -- start 2024-04-21T08:47:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaf504c740 1 -- start start 2024-04-21T08:47:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x56289885d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898858e80 0x562898854700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562898861a00 0x562898860de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaf504c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec1bc790 con 0x5628988584d0 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaf504c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec1ba060 con 0x562898861a00 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaf504c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec1b7ed0 con 0x562898858e80 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x56289885d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x56289885d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48014/0 (socket says 172.21.15.94:48014) 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaebfff640 1 -- 172.21.15.94:0/916914949 learned_addr learned my addr 172.21.15.94:0/916914949 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaebfff640 1 -- 172.21.15.94:0/916914949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898858e80 msgr2=0x562898854700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaebfff640 1 --2- 172.21.15.94:0/916914949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898858e80 0x562898854700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaebfff640 1 -- 172.21.15.94:0/916914949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562898861a00 msgr2=0x562898860de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaebfff640 1 --2- 172.21.15.94:0/916914949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562898861a00 0x562898860de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaebfff640 1 -- 172.21.15.94:0/916914949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56289886e3a0 con 0x5628988584d0 2024-04-21T08:47:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcaebfff640 1 --2- 172.21.15.94:0/916914949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x56289885d810 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fcadc0ce100 tx=0x7fcadc010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93006294b090dec1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcad1ffb640 1 -- 172.21.15.94:0/916914949 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc020070 con 0x5628988584d0 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcad1ffb640 1 -- 172.21.15.94:0/916914949 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcadc09ca50 con 0x5628988584d0 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.828+0000 7fcad1ffb640 1 -- 172.21.15.94:0/916914949 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc09c350 con 0x5628988584d0 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.829+0000 7fcaf504c740 1 -- 172.21.15.94:0/916914949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 msgr2=0x56289885d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.829+0000 7fcaf504c740 1 --2- 172.21.15.94:0/916914949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x56289885d810 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fcadc0ce100 tx=0x7fcadc010cb0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.829+0000 7fcaf504c740 1 -- 172.21.15.94:0/916914949 shutdown_connections 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.829+0000 7fcaf504c740 1 --2- 172.21.15.94:0/916914949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898858e80 0x562898854700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.829+0000 7fcaf504c740 1 --2- 172.21.15.94:0/916914949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562898861a00 0x562898860de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.829+0000 7fcaf504c740 1 --2- 172.21.15.94:0/916914949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5628988584d0 0x56289885d810 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.829+0000 7fcaf504c740 1 -- 172.21.15.94:0/916914949 >> 172.21.15.94:0/916914949 conn(0x562898852ba0 msgr2=0x562898853ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.829+0000 7fcaf504c740 1 -- 172.21.15.94:0/916914949 shutdown_connections 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.829+0000 7fcaf504c740 1 -- 172.21.15.94:0/916914949 wait complete. 2024-04-21T08:47:36.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.829+0000 7fcaf504c740 1 -- start start 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaf504c740 1 Processor -- start 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5628988584d0 0x5628988773d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898874760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaf504c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x562898874ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaf504c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec1b8520 con 0x562898858e80 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaf504c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec1b9990 con 0x5628988584d0 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaf504c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec1b8bf0 con 0x562898861a00 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaf504c740 10 client.0 _ll_register_callbacks cb 0x7fcadc00df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaeb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898874760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaeb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898874760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48026/0 (socket says 172.21.15.94:48026) 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaeb7fe640 1 -- 172.21.15.94:0/4279886946 learned_addr learned my addr 172.21.15.94:0/4279886946 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaeb7fe640 1 -- 172.21.15.94:0/4279886946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 msgr2=0x562898874ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaeb7fe640 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x562898874ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaeb7fe640 1 -- 172.21.15.94:0/4279886946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5628988584d0 msgr2=0x5628988773d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaeb7fe640 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5628988584d0 0x5628988773d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaeb7fe640 1 -- 172.21.15.94:0/4279886946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628988d6c80 con 0x562898858e80 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaeb7fe640 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898874760 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x562898864b50 tx=0x7fcae0009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a32851b6ce0f0c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcacb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcacb7fe640 1 -- 172.21.15.94:0/4279886946 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0013d40 con 0x562898858e80 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcacb7fe640 1 -- 172.21.15.94:0/4279886946 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7fcae001b3b0 con 0x562898858e80 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.830+0000 7fcaf504c740 1 -- 172.21.15.94:0/4279886946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcadc09ca50 con 0x562898858e80 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcacb7fe640 1 -- 172.21.15.94:0/4279886946 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcae002b050 con 0x562898858e80 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcaf504c740 10 client.17613 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcaf504c740 1 -- 172.21.15.94:0/4279886946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5628988d7e50 con 0x562898858e80 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcacb7fe640 1 -- 172.21.15.94:0/4279886946 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcadc09ca50 con 0x562898858e80 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcacb7fe640 1 -- 172.21.15.94:0/4279886946 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fcae000f040 con 0x562898858e80 2024-04-21T08:47:36.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcaf504c740 10 client.17613 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcaf504c740 10 client.17613 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcaf504c740 20 client.17613 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcaf504c740 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5628988d85b0 0x5628988da9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcaf504c740 4 client.17613 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcaf504c740 1 -- 172.21.15.94:0/4279886946 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628987cac10 con 0x5628988d85b0 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.831+0000 7fcaebfff640 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5628988d85b0 0x5628988da9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcacb7fe640 10 client.17613 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaebfff640 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5628988d85b0 0x5628988da9d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3a1606758facbd58 server_cookie=c2a010922ac2d854 in_seq=0 out_seq=0 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcacb7fe640 1 -- 172.21.15.94:0/4279886946 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628987cac10 con 0x5628988d85b0 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcacb7fe640 10 client.17613 handle_command_reply: tid=0 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 client.17613 shutdown 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 -- 172.21.15.94:0/4279886946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 msgr2=0x562898874760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898874760 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x562898864b50 tx=0x7fcae0009cb0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 -- 172.21.15.94:0/4279886946 shutdown_connections 2024-04-21T08:47:36.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x5628988d85b0 0x5628988da9d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562898861a00 0x562898874ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562898858e80 0x562898874760 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 --2- 172.21.15.94:0/4279886946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5628988584d0 0x5628988773d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcacb7fe640 0 client.17613 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:47:36.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 -- 172.21.15.94:0/4279886946 >> 172.21.15.94:0/4279886946 conn(0x562898852ba0 msgr2=0x56289886d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 -- 172.21.15.94:0/4279886946 shutdown_connections 2024-04-21T08:47:36.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 1 -- 172.21.15.94:0/4279886946 wait complete. 2024-04-21T08:47:36.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.832+0000 7fcaf504c740 20 client.17613 trim_cache size 0 max 16384 2024-04-21T08:47:36.838 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:47:36.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 -- 172.21.15.94:0/3378207799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcad40a3d20 msgr2=0x7fcad40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcad40a3d20 0x7fcad40a61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcae4005cc0 tx=0x7fcae402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 -- 172.21.15.94:0/3378207799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaec10c970 msgr2=0x7fcaec1a3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaec10c970 0x7fcaec1a3160 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00d950 tx=0x7fcadc00de10 comp rx=0 tx=0).stop 2024-04-21T08:47:36.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 -- 172.21.15.94:0/3378207799 shutdown_connections 2024-04-21T08:47:36.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcad40a3d20 0x7fcad40a61e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcaec10d330 0x7fcaec1a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaec112930 0x7fcaec1a7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 --2- 172.21.15.94:0/3378207799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaec10c970 0x7fcaec1a3160 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 -- 172.21.15.94:0/3378207799 >> 172.21.15.94:0/3378207799 conn(0x7fcaec06cfc0 msgr2=0x7fcaec10f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 -- 172.21.15.94:0/3378207799 shutdown_connections 2024-04-21T08:47:36.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.833+0000 7fcaf2462640 1 -- 172.21.15.94:0/3378207799 wait complete. 2024-04-21T08:47:36.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:47:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.944+0000 7fa557304640 1 -- 172.21.15.94:0/3085391714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa550102b40 msgr2=0x7fa550106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.944+0000 7fa557304640 1 --2- 172.21.15.94:0/3085391714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa550102b40 0x7fa550106b80 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fa540009900 tx=0x7fa54002bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.944+0000 7fa557304640 1 -- 172.21.15.94:0/3085391714 shutdown_connections 2024-04-21T08:47:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.944+0000 7fa557304640 1 --2- 172.21.15.94:0/3085391714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa550107260 0x7fa55010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.944+0000 7fa557304640 1 --2- 172.21.15.94:0/3085391714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa550102b40 0x7fa550106b80 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.944+0000 7fa557304640 1 --2- 172.21.15.94:0/3085391714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa550102190 0x7fa550102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.944+0000 7fa557304640 1 -- 172.21.15.94:0/3085391714 >> 172.21.15.94:0/3085391714 conn(0x7fa5500793a0 msgr2=0x7fa5500797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa557304640 1 -- 172.21.15.94:0/3085391714 shutdown_connections 2024-04-21T08:47:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa557304640 1 -- 172.21.15.94:0/3085391714 wait complete. 2024-04-21T08:47:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa557304640 1 Processor -- start 2024-04-21T08:47:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa557304640 1 -- start start 2024-04-21T08:47:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa557304640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa550102190 0x7fa55019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa557304640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa550107260 0x7fa55019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa557304640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5501985e0 0x7fa550198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa557304640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa550112180 con 0x7fa550102190 2024-04-21T08:47:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa557304640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa550112000 con 0x7fa5501985e0 2024-04-21T08:47:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa557304640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa550112300 con 0x7fa550107260 2024-04-21T08:47:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa5550a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa550102190 0x7fa55019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa5550a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa550102190 0x7fa55019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48036/0 (socket says 172.21.15.94:48036) 2024-04-21T08:47:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa5550a0640 1 -- 172.21.15.94:0/3166476111 learned_addr learned my addr 172.21.15.94:0/3166476111 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.945+0000 7fa55489f640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa550107260 0x7fa55019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa5558a1640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5501985e0 0x7fa550198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa5550a0640 1 -- 172.21.15.94:0/3166476111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa550107260 msgr2=0x7fa55019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa5550a0640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa550107260 0x7fa55019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa5550a0640 1 -- 172.21.15.94:0/3166476111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5501985e0 msgr2=0x7fa550198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa5550a0640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5501985e0 0x7fa550198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa5550a0640 1 -- 172.21.15.94:0/3166476111 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa550199100 con 0x7fa550102190 2024-04-21T08:47:36.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa5550a0640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa550102190 0x7fa55019e3e0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00e9f0 tx=0x7fa54c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa5467fc640 1 -- 172.21.15.94:0/3166476111 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c00cd80 con 0x7fa550102190 2024-04-21T08:47:36.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa5467fc640 1 -- 172.21.15.94:0/3166476111 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa54c0108e0 con 0x7fa550102190 2024-04-21T08:47:36.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa557304640 1 -- 172.21.15.94:0/3166476111 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5501993f0 con 0x7fa550102190 2024-04-21T08:47:36.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa5467fc640 1 -- 172.21.15.94:0/3166476111 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c018980 con 0x7fa550102190 2024-04-21T08:47:36.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.946+0000 7fa557304640 1 -- 172.21.15.94:0/3166476111 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5501a55f0 con 0x7fa550102190 2024-04-21T08:47:36.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa5467fc640 1 -- 172.21.15.94:0/3166476111 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa54c010430 con 0x7fa550102190 2024-04-21T08:47:36.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa559fd1740 1 Processor -- start 2024-04-21T08:47:36.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa559fd1740 1 -- start start 2024-04-21T08:47:36.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 0x55cd7a5438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a54c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a54ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa559fd1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55cd7a4a81f0 con 0x55cd7a54ca00 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa559fd1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cd7a4454b0 con 0x55cd7a543e80 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa559fd1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cd7a4a41b0 con 0x55cd7a5434d0 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa55489f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a54ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa55489f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a54ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48046/0 (socket says 172.21.15.94:48046) 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.949+0000 7fa55489f640 1 -- 172.21.15.94:0/508056123 learned_addr learned my addr 172.21.15.94:0/508056123 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa5558a1640 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 0x55cd7a5438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa5550a0640 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a54c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa5467fc640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa53c0a3d20 0x7fa53c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa55489f640 1 -- 172.21.15.94:0/508056123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 msgr2=0x55cd7a5438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa55489f640 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 0x55cd7a5438b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa55489f640 1 -- 172.21.15.94:0/508056123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 msgr2=0x55cd7a54c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa55489f640 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a54c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa55489f640 1 -- 172.21.15.94:0/508056123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd7a4371b0 con 0x55cd7a54ca00 2024-04-21T08:47:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa5558a1640 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 0x55cd7a5438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa5550a0640 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a54c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa5558a1640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa53c0a3d20 0x7fa53c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa5467fc640 1 -- 172.21.15.94:0/3166476111 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7fa54c0d0fc0 con 0x7fa550102190 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa55489f640 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a54ee50 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fa540009900 tx=0x7fa540005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5323956f4089b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa53b7fe640 1 -- 172.21.15.94:0/508056123 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540045020 con 0x55cd7a54ca00 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa5558a1640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa53c0a3d20 0x7fa53c0a61e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa548005fd0 tx=0x7fa548004a40 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa53b7fe640 1 -- 172.21.15.94:0/508056123 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa54002c7e0 con 0x55cd7a54ca00 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.950+0000 7fa53b7fe640 1 -- 172.21.15.94:0/508056123 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400046c0 con 0x55cd7a54ca00 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 -- 172.21.15.94:0/508056123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 msgr2=0x55cd7a54ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a54ee50 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fa540009900 tx=0x7fa540005bf0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 -- 172.21.15.94:0/508056123 shutdown_connections 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a54ee50 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a54c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 --2- 172.21.15.94:0/508056123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 0x55cd7a5438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 -- 172.21.15.94:0/508056123 >> 172.21.15.94:0/508056123 conn(0x55cd7a53dba0 msgr2=0x55cd7a53df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 -- 172.21.15.94:0/508056123 shutdown_connections 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 -- 172.21.15.94:0/508056123 wait complete. 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 -- start start 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.951+0000 7fa559fd1740 1 Processor -- start 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 0x55cd7a54c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a549730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a549e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa559fd1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55cd7a54f910 con 0x55cd7a54ca00 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa559fd1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cd7a461500 con 0x55cd7a543e80 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa559fd1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cd7a54fa90 con 0x55cd7a5434d0 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa559fd1740 10 client.0 _ll_register_callbacks cb 0x55cd7a42fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa5558a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a549e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa5558a1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a549e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48058/0 (socket says 172.21.15.94:48058) 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa5558a1640 1 -- 172.21.15.94:0/2054113670 learned_addr learned my addr 172.21.15.94:0/2054113670 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa5550a0640 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 0x55cd7a54c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa55489f640 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a549730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa5558a1640 1 -- 172.21.15.94:0/2054113670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 msgr2=0x55cd7a54c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa5558a1640 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 0x55cd7a54c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa5558a1640 1 -- 172.21.15.94:0/2054113670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 msgr2=0x55cd7a549730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa5558a1640 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a549730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa5558a1640 1 -- 172.21.15.94:0/2054113670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd7a1ba3f0 con 0x55cd7a54ca00 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa55489f640 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a549730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa5558a1640 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a549e70 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fa548012990 tx=0x7fa548004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8311e0b00929a3c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa538ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa538ff9640 1 -- 172.21.15.94:0/2054113670 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54801dd70 con 0x55cd7a54ca00 2024-04-21T08:47:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa538ff9640 1 -- 172.21.15.94:0/2054113670 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7fa548027470 con 0x55cd7a54ca00 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.952+0000 7fa559fd1740 1 -- 172.21.15.94:0/2054113670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa54002c7e0 con 0x55cd7a54ca00 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa538ff9640 1 -- 172.21.15.94:0/2054113670 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa548030050 con 0x55cd7a54ca00 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa559fd1740 10 client.17625 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa559fd1740 1 -- 172.21.15.94:0/2054113670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cd7a2847d0 con 0x55cd7a54ca00 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa538ff9640 1 -- 172.21.15.94:0/2054113670 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa54002c7e0 con 0x55cd7a54ca00 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa538ff9640 1 -- 172.21.15.94:0/2054113670 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa54801b4d0 con 0x55cd7a54ca00 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa559fd1740 10 client.17625 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa559fd1740 10 client.17625 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa559fd1740 20 client.17625 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa559fd1740 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55cd7a55cd20 0x55cd7a55f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa559fd1740 4 client.17625 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa559fd1740 1 -- 172.21.15.94:0/2054113670 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd7a447930 con 0x55cd7a55cd20 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.953+0000 7fa5550a0640 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55cd7a55cd20 0x55cd7a55f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.954+0000 7fa538ff9640 10 client.17625 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.954+0000 7fa5550a0640 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55cd7a55cd20 0x55cd7a55f180 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb77fdcb97918e75 server_cookie=f8c61382c5417e2f in_seq=0 out_seq=0 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.954+0000 7fa538ff9640 1 -- 172.21.15.94:0/2054113670 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd7a447930 con 0x55cd7a55cd20 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.954+0000 7fa538ff9640 10 client.17625 handle_command_reply: tid=0 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.954+0000 7fa559fd1740 1 client.17625 shutdown 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 1 -- 172.21.15.94:0/2054113670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 msgr2=0x55cd7a549e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a549e70 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fa548012990 tx=0x7fa548004ab0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 1 -- 172.21.15.94:0/2054113670 shutdown_connections 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55cd7a55cd20 0x55cd7a55f180 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a54ca00 0x55cd7a549e70 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a549730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 1 --2- 172.21.15.94:0/2054113670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a5434d0 0x55cd7a54c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa538ff9640 0 client.17625 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 1 -- 172.21.15.94:0/2054113670 >> 172.21.15.94:0/2054113670 conn(0x55cd7a53dba0 msgr2=0x55cd7a53f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 1 -- 172.21.15.94:0/2054113670 shutdown_connections 2024-04-21T08:47:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 1 -- 172.21.15.94:0/2054113670 wait complete. 2024-04-21T08:47:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.955+0000 7fa559fd1740 20 client.17625 trim_cache size 0 max 16384 2024-04-21T08:47:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa559fd1740 1 Processor -- start 2024-04-21T08:47:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa559fd1740 1 -- start start 2024-04-21T08:47:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a5434d0 0x55cd7a5587d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a543e80 0x55cd7a564a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a54ca00 0x55cd7a564f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa559fd1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa5501b1d40 con 0x55cd7a543e80 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa559fd1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa5501af610 con 0x55cd7a5434d0 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa559fd1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa5501ad480 con 0x55cd7a54ca00 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa55489f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a543e80 0x55cd7a564a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa55489f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a543e80 0x55cd7a564a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48060/0 (socket says 172.21.15.94:48060) 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa55489f640 1 -- 172.21.15.94:0/2802388122 learned_addr learned my addr 172.21.15.94:0/2802388122 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa55489f640 1 -- 172.21.15.94:0/2802388122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a54ca00 msgr2=0x55cd7a564f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa55489f640 1 --2- 172.21.15.94:0/2802388122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a54ca00 0x55cd7a564f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa55489f640 1 -- 172.21.15.94:0/2802388122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a5434d0 msgr2=0x55cd7a5587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa55489f640 1 --2- 172.21.15.94:0/2802388122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a5434d0 0x55cd7a5587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.968+0000 7fa55489f640 1 -- 172.21.15.94:0/2802388122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd7a5486e0 con 0x55cd7a543e80 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa55489f640 1 --2- 172.21.15.94:0/2802388122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a543e80 0x55cd7a564a50 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fa540005e90 tx=0x7fa54002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3090c8b020c19ba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa53affd640 1 -- 172.21.15.94:0/2802388122 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540045030 con 0x55cd7a543e80 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa53affd640 1 -- 172.21.15.94:0/2802388122 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5400053f0 con 0x55cd7a543e80 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa53affd640 1 -- 172.21.15.94:0/2802388122 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540003850 con 0x55cd7a543e80 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa559fd1740 1 -- 172.21.15.94:0/2802388122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a543e80 msgr2=0x55cd7a564a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa559fd1740 1 --2- 172.21.15.94:0/2802388122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a543e80 0x55cd7a564a50 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fa540005e90 tx=0x7fa54002bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa559fd1740 1 -- 172.21.15.94:0/2802388122 shutdown_connections 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa559fd1740 1 --2- 172.21.15.94:0/2802388122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a54ca00 0x55cd7a564f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa559fd1740 1 --2- 172.21.15.94:0/2802388122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a543e80 0x55cd7a564a50 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa559fd1740 1 --2- 172.21.15.94:0/2802388122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a5434d0 0x55cd7a5587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa559fd1740 1 -- 172.21.15.94:0/2802388122 >> 172.21.15.94:0/2802388122 conn(0x55cd7a53dba0 msgr2=0x55cd7a53cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa559fd1740 1 -- 172.21.15.94:0/2802388122 shutdown_connections 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.969+0000 7fa559fd1740 1 -- 172.21.15.94:0/2802388122 wait complete. 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa559fd1740 1 -- start start 2024-04-21T08:47:36.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa559fd1740 1 Processor -- start 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a5434d0 0x55cd7a554e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a5511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa559fd1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a54ca00 0x55cd7a551700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa559fd1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa5501adad0 con 0x55cd7a5434d0 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa559fd1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa5501aef40 con 0x55cd7a543e80 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa559fd1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa5501ae1a0 con 0x55cd7a54ca00 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa559fd1740 10 client.0 _ll_register_callbacks cb 0x7fa5480124b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa5550a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a5434d0 0x55cd7a554e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa5550a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a5434d0 0x55cd7a554e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48074/0 (socket says 172.21.15.94:48074) 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa5550a0640 1 -- 172.21.15.94:0/1754328357 learned_addr learned my addr 172.21.15.94:0/1754328357 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa55489f640 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a5511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa5550a0640 1 -- 172.21.15.94:0/1754328357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a54ca00 msgr2=0x55cd7a551700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa5550a0640 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a54ca00 0x55cd7a551700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa5550a0640 1 -- 172.21.15.94:0/1754328357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 msgr2=0x55cd7a5511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa5550a0640 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a5511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa5550a0640 1 -- 172.21.15.94:0/1754328357 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd7a551cc0 con 0x55cd7a5434d0 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa5550a0640 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a5434d0 0x55cd7a554e40 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0097b0 tx=0x7fa54c018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcdb40b349bc65a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.970+0000 7fa538ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa538ff9640 1 -- 172.21.15.94:0/1754328357 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c09de60 con 0x55cd7a5434d0 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa538ff9640 1 -- 172.21.15.94:0/1754328357 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7fa54c010a90 con 0x55cd7a5434d0 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa559fd1740 1 -- 172.21.15.94:0/1754328357 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5400053f0 con 0x55cd7a5434d0 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa538ff9640 1 -- 172.21.15.94:0/1754328357 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa54c0a0ee0 con 0x55cd7a5434d0 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa559fd1740 10 client.17631 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa559fd1740 1 -- 172.21.15.94:0/1754328357 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cd7a5c4060 con 0x55cd7a5434d0 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa538ff9640 1 -- 172.21.15.94:0/1754328357 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5400053f0 con 0x55cd7a5434d0 2024-04-21T08:47:36.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa538ff9640 1 -- 172.21.15.94:0/1754328357 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa54c0991a0 con 0x55cd7a5434d0 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa559fd1740 10 client.17631 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa559fd1740 10 client.17631 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa559fd1740 20 client.17631 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa559fd1740 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55cd7a5c4760 0x55cd7a5c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa559fd1740 4 client.17631 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.971+0000 7fa559fd1740 1 -- 172.21.15.94:0/1754328357 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cd7a4b5c10 con 0x55cd7a5c4760 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa55489f640 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55cd7a5c4760 0x55cd7a5c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa538ff9640 10 client.17631 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa55489f640 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55cd7a5c4760 0x55cd7a5c6b80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=356b0dca39015621 server_cookie=57934e7931be3c0d in_seq=0 out_seq=0 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa538ff9640 1 -- 172.21.15.94:0/1754328357 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cd7a4b5c10 con 0x55cd7a5c4760 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa538ff9640 10 client.17631 handle_command_reply: tid=0 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 client.17631 shutdown 2024-04-21T08:47:36.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 -- 172.21.15.94:0/1754328357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a5434d0 msgr2=0x55cd7a554e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a5434d0 0x55cd7a554e40 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0097b0 tx=0x7fa54c018420 comp rx=0 tx=0).stop 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 -- 172.21.15.94:0/1754328357 shutdown_connections 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cd7a54ca00 0x55cd7a551700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55cd7a5c4760 0x55cd7a5c6b80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7a543e80 0x55cd7a5511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 --2- 172.21.15.94:0/1754328357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cd7a5434d0 0x55cd7a554e40 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa538ff9640 0 client.17631 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 -- 172.21.15.94:0/1754328357 >> 172.21.15.94:0/1754328357 conn(0x55cd7a53dba0 msgr2=0x55cd7a544260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 -- 172.21.15.94:0/1754328357 shutdown_connections 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.972+0000 7fa559fd1740 1 -- 172.21.15.94:0/1754328357 wait complete. 2024-04-21T08:47:36.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa559fd1740 20 client.17631 trim_cache size 0 max 16384 2024-04-21T08:47:36.980 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:47:36.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 -- 172.21.15.94:0/3166476111 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa53c0a3d20 msgr2=0x7fa53c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa53c0a3d20 0x7fa53c0a61e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa548005fd0 tx=0x7fa548004a40 comp rx=0 tx=0).stop 2024-04-21T08:47:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 -- 172.21.15.94:0/3166476111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa550102190 msgr2=0x7fa55019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa550102190 0x7fa55019e3e0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00e9f0 tx=0x7fa54c00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 -- 172.21.15.94:0/3166476111 shutdown_connections 2024-04-21T08:47:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa53c0a3d20 0x7fa53c0a61e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa550107260 0x7fa55019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5501985e0 0x7fa550198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 --2- 172.21.15.94:0/3166476111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa550102190 0x7fa55019e3e0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 -- 172.21.15.94:0/3166476111 >> 172.21.15.94:0/3166476111 conn(0x7fa5500793a0 msgr2=0x7fa550100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 -- 172.21.15.94:0/3166476111 shutdown_connections 2024-04-21T08:47:36.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:36.973+0000 7fa557304640 1 -- 172.21.15.94:0/3166476111 wait complete. 2024-04-21T08:47:36.982 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:36 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/595109039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:36.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:47:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.084+0000 7f4ab8406640 1 -- 172.21.15.94:0/2620274289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab01072f0 msgr2=0x7f4ab010dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.084+0000 7f4ab8406640 1 --2- 172.21.15.94:0/2620274289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab01072f0 0x7f4ab010dd20 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009900 tx=0x7f4aa802bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.084+0000 7f4ab8406640 1 -- 172.21.15.94:0/2620274289 shutdown_connections 2024-04-21T08:47:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.084+0000 7f4ab8406640 1 --2- 172.21.15.94:0/2620274289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ab01027e0 0x7f4ab0106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.084+0000 7f4ab8406640 1 --2- 172.21.15.94:0/2620274289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab01072f0 0x7f4ab010dd20 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.084+0000 7f4ab8406640 1 --2- 172.21.15.94:0/2620274289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ab0101e30 0x7f4ab0102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.084+0000 7f4ab8406640 1 -- 172.21.15.94:0/2620274289 >> 172.21.15.94:0/2620274289 conn(0x7f4ab0076150 msgr2=0x7f4ab0076560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.084+0000 7f4ab8406640 1 -- 172.21.15.94:0/2620274289 shutdown_connections 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.084+0000 7f4ab8406640 1 -- 172.21.15.94:0/2620274289 wait complete. 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab8406640 1 Processor -- start 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab8406640 1 -- start start 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab8406640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab0101e30 0x7f4ab0199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab8406640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ab01072f0 0x7f4ab0199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab8406640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ab0199fd0 0x7f4ab01a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab8406640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01110c0 con 0x7f4ab0101e30 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab8406640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab0110f40 con 0x7f4ab01072f0 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab8406640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab0111240 con 0x7f4ab0199fd0 2024-04-21T08:47:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab61a2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab0101e30 0x7f4ab0199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab61a2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab0101e30 0x7f4ab0199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48086/0 (socket says 172.21.15.94:48086) 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab61a2640 1 -- 172.21.15.94:0/766190645 learned_addr learned my addr 172.21.15.94:0/766190645 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab59a1640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ab01072f0 0x7f4ab0199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ab0199fd0 0x7f4ab01a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab61a2640 1 -- 172.21.15.94:0/766190645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ab0199fd0 msgr2=0x7f4ab01a4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ab0199fd0 0x7f4ab01a4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab61a2640 1 -- 172.21.15.94:0/766190645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ab01072f0 msgr2=0x7f4ab0199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ab01072f0 0x7f4ab0199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab61a2640 1 -- 172.21.15.94:0/766190645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab01a4d50 con 0x7f4ab0101e30 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab59a1640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ab01072f0 0x7f4ab0199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab0101e30 0x7f4ab0199500 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00cce0 tx=0x7f4aac00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.085+0000 7f4aa77fe640 1 -- 172.21.15.94:0/766190645 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aac00f650 con 0x7f4ab0101e30 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.086+0000 7f4aa77fe640 1 -- 172.21.15.94:0/766190645 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aac00fca0 con 0x7f4ab0101e30 2024-04-21T08:47:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.086+0000 7f4ab8406640 1 -- 172.21.15.94:0/766190645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab01a4fe0 con 0x7f4ab0101e30 2024-04-21T08:47:37.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.086+0000 7f4aa77fe640 1 -- 172.21.15.94:0/766190645 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aac018b40 con 0x7f4ab0101e30 2024-04-21T08:47:37.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.086+0000 7f4ab8406640 1 -- 172.21.15.94:0/766190645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab01a5500 con 0x7f4ab0101e30 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.088+0000 7f4aa77fe640 1 -- 172.21.15.94:0/766190645 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f4aac00f7f0 con 0x7f4ab0101e30 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.088+0000 7f4abaff9740 1 Processor -- start 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.088+0000 7f4abaff9740 1 -- start start 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b0e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b17430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b17a00 0x55ace7b19e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4abaff9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ace7a731f0 con 0x55ace7b0e4d0 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4abaff9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ace7a104b0 con 0x55ace7b0ee80 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4abaff9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ace7a6f1b0 con 0x55ace7b17a00 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4aa77fe640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a9c0a3d20 0x7f4a9c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab69a3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b0e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab69a3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b0e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48098/0 (socket says 172.21.15.94:48098) 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab69a3640 1 -- 172.21.15.94:0/2681264154 learned_addr learned my addr 172.21.15.94:0/2681264154 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/2681264154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b17430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4aa77fe640 1 -- 172.21.15.94:0/766190645 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7f4aac0d1090 con 0x7f4ab0101e30 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a9c0a3d20 0x7f4a9c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab59a1640 1 --2- 172.21.15.94:0/2681264154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b17a00 0x55ace7b19e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab69a3640 1 -- 172.21.15.94:0/2681264154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b17a00 msgr2=0x55ace7b19e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/2681264154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b17a00 0x55ace7b19e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab69a3640 1 -- 172.21.15.94:0/2681264154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 msgr2=0x55ace7b17430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/2681264154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b17430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab69a3640 1 -- 172.21.15.94:0/2681264154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ace7a021b0 con 0x55ace7b0e4d0 2024-04-21T08:47:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/2681264154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b0e8b0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8005ec0 tx=0x7f4aa8004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563e85dbec93a4a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.089+0000 7f4aa4bfe640 1 -- 172.21.15.94:0/2681264154 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa802cb80 con 0x55ace7b0e4d0 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4aa4bfe640 1 -- 172.21.15.94:0/2681264154 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aa8005410 con 0x55ace7b0e4d0 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4aa4bfe640 1 -- 172.21.15.94:0/2681264154 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8003850 con 0x55ace7b0e4d0 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4abaff9740 1 -- 172.21.15.94:0/2681264154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 msgr2=0x55ace7b0e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a9c0a3d20 0x7f4a9c0a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8006010 tx=0x7f4aa8048000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4abaff9740 1 --2- 172.21.15.94:0/2681264154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b0e8b0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8005ec0 tx=0x7f4aa8004560 comp rx=0 tx=0).stop 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4abaff9740 1 -- 172.21.15.94:0/2681264154 shutdown_connections 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4abaff9740 1 --2- 172.21.15.94:0/2681264154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b17a00 0x55ace7b19e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4abaff9740 1 --2- 172.21.15.94:0/2681264154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b17430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4abaff9740 1 --2- 172.21.15.94:0/2681264154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b0e8b0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4abaff9740 1 -- 172.21.15.94:0/2681264154 >> 172.21.15.94:0/2681264154 conn(0x55ace7b08ba0 msgr2=0x55ace7b08f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4abaff9740 1 -- 172.21.15.94:0/2681264154 shutdown_connections 2024-04-21T08:47:37.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.090+0000 7f4abaff9740 1 -- 172.21.15.94:0/2681264154 wait complete. 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4abaff9740 1 -- start start 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4abaff9740 1 Processor -- start 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0e4d0 0x55ace7b17300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b14690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b14c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4abaff9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ace7b1a970 con 0x55ace7b17a00 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4abaff9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ace7a2c500 con 0x55ace7b0ee80 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4abaff9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ace7b1aaf0 con 0x55ace7b0e4d0 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4abaff9740 10 client.0 _ll_register_callbacks cb 0x55ace79fab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab69a3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b14c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab69a3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b14c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48114/0 (socket says 172.21.15.94:48114) 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab69a3640 1 -- 172.21.15.94:0/3043220487 learned_addr learned my addr 172.21.15.94:0/3043220487 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:37.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0e4d0 0x55ace7b17300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab69a3640 1 -- 172.21.15.94:0/3043220487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0e4d0 msgr2=0x55ace7b17300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0e4d0 0x55ace7b17300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab69a3640 1 -- 172.21.15.94:0/3043220487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 msgr2=0x55ace7b14690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab59a1640 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b14690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b14690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab69a3640 1 -- 172.21.15.94:0/3043220487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ace77853f0 con 0x55ace7b17a00 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.091+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0e4d0 0x55ace7b17300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b14c10 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8005ab0 tx=0x7f4aa802c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ceded0f1d15d44f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4a99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3043220487 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa805a070 con 0x55ace7b17a00 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3043220487 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7f4aa80037f0 con 0x55ace7b17a00 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4abaff9740 1 -- 172.21.15.94:0/3043220487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4aa8005410 con 0x55ace7b17a00 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3043220487 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4aa8065050 con 0x55ace7b17a00 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4abaff9740 10 client.17643 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4abaff9740 1 -- 172.21.15.94:0/3043220487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ace784f7d0 con 0x55ace7b17a00 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3043220487 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aa8005410 con 0x55ace7b17a00 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3043220487 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4aa8043450 con 0x55ace7b17a00 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4abaff9740 10 client.17643 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4abaff9740 10 client.17643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4abaff9740 20 client.17643 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ace7b27a80 0x55ace7b29e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4abaff9740 4 client.17643 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4abaff9740 1 -- 172.21.15.94:0/3043220487 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ace7a12930 con 0x55ace7b27a80 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.092+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ace7b27a80 0x55ace7b29e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.093+0000 7f4a99ffb640 10 client.17643 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.093+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ace7b27a80 0x55ace7b29e70 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75c336bcef724aac server_cookie=8c107d3aecdbbf45 in_seq=0 out_seq=0 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.093+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3043220487 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ace7a12930 con 0x55ace7b27a80 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.093+0000 7f4a99ffb640 10 client.17643 handle_command_reply: tid=0 2024-04-21T08:47:37.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.093+0000 7f4abaff9740 1 client.17643 shutdown 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 1 -- 172.21.15.94:0/3043220487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 msgr2=0x55ace7b14c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b14c10 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8005ab0 tx=0x7f4aa802c860 comp rx=0 tx=0).stop 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 1 -- 172.21.15.94:0/3043220487 shutdown_connections 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ace7b27a80 0x55ace7b29e70 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b14c10 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b14690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3043220487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0e4d0 0x55ace7b17300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 1 -- 172.21.15.94:0/3043220487 >> 172.21.15.94:0/3043220487 conn(0x55ace7b08ba0 msgr2=0x55ace7b0a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4a99ffb640 0 client.17643 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 1 -- 172.21.15.94:0/3043220487 shutdown_connections 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 1 -- 172.21.15.94:0/3043220487 wait complete. 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.094+0000 7f4abaff9740 20 client.17643 trim_cache size 0 max 16384 2024-04-21T08:47:37.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4abaff9740 1 Processor -- start 2024-04-21T08:47:37.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4abaff9740 1 -- start start 2024-04-21T08:47:37.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0e4d0 0x55ace7b23830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b0fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b16dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4abaff9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01b1ce0 con 0x55ace7b17a00 2024-04-21T08:47:37.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4abaff9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01af5b0 con 0x55ace7b0ee80 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4abaff9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01ad420 con 0x55ace7b0e4d0 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4ab69a3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b16dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4ab69a3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b16dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48118/0 (socket says 172.21.15.94:48118) 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4ab69a3640 1 -- 172.21.15.94:0/2793712314 learned_addr learned my addr 172.21.15.94:0/2793712314 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4ab69a3640 1 -- 172.21.15.94:0/2793712314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0e4d0 msgr2=0x55ace7b23830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/2793712314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0e4d0 0x55ace7b23830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4ab69a3640 1 -- 172.21.15.94:0/2793712314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 msgr2=0x55ace7b0fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/2793712314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b0fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4ab69a3640 1 -- 172.21.15.94:0/2793712314 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ace7b14ee0 con 0x55ace7b17a00 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.107+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/2793712314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b16dc0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f4aa80438a0 tx=0x7f4aa80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945a650ea3da0568 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4a9bfff640 1 -- 172.21.15.94:0/2793712314 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa805a070 con 0x55ace7b17a00 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4a9bfff640 1 -- 172.21.15.94:0/2793712314 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aa8004b70 con 0x55ace7b17a00 2024-04-21T08:47:37.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4a9bfff640 1 -- 172.21.15.94:0/2793712314 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8043e60 con 0x55ace7b17a00 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4abaff9740 1 -- 172.21.15.94:0/2793712314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 msgr2=0x55ace7b16dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4abaff9740 1 --2- 172.21.15.94:0/2793712314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b16dc0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f4aa80438a0 tx=0x7f4aa80045e0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4abaff9740 1 -- 172.21.15.94:0/2793712314 shutdown_connections 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4abaff9740 1 --2- 172.21.15.94:0/2793712314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b17a00 0x55ace7b16dc0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4abaff9740 1 --2- 172.21.15.94:0/2793712314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b0ee80 0x55ace7b0fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4abaff9740 1 --2- 172.21.15.94:0/2793712314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0e4d0 0x55ace7b23830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4abaff9740 1 -- 172.21.15.94:0/2793712314 >> 172.21.15.94:0/2793712314 conn(0x55ace7b08ba0 msgr2=0x55ace7b0f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4abaff9740 1 -- 172.21.15.94:0/2793712314 shutdown_connections 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4abaff9740 1 -- 172.21.15.94:0/2793712314 wait complete. 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.108+0000 7f4abaff9740 1 -- start start 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4abaff9740 1 Processor -- start 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b1fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0ee80 0x55ace7b20280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4abaff9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b17a00 0x55ace7b10ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4abaff9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01ada70 con 0x55ace7b0e4d0 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4abaff9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01aeee0 con 0x55ace7b17a00 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4abaff9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01ae140 con 0x55ace7b0ee80 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4abaff9740 10 client.0 _ll_register_callbacks cb 0x7f4aa80043b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab61a2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b1fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab61a2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b1fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48134/0 (socket says 172.21.15.94:48134) 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab61a2640 1 -- 172.21.15.94:0/3270767267 learned_addr learned my addr 172.21.15.94:0/3270767267 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b17a00 0x55ace7b10ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab59a1640 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0ee80 0x55ace7b20280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab61a2640 1 -- 172.21.15.94:0/3270767267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0ee80 msgr2=0x55ace7b20280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0ee80 0x55ace7b20280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab61a2640 1 -- 172.21.15.94:0/3270767267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b17a00 msgr2=0x55ace7b10ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b17a00 0x55ace7b10ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab61a2640 1 -- 172.21.15.94:0/3270767267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ace7b116e0 con 0x55ace7b0e4d0 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab59a1640 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0ee80 0x55ace7b20280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab69a3640 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b17a00 0x55ace7b10ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4ab61a2640 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b1fd40 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0995d0 tx=0x7f4aac007940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d167dafead18f05a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4a99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.109+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3270767267 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aac09cc00 con 0x55ace7b0e4d0 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4abaff9740 1 -- 172.21.15.94:0/3270767267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4aa8004b70 con 0x55ace7b0e4d0 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3270767267 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7f4aac018d00 con 0x55ace7b0e4d0 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3270767267 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4aac0a0fb0 con 0x55ace7b0e4d0 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4abaff9740 10 client.17649 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4abaff9740 1 -- 172.21.15.94:0/3270767267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ace7b8d8c0 con 0x55ace7b0e4d0 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3270767267 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aa8004b70 con 0x55ace7b0e4d0 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3270767267 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4aac099270 con 0x55ace7b0e4d0 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4abaff9740 10 client.17649 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:37.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4abaff9740 10 client.17649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4abaff9740 20 client.17649 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ace7b8dfc0 0x55ace7b903b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4abaff9740 4 client.17649 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4abaff9740 1 -- 172.21.15.94:0/3270767267 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ace7a80c10 con 0x55ace7b8dfc0 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.110+0000 7f4ab59a1640 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ace7b8dfc0 0x55ace7b903b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4a99ffb640 10 client.17649 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab59a1640 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ace7b8dfc0 0x55ace7b903b0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7acd7f022766c46f server_cookie=2fa3029faf7714c4 in_seq=0 out_seq=0 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4a99ffb640 1 -- 172.21.15.94:0/3270767267 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ace7a80c10 con 0x55ace7b8dfc0 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4a99ffb640 10 client.17649 handle_command_reply: tid=0 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 client.17649 shutdown 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 -- 172.21.15.94:0/3270767267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 msgr2=0x55ace7b1fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b1fd40 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0995d0 tx=0x7f4aac007940 comp rx=0 tx=0).stop 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 -- 172.21.15.94:0/3270767267 shutdown_connections 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55ace7b8dfc0 0x55ace7b903b0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ace7b0ee80 0x55ace7b20280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4a99ffb640 0 client.17649 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:37.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ace7b17a00 0x55ace7b10ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 --2- 172.21.15.94:0/3270767267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ace7b0e4d0 0x55ace7b1fd40 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 -- 172.21.15.94:0/3270767267 >> 172.21.15.94:0/3270767267 conn(0x55ace7b08ba0 msgr2=0x55ace7b136e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 -- 172.21.15.94:0/3270767267 shutdown_connections 2024-04-21T08:47:37.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 1 -- 172.21.15.94:0/3270767267 wait complete. 2024-04-21T08:47:37.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4abaff9740 20 client.17649 trim_cache size 0 max 16384 2024-04-21T08:47:37.118 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:47:37.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab8406640 1 -- 172.21.15.94:0/766190645 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a9c0a3d20 msgr2=0x7f4a9c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab8406640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a9c0a3d20 0x7f4a9c0a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8006010 tx=0x7f4aa8048000 comp rx=0 tx=0).stop 2024-04-21T08:47:37.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab8406640 1 -- 172.21.15.94:0/766190645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab0101e30 msgr2=0x7f4ab0199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab8406640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab0101e30 0x7f4ab0199500 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00cce0 tx=0x7f4aac00a620 comp rx=0 tx=0).stop 2024-04-21T08:47:37.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab8406640 1 -- 172.21.15.94:0/766190645 shutdown_connections 2024-04-21T08:47:37.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab8406640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f4a9c0a3d20 0x7f4a9c0a61e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab8406640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ab0199fd0 0x7f4ab01a4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab8406640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ab01072f0 0x7f4ab0199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab8406640 1 --2- 172.21.15.94:0/766190645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab0101e30 0x7f4ab0199500 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.111+0000 7f4ab8406640 1 -- 172.21.15.94:0/766190645 >> 172.21.15.94:0/766190645 conn(0x7f4ab0076150 msgr2=0x7f4ab00ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.112+0000 7f4ab8406640 1 -- 172.21.15.94:0/766190645 shutdown_connections 2024-04-21T08:47:37.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:37.112+0000 7f4ab8406640 1 -- 172.21.15.94:0/766190645 wait complete. 2024-04-21T08:47:37.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:36 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/595109039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:37.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:36 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/595109039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:38.123 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:47:38.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:47:38.123 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:47:38.148 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:37 smithi094 ceph-mon[27988]: pgmap v234: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:47:38.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:37 smithi111 ceph-mon[31493]: pgmap v234: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:47:38.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.242+0000 7fda81ba4640 1 -- 172.21.15.94:0/685080884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7c102190 msgr2=0x7fda7c102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.242+0000 7fda81ba4640 1 --2- 172.21.15.94:0/685080884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7c102190 0x7fda7c102570 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009900 tx=0x7fda6c02bc50 comp rx=0 tx=0).stop 2024-04-21T08:47:38.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.242+0000 7fda81ba4640 1 -- 172.21.15.94:0/685080884 shutdown_connections 2024-04-21T08:47:38.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.242+0000 7fda81ba4640 1 --2- 172.21.15.94:0/685080884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda7c107260 0x7fda7c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.242+0000 7fda81ba4640 1 --2- 172.21.15.94:0/685080884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda7c102b40 0x7fda7c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.242+0000 7fda81ba4640 1 --2- 172.21.15.94:0/685080884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7c102190 0x7fda7c102570 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.242+0000 7fda81ba4640 1 -- 172.21.15.94:0/685080884 >> 172.21.15.94:0/685080884 conn(0x7fda7c0793a0 msgr2=0x7fda7c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.242+0000 7fda81ba4640 1 -- 172.21.15.94:0/685080884 shutdown_connections 2024-04-21T08:47:38.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.242+0000 7fda81ba4640 1 -- 172.21.15.94:0/685080884 wait complete. 2024-04-21T08:47:38.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.242+0000 7fda81ba4640 1 Processor -- start 2024-04-21T08:47:38.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda81ba4640 1 -- start start 2024-04-21T08:47:38.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda81ba4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7c102190 0x7fda7c19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda81ba4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda7c107260 0x7fda7c19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda81ba4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda7c198590 0x7fda7c198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda81ba4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c1121d0 con 0x7fda7c102190 2024-04-21T08:47:38.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda81ba4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c112050 con 0x7fda7c107260 2024-04-21T08:47:38.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda81ba4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c112350 con 0x7fda7c198590 2024-04-21T08:47:38.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7c102190 0x7fda7c19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7c102190 0x7fda7c19e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48158/0 (socket says 172.21.15.94:48158) 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7b7fe640 1 -- 172.21.15.94:0/1804443209 learned_addr learned my addr 172.21.15.94:0/1804443209 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7affd640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda7c107260 0x7fda7c19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7bfff640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda7c198590 0x7fda7c198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7b7fe640 1 -- 172.21.15.94:0/1804443209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda7c198590 msgr2=0x7fda7c198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda7c198590 0x7fda7c198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7b7fe640 1 -- 172.21.15.94:0/1804443209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda7c107260 msgr2=0x7fda7c19e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda7c107260 0x7fda7c19e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7b7fe640 1 -- 172.21.15.94:0/1804443209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7c1990b0 con 0x7fda7c102190 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7affd640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda7c107260 0x7fda7c19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7c102190 0x7fda7c19e3e0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fda6c000c00 tx=0x7fda6c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.243+0000 7fda78ff9640 1 -- 172.21.15.94:0/1804443209 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c043020 con 0x7fda7c102190 2024-04-21T08:47:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.244+0000 7fda78ff9640 1 -- 172.21.15.94:0/1804443209 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda6c0052b0 con 0x7fda7c102190 2024-04-21T08:47:38.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.244+0000 7fda81ba4640 1 -- 172.21.15.94:0/1804443209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda7c199340 con 0x7fda7c102190 2024-04-21T08:47:38.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.244+0000 7fda78ff9640 1 -- 172.21.15.94:0/1804443209 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c003b70 con 0x7fda7c102190 2024-04-21T08:47:38.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.244+0000 7fda81ba4640 1 -- 172.21.15.94:0/1804443209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda7c1a5590 con 0x7fda7c102190 2024-04-21T08:47:38.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda78ff9640 1 -- 172.21.15.94:0/1804443209 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fda6c046020 con 0x7fda7c102190 2024-04-21T08:47:38.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda84793740 1 Processor -- start 2024-04-21T08:47:38.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda84793740 1 -- start start 2024-04-21T08:47:38.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda84793740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 0x55981dfa0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda84793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 0x55981dfa9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda78ff9640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda640a3d20 0x7fda640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda84793740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa9e10 0x55981dfac260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda84793740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55981df04f00 con 0x55981dfa9e10 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda84793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55981dea2590 con 0x55981dfa1290 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda84793740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55981df00ec0 con 0x55981dfa08e0 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda7bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa9e10 0x55981dfac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.246+0000 7fda7affd640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda640a3d20 0x7fda640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa9e10 0x55981dfac260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48162/0 (socket says 172.21.15.94:48162) 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7bfff640 1 -- 172.21.15.94:0/3197919758 learned_addr learned my addr 172.21.15.94:0/3197919758 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda78ff9640 1 -- 172.21.15.94:0/1804443209 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7fda6c0f2f00 con 0x7fda7c102190 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7bfff640 1 --2- 172.21.15.94:0/3197919758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 0x55981dfa0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/3197919758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 0x55981dfa9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7bfff640 1 -- 172.21.15.94:0/3197919758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 msgr2=0x55981dfa0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7bfff640 1 --2- 172.21.15.94:0/3197919758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 0x55981dfa0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7bfff640 1 -- 172.21.15.94:0/3197919758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 msgr2=0x55981dfa9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7bfff640 1 --2- 172.21.15.94:0/3197919758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 0x55981dfa9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7affd640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda640a3d20 0x7fda640a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fda7c1018e0 tx=0x7fda70005d30 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7bfff640 1 -- 172.21.15.94:0/3197919758 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55981dcdf7e0 con 0x55981dfa9e10 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda7bfff640 1 --2- 172.21.15.94:0/3197919758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa9e10 0x55981dfac260 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fda74014a20 tx=0x7fda74009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678606628203f268 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda61ffb640 1 -- 172.21.15.94:0/3197919758 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74014270 con 0x55981dfa9e10 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda61ffb640 1 -- 172.21.15.94:0/3197919758 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda7400f700 con 0x55981dfa9e10 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.247+0000 7fda61ffb640 1 -- 172.21.15.94:0/3197919758 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74005280 con 0x55981dfa9e10 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.248+0000 7fda84793740 1 -- 172.21.15.94:0/3197919758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa9e10 msgr2=0x55981dfac260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.248+0000 7fda84793740 1 --2- 172.21.15.94:0/3197919758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa9e10 0x55981dfac260 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fda74014a20 tx=0x7fda74009600 comp rx=0 tx=0).stop 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.248+0000 7fda84793740 1 -- 172.21.15.94:0/3197919758 shutdown_connections 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.248+0000 7fda84793740 1 --2- 172.21.15.94:0/3197919758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa9e10 0x55981dfac260 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.248+0000 7fda84793740 1 --2- 172.21.15.94:0/3197919758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 0x55981dfa9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.248+0000 7fda84793740 1 --2- 172.21.15.94:0/3197919758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 0x55981dfa0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.248+0000 7fda84793740 1 -- 172.21.15.94:0/3197919758 >> 172.21.15.94:0/3197919758 conn(0x55981df9afb0 msgr2=0x55981df9b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.248+0000 7fda84793740 1 -- 172.21.15.94:0/3197919758 shutdown_connections 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.248+0000 7fda84793740 1 -- 172.21.15.94:0/3197919758 wait complete. 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.248+0000 7fda84793740 1 -- start start 2024-04-21T08:47:38.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda84793740 1 Processor -- start 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda84793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa08e0 0x55981dfa6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda84793740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfa72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda84793740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa9e10 0x55981dfa7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda84793740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55981dfacd80 con 0x55981dfa1290 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda84793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55981debe580 con 0x55981dfa08e0 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda84793740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55981dfacf00 con 0x55981dfa9e10 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda84793740 10 client.0 _ll_register_callbacks cb 0x55981dec4650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda7affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfa72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda7affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfa72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48178/0 (socket says 172.21.15.94:48178) 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda7affd640 1 -- 172.21.15.94:0/1735629435 learned_addr learned my addr 172.21.15.94:0/1735629435 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa08e0 0x55981dfa6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda7affd640 1 -- 172.21.15.94:0/1735629435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa9e10 msgr2=0x55981dfa7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda7affd640 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa9e10 0x55981dfa7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda7affd640 1 -- 172.21.15.94:0/1735629435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa08e0 msgr2=0x55981dfa6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda7affd640 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa08e0 0x55981dfa6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda7affd640 1 -- 172.21.15.94:0/1735629435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55981dd3ff70 con 0x55981dfa1290 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda7affd640 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfa72f0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fda7000f9a0 tx=0x7fda700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21acb36b9bbc856a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.249+0000 7fda5affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda5affd640 1 -- 172.21.15.94:0/1735629435 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70020070 con 0x55981dfa1290 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda5affd640 1 -- 172.21.15.94:0/1735629435 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7fda7000ecc0 con 0x55981dfa1290 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda84793740 1 -- 172.21.15.94:0/1735629435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda7400f700 con 0x55981dfa1290 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda5affd640 1 -- 172.21.15.94:0/1735629435 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fda7002c050 con 0x55981dfa1290 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda84793740 10 client.17661 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda84793740 1 -- 172.21.15.94:0/1735629435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55981dbd7170 con 0x55981dfa1290 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda5affd640 1 -- 172.21.15.94:0/1735629435 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda7400f700 con 0x55981dfa1290 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda5affd640 1 -- 172.21.15.94:0/1735629435 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fda700243b0 con 0x55981dfa1290 2024-04-21T08:47:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda84793740 10 client.17661 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda84793740 10 client.17661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda84793740 20 client.17661 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda84793740 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55981dfba360 0x55981dfbc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda84793740 4 client.17661 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.250+0000 7fda84793740 1 -- 172.21.15.94:0/1735629435 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55981df12920 con 0x55981dfba360 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.251+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55981dfba360 0x55981dfbc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.251+0000 7fda5affd640 10 client.17661 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.251+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55981dfba360 0x55981dfbc800 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6650d32eefab89e server_cookie=1bab2d6597b6af85 in_seq=0 out_seq=0 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda5affd640 1 -- 172.21.15.94:0/1735629435 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55981df12920 con 0x55981dfba360 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda5affd640 10 client.17661 handle_command_reply: tid=0 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 client.17661 shutdown 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 -- 172.21.15.94:0/1735629435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 msgr2=0x55981dfa72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfa72f0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fda7000f9a0 tx=0x7fda700046c0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 -- 172.21.15.94:0/1735629435 shutdown_connections 2024-04-21T08:47:38.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55981dfba360 0x55981dfbc800 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa9e10 0x55981dfa7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda5affd640 0 client.17661 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:38.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfa72f0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 --2- 172.21.15.94:0/1735629435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa08e0 0x55981dfa6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 -- 172.21.15.94:0/1735629435 >> 172.21.15.94:0/1735629435 conn(0x55981df9afb0 msgr2=0x55981df9cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 -- 172.21.15.94:0/1735629435 shutdown_connections 2024-04-21T08:47:38.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 1 -- 172.21.15.94:0/1735629435 wait complete. 2024-04-21T08:47:38.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.252+0000 7fda84793740 20 client.17661 trim_cache size 0 max 16384 2024-04-21T08:47:38.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:37 smithi173 ceph-mon[31428]: pgmap v234: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:47:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda84793740 1 Processor -- start 2024-04-21T08:47:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda84793740 1 -- start start 2024-04-21T08:47:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda84793740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 0x55981dfa94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda84793740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfb5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda84793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa9e10 0x55981dfc20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda84793740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c1b1ea0 con 0x55981dfa1290 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda84793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c1af770 con 0x55981dfa9e10 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda84793740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c1ad5e0 con 0x55981dfa08e0 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda7affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfb5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda7affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfb5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48188/0 (socket says 172.21.15.94:48188) 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda7affd640 1 -- 172.21.15.94:0/1334638097 learned_addr learned my addr 172.21.15.94:0/1334638097 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda7affd640 1 -- 172.21.15.94:0/1334638097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 msgr2=0x55981dfa94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 0x55981dfa94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda7bfff640 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa9e10 0x55981dfc20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda7affd640 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 0x55981dfa94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda7affd640 1 -- 172.21.15.94:0/1334638097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa9e10 msgr2=0x55981dfc20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda7affd640 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa9e10 0x55981dfc20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.270+0000 7fda7affd640 1 -- 172.21.15.94:0/1334638097 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55981df9b7f0 con 0x55981dfa1290 2024-04-21T08:47:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 0x55981dfa94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda7bfff640 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa9e10 0x55981dfc20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda7affd640 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfb5c40 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fda700058c0 tx=0x7fda70005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a126a91f5ea7e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda617fa640 1 -- 172.21.15.94:0/1334638097 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70034030 con 0x55981dfa1290 2024-04-21T08:47:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda617fa640 1 -- 172.21.15.94:0/1334638097 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda7000ddf0 con 0x55981dfa1290 2024-04-21T08:47:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda617fa640 1 -- 172.21.15.94:0/1334638097 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70020050 con 0x55981dfa1290 2024-04-21T08:47:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda84793740 1 -- 172.21.15.94:0/1334638097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 msgr2=0x55981dfb5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda84793740 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfb5c40 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fda700058c0 tx=0x7fda70005ee0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda84793740 1 -- 172.21.15.94:0/1334638097 shutdown_connections 2024-04-21T08:47:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda84793740 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa9e10 0x55981dfc20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda84793740 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa1290 0x55981dfb5c40 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda84793740 1 --2- 172.21.15.94:0/1334638097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa08e0 0x55981dfa94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda84793740 1 -- 172.21.15.94:0/1334638097 >> 172.21.15.94:0/1334638097 conn(0x55981df9afb0 msgr2=0x55981dfa7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.271+0000 7fda84793740 1 -- 172.21.15.94:0/1334638097 shutdown_connections 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda84793740 1 -- 172.21.15.94:0/1334638097 wait complete. 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda84793740 1 -- start start 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda84793740 1 Processor -- start 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda84793740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa08e0 0x55981dfae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda84793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 0x55981dfae850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda84793740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa9e10 0x55981e01f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda84793740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c1adc30 con 0x55981dfa08e0 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda84793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c1af0a0 con 0x55981dfa1290 2024-04-21T08:47:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda84793740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c1ae300 con 0x55981dfa9e10 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda84793740 10 client.0 _ll_register_callbacks cb 0x7fda70013070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda7b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa08e0 0x55981dfae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda7b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa08e0 0x55981dfae310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48196/0 (socket says 172.21.15.94:48196) 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda7b7fe640 1 -- 172.21.15.94:0/17638907 learned_addr learned my addr 172.21.15.94:0/17638907 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.272+0000 7fda7bfff640 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa9e10 0x55981e01f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda7affd640 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 0x55981dfae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda7b7fe640 1 -- 172.21.15.94:0/17638907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa9e10 msgr2=0x55981e01f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa9e10 0x55981e01f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda7b7fe640 1 -- 172.21.15.94:0/17638907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 msgr2=0x55981dfae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 0x55981dfae850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda7b7fe640 1 -- 172.21.15.94:0/17638907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55981e01ff00 con 0x55981dfa08e0 2024-04-21T08:47:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda7affd640 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 0x55981dfae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda7b7fe640 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa08e0 0x55981dfae310 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0f0820 tx=0x7fda6c0bea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7abde2a8a9e4d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda5affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda5affd640 1 -- 172.21.15.94:0/17638907 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c0c2f30 con 0x55981dfa08e0 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda5affd640 1 -- 172.21.15.94:0/17638907 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7fda6c0df0b0 con 0x55981dfa08e0 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda84793740 1 -- 172.21.15.94:0/17638907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda7000ddf0 con 0x55981dfa08e0 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda5affd640 1 -- 172.21.15.94:0/17638907 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fda6c0c77f0 con 0x55981dfa08e0 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda84793740 10 client.17667 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda84793740 1 -- 172.21.15.94:0/17638907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55981e0211f0 con 0x55981dfa08e0 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda5affd640 1 -- 172.21.15.94:0/17638907 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda7000ddf0 con 0x55981dfa08e0 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.273+0000 7fda5affd640 1 -- 172.21.15.94:0/17638907 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fda6c0c7360 con 0x55981dfa08e0 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.274+0000 7fda84793740 10 client.17667 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.274+0000 7fda84793740 10 client.17667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.274+0000 7fda84793740 20 client.17667 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.274+0000 7fda84793740 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55981e021920 0x55981e023d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.274+0000 7fda84793740 4 client.17667 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:47:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.274+0000 7fda84793740 1 -- 172.21.15.94:0/17638907 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55981df0c790 con 0x55981e021920 2024-04-21T08:47:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.274+0000 7fda7affd640 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55981e021920 0x55981e023d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.274+0000 7fda7affd640 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55981e021920 0x55981e023d40 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f0163bd2c712ebc server_cookie=e3bd82d6cebf513e in_seq=0 out_seq=0 2024-04-21T08:47:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.274+0000 7fda5affd640 10 client.17667 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.275+0000 7fda5affd640 1 -- 172.21.15.94:0/17638907 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55981df0c790 con 0x55981e021920 2024-04-21T08:47:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.275+0000 7fda5affd640 10 client.17667 handle_command_reply: tid=0 2024-04-21T08:47:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.275+0000 7fda84793740 1 client.17667 shutdown 2024-04-21T08:47:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 1 -- 172.21.15.94:0/17638907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa08e0 msgr2=0x55981dfae310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa08e0 0x55981dfae310 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0f0820 tx=0x7fda6c0bea10 comp rx=0 tx=0).stop 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 1 -- 172.21.15.94:0/17638907 shutdown_connections 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55981e021920 0x55981e023d40 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55981dfa9e10 0x55981e01f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda5affd640 0 client.17667 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55981dfa1290 0x55981dfae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 1 --2- 172.21.15.94:0/17638907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55981dfa08e0 0x55981dfae310 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 1 -- 172.21.15.94:0/17638907 >> 172.21.15.94:0/17638907 conn(0x55981df9afb0 msgr2=0x55981df99780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 1 -- 172.21.15.94:0/17638907 shutdown_connections 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 1 -- 172.21.15.94:0/17638907 wait complete. 2024-04-21T08:47:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda84793740 20 client.17667 trim_cache size 0 max 16384 2024-04-21T08:47:38.286 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:47:38.286 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:47:38.286 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "e534e717-c38f-4b74-9282-53646dcaca97", 2024-04-21T08:47:38.286 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:47:38.286 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda81ba4640 1 -- 172.21.15.94:0/1804443209 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda640a3d20 msgr2=0x7fda640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda81ba4640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda640a3d20 0x7fda640a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fda7c1018e0 tx=0x7fda70005d30 comp rx=0 tx=0).stop 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda81ba4640 1 -- 172.21.15.94:0/1804443209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7c102190 msgr2=0x7fda7c19e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda81ba4640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7c102190 0x7fda7c19e3e0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fda6c000c00 tx=0x7fda6c004040 comp rx=0 tx=0).stop 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda81ba4640 1 -- 172.21.15.94:0/1804443209 shutdown_connections 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda81ba4640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fda640a3d20 0x7fda640a61e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda81ba4640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda7c198590 0x7fda7c198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda81ba4640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda7c107260 0x7fda7c19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda81ba4640 1 --2- 172.21.15.94:0/1804443209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7c102190 0x7fda7c19e3e0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.276+0000 7fda81ba4640 1 -- 172.21.15.94:0/1804443209 >> 172.21.15.94:0/1804443209 conn(0x7fda7c0793a0 msgr2=0x7fda7c0fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.277+0000 7fda81ba4640 1 -- 172.21.15.94:0/1804443209 shutdown_connections 2024-04-21T08:47:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.277+0000 7fda81ba4640 1 -- 172.21.15.94:0/1804443209 wait complete. 2024-04-21T08:47:38.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:47:38.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:47:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.429+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/453888960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7c4102b50 msgr2=0x7ff7c4106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.429+0000 7ff7cb7b2640 1 --2- 172.21.15.94:0/453888960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7c4102b50 0x7ff7c4106b90 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009900 tx=0x7ff7b402bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:38.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/453888960 shutdown_connections 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 --2- 172.21.15.94:0/453888960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c4107270 0x7ff7c410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 --2- 172.21.15.94:0/453888960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7c4102b50 0x7ff7c4106b90 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 --2- 172.21.15.94:0/453888960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c41021a0 0x7ff7c4102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/453888960 >> 172.21.15.94:0/453888960 conn(0x7ff7c40793e0 msgr2=0x7ff7c40797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/453888960 shutdown_connections 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/453888960 wait complete. 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 Processor -- start 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 -- start start 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7c41021a0 0x7ff7c4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c4107270 0x7ff7c4198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c419cfd0 0x7ff7c419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c4112190 con 0x7ff7c41021a0 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c4112010 con 0x7ff7c4107270 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7cb7b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c4112310 con 0x7ff7c419cfd0 2024-04-21T08:47:38.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.430+0000 7ff7c954e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7c41021a0 0x7ff7c4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7c954e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7c41021a0 0x7ff7c4198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48216/0 (socket says 172.21.15.94:48216) 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7c954e640 1 -- 172.21.15.94:0/656089173 learned_addr learned my addr 172.21.15.94:0/656089173 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7c8d4d640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c4107270 0x7ff7c4198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7c954e640 1 -- 172.21.15.94:0/656089173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c419cfd0 msgr2=0x7ff7c419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7c954e640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c419cfd0 0x7ff7c419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7c954e640 1 -- 172.21.15.94:0/656089173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c4107270 msgr2=0x7ff7c4198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7c954e640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c4107270 0x7ff7c4198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7c954e640 1 -- 172.21.15.94:0/656089173 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c419dac0 con 0x7ff7c41021a0 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7c954e640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7c41021a0 0x7ff7c4198710 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000d950 tx=0x7ff7c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7ba7fc640 1 -- 172.21.15.94:0/656089173 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0017070 con 0x7ff7c41021a0 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7ba7fc640 1 -- 172.21.15.94:0/656089173 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c0009c30 con 0x7ff7c41021a0 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/656089173 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c419ddb0 con 0x7ff7c41021a0 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7ba7fc640 1 -- 172.21.15.94:0/656089173 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c000f620 con 0x7ff7c41021a0 2024-04-21T08:47:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.431+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/656089173 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c41a5570 con 0x7ff7c41021a0 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.433+0000 7ff7ba7fc640 1 -- 172.21.15.94:0/656089173 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff7c0010460 con 0x7ff7c41021a0 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7ce3a4740 1 Processor -- start 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7ce3a4740 1 -- start start 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839580 0x558770839960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839ea0 0x558770842540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x558770844e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55877079e1e0 con 0x558770839ea0 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55877073b390 con 0x558770839580 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55877079a1a0 con 0x558770842a80 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7c9d4f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839580 0x558770839960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7c9d4f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839580 0x558770839960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:45340/0 (socket says 172.21.15.94:45340) 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7c954e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839ea0 0x558770842540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.434+0000 7ff7c9d4f640 1 -- 172.21.15.94:0/2504499610 learned_addr learned my addr 172.21.15.94:0/2504499610 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7c8d4d640 1 --2- 172.21.15.94:0/2504499610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x558770844e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7ba7fc640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7b00a3d20 0x7ff7b00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7c954e640 1 -- 172.21.15.94:0/2504499610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 msgr2=0x558770844e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7c954e640 1 --2- 172.21.15.94:0/2504499610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x558770844e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7c954e640 1 -- 172.21.15.94:0/2504499610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839580 msgr2=0x558770839960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7c954e640 1 --2- 172.21.15.94:0/2504499610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839580 0x558770839960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7c954e640 1 -- 172.21.15.94:0/2504499610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55877057a7d0 con 0x558770839ea0 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7c9d4f640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7b00a3d20 0x7ff7b00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7ba7fc640 1 -- 172.21.15.94:0/656089173 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7ff7c00d0920 con 0x7ff7c41021a0 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7c954e640 1 --2- 172.21.15.94:0/2504499610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839ea0 0x558770842540 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00cdc40 tx=0x7ff7c0091920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d02e5757e4225ef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.435+0000 7ff7c9d4f640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7b00a3d20 0x7ff7b00a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc0092f0 tx=0x7ff7bc005bd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.436+0000 7ff7af7fe640 1 -- 172.21.15.94:0/2504499610 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c009bfd0 con 0x558770839ea0 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.436+0000 7ff7af7fe640 1 -- 172.21.15.94:0/2504499610 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c0066fa0 con 0x558770839ea0 2024-04-21T08:47:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.436+0000 7ff7af7fe640 1 -- 172.21.15.94:0/2504499610 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c006bde0 con 0x558770839ea0 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.436+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/2504499610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839ea0 msgr2=0x558770842540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.436+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/2504499610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839ea0 0x558770842540 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00cdc40 tx=0x7ff7c0091920 comp rx=0 tx=0).stop 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.436+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/2504499610 shutdown_connections 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.436+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/2504499610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x558770844e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.436+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/2504499610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839ea0 0x558770842540 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.436+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/2504499610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839580 0x558770839960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.436+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/2504499610 >> 172.21.15.94:0/2504499610 conn(0x558770833cb0 msgr2=0x558770834090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/2504499610 shutdown_connections 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/2504499610 wait complete. 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 1 -- start start 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 1 Processor -- start 2024-04-21T08:47:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x55877083fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x558770840060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x5587708405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x558770845810 con 0x558770839580 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5587707572f0 con 0x558770839ea0 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558770845990 con 0x558770842a80 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7ce3a4740 10 client.0 _ll_register_callbacks cb 0x558770764580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7c954e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x55877083fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7c954e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x55877083fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48242/0 (socket says 172.21.15.94:48242) 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.437+0000 7ff7c954e640 1 -- 172.21.15.94:0/371322581 learned_addr learned my addr 172.21.15.94:0/371322581 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7c9d4f640 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x5587708405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7c8d4d640 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x558770840060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7c954e640 1 -- 172.21.15.94:0/371322581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 msgr2=0x5587708405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7c954e640 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x5587708405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7c954e640 1 -- 172.21.15.94:0/371322581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 msgr2=0x558770840060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7c954e640 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x558770840060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7c954e640 1 -- 172.21.15.94:0/371322581 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587704b03f0 con 0x558770839580 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7c954e640 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x55877083fb20 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0000c00 tx=0x7ff7c00097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24c38d1ab1ef8d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7acff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7acff9640 1 -- 172.21.15.94:0/371322581 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c009bdf0 con 0x558770839580 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7acff9640 1 -- 172.21.15.94:0/371322581 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7ff7c00d6130 con 0x558770839580 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/371322581 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c0066fa0 con 0x558770839580 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7acff9640 1 -- 172.21.15.94:0/371322581 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff7c009cb90 con 0x558770839580 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7ce3a4740 10 client.17679 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/371322581 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5587704700f0 con 0x558770839580 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7acff9640 1 -- 172.21.15.94:0/371322581 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c0066fa0 con 0x558770839580 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.438+0000 7ff7acff9640 1 -- 172.21.15.94:0/371322581 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff7c0098b00 con 0x558770839580 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.439+0000 7ff7ce3a4740 10 client.17679 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.439+0000 7ff7ce3a4740 10 client.17679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.439+0000 7ff7ce3a4740 20 client.17679 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.439+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558770852900 0x558770854cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.439+0000 7ff7ce3a4740 4 client.17679 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.439+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/371322581 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587707abc00 con 0x558770852900 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.439+0000 7ff7c8d4d640 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558770852900 0x558770854cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.440+0000 7ff7c8d4d640 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558770852900 0x558770854cf0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cdb4bdf6f38a28a server_cookie=56e8ab192ed0e19e in_seq=0 out_seq=0 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.440+0000 7ff7acff9640 10 client.17679 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.440+0000 7ff7acff9640 1 -- 172.21.15.94:0/371322581 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5587707abc00 con 0x558770852900 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.440+0000 7ff7acff9640 10 client.17679 handle_command_reply: tid=0 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.440+0000 7ff7ce3a4740 1 client.17679 shutdown 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/371322581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 msgr2=0x55877083fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x55877083fb20 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0000c00 tx=0x7ff7c00097b0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/371322581 shutdown_connections 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558770852900 0x558770854cf0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x5587708405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x558770840060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7acff9640 0 client.17679 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:38.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/371322581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x55877083fb20 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/371322581 >> 172.21.15.94:0/371322581 conn(0x558770833cb0 msgr2=0x558770835a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/371322581 shutdown_connections 2024-04-21T08:47:38.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/371322581 wait complete. 2024-04-21T08:47:38.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.441+0000 7ff7ce3a4740 20 client.17679 trim_cache size 0 max 16384 2024-04-21T08:47:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7ce3a4740 1 Processor -- start 2024-04-21T08:47:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7ce3a4740 1 -- start start 2024-04-21T08:47:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770839580 0x558770842250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x55877083f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770842a80 0x558770857d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41b1e30 con 0x558770842a80 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41af700 con 0x558770839ea0 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41ad570 con 0x558770839580 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7c9d4f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770842a80 0x558770857d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7c9d4f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770842a80 0x558770857d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48246/0 (socket says 172.21.15.94:48246) 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.458+0000 7ff7c9d4f640 1 -- 172.21.15.94:0/1501501255 learned_addr learned my addr 172.21.15.94:0/1501501255 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7c954e640 1 --2- 172.21.15.94:0/1501501255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770839580 0x558770842250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7c9d4f640 1 -- 172.21.15.94:0/1501501255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770839580 msgr2=0x558770842250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7c9d4f640 1 --2- 172.21.15.94:0/1501501255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770839580 0x558770842250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7c9d4f640 1 -- 172.21.15.94:0/1501501255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 msgr2=0x55877083f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7c8d4d640 1 --2- 172.21.15.94:0/1501501255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x55877083f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7c9d4f640 1 --2- 172.21.15.94:0/1501501255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x55877083f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7c9d4f640 1 -- 172.21.15.94:0/1501501255 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558770839cd0 con 0x558770842a80 2024-04-21T08:47:38.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7c9d4f640 1 --2- 172.21.15.94:0/1501501255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770842a80 0x558770857d10 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc003c30 tx=0x7ff7bc009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9ffd2fb0cf2d40 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7aeffd640 1 -- 172.21.15.94:0/1501501255 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc026020 con 0x558770842a80 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7aeffd640 1 -- 172.21.15.94:0/1501501255 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7bc003c60 con 0x558770842a80 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.459+0000 7ff7aeffd640 1 -- 172.21.15.94:0/1501501255 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc022d40 con 0x558770842a80 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/1501501255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770842a80 msgr2=0x558770857d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/1501501255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770842a80 0x558770857d10 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc003c30 tx=0x7ff7bc009990 comp rx=0 tx=0).stop 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/1501501255 shutdown_connections 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/1501501255 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770842a80 0x558770857d10 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/1501501255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x55877083f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/1501501255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770839580 0x558770842250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/1501501255 >> 172.21.15.94:0/1501501255 conn(0x558770833cb0 msgr2=0x55877084e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/1501501255 shutdown_connections 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/1501501255 wait complete. 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 -- start start 2024-04-21T08:47:38.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 Processor -- start 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x558770855b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x558770856080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x5587708b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41adbc0 con 0x558770839580 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41af030 con 0x558770839ea0 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41ae290 con 0x558770842a80 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.460+0000 7ff7ce3a4740 10 client.0 _ll_register_callbacks cb 0x7ff7bc009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7c954e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x558770855b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7c954e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x558770855b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48252/0 (socket says 172.21.15.94:48252) 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7c954e640 1 -- 172.21.15.94:0/4256414027 learned_addr learned my addr 172.21.15.94:0/4256414027 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7c9d4f640 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x5587708b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7c954e640 1 -- 172.21.15.94:0/4256414027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 msgr2=0x5587708b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7c954e640 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x5587708b7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7c954e640 1 -- 172.21.15.94:0/4256414027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 msgr2=0x558770856080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7c954e640 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x558770856080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7c954e640 1 -- 172.21.15.94:0/4256414027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587708b8020 con 0x558770839580 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7c954e640 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x558770855b40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00b0e40 tx=0x7ff7c0009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0675107bc56f464 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7acff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7acff9640 1 -- 172.21.15.94:0/4256414027 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c009bdf0 con 0x558770839580 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7acff9640 1 -- 172.21.15.94:0/4256414027 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9599+0+0 (secure 0 0 0) 0x7ff7c006e1c0 con 0x558770839580 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/4256414027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7bc003c60 con 0x558770839580 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7acff9640 1 -- 172.21.15.94:0/4256414027 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff7c009c260 con 0x558770839580 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7ce3a4740 10 client.17685 fetch_fsmap learned FSMap version 24 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/4256414027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5587708b8020 con 0x558770839580 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7acff9640 1 -- 172.21.15.94:0/4256414027 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7bc003c60 con 0x558770839580 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7acff9640 1 -- 172.21.15.94:0/4256414027 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff7c0010750 con 0x558770839580 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7ce3a4740 10 client.17685 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7ce3a4740 10 client.17685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:38.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7ce3a4740 20 client.17685 populate_metadata read hostname 'smithi094' 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.461+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5587708b94d0 0x5587708bb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.462+0000 7ff7ce3a4740 4 client.17685 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.462+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/4256414027 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5587707a5a70 con 0x5587708b94d0 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.462+0000 7ff7c8d4d640 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5587708b94d0 0x5587708bb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.462+0000 7ff7acff9640 10 client.17685 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.462+0000 7ff7c8d4d640 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5587708b94d0 0x5587708bb8e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85241659f3e0d821 server_cookie=60833aa20cf18412 in_seq=0 out_seq=0 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.462+0000 7ff7acff9640 1 -- 172.21.15.94:0/4256414027 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5587707a5a70 con 0x5587708b94d0 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7acff9640 10 client.17685 handle_command_reply: tid=0 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 client.17685 shutdown 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/4256414027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 msgr2=0x558770855b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x558770855b40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00b0e40 tx=0x7ff7c0009740 comp rx=0 tx=0).stop 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/4256414027 shutdown_connections 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5587708b94d0 0x5587708bb8e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558770842a80 0x5587708b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558770839ea0 0x558770856080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 --2- 172.21.15.94:0/4256414027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558770839580 0x558770855b40 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7acff9640 0 client.17685 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:47:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/4256414027 >> 172.21.15.94:0/4256414027 conn(0x558770833cb0 msgr2=0x55877083fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/4256414027 shutdown_connections 2024-04-21T08:47:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 1 -- 172.21.15.94:0/4256414027 wait complete. 2024-04-21T08:47:38.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7ce3a4740 20 client.17685 trim_cache size 0 max 16384 2024-04-21T08:47:38.469 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:47:38.469 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:47:38.469 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:47:38.470 INFO:teuthology.orchestra.run.smithi094.stdout: "e534e717-c38f-4b74-9282-53646dcaca97": { 2024-04-21T08:47:38.470 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:47:38.470 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "e534e717-c38f-4b74-9282-53646dcaca97", 2024-04-21T08:47:38.470 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:47:38.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:47:38.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:47:38.470 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:47:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/656089173 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7b00a3d20 msgr2=0x7ff7b00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7b00a3d20 0x7ff7b00a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc0092f0 tx=0x7ff7bc005bd0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/656089173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7c41021a0 msgr2=0x7ff7c4198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7c41021a0 0x7ff7c4198710 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000d950 tx=0x7ff7c000de10 comp rx=0 tx=0).stop 2024-04-21T08:47:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/656089173 shutdown_connections 2024-04-21T08:47:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff7b00a3d20 0x7ff7b00a61e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c419cfd0 0x7ff7c419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c4107270 0x7ff7c4198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 --2- 172.21.15.94:0/656089173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7c41021a0 0x7ff7c4198710 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/656089173 >> 172.21.15.94:0/656089173 conn(0x7ff7c40793e0 msgr2=0x7ff7c40fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/656089173 shutdown_connections 2024-04-21T08:47:38.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:47:38.463+0000 7ff7cb7b2640 1 -- 172.21.15.94:0/656089173 wait complete. 2024-04-21T08:47:38.474 INFO:tasks.cephfs.filesystem:scrub status for tag:e534e717-c38f-4b74-9282-53646dcaca97 - {'path': '/', 'tag': 'e534e717-c38f-4b74-9282-53646dcaca97', 'options': 'recursive,force'} 2024-04-21T08:47:39.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:38 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:47:39.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:38 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:39.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:38 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:47:39.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:38 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:39.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:38 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:47:39.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:38 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:39.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:38 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:47:39.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:38 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:39.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:38 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:47:39.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:38 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:39.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:38 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:47:39.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:38 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:40.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:39 smithi111 ceph-mon[31493]: pgmap v235: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:40.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:39 smithi094 ceph-mon[27988]: pgmap v235: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:40.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:39 smithi173 ceph-mon[31428]: pgmap v235: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:42.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:41 smithi111 ceph-mon[31493]: pgmap v236: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:47:42.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:41 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:47:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:41 smithi094 ceph-mon[27988]: pgmap v236: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:47:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:41 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:47:42.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:41 smithi173 ceph-mon[31428]: pgmap v236: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:47:42.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:41 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:47:44.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:43 smithi111 ceph-mon[31493]: pgmap v237: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:47:44.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:43 smithi094 ceph-mon[27988]: pgmap v237: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:47:44.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:43 smithi173 ceph-mon[31428]: pgmap v237: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:47:46.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:45 smithi111 ceph-mon[31493]: pgmap v238: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:47:46.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:45 smithi094 ceph-mon[27988]: pgmap v238: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:47:46.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:45 smithi173 ceph-mon[31428]: pgmap v238: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:47:47.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:46 smithi111 ceph-mon[31493]: pgmap v239: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:47:47.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:46 smithi094 ceph-mon[27988]: pgmap v239: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:47:47.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:46 smithi173 ceph-mon[31428]: pgmap v239: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:47:49.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:49 smithi111 ceph-mon[31493]: pgmap v240: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:47:49.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:49 smithi094 ceph-mon[27988]: pgmap v240: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:47:49.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:49 smithi173 ceph-mon[31428]: pgmap v240: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:47:51.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:51 smithi111 ceph-mon[31493]: pgmap v241: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:47:51.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:51 smithi094 ceph-mon[27988]: pgmap v241: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:47:51.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:51 smithi173 ceph-mon[31428]: pgmap v241: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:47:53.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:53 smithi111 ceph-mon[31493]: pgmap v242: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:53.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:53 smithi094 ceph-mon[27988]: pgmap v242: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:53.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:53 smithi173 ceph-mon[31428]: pgmap v242: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:47:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:55 smithi111 ceph-mon[31493]: pgmap v243: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:47:55.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:55 smithi094 ceph-mon[27988]: pgmap v243: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:47:55.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:55 smithi173 ceph-mon[31428]: pgmap v243: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:47:58.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:57 smithi111 ceph-mon[31493]: pgmap v244: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:47:58.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:57 smithi094 ceph-mon[27988]: pgmap v244: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:47:58.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:57 smithi173 ceph-mon[31428]: pgmap v244: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:48:00.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:47:59 smithi111 ceph-mon[31493]: pgmap v245: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:00.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:47:59 smithi094 ceph-mon[27988]: pgmap v245: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:00.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:47:59 smithi173 ceph-mon[31428]: pgmap v245: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:02.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:01 smithi111 ceph-mon[31493]: pgmap v246: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:48:02.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:01 smithi094 ceph-mon[27988]: pgmap v246: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:48:02.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:01 smithi173 ceph-mon[31428]: pgmap v246: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:48:04.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:03 smithi111 ceph-mon[31493]: pgmap v247: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:04.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:03 smithi094 ceph-mon[27988]: pgmap v247: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:04.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:03 smithi173 ceph-mon[31428]: pgmap v247: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:06.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:05 smithi111 ceph-mon[31493]: pgmap v248: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-21T08:48:06.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:05 smithi094 ceph-mon[27988]: pgmap v248: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-21T08:48:06.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:05 smithi173 ceph-mon[31428]: pgmap v248: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-21T08:48:08.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:07 smithi111 ceph-mon[31493]: pgmap v249: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:48:08.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:07 smithi111 ceph-mon[31493]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:48:08.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:07 smithi094 ceph-mon[27988]: pgmap v249: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:48:08.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:07 smithi094 ceph-mon[27988]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:48:08.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:07 smithi173 ceph-mon[31428]: pgmap v249: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:48:08.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:07 smithi173 ceph-mon[31428]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:48:08.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:48:08.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.582+0000 7f5074a81640 1 -- 172.21.15.94:0/2650648906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50701018d0 msgr2=0x7f5070105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.582+0000 7f5074a81640 1 --2- 172.21.15.94:0/2650648906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50701018d0 0x7f5070105d90 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f5058009900 tx=0x7f505802bd60 comp rx=0 tx=0).stop 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.582+0000 7f5074a81640 1 -- 172.21.15.94:0/2650648906 shutdown_connections 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.582+0000 7f5074a81640 1 --2- 172.21.15.94:0/2650648906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50701062d0 0x7f5070106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.582+0000 7f5074a81640 1 --2- 172.21.15.94:0/2650648906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50701018d0 0x7f5070105d90 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.582+0000 7f5074a81640 1 --2- 172.21.15.94:0/2650648906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5070100fb0 0x7f5070101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.582+0000 7f5074a81640 1 -- 172.21.15.94:0/2650648906 >> 172.21.15.94:0/2650648906 conn(0x7f5070078000 msgr2=0x7f5070078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.582+0000 7f5074a81640 1 -- 172.21.15.94:0/2650648906 shutdown_connections 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.582+0000 7f5074a81640 1 -- 172.21.15.94:0/2650648906 wait complete. 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f5074a81640 1 Processor -- start 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f5074a81640 1 -- start start 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f5074a81640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5070100fb0 0x7f507019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f5074a81640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50701018d0 0x7f507019f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f5074a81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50701062d0 0x7f50701a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f5074a81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5070118790 con 0x7f50701062d0 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f5074a81640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5070118610 con 0x7f50701018d0 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f5074a81640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5070118910 con 0x7f5070100fb0 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f506ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50701062d0 0x7f50701a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f506ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50701062d0 0x7f50701a3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36600/0 (socket says 172.21.15.94:36600) 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f506ed9d640 1 -- 172.21.15.94:0/2550317274 learned_addr learned my addr 172.21.15.94:0/2550317274 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f506e59c640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5070100fb0 0x7f507019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.583+0000 7f5067fff640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50701018d0 0x7f507019f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f506ed9d640 1 -- 172.21.15.94:0/2550317274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5070100fb0 msgr2=0x7f507019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f506ed9d640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5070100fb0 0x7f507019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f506ed9d640 1 -- 172.21.15.94:0/2550317274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50701018d0 msgr2=0x7f507019f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f506ed9d640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50701018d0 0x7f507019f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f506ed9d640 1 -- 172.21.15.94:0/2550317274 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50701a3d60 con 0x7f50701062d0 2024-04-21T08:48:08.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f506ed9d640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50701062d0 0x7f50701a3630 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f506800eb10 tx=0x7f506800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f50677fe640 1 -- 172.21.15.94:0/2550317274 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068004b20 con 0x7f50701062d0 2024-04-21T08:48:08.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f50677fe640 1 -- 172.21.15.94:0/2550317274 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5068004cc0 con 0x7f50701062d0 2024-04-21T08:48:08.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f5074a81640 1 -- 172.21.15.94:0/2550317274 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50701a4050 con 0x7f50701062d0 2024-04-21T08:48:08.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f50677fe640 1 -- 172.21.15.94:0/2550317274 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068010670 con 0x7f50701062d0 2024-04-21T08:48:08.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.584+0000 7f5074a81640 1 -- 172.21.15.94:0/2550317274 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50701ab8e0 con 0x7f50701062d0 2024-04-21T08:48:08.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.586+0000 7f50677fe640 1 -- 172.21.15.94:0/2550317274 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5068005020 con 0x7f50701062d0 2024-04-21T08:48:08.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f507767c740 1 Processor -- start 2024-04-21T08:48:08.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f507767c740 1 -- start start 2024-04-21T08:48:08.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f507767c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af45960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f507767c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45ea0 0x560a5af4e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f507767c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af4ea80 0x560a5af50e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f507767c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x560a5aeaa1e0 con 0x560a5af45ea0 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f507767c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560a5ae47390 con 0x560a5af4ea80 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f507767c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560a5aea61a0 con 0x560a5af45580 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f50677fe640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f505c0a3d20 0x7f505c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f506e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45ea0 0x560a5af4e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f506e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45ea0 0x560a5af4e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36602/0 (socket says 172.21.15.94:36602) 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f506e59c640 1 -- 172.21.15.94:0/388205315 learned_addr learned my addr 172.21.15.94:0/388205315 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f5067fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af45960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f5067fff640 1 --2- 172.21.15.94:0/388205315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af4ea80 0x560a5af50e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f506ed9d640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f505c0a3d20 0x7f505c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f506e59c640 1 -- 172.21.15.94:0/388205315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 msgr2=0x560a5af45960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f506e59c640 1 --2- 172.21.15.94:0/388205315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af45960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f506e59c640 1 -- 172.21.15.94:0/388205315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af4ea80 msgr2=0x560a5af50e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f506e59c640 1 --2- 172.21.15.94:0/388205315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af4ea80 0x560a5af50e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f506e59c640 1 -- 172.21.15.94:0/388205315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a5ac867d0 con 0x560a5af45ea0 2024-04-21T08:48:08.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.587+0000 7f5067fff640 1 --2- 172.21.15.94:0/388205315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af4ea80 0x560a5af50e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.588+0000 7f50677fe640 1 -- 172.21.15.94:0/2550317274 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f50680d11d0 con 0x7f50701062d0 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.588+0000 7f506e59c640 1 --2- 172.21.15.94:0/388205315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45ea0 0x560a5af4e540 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f5060009cf0 tx=0x7f5060011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76da25ddfc714e54 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.588+0000 7f506ed9d640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f505c0a3d20 0x7f505c0a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f50680990e0 tx=0x7f5068000f40 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.588+0000 7f5064ff9640 1 -- 172.21.15.94:0/388205315 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5060004230 con 0x560a5af45ea0 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.588+0000 7f5064ff9640 1 -- 172.21.15.94:0/388205315 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50600043d0 con 0x560a5af45ea0 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.588+0000 7f5064ff9640 1 -- 172.21.15.94:0/388205315 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5060019880 con 0x560a5af45ea0 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 -- 172.21.15.94:0/388205315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45ea0 msgr2=0x560a5af4e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 --2- 172.21.15.94:0/388205315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45ea0 0x560a5af4e540 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f5060009cf0 tx=0x7f5060011810 comp rx=0 tx=0).stop 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 -- 172.21.15.94:0/388205315 shutdown_connections 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 --2- 172.21.15.94:0/388205315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af4ea80 0x560a5af50e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 --2- 172.21.15.94:0/388205315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45ea0 0x560a5af4e540 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 --2- 172.21.15.94:0/388205315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af45960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 -- 172.21.15.94:0/388205315 >> 172.21.15.94:0/388205315 conn(0x560a5af3fcb0 msgr2=0x560a5af40090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 -- 172.21.15.94:0/388205315 shutdown_connections 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 -- 172.21.15.94:0/388205315 wait complete. 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 -- start start 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 Processor -- start 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af4baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af4bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af4c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x560a5af51870 con 0x560a5af4ea80 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560a5ae632f0 con 0x560a5af45ea0 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560a5af519f0 con 0x560a5af45580 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f507767c740 10 client.0 _ll_register_callbacks cb 0x560a5ae70580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f5067fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af4c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f5067fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af4c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36618/0 (socket says 172.21.15.94:36618) 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f5067fff640 1 -- 172.21.15.94:0/1478550441 learned_addr learned my addr 172.21.15.94:0/1478550441 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f506e59c640 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af4bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.589+0000 7f5067fff640 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af4baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5067fff640 1 -- 172.21.15.94:0/1478550441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 msgr2=0x560a5af4baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5067fff640 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af4baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5067fff640 1 -- 172.21.15.94:0/1478550441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 msgr2=0x560a5af4bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5067fff640 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af4bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5067fff640 1 -- 172.21.15.94:0/1478550441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a5abbc3f0 con 0x560a5af4ea80 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5067fff640 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af4c520 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f505802c2a0 tx=0x7f5058009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc2aa46c36902a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5055ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5055ffb640 1 -- 172.21.15.94:0/1478550441 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058004480 con 0x560a5af4ea80 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5055ffb640 1 -- 172.21.15.94:0/1478550441 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f5058003a30 con 0x560a5af4ea80 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f507767c740 1 -- 172.21.15.94:0/1478550441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50600043d0 con 0x560a5af4ea80 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5055ffb640 1 -- 172.21.15.94:0/1478550441 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5058054050 con 0x560a5af4ea80 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f507767c740 10 client.17697 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:08.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f507767c740 1 -- 172.21.15.94:0/1478550441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560a5ab7c0f0 con 0x560a5af4ea80 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5055ffb640 1 -- 172.21.15.94:0/1478550441 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50600043d0 con 0x560a5af4ea80 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.590+0000 7f5055ffb640 1 -- 172.21.15.94:0/1478550441 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5058043ce0 con 0x560a5af4ea80 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.591+0000 7f507767c740 10 client.17697 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.591+0000 7f507767c740 10 client.17697 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.591+0000 7f507767c740 20 client.17697 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.591+0000 7f507767c740 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560a5af5ea50 0x560a5af60e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.591+0000 7f507767c740 4 client.17697 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.591+0000 7f507767c740 1 -- 172.21.15.94:0/1478550441 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a5aeb7c00 con 0x560a5af5ea50 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.591+0000 7f506e59c640 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560a5af5ea50 0x560a5af60e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.591+0000 7f5055ffb640 10 client.17697 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.591+0000 7f506e59c640 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560a5af5ea50 0x560a5af60e60 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9f7da04f6395693 server_cookie=df27a5574b15072f in_seq=0 out_seq=0 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f5055ffb640 1 -- 172.21.15.94:0/1478550441 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a5aeb7c00 con 0x560a5af5ea50 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f5055ffb640 10 client.17697 handle_command_reply: tid=0 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 client.17697 shutdown 2024-04-21T08:48:08.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 -- 172.21.15.94:0/1478550441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 msgr2=0x560a5af4c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af4c520 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f505802c2a0 tx=0x7f5058009260 comp rx=0 tx=0).stop 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 -- 172.21.15.94:0/1478550441 shutdown_connections 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560a5af5ea50 0x560a5af60e60 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af4c520 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af4bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 --2- 172.21.15.94:0/1478550441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af4baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f5055ffb640 0 client.17697 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 -- 172.21.15.94:0/1478550441 >> 172.21.15.94:0/1478550441 conn(0x560a5af3fcb0 msgr2=0x560a5af41a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 -- 172.21.15.94:0/1478550441 shutdown_connections 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.592+0000 7f507767c740 1 -- 172.21.15.94:0/1478550441 wait complete. 2024-04-21T08:48:08.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.593+0000 7f507767c740 20 client.17697 trim_cache size 0 max 16384 2024-04-21T08:48:08.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.609+0000 7f507767c740 1 Processor -- start 2024-04-21T08:48:08.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.609+0000 7f507767c740 1 -- start start 2024-04-21T08:48:08.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.609+0000 7f507767c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45580 0x560a5af5a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.609+0000 7f507767c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af4b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.609+0000 7f507767c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af4ea80 0x560a5af4c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f507767c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b85c0 con 0x560a5af45580 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f507767c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b5e90 con 0x560a5af45ea0 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f507767c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b3d00 con 0x560a5af4ea80 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5067fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45580 0x560a5af5a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5067fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45580 0x560a5af5a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36622/0 (socket says 172.21.15.94:36622) 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5067fff640 1 -- 172.21.15.94:0/1235633864 learned_addr learned my addr 172.21.15.94:0/1235633864 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f506e59c640 1 --2- 172.21.15.94:0/1235633864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af4b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5067fff640 1 --2- 172.21.15.94:0/1235633864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af4ea80 0x560a5af4c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5067fff640 1 -- 172.21.15.94:0/1235633864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af4ea80 msgr2=0x560a5af4c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5067fff640 1 --2- 172.21.15.94:0/1235633864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af4ea80 0x560a5af4c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5067fff640 1 -- 172.21.15.94:0/1235633864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 msgr2=0x560a5af4b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5067fff640 1 --2- 172.21.15.94:0/1235633864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af4b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5067fff640 1 -- 172.21.15.94:0/1235633864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a5af49ff0 con 0x560a5af45580 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5067fff640 1 --2- 172.21.15.94:0/1235633864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45580 0x560a5af5a730 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f5058005eb0 tx=0x7f505802bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46658d95d0cfbc85 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5057fff640 1 -- 172.21.15.94:0/1235633864 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505805d070 con 0x560a5af45580 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5057fff640 1 -- 172.21.15.94:0/1235633864 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50580043c0 con 0x560a5af45580 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.610+0000 7f5057fff640 1 -- 172.21.15.94:0/1235633864 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058005030 con 0x560a5af45580 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 -- 172.21.15.94:0/1235633864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45580 msgr2=0x560a5af5a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 --2- 172.21.15.94:0/1235633864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45580 0x560a5af5a730 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f5058005eb0 tx=0x7f505802bd60 comp rx=0 tx=0).stop 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 -- 172.21.15.94:0/1235633864 shutdown_connections 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 --2- 172.21.15.94:0/1235633864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af4ea80 0x560a5af4c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 --2- 172.21.15.94:0/1235633864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af4b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 --2- 172.21.15.94:0/1235633864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af45580 0x560a5af5a730 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 -- 172.21.15.94:0/1235633864 >> 172.21.15.94:0/1235633864 conn(0x560a5af3fcb0 msgr2=0x560a5af4ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 -- 172.21.15.94:0/1235633864 shutdown_connections 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 -- 172.21.15.94:0/1235633864 wait complete. 2024-04-21T08:48:08.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 -- start start 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 Processor -- start 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af48740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af48c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af491c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b4350 con 0x560a5af4ea80 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b57c0 con 0x560a5af45ea0 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f50701b4a20 con 0x560a5af45580 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f507767c740 10 client.0 _ll_register_callbacks cb 0x7f5058009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f5067fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f506e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af48c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f5067fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af491c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36624/0 (socket says 172.21.15.94:36624) 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f506e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af48c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:32880/0 (socket says 172.21.15.94:32880) 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.611+0000 7f5067fff640 1 -- 172.21.15.94:0/2385144676 learned_addr learned my addr 172.21.15.94:0/2385144676 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5067fff640 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af48740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5067fff640 1 -- 172.21.15.94:0/2385144676 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 msgr2=0x560a5af48740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5067fff640 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af48740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5067fff640 1 -- 172.21.15.94:0/2385144676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 msgr2=0x560a5af48c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5067fff640 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af48c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5067fff640 1 -- 172.21.15.94:0/2385144676 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a5afc3fa0 con 0x560a5af4ea80 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5067fff640 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af491c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f50580096c0 tx=0x7f505802c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd33590b176bc127 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5055ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5055ffb640 1 -- 172.21.15.94:0/2385144676 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505805d040 con 0x560a5af4ea80 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5055ffb640 1 -- 172.21.15.94:0/2385144676 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f5058034c10 con 0x560a5af4ea80 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f507767c740 1 -- 172.21.15.94:0/2385144676 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50580043c0 con 0x560a5af4ea80 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5055ffb640 1 -- 172.21.15.94:0/2385144676 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f505806c050 con 0x560a5af4ea80 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f507767c740 10 client.17703 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f507767c740 1 -- 172.21.15.94:0/2385144676 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560a5afc5280 con 0x560a5af4ea80 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5055ffb640 1 -- 172.21.15.94:0/2385144676 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50580043c0 con 0x560a5af4ea80 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f5055ffb640 1 -- 172.21.15.94:0/2385144676 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5058063040 con 0x560a5af4ea80 2024-04-21T08:48:08.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.612+0000 7f507767c740 10 client.17703 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 10 client.17703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 20 client.17703 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560a5afc5920 0x560a5afc7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 4 client.17703 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 -- 172.21.15.94:0/2385144676 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560a5aeb1a70 con 0x560a5afc5920 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f506e59c640 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560a5afc5920 0x560a5afc7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f506e59c640 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560a5afc5920 0x560a5afc7d40 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9def946b405dd50 server_cookie=d972c56e96380cfa in_seq=0 out_seq=0 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f5055ffb640 10 client.17703 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f5055ffb640 1 -- 172.21.15.94:0/2385144676 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560a5aeb1a70 con 0x560a5afc5920 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f5055ffb640 10 client.17703 handle_command_reply: tid=0 2024-04-21T08:48:08.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 client.17703 shutdown 2024-04-21T08:48:08.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 -- 172.21.15.94:0/2385144676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 msgr2=0x560a5af491c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af491c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f50580096c0 tx=0x7f505802c870 comp rx=0 tx=0).stop 2024-04-21T08:48:08.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 -- 172.21.15.94:0/2385144676 shutdown_connections 2024-04-21T08:48:08.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x560a5afc5920 0x560a5afc7d40 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a5af4ea80 0x560a5af491c0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a5af45ea0 0x560a5af48c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 --2- 172.21.15.94:0/2385144676 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a5af45580 0x560a5af48740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f5055ffb640 0 client.17703 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:08.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.613+0000 7f507767c740 1 -- 172.21.15.94:0/2385144676 >> 172.21.15.94:0/2385144676 conn(0x560a5af3fcb0 msgr2=0x560a5af4c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f507767c740 1 -- 172.21.15.94:0/2385144676 shutdown_connections 2024-04-21T08:48:08.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f507767c740 1 -- 172.21.15.94:0/2385144676 wait complete. 2024-04-21T08:48:08.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f507767c740 20 client.17703 trim_cache size 0 max 16384 2024-04-21T08:48:08.623 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:48:08.623 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:48:08.623 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:48:08.623 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 -- 172.21.15.94:0/2550317274 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f505c0a3d20 msgr2=0x7f505c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f505c0a3d20 0x7f505c0a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f50680990e0 tx=0x7f5068000f40 comp rx=0 tx=0).stop 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 -- 172.21.15.94:0/2550317274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50701062d0 msgr2=0x7f50701a3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50701062d0 0x7f50701a3630 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f506800eb10 tx=0x7f506800c2d0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 -- 172.21.15.94:0/2550317274 shutdown_connections 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f505c0a3d20 0x7f505c0a61e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50701062d0 0x7f50701a3630 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50701018d0 0x7f507019f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 --2- 172.21.15.94:0/2550317274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5070100fb0 0x7f507019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 -- 172.21.15.94:0/2550317274 >> 172.21.15.94:0/2550317274 conn(0x7f5070078000 msgr2=0x7f5070103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 -- 172.21.15.94:0/2550317274 shutdown_connections 2024-04-21T08:48:08.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.614+0000 7f5074a81640 1 -- 172.21.15.94:0/2550317274 wait complete. 2024-04-21T08:48:08.626 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:48:08.626 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:08.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.745+0000 7fe2d21f4640 1 -- 172.21.15.94:0/264545176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc107fd0 msgr2=0x7fe2cc1083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.745+0000 7fe2d21f4640 1 --2- 172.21.15.94:0/264545176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc107fd0 0x7fe2cc1083b0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009930 tx=0x7fe2bc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:08.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.745+0000 7fe2d21f4640 1 -- 172.21.15.94:0/264545176 shutdown_connections 2024-04-21T08:48:08.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.745+0000 7fe2d21f4640 1 --2- 172.21.15.94:0/264545176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2cc069aa0 0x7fe2cc10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.745+0000 7fe2d21f4640 1 --2- 172.21.15.94:0/264545176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0690e0 0x7fe2cc069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.745+0000 7fe2d21f4640 1 --2- 172.21.15.94:0/264545176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc107fd0 0x7fe2cc1083b0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.745+0000 7fe2d21f4640 1 -- 172.21.15.94:0/264545176 >> 172.21.15.94:0/264545176 conn(0x7fe2cc077e30 msgr2=0x7fe2cc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.745+0000 7fe2d21f4640 1 -- 172.21.15.94:0/264545176 shutdown_connections 2024-04-21T08:48:08.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.745+0000 7fe2d21f4640 1 -- 172.21.15.94:0/264545176 wait complete. 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2d21f4640 1 Processor -- start 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2d21f4640 1 -- start start 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2d21f4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0690e0 0x7fe2cc101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2d21f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2cc069aa0 0x7fe2cc1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2d21f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc107fd0 0x7fe2cc101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2d21f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe2cc112140 con 0x7fe2cc107fd0 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2d21f4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe2cc111fc0 con 0x7fe2cc0690e0 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2d21f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2cc1122c0 con 0x7fe2cc069aa0 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2cbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc107fd0 0x7fe2cc101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2cbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc107fd0 0x7fe2cc101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36636/0 (socket says 172.21.15.94:36636) 2024-04-21T08:48:08.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2cbfff640 1 -- 172.21.15.94:0/410204714 learned_addr learned my addr 172.21.15.94:0/410204714 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:08.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2cb7fe640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0690e0 0x7fe2cc101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2caffd640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2cc069aa0 0x7fe2cc1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2cbfff640 1 -- 172.21.15.94:0/410204714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2cc069aa0 msgr2=0x7fe2cc1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2cc069aa0 0x7fe2cc1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2cbfff640 1 -- 172.21.15.94:0/410204714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0690e0 msgr2=0x7fe2cc101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0690e0 0x7fe2cc101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.746+0000 7fe2cbfff640 1 -- 172.21.15.94:0/410204714 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2cc1a6b80 con 0x7fe2cc107fd0 2024-04-21T08:48:08.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.747+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc107fd0 0x7fe2cc101cf0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400eb10 tx=0x7fe2c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.747+0000 7fe2c8ff9640 1 -- 172.21.15.94:0/410204714 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4004b20 con 0x7fe2cc107fd0 2024-04-21T08:48:08.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.747+0000 7fe2c8ff9640 1 -- 172.21.15.94:0/410204714 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2c4004cc0 con 0x7fe2cc107fd0 2024-04-21T08:48:08.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.747+0000 7fe2d21f4640 1 -- 172.21.15.94:0/410204714 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2cc1a6e70 con 0x7fe2cc107fd0 2024-04-21T08:48:08.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.747+0000 7fe2c8ff9640 1 -- 172.21.15.94:0/410204714 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4010670 con 0x7fe2cc107fd0 2024-04-21T08:48:08.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.747+0000 7fe2d21f4640 1 -- 172.21.15.94:0/410204714 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2cc1a7390 con 0x7fe2cc107fd0 2024-04-21T08:48:08.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.748+0000 7fe2d21f4640 1 -- 172.21.15.94:0/410204714 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2cc068de0 con 0x7fe2cc107fd0 2024-04-21T08:48:08.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.749+0000 7fe2c8ff9640 1 -- 172.21.15.94:0/410204714 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fe2c4005020 con 0x7fe2cc107fd0 2024-04-21T08:48:08.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.749+0000 7fe2c8ff9640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2b40a3d20 0x7fe2b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.750+0000 7fe2c8ff9640 1 -- 172.21.15.94:0/410204714 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7fe2c40d1160 con 0x7fe2cc107fd0 2024-04-21T08:48:08.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.750+0000 7fe2cb7fe640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2b40a3d20 0x7fe2b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.750+0000 7fe2cb7fe640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2b40a3d20 0x7fe2b40a61e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc005ab0 tx=0x7fe2bc005fd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.754+0000 7fe2c8ff9640 1 -- 172.21.15.94:0/410204714 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2c409d370 con 0x7fe2cc107fd0 2024-04-21T08:48:08.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.983+0000 7fe2d21f4640 1 -- 172.21.15.94:0/410204714 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2cc1027a0 con 0x7fe2cc107fd0 2024-04-21T08:48:08.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.984+0000 7fe2c8ff9640 1 -- 172.21.15.94:0/410204714 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7fe2cc1027a0 con 0x7fe2cc107fd0 2024-04-21T08:48:08.984 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:48:08.984 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:48:08.984 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:48:08.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 -- 172.21.15.94:0/410204714 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2b40a3d20 msgr2=0x7fe2b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2b40a3d20 0x7fe2b40a61e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc005ab0 tx=0x7fe2bc005fd0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 -- 172.21.15.94:0/410204714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc107fd0 msgr2=0x7fe2cc101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc107fd0 0x7fe2cc101cf0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400eb10 tx=0x7fe2c400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 -- 172.21.15.94:0/410204714 shutdown_connections 2024-04-21T08:48:08.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2b40a3d20 0x7fe2b40a61e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2cc069aa0 0x7fe2cc1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc107fd0 0x7fe2cc101cf0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 --2- 172.21.15.94:0/410204714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0690e0 0x7fe2cc101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 -- 172.21.15.94:0/410204714 >> 172.21.15.94:0/410204714 conn(0x7fe2cc077e30 msgr2=0x7fe2cc100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 -- 172.21.15.94:0/410204714 shutdown_connections 2024-04-21T08:48:08.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:08.986+0000 7fe2d21f4640 1 -- 172.21.15.94:0/410204714 wait complete. 2024-04-21T08:48:08.998 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:48:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.100+0000 7f5e61a43640 1 -- 172.21.15.94:0/3813685605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e5c1015a0 msgr2=0x7f5e5c101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.100+0000 7f5e61a43640 1 --2- 172.21.15.94:0/3813685605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e5c1015a0 0x7f5e5c101980 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f5e54009ca0 tx=0x7f5e5402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.100+0000 7f5e61a43640 1 -- 172.21.15.94:0/3813685605 shutdown_connections 2024-04-21T08:48:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.100+0000 7f5e61a43640 1 --2- 172.21.15.94:0/3813685605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e5c106150 0x7f5e5c10dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.100+0000 7f5e61a43640 1 --2- 172.21.15.94:0/3813685605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c101f50 0x7f5e5c105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.100+0000 7f5e61a43640 1 --2- 172.21.15.94:0/3813685605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e5c1015a0 0x7f5e5c101980 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.100+0000 7f5e61a43640 1 -- 172.21.15.94:0/3813685605 >> 172.21.15.94:0/3813685605 conn(0x7f5e5c0769d0 msgr2=0x7f5e5c076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.101+0000 7f5e61a43640 1 -- 172.21.15.94:0/3813685605 shutdown_connections 2024-04-21T08:48:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.101+0000 7f5e61a43640 1 -- 172.21.15.94:0/3813685605 wait complete. 2024-04-21T08:48:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.101+0000 7f5e61a43640 1 Processor -- start 2024-04-21T08:48:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.101+0000 7f5e61a43640 1 -- start start 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.101+0000 7f5e61a43640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c1015a0 0x7f5e5c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.101+0000 7f5e61a43640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e5c106150 0x7f5e5c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.101+0000 7f5e61a43640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e5c19d010 0x7f5e5c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.101+0000 7f5e61a43640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c1110c0 con 0x7f5e5c19d010 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.101+0000 7f5e61a43640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c110f40 con 0x7f5e5c1015a0 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e61a43640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c111240 con 0x7f5e5c106150 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e5c19d010 0x7f5e5c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5affd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c1015a0 0x7f5e5c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e5c19d010 0x7f5e5c19d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36654/0 (socket says 172.21.15.94:36654) 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5b7fe640 1 -- 172.21.15.94:0/2779195897 learned_addr learned my addr 172.21.15.94:0/2779195897 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e5c106150 0x7f5e5c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5affd640 1 -- 172.21.15.94:0/2779195897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e5c106150 msgr2=0x7f5e5c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5affd640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e5c106150 0x7f5e5c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5affd640 1 -- 172.21.15.94:0/2779195897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e5c19d010 msgr2=0x7f5e5c19d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5affd640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e5c19d010 0x7f5e5c19d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5affd640 1 -- 172.21.15.94:0/2779195897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5c19db00 con 0x7f5e5c1015a0 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e5affd640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c1015a0 0x7f5e5c198750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5e54006010 tx=0x7f5e540346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e60a41640 1 -- 172.21.15.94:0/2779195897 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e54041020 con 0x7f5e5c1015a0 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e61a43640 1 -- 172.21.15.94:0/2779195897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e5c19dd90 con 0x7f5e5c1015a0 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e60a41640 1 -- 172.21.15.94:0/2779195897 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e54032970 con 0x7f5e5c1015a0 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.102+0000 7f5e60a41640 1 -- 172.21.15.94:0/2779195897 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e54039040 con 0x7f5e5c1015a0 2024-04-21T08:48:09.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.103+0000 7f5e61a43640 1 -- 172.21.15.94:0/2779195897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e5c1a5550 con 0x7f5e5c1015a0 2024-04-21T08:48:09.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e6463b740 1 Processor -- start 2024-04-21T08:48:09.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e6463b740 1 -- start start 2024-04-21T08:48:09.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f07f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 0x562b1f088820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f088e40 0x562b1f0892c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e60a41640 1 -- 172.21.15.94:0/2779195897 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5e54044020 con 0x7f5e5c1015a0 2024-04-21T08:48:09.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e6463b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x562b1efe41f0 con 0x562b1f088e40 2024-04-21T08:48:09.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e6463b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562b1ef814b0 con 0x562b1f07fce0 2024-04-21T08:48:09.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e6463b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562b1efe01b0 con 0x562b1f07f330 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e5a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f088e40 0x562b1f0892c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e5b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f07f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e5b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f07f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:51120/0 (socket says 172.21.15.94:51120) 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e5b7fe640 1 -- 172.21.15.94:0/2877626170 learned_addr learned my addr 172.21.15.94:0/2877626170 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.105+0000 7f5e5affd640 1 --2- 172.21.15.94:0/2877626170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 0x562b1f088820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e5b7fe640 1 -- 172.21.15.94:0/2877626170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 msgr2=0x562b1f088820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/2877626170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 0x562b1f088820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e5b7fe640 1 -- 172.21.15.94:0/2877626170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f088e40 msgr2=0x562b1f0892c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/2877626170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f088e40 0x562b1f0892c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e5b7fe640 1 -- 172.21.15.94:0/2877626170 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b1ef731b0 con 0x562b1f07f330 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/2877626170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f07f710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009d20 tx=0x7f5e4c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3c54a156bf436e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e60a41640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e440a3d20 0x7f5e440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e41ffb640 1 -- 172.21.15.94:0/2877626170 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c01e070 con 0x562b1f07f330 2024-04-21T08:48:09.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e5affd640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e440a3d20 0x7f5e440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e41ffb640 1 -- 172.21.15.94:0/2877626170 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e4c019ae0 con 0x562b1f07f330 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e41ffb640 1 -- 172.21.15.94:0/2877626170 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c00fa20 con 0x562b1f07f330 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.106+0000 7f5e60a41640 1 -- 172.21.15.94:0/2779195897 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f5e540f27a0 con 0x7f5e5c1015a0 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.107+0000 7f5e5affd640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e440a3d20 0x7f5e440a61e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5e540efc20 tx=0x7f5e540f8550 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.107+0000 7f5e6463b740 1 -- 172.21.15.94:0/2877626170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 msgr2=0x562b1f07f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.107+0000 7f5e6463b740 1 --2- 172.21.15.94:0/2877626170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f07f710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009d20 tx=0x7f5e4c011810 comp rx=0 tx=0).stop 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.107+0000 7f5e6463b740 1 -- 172.21.15.94:0/2877626170 shutdown_connections 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.107+0000 7f5e6463b740 1 --2- 172.21.15.94:0/2877626170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f088e40 0x562b1f0892c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.107+0000 7f5e6463b740 1 --2- 172.21.15.94:0/2877626170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 0x562b1f088820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.107+0000 7f5e6463b740 1 --2- 172.21.15.94:0/2877626170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f07f710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.107+0000 7f5e6463b740 1 -- 172.21.15.94:0/2877626170 >> 172.21.15.94:0/2877626170 conn(0x562b1f079ba0 msgr2=0x562b1f079f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.107+0000 7f5e6463b740 1 -- 172.21.15.94:0/2877626170 shutdown_connections 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.107+0000 7f5e6463b740 1 -- 172.21.15.94:0/2877626170 wait complete. 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e6463b740 1 -- start start 2024-04-21T08:48:09.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e6463b740 1 Processor -- start 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f08f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 0x562b1f08ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e6463b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x562b1f08fdc0 con 0x562b1f07fce0 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e6463b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562b1ef9d500 con 0x562b1f088e40 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e6463b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562b1f08ff40 con 0x562b1f07f330 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e5a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e5a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36668/0 (socket says 172.21.15.94:36668) 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e5a7fc640 1 -- 172.21.15.94:0/357535764 learned_addr learned my addr 172.21.15.94:0/357535764 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e5b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f08f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e6463b740 10 client.0 _ll_register_callbacks cb 0x562b1ef6bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.108+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 0x562b1f08ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e5a7fc640 1 -- 172.21.15.94:0/357535764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 msgr2=0x562b1f08f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f08f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e5a7fc640 1 -- 172.21.15.94:0/357535764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 msgr2=0x562b1f08ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 0x562b1f08ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e5a7fc640 1 -- 172.21.15.94:0/357535764 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b1ecf63f0 con 0x562b1f07fce0 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08c540 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000bad0 tx=0x7f5e5000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274195e2e8a9282f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e3affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e3affd640 1 -- 172.21.15.94:0/357535764 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50003e20 con 0x562b1f07fce0 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e6463b740 1 -- 172.21.15.94:0/357535764 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e4c019ae0 con 0x562b1f07fce0 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e3affd640 1 -- 172.21.15.94:0/357535764 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f5e50011440 con 0x562b1f07fce0 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e3affd640 1 -- 172.21.15.94:0/357535764 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e50011640 con 0x562b1f07fce0 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e6463b740 10 client.17721 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e6463b740 1 -- 172.21.15.94:0/357535764 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562b1edc07d0 con 0x562b1f07fce0 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.109+0000 7f5e3affd640 1 -- 172.21.15.94:0/357535764 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e4c019ae0 con 0x562b1f07fce0 2024-04-21T08:48:09.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.110+0000 7f5e3affd640 1 -- 172.21.15.94:0/357535764 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5e5001b8b0 con 0x562b1f07fce0 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.110+0000 7f5e6463b740 10 client.17721 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.110+0000 7f5e6463b740 10 client.17721 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.110+0000 7f5e6463b740 20 client.17721 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.110+0000 7f5e6463b740 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562b1f09d0b0 0x562b1f09f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.110+0000 7f5e6463b740 4 client.17721 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.110+0000 7f5e6463b740 1 -- 172.21.15.94:0/357535764 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b1ef83930 con 0x562b1f09d0b0 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.110+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562b1f09d0b0 0x562b1f09f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.110+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562b1f09d0b0 0x562b1f09f4d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54760a665172d25d server_cookie=93d3d815a2d1d9e4 in_seq=0 out_seq=0 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.110+0000 7f5e3affd640 10 client.17721 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e3affd640 1 -- 172.21.15.94:0/357535764 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562b1ef83930 con 0x562b1f09d0b0 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e3affd640 10 client.17721 handle_command_reply: tid=0 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 client.17721 shutdown 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 -- 172.21.15.94:0/357535764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 msgr2=0x562b1f08c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08c540 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000bad0 tx=0x7f5e5000bf90 comp rx=0 tx=0).stop 2024-04-21T08:48:09.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 -- 172.21.15.94:0/357535764 shutdown_connections 2024-04-21T08:48:09.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562b1f09d0b0 0x562b1f09f4d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 0x562b1f08ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08c540 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e3affd640 0 client.17721 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:48:09.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 --2- 172.21.15.94:0/357535764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f08f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 -- 172.21.15.94:0/357535764 >> 172.21.15.94:0/357535764 conn(0x562b1f079ba0 msgr2=0x562b1f07b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 -- 172.21.15.94:0/357535764 shutdown_connections 2024-04-21T08:48:09.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 1 -- 172.21.15.94:0/357535764 wait complete. 2024-04-21T08:48:09.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.111+0000 7f5e6463b740 20 client.17721 trim_cache size 0 max 16384 2024-04-21T08:48:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e6463b740 1 Processor -- start 2024-04-21T08:48:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e6463b740 1 -- start start 2024-04-21T08:48:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f098cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 0x562b1f078a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e6463b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c1afd20 con 0x562b1f07fce0 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e6463b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c1ad5f0 con 0x562b1f088e40 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e6463b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c1ab460 con 0x562b1f07f330 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36676/0 (socket says 172.21.15.94:36676) 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5a7fc640 1 -- 172.21.15.94:0/1566722624 learned_addr learned my addr 172.21.15.94:0/1566722624 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/1566722624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f098cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/1566722624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 0x562b1f078a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5a7fc640 1 -- 172.21.15.94:0/1566722624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 msgr2=0x562b1f098cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/1566722624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f098cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5a7fc640 1 -- 172.21.15.94:0/1566722624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 msgr2=0x562b1f078a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/1566722624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 0x562b1f078a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5a7fc640 1 -- 172.21.15.94:0/1566722624 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b1f0867f0 con 0x562b1f07fce0 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.133+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/1566722624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 0x562b1f078a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.134+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/1566722624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08b6a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5e5001cff0 tx=0x7f5e50005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d1b50f8c489b41 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.134+0000 7f5e417fa640 1 -- 172.21.15.94:0/1566722624 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50010440 con 0x562b1f07fce0 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.134+0000 7f5e417fa640 1 -- 172.21.15.94:0/1566722624 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e500054f0 con 0x562b1f07fce0 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.134+0000 7f5e417fa640 1 -- 172.21.15.94:0/1566722624 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5001b420 con 0x562b1f07fce0 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.134+0000 7f5e6463b740 1 -- 172.21.15.94:0/1566722624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 msgr2=0x562b1f08b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.134+0000 7f5e6463b740 1 --2- 172.21.15.94:0/1566722624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08b6a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5e5001cff0 tx=0x7f5e50005cf0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.134+0000 7f5e6463b740 1 -- 172.21.15.94:0/1566722624 shutdown_connections 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.134+0000 7f5e6463b740 1 --2- 172.21.15.94:0/1566722624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f088e40 0x562b1f078a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.134+0000 7f5e6463b740 1 --2- 172.21.15.94:0/1566722624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07fce0 0x562b1f08b6a0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 --2- 172.21.15.94:0/1566722624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f07f330 0x562b1f098cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 -- 172.21.15.94:0/1566722624 >> 172.21.15.94:0/1566722624 conn(0x562b1f079ba0 msgr2=0x562b1f078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 -- 172.21.15.94:0/1566722624 shutdown_connections 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 -- 172.21.15.94:0/1566722624 wait complete. 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 -- start start 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 Processor -- start 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07f330 0x562b1f095580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 0x562b1f08df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f088e40 0x562b1f08e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c1abab0 con 0x562b1f07f330 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c1acf20 con 0x562b1f07fce0 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c1ac180 con 0x562b1f088e40 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e6463b740 10 client.0 _ll_register_callbacks cb 0x7f5e50004240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e5b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07f330 0x562b1f095580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e5b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07f330 0x562b1f095580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36688/0 (socket says 172.21.15.94:36688) 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e5b7fe640 1 -- 172.21.15.94:0/1115174067 learned_addr learned my addr 172.21.15.94:0/1115174067 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.135+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 0x562b1f08df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f088e40 0x562b1f08e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e5b7fe640 1 -- 172.21.15.94:0/1115174067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f088e40 msgr2=0x562b1f08e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f088e40 0x562b1f08e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e5b7fe640 1 -- 172.21.15.94:0/1115174067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 msgr2=0x562b1f08df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 0x562b1f08df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e5b7fe640 1 -- 172.21.15.94:0/1115174067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b1f08ec60 con 0x562b1f07f330 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 0x562b1f08df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e5b7fe640 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07f330 0x562b1f095580 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00f530 tx=0x7f5e500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5f2f1ba8a0b9ea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e3affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e3affd640 1 -- 172.21.15.94:0/1115174067 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c033040 con 0x562b1f07f330 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e3affd640 1 -- 172.21.15.94:0/1115174067 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f5e4c005040 con 0x562b1f07f330 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e6463b740 1 -- 172.21.15.94:0/1115174067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e500054f0 con 0x562b1f07f330 2024-04-21T08:48:09.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e3affd640 1 -- 172.21.15.94:0/1115174067 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e4c040050 con 0x562b1f07f330 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e6463b740 10 client.17727 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e6463b740 1 -- 172.21.15.94:0/1115174067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562b1f1035c0 con 0x562b1f07f330 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e3affd640 1 -- 172.21.15.94:0/1115174067 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e500054f0 con 0x562b1f07f330 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e3affd640 1 -- 172.21.15.94:0/1115174067 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5e4c024040 con 0x562b1f07f330 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e6463b740 10 client.17727 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e6463b740 10 client.17727 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.136+0000 7f5e6463b740 20 client.17727 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562b1f103aa0 0x562b1f105e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 4 client.17727 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 1 -- 172.21.15.94:0/1115174067 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562b1f08ce10 con 0x562b1f103aa0 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562b1f103aa0 0x562b1f105e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e5a7fc640 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562b1f103aa0 0x562b1f105e90 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac5bc624c8ee707 server_cookie=ed54fbbb97657c47 in_seq=0 out_seq=0 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e3affd640 10 client.17727 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e3affd640 1 -- 172.21.15.94:0/1115174067 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562b1f08ce10 con 0x562b1f103aa0 2024-04-21T08:48:09.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e3affd640 10 client.17727 handle_command_reply: tid=0 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 1 client.17727 shutdown 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 1 -- 172.21.15.94:0/1115174067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07f330 msgr2=0x562b1f095580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07f330 0x562b1f095580 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00f530 tx=0x7f5e500042f0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 1 -- 172.21.15.94:0/1115174067 shutdown_connections 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x562b1f103aa0 0x562b1f105e90 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b1f088e40 0x562b1f08e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b1f07fce0 0x562b1f08df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.137+0000 7f5e6463b740 1 --2- 172.21.15.94:0/1115174067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562b1f07f330 0x562b1f095580 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e6463b740 1 -- 172.21.15.94:0/1115174067 >> 172.21.15.94:0/1115174067 conn(0x562b1f079ba0 msgr2=0x562b1f078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e3affd640 0 client.17727 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e6463b740 1 -- 172.21.15.94:0/1115174067 shutdown_connections 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e6463b740 1 -- 172.21.15.94:0/1115174067 wait complete. 2024-04-21T08:48:09.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e6463b740 20 client.17727 trim_cache size 0 max 16384 2024-04-21T08:48:09.145 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:48:09.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 -- 172.21.15.94:0/2779195897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e440a3d20 msgr2=0x7f5e440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e440a3d20 0x7f5e440a61e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5e540efc20 tx=0x7f5e540f8550 comp rx=0 tx=0).stop 2024-04-21T08:48:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 -- 172.21.15.94:0/2779195897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c1015a0 msgr2=0x7f5e5c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c1015a0 0x7f5e5c198750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5e54006010 tx=0x7f5e540346f0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 -- 172.21.15.94:0/2779195897 shutdown_connections 2024-04-21T08:48:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5e440a3d20 0x7f5e440a61e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e5c106150 0x7f5e5c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e5c19d010 0x7f5e5c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 --2- 172.21.15.94:0/2779195897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c1015a0 0x7f5e5c198750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 -- 172.21.15.94:0/2779195897 >> 172.21.15.94:0/2779195897 conn(0x7f5e5c0769d0 msgr2=0x7f5e5c06a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 -- 172.21.15.94:0/2779195897 shutdown_connections 2024-04-21T08:48:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.138+0000 7f5e61a43640 1 -- 172.21.15.94:0/2779195897 wait complete. 2024-04-21T08:48:09.148 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:08 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:09.149 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:48:09.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:08 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:09.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.252+0000 7fe42348e640 1 -- 172.21.15.94:0/3595777240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe41c101ac0 msgr2=0x7fe41c105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.252+0000 7fe42348e640 1 --2- 172.21.15.94:0/3595777240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe41c101ac0 0x7fe41c105b00 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009900 tx=0x7fe40c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:09.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.252+0000 7fe42348e640 1 -- 172.21.15.94:0/3595777240 shutdown_connections 2024-04-21T08:48:09.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.252+0000 7fe42348e640 1 --2- 172.21.15.94:0/3595777240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41c1061e0 0x7fe41c10ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.252+0000 7fe42348e640 1 --2- 172.21.15.94:0/3595777240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe41c101ac0 0x7fe41c105b00 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.252+0000 7fe42348e640 1 --2- 172.21.15.94:0/3595777240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe41c101110 0x7fe41c1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.252+0000 7fe42348e640 1 -- 172.21.15.94:0/3595777240 >> 172.21.15.94:0/3595777240 conn(0x7fe41c077f80 msgr2=0x7fe41c078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.252+0000 7fe42348e640 1 -- 172.21.15.94:0/3595777240 shutdown_connections 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.252+0000 7fe42348e640 1 -- 172.21.15.94:0/3595777240 wait complete. 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42348e640 1 Processor -- start 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42348e640 1 -- start start 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42348e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe41c101110 0x7fe41c199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42348e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe41c1061e0 0x7fe41c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42348e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41c19a150 0x7fe41c1a4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42348e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c111120 con 0x7fe41c101110 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42348e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c110fa0 con 0x7fe41c19a150 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42348e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c1112a0 con 0x7fe41c1061e0 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42122a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe41c101110 0x7fe41c199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42122a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe41c101110 0x7fe41c199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36694/0 (socket says 172.21.15.94:36694) 2024-04-21T08:48:09.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42122a640 1 -- 172.21.15.94:0/940888583 learned_addr learned my addr 172.21.15.94:0/940888583 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe420a29640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe41c1061e0 0x7fe41c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe421a2b640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41c19a150 0x7fe41c1a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42122a640 1 -- 172.21.15.94:0/940888583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe41c1061e0 msgr2=0x7fe41c199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42122a640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe41c1061e0 0x7fe41c199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42122a640 1 -- 172.21.15.94:0/940888583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41c19a150 msgr2=0x7fe41c1a4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42122a640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41c19a150 0x7fe41c1a4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.253+0000 7fe42122a640 1 -- 172.21.15.94:0/940888583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe41c1a4ed0 con 0x7fe41c101110 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.254+0000 7fe42122a640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe41c101110 0x7fe41c199680 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe41800d770 tx=0x7fe41800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.254+0000 7fe4127fc640 1 -- 172.21.15.94:0/940888583 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418017070 con 0x7fe41c101110 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.254+0000 7fe42348e640 1 -- 172.21.15.94:0/940888583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe41c1a51c0 con 0x7fe41c101110 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.254+0000 7fe4127fc640 1 -- 172.21.15.94:0/940888583 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4180054d0 con 0x7fe41c101110 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.254+0000 7fe4127fc640 1 -- 172.21.15.94:0/940888583 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418004c30 con 0x7fe41c101110 2024-04-21T08:48:09.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.254+0000 7fe42348e640 1 -- 172.21.15.94:0/940888583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe41c1a56e0 con 0x7fe41c101110 2024-04-21T08:48:09.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:08 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:09.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42607d740 1 Processor -- start 2024-04-21T08:48:09.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42607d740 1 -- start start 2024-04-21T08:48:09.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe4127fc640 1 -- 172.21.15.94:0/940888583 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fe418004e10 con 0x7fe41c101110 2024-04-21T08:48:09.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42607d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 0x55c8440678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42607d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 0x55c844070430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42607d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c844072e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42607d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c843fcc1f0 con 0x55c844070a00 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42607d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c843f694b0 con 0x55c844067e80 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42607d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c843fc81b0 con 0x55c8440674d0 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42122a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 0x55c844070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe420a29640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c844072e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42122a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 0x55c844070430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:32974/0 (socket says 172.21.15.94:32974) 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42122a640 1 -- 172.21.15.94:0/51654423 learned_addr learned my addr 172.21.15.94:0/51654423 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe420a29640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c844072e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36698/0 (socket says 172.21.15.94:36698) 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe421a2b640 1 --2- 172.21.15.94:0/51654423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 0x55c8440678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42122a640 1 -- 172.21.15.94:0/51654423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 msgr2=0x55c8440678b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42122a640 1 --2- 172.21.15.94:0/51654423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 0x55c8440678b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42122a640 1 -- 172.21.15.94:0/51654423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 msgr2=0x55c844072e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42122a640 1 --2- 172.21.15.94:0/51654423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c844072e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe42122a640 1 -- 172.21.15.94:0/51654423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c843f5b1b0 con 0x55c844067e80 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.257+0000 7fe4127fc640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe4080a3da0 0x7fe4080a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.258+0000 7fe42122a640 1 --2- 172.21.15.94:0/51654423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 0x55c844070430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe4180cd890 tx=0x7fe4180d8690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1064326d02c4acf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.258+0000 7fe421a2b640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe4080a3da0 0x7fe4080a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.258+0000 7fe4127fc640 1 -- 172.21.15.94:0/940888583 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7fe4180d0660 con 0x7fe41c101110 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.258+0000 7fe4077fe640 1 -- 172.21.15.94:0/51654423 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4180984c0 con 0x55c844067e80 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.258+0000 7fe4077fe640 1 -- 172.21.15.94:0/51654423 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe418005680 con 0x55c844067e80 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.258+0000 7fe4077fe640 1 -- 172.21.15.94:0/51654423 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41809bb80 con 0x55c844067e80 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 -- 172.21.15.94:0/51654423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 msgr2=0x55c844070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe421a2b640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe4080a3da0 0x7fe4080a6260 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe414005fd0 tx=0x7fe414009500 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 --2- 172.21.15.94:0/51654423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 0x55c844070430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe4180cd890 tx=0x7fe4180d8690 comp rx=0 tx=0).stop 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 -- 172.21.15.94:0/51654423 shutdown_connections 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 --2- 172.21.15.94:0/51654423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c844072e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 --2- 172.21.15.94:0/51654423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 0x55c844070430 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 --2- 172.21.15.94:0/51654423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 0x55c8440678b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 -- 172.21.15.94:0/51654423 >> 172.21.15.94:0/51654423 conn(0x55c844061ba0 msgr2=0x55c844061f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 -- 172.21.15.94:0/51654423 shutdown_connections 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 -- 172.21.15.94:0/51654423 wait complete. 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 -- start start 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.259+0000 7fe42607d740 1 Processor -- start 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42607d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 0x55c844070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42607d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 0x55c84406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42607d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42607d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c844073970 con 0x55c844070a00 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42607d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c843f85500 con 0x55c844067e80 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42607d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c844073af0 con 0x55c8440674d0 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42607d740 10 client.0 _ll_register_callbacks cb 0x55c843f53b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe421a2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe421a2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36700/0 (socket says 172.21.15.94:36700) 2024-04-21T08:48:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42122a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 0x55c844070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe421a2b640 1 -- 172.21.15.94:0/907318058 learned_addr learned my addr 172.21.15.94:0/907318058 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe420a29640 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 0x55c84406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42122a640 1 -- 172.21.15.94:0/907318058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 msgr2=0x55c84406d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42122a640 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 0x55c84406d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42122a640 1 -- 172.21.15.94:0/907318058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 msgr2=0x55c84406db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42122a640 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42122a640 1 -- 172.21.15.94:0/907318058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c843cde3f0 con 0x55c8440674d0 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe421a2b640 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42122a640 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 0x55c844070260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe418009e30 tx=0x7fe4180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82e508ba89ae4738 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe404ff9640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe404ff9640 1 -- 172.21.15.94:0/907318058 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418020040 con 0x55c8440674d0 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe404ff9640 1 -- 172.21.15.94:0/907318058 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7fe41809a520 con 0x55c8440674d0 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.260+0000 7fe42607d740 1 -- 172.21.15.94:0/907318058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe418005680 con 0x55c8440674d0 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe404ff9640 1 -- 172.21.15.94:0/907318058 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe4180ba050 con 0x55c8440674d0 2024-04-21T08:48:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe42607d740 10 client.26524 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe42607d740 1 -- 172.21.15.94:0/907318058 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c843da87d0 con 0x55c8440674d0 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe404ff9640 1 -- 172.21.15.94:0/907318058 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe418005680 con 0x55c8440674d0 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe404ff9640 1 -- 172.21.15.94:0/907318058 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe4180cdba0 con 0x55c8440674d0 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe42607d740 10 client.26524 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe42607d740 10 client.26524 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe42607d740 20 client.26524 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe42607d740 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c844080a30 0x55c844082e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe42607d740 4 client.26524 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe42607d740 1 -- 172.21.15.94:0/907318058 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c843f6b930 con 0x55c844080a30 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.261+0000 7fe420a29640 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c844080a30 0x55c844082e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.262+0000 7fe420a29640 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c844080a30 0x55c844082e20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bdaaacb929bba9dd server_cookie=e10d26624874f74f in_seq=0 out_seq=0 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.262+0000 7fe404ff9640 10 client.26524 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.262+0000 7fe404ff9640 1 -- 172.21.15.94:0/907318058 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c843f6b930 con 0x55c844080a30 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.262+0000 7fe404ff9640 10 client.26524 handle_command_reply: tid=0 2024-04-21T08:48:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.262+0000 7fe42607d740 1 client.26524 shutdown 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 1 -- 172.21.15.94:0/907318058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 msgr2=0x55c844070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 0x55c844070260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe418009e30 tx=0x7fe4180046c0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 1 -- 172.21.15.94:0/907318058 shutdown_connections 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c844080a30 0x55c844082e20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe404ff9640 0 client.26524 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c844067e80 0x55c84406d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 1 --2- 172.21.15.94:0/907318058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c8440674d0 0x55c844070260 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 1 -- 172.21.15.94:0/907318058 >> 172.21.15.94:0/907318058 conn(0x55c844061ba0 msgr2=0x55c8440638f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 1 -- 172.21.15.94:0/907318058 shutdown_connections 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 1 -- 172.21.15.94:0/907318058 wait complete. 2024-04-21T08:48:09.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.263+0000 7fe42607d740 20 client.26524 trim_cache size 0 max 16384 2024-04-21T08:48:09.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.289+0000 7fe42607d740 1 Processor -- start 2024-04-21T08:48:09.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.289+0000 7fe42607d740 1 -- start start 2024-04-21T08:48:09.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe42607d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8440674d0 0x55c84406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe42607d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c844067e80 0x55c844070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe42607d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe42607d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c1b1f90 con 0x55c844070a00 2024-04-21T08:48:09.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe42607d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c1af860 con 0x55c8440674d0 2024-04-21T08:48:09.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe42607d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c1ad6d0 con 0x55c844067e80 2024-04-21T08:48:09.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe421a2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe421a2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36716/0 (socket says 172.21.15.94:36716) 2024-04-21T08:48:09.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe421a2b640 1 -- 172.21.15.94:0/2131181299 learned_addr learned my addr 172.21.15.94:0/2131181299 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe42122a640 1 --2- 172.21.15.94:0/2131181299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8440674d0 0x55c84406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe420a29640 1 --2- 172.21.15.94:0/2131181299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c844067e80 0x55c844070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe421a2b640 1 -- 172.21.15.94:0/2131181299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c844067e80 msgr2=0x55c844070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe421a2b640 1 --2- 172.21.15.94:0/2131181299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c844067e80 0x55c844070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe421a2b640 1 -- 172.21.15.94:0/2131181299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8440674d0 msgr2=0x55c84406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe421a2b640 1 --2- 172.21.15.94:0/2131181299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8440674d0 0x55c84406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe421a2b640 1 -- 172.21.15.94:0/2131181299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c84405c070 con 0x55c844070a00 2024-04-21T08:48:09.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.290+0000 7fe421a2b640 1 --2- 172.21.15.94:0/2131181299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406a890 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fe414011dd0 tx=0x7fe41401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94638e87d6e37f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.291+0000 7fe406ffd640 1 -- 172.21.15.94:0/2131181299 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4140047f0 con 0x55c844070a00 2024-04-21T08:48:09.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.291+0000 7fe406ffd640 1 -- 172.21.15.94:0/2131181299 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe41400fe30 con 0x55c844070a00 2024-04-21T08:48:09.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.291+0000 7fe406ffd640 1 -- 172.21.15.94:0/2131181299 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41401e430 con 0x55c844070a00 2024-04-21T08:48:09.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 -- 172.21.15.94:0/2131181299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 msgr2=0x55c84406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 --2- 172.21.15.94:0/2131181299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406a890 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fe414011dd0 tx=0x7fe41401a000 comp rx=0 tx=0).stop 2024-04-21T08:48:09.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 -- 172.21.15.94:0/2131181299 shutdown_connections 2024-04-21T08:48:09.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 --2- 172.21.15.94:0/2131181299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c844067e80 0x55c844070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 --2- 172.21.15.94:0/2131181299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c84406a890 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 --2- 172.21.15.94:0/2131181299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8440674d0 0x55c84406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 -- 172.21.15.94:0/2131181299 >> 172.21.15.94:0/2131181299 conn(0x55c844061ba0 msgr2=0x55c844060c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 -- 172.21.15.94:0/2131181299 shutdown_connections 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 -- 172.21.15.94:0/2131181299 wait complete. 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 -- start start 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.292+0000 7fe42607d740 1 Processor -- start 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe42607d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8440674d0 0x55c844083440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe42607d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c844067e80 0x55c844083980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe42607d740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c8440debd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe42607d740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c1add20 con 0x55c844070a00 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe42607d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c1af190 con 0x55c8440674d0 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe42607d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c1ae3f0 con 0x55c844067e80 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe42607d740 10 client.0 _ll_register_callbacks cb 0x7fe414009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe421a2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c8440debd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe421a2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c8440debd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36726/0 (socket says 172.21.15.94:36726) 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe421a2b640 1 -- 172.21.15.94:0/1140037282 learned_addr learned my addr 172.21.15.94:0/1140037282 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe420a29640 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c844067e80 0x55c844083980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe42122a640 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8440674d0 0x55c844083440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe421a2b640 1 -- 172.21.15.94:0/1140037282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c844067e80 msgr2=0x55c844083980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe421a2b640 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c844067e80 0x55c844083980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe421a2b640 1 -- 172.21.15.94:0/1140037282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8440674d0 msgr2=0x55c844083440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe421a2b640 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8440674d0 0x55c844083440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe421a2b640 1 -- 172.21.15.94:0/1140037282 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8440df240 con 0x55c844070a00 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe421a2b640 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c8440debd0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe414029ad0 tx=0x7fe4140047f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a772262ad08980bb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe404ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe404ff9640 1 -- 172.21.15.94:0/1140037282 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414029480 con 0x55c844070a00 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe404ff9640 1 -- 172.21.15.94:0/1140037282 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7fe414029620 con 0x55c844070a00 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.293+0000 7fe42607d740 1 -- 172.21.15.94:0/1140037282 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe41400fe30 con 0x55c844070a00 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe404ff9640 1 -- 172.21.15.94:0/1140037282 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe414029820 con 0x55c844070a00 2024-04-21T08:48:09.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe42607d740 10 client.17745 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe42607d740 1 -- 172.21.15.94:0/1140037282 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c8440e6700 con 0x55c844070a00 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe404ff9640 1 -- 172.21.15.94:0/1140037282 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe41400fe30 con 0x55c844070a00 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe404ff9640 1 -- 172.21.15.94:0/1140037282 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe414030020 con 0x55c844070a00 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe42607d740 10 client.17745 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe42607d740 10 client.17745 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe42607d740 20 client.17745 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe42607d740 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c8440e6d70 0x55c8440e9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe42607d740 4 client.17745 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe42607d740 1 -- 172.21.15.94:0/1140037282 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c843fd9c10 con 0x55c8440e6d70 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe42122a640 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c8440e6d70 0x55c8440e9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe404ff9640 10 client.17745 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.294+0000 7fe42122a640 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c8440e6d70 0x55c8440e9190 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c7d0bcb712eb8f59 server_cookie=7fc653df6c447c4 in_seq=0 out_seq=0 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe404ff9640 1 -- 172.21.15.94:0/1140037282 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c843fd9c10 con 0x55c8440e6d70 2024-04-21T08:48:09.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe404ff9640 10 client.17745 handle_command_reply: tid=0 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 client.17745 shutdown 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 -- 172.21.15.94:0/1140037282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 msgr2=0x55c8440debd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c8440debd0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe414029ad0 tx=0x7fe4140047f0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 -- 172.21.15.94:0/1140037282 shutdown_connections 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c844067e80 0x55c844083980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c8440e6d70 0x55c8440e9190 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c844070a00 0x55c8440debd0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe404ff9640 0 client.17745 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 --2- 172.21.15.94:0/1140037282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8440674d0 0x55c844083440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 -- 172.21.15.94:0/1140037282 >> 172.21.15.94:0/1140037282 conn(0x55c844061ba0 msgr2=0x55c84407bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 -- 172.21.15.94:0/1140037282 shutdown_connections 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 1 -- 172.21.15.94:0/1140037282 wait complete. 2024-04-21T08:48:09.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.295+0000 7fe42607d740 20 client.17745 trim_cache size 0 max 16384 2024-04-21T08:48:09.302 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:48:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 -- 172.21.15.94:0/940888583 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe4080a3da0 msgr2=0x7fe4080a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe4080a3da0 0x7fe4080a6260 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe414005fd0 tx=0x7fe414009500 comp rx=0 tx=0).stop 2024-04-21T08:48:09.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 -- 172.21.15.94:0/940888583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe41c101110 msgr2=0x7fe41c199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe41c101110 0x7fe41c199680 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe41800d770 tx=0x7fe41800dc30 comp rx=0 tx=0).stop 2024-04-21T08:48:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 -- 172.21.15.94:0/940888583 shutdown_connections 2024-04-21T08:48:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe4080a3da0 0x7fe4080a6260 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe41c1061e0 0x7fe41c199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41c19a150 0x7fe41c1a4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 --2- 172.21.15.94:0/940888583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe41c101110 0x7fe41c199680 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 -- 172.21.15.94:0/940888583 >> 172.21.15.94:0/940888583 conn(0x7fe41c077f80 msgr2=0x7fe41c0ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 -- 172.21.15.94:0/940888583 shutdown_connections 2024-04-21T08:48:09.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.296+0000 7fe42348e640 1 -- 172.21.15.94:0/940888583 wait complete. 2024-04-21T08:48:09.310 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:48:09.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.413+0000 7f4461fa4640 1 -- 172.21.15.94:0/382304394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f445c106160 msgr2=0x7f445c10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.413+0000 7f4461fa4640 1 --2- 172.21.15.94:0/382304394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f445c106160 0x7f445c10dd30 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f4454009ca0 tx=0x7f445402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.413+0000 7f4461fa4640 1 -- 172.21.15.94:0/382304394 shutdown_connections 2024-04-21T08:48:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.413+0000 7f4461fa4640 1 --2- 172.21.15.94:0/382304394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f445c106160 0x7f445c10dd30 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.413+0000 7f4461fa4640 1 --2- 172.21.15.94:0/382304394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445c101a40 0x7f445c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.413+0000 7f4461fa4640 1 --2- 172.21.15.94:0/382304394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f445c101090 0x7f445c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.413+0000 7f4461fa4640 1 -- 172.21.15.94:0/382304394 >> 172.21.15.94:0/382304394 conn(0x7f445c077f50 msgr2=0x7f445c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.413+0000 7f4461fa4640 1 -- 172.21.15.94:0/382304394 shutdown_connections 2024-04-21T08:48:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.413+0000 7f4461fa4640 1 -- 172.21.15.94:0/382304394 wait complete. 2024-04-21T08:48:09.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f4461fa4640 1 Processor -- start 2024-04-21T08:48:09.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f4461fa4640 1 -- start start 2024-04-21T08:48:09.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f4461fa4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f445c101090 0x7f445c1973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f4461fa4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445c106160 0x7f445c197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f4461fa4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f445c197e60 0x7f445c1a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f4461fa4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f445c1110d0 con 0x7f445c101090 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f4461fa4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f445c110f50 con 0x7f445c106160 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f4461fa4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f445c111250 con 0x7f445c197e60 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f445b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f445c101090 0x7f445c1973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f445b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f445c101090 0x7f445c1973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36740/0 (socket says 172.21.15.94:36740) 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f445b7fe640 1 -- 172.21.15.94:0/3303111089 learned_addr learned my addr 172.21.15.94:0/3303111089 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f445bfff640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f445c197e60 0x7f445c1a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.414+0000 7f445affd640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445c106160 0x7f445c197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f445b7fe640 1 -- 172.21.15.94:0/3303111089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f445c197e60 msgr2=0x7f445c1a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f445b7fe640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f445c197e60 0x7f445c1a6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f445b7fe640 1 -- 172.21.15.94:0/3303111089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445c106160 msgr2=0x7f445c197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f445b7fe640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445c106160 0x7f445c197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f445b7fe640 1 -- 172.21.15.94:0/3303111089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445c1a6ff0 con 0x7f445c101090 2024-04-21T08:48:09.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f445bfff640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f445c197e60 0x7f445c1a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:09.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f445affd640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445c106160 0x7f445c197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:09.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f445b7fe640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f445c101090 0x7f445c1973e0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e9f0 tx=0x7f444c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f4458ff9640 1 -- 172.21.15.94:0/3303111089 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c00cd80 con 0x7f445c101090 2024-04-21T08:48:09.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f4458ff9640 1 -- 172.21.15.94:0/3303111089 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f444c0054d0 con 0x7f445c101090 2024-04-21T08:48:09.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f4461fa4640 1 -- 172.21.15.94:0/3303111089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f445c1a72e0 con 0x7f445c101090 2024-04-21T08:48:09.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f4458ff9640 1 -- 172.21.15.94:0/3303111089 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c010690 con 0x7f445c101090 2024-04-21T08:48:09.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.415+0000 7f4461fa4640 1 -- 172.21.15.94:0/3303111089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f445c1a77b0 con 0x7f445c101090 2024-04-21T08:48:09.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.417+0000 7f4464b96740 1 Processor -- start 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.417+0000 7f4464b96740 1 -- start start 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.417+0000 7f4458ff9640 1 -- 172.21.15.94:0/3303111089 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f444c010830 con 0x7f445c101090 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f4464b96740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3f4d0 0x55fbeff3f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f4464b96740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f4464b96740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 0x55fbeff4ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f4464b96740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55fbefea41f0 con 0x55fbeff3fe80 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f4464b96740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fbefe414b0 con 0x55fbeff48a00 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f4464b96740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fbefea01b0 con 0x55fbeff3f4d0 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36752/0 (socket says 172.21.15.94:36752) 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445b7fe640 1 -- 172.21.15.94:0/891162258 learned_addr learned my addr 172.21.15.94:0/891162258 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445bfff640 1 --2- 172.21.15.94:0/891162258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3f4d0 0x55fbeff3f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445affd640 1 --2- 172.21.15.94:0/891162258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 0x55fbeff4ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445b7fe640 1 -- 172.21.15.94:0/891162258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3f4d0 msgr2=0x55fbeff3f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445b7fe640 1 --2- 172.21.15.94:0/891162258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3f4d0 0x55fbeff3f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445b7fe640 1 -- 172.21.15.94:0/891162258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 msgr2=0x55fbeff4ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445b7fe640 1 --2- 172.21.15.94:0/891162258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 0x55fbeff4ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445b7fe640 1 -- 172.21.15.94:0/891162258 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbefe331b0 con 0x55fbeff3fe80 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445affd640 1 --2- 172.21.15.94:0/891162258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 0x55fbeff4ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f4458ff9640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44440a3d20 0x7f44440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445bfff640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44440a3d20 0x7f44440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.418+0000 7f445b7fe640 1 --2- 172.21.15.94:0/891162258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48430 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f444c0ce3a0 tx=0x7f444c0d9320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e407b5afdc3fc75 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.419+0000 7f4458ff9640 1 -- 172.21.15.94:0/3303111089 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f444c0d1170 con 0x7f445c101090 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.419+0000 7f4441ffb640 1 -- 172.21.15.94:0/891162258 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c09ca50 con 0x55fbeff3fe80 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.419+0000 7f4441ffb640 1 -- 172.21.15.94:0/891162258 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f444c005680 con 0x55fbeff3fe80 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.419+0000 7f4441ffb640 1 -- 172.21.15.94:0/891162258 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c09c5e0 con 0x55fbeff3fe80 2024-04-21T08:48:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 -- 172.21.15.94:0/891162258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 msgr2=0x55fbeff48430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 --2- 172.21.15.94:0/891162258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48430 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f444c0ce3a0 tx=0x7f444c0d9320 comp rx=0 tx=0).stop 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f445bfff640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44440a3d20 0x7f44440a61e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4454004af0 tx=0x7f4454005c50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 -- 172.21.15.94:0/891162258 shutdown_connections 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 --2- 172.21.15.94:0/891162258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 0x55fbeff4ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 --2- 172.21.15.94:0/891162258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48430 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 --2- 172.21.15.94:0/891162258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3f4d0 0x55fbeff3f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 -- 172.21.15.94:0/891162258 >> 172.21.15.94:0/891162258 conn(0x55fbeff39ba0 msgr2=0x55fbeff39f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 -- 172.21.15.94:0/891162258 shutdown_connections 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 -- 172.21.15.94:0/891162258 wait complete. 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 -- start start 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 Processor -- start 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3f4d0 0x55fbeff48290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3fe80 0x55fbeff45620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 0x55fbeff45b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55fbeff4b970 con 0x55fbeff3f4d0 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fbefe5d500 con 0x55fbeff48a00 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.420+0000 7f4464b96740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fbeff4baf0 con 0x55fbeff3fe80 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3f4d0 0x55fbeff48290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3f4d0 0x55fbeff48290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36756/0 (socket says 172.21.15.94:36756) 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f4464b96740 10 client.0 _ll_register_callbacks cb 0x55fbefe2bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445b7fe640 1 -- 172.21.15.94:0/1230363573 learned_addr learned my addr 172.21.15.94:0/1230363573 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445bfff640 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 0x55fbeff45b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445affd640 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3fe80 0x55fbeff45620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445b7fe640 1 -- 172.21.15.94:0/1230363573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3fe80 msgr2=0x55fbeff45620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445b7fe640 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3fe80 0x55fbeff45620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445b7fe640 1 -- 172.21.15.94:0/1230363573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 msgr2=0x55fbeff45b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445b7fe640 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 0x55fbeff45b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445b7fe640 1 -- 172.21.15.94:0/1230363573 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbefbb63f0 con 0x55fbeff3f4d0 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f445b7fe640 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3f4d0 0x55fbeff48290 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097d0 tx=0x7f444c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d36284488450db1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f443affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:09.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f443affd640 1 -- 172.21.15.94:0/1230363573 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c09c7c0 con 0x55fbeff3f4d0 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f443affd640 1 -- 172.21.15.94:0/1230363573 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f444c0bd110 con 0x55fbeff3f4d0 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f4464b96740 1 -- 172.21.15.94:0/1230363573 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f444c005680 con 0x55fbeff3f4d0 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f443affd640 1 -- 172.21.15.94:0/1230363573 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f444c09c5e0 con 0x55fbeff3f4d0 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f4464b96740 10 client.17757 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f4464b96740 1 -- 172.21.15.94:0/1230363573 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fbefc807d0 con 0x55fbeff3f4d0 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.421+0000 7f443affd640 1 -- 172.21.15.94:0/1230363573 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f444c005680 con 0x55fbeff3f4d0 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.422+0000 7f443affd640 1 -- 172.21.15.94:0/1230363573 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f444c09cee0 con 0x55fbeff3f4d0 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.422+0000 7f4464b96740 10 client.17757 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.422+0000 7f4464b96740 10 client.17757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.422+0000 7f4464b96740 20 client.17757 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.422+0000 7f4464b96740 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fbeff589f0 0x55fbeff5ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.422+0000 7f4464b96740 4 client.17757 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.422+0000 7f4464b96740 1 -- 172.21.15.94:0/1230363573 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbefe43930 con 0x55fbeff589f0 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.422+0000 7f445affd640 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fbeff589f0 0x55fbeff5ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.422+0000 7f445affd640 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fbeff589f0 0x55fbeff5ade0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88229fce91cb3460 server_cookie=48d194b64baf6fe6 in_seq=0 out_seq=0 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.422+0000 7f443affd640 10 client.17757 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f443affd640 1 -- 172.21.15.94:0/1230363573 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fbefe43930 con 0x55fbeff589f0 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f443affd640 10 client.17757 handle_command_reply: tid=0 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 client.17757 shutdown 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 -- 172.21.15.94:0/1230363573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3f4d0 msgr2=0x55fbeff48290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3f4d0 0x55fbeff48290 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097d0 tx=0x7f444c0048f0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 -- 172.21.15.94:0/1230363573 shutdown_connections 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fbeff589f0 0x55fbeff5ade0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3fe80 0x55fbeff45620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff48a00 0x55fbeff45b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f443affd640 0 client.17757 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 --2- 172.21.15.94:0/1230363573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3f4d0 0x55fbeff48290 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 -- 172.21.15.94:0/1230363573 >> 172.21.15.94:0/1230363573 conn(0x55fbeff39ba0 msgr2=0x55fbeff3b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 -- 172.21.15.94:0/1230363573 shutdown_connections 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 1 -- 172.21.15.94:0/1230363573 wait complete. 2024-04-21T08:48:09.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.423+0000 7f4464b96740 20 client.17757 trim_cache size 0 max 16384 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.436+0000 7f4464b96740 1 Processor -- start 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.436+0000 7f4464b96740 1 -- start start 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.436+0000 7f4464b96740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 0x55fbeff47d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.436+0000 7f4464b96740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.436+0000 7f4464b96740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff48a00 0x55fbeff5de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.436+0000 7f4464b96740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f445c1b4160 con 0x55fbeff3fe80 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f4464b96740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f445c1b1a30 con 0x55fbeff3f4d0 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f4464b96740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f445c1af8a0 con 0x55fbeff48a00 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36772/0 (socket says 172.21.15.94:36772) 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445affd640 1 -- 172.21.15.94:0/987982828 learned_addr learned my addr 172.21.15.94:0/987982828 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445b7fe640 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 0x55fbeff47d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445bfff640 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff48a00 0x55fbeff5de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445affd640 1 -- 172.21.15.94:0/987982828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff48a00 msgr2=0x55fbeff5de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445affd640 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff48a00 0x55fbeff5de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445affd640 1 -- 172.21.15.94:0/987982828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 msgr2=0x55fbeff47d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445affd640 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 0x55fbeff47d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445affd640 1 -- 172.21.15.94:0/987982828 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbeff45ec0 con 0x55fbeff3fe80 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445bfff640 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff48a00 0x55fbeff5de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445b7fe640 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 0x55fbeff47d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f445affd640 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48290 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f445000b370 tx=0x7f4450014900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be08d54173b8696 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f44417fa640 1 -- 172.21.15.94:0/987982828 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4450004030 con 0x55fbeff3fe80 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f44417fa640 1 -- 172.21.15.94:0/987982828 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44500041d0 con 0x55fbeff3fe80 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.437+0000 7f44417fa640 1 -- 172.21.15.94:0/987982828 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445001e2b0 con 0x55fbeff3fe80 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 -- 172.21.15.94:0/987982828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 msgr2=0x55fbeff48290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48290 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f445000b370 tx=0x7f4450014900 comp rx=0 tx=0).stop 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 -- 172.21.15.94:0/987982828 shutdown_connections 2024-04-21T08:48:09.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff48a00 0x55fbeff5de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff3fe80 0x55fbeff48290 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 --2- 172.21.15.94:0/987982828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 0x55fbeff47d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 -- 172.21.15.94:0/987982828 >> 172.21.15.94:0/987982828 conn(0x55fbeff39ba0 msgr2=0x55fbeff38c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 -- 172.21.15.94:0/987982828 shutdown_connections 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 -- 172.21.15.94:0/987982828 wait complete. 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 -- start start 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 Processor -- start 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 0x55fbeff5b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3fe80 0x55fbeff5beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff48a00 0x55fbeff5c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f445c1afef0 con 0x55fbeff48a00 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f445c1b1360 con 0x55fbeff3f4d0 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.438+0000 7f4464b96740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f445c1b05c0 con 0x55fbeff3fe80 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f4464b96740 10 client.0 _ll_register_callbacks cb 0x7f444c098fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:09.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff48a00 0x55fbeff5c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445b7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 0x55fbeff5b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff48a00 0x55fbeff5c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36788/0 (socket says 172.21.15.94:36788) 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445b7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 0x55fbeff5b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.94:33058/0 (socket says 172.21.15.94:33058) 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445bfff640 1 -- 172.21.15.94:0/1112497204 learned_addr learned my addr 172.21.15.94:0/1112497204 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445affd640 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3fe80 0x55fbeff5beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445bfff640 1 -- 172.21.15.94:0/1112497204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3fe80 msgr2=0x55fbeff5beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445bfff640 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3fe80 0x55fbeff5beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445bfff640 1 -- 172.21.15.94:0/1112497204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 msgr2=0x55fbeff5b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445bfff640 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 0x55fbeff5b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445bfff640 1 -- 172.21.15.94:0/1112497204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbeffbdca0 con 0x55fbeff48a00 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f445bfff640 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff48a00 0x55fbeff5c3f0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f4454036000 tx=0x7f44540095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6e429f6ff93a25 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f443affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f443affd640 1 -- 172.21.15.94:0/1112497204 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4454039040 con 0x55fbeff48a00 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f4464b96740 1 -- 172.21.15.94:0/1112497204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44500041d0 con 0x55fbeff48a00 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f443affd640 1 -- 172.21.15.94:0/1112497204 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f44540040f0 con 0x55fbeff48a00 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f443affd640 1 -- 172.21.15.94:0/1112497204 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f44540042f0 con 0x55fbeff48a00 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f4464b96740 10 client.17763 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.439+0000 7f4464b96740 1 -- 172.21.15.94:0/1112497204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fbeffbef10 con 0x55fbeff48a00 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f443affd640 1 -- 172.21.15.94:0/1112497204 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44500041d0 con 0x55fbeff48a00 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f443affd640 1 -- 172.21.15.94:0/1112497204 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4454034cd0 con 0x55fbeff48a00 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f4464b96740 10 client.17763 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f4464b96740 10 client.17763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f4464b96740 20 client.17763 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f4464b96740 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fbeffbf690 0x55fbeffc1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f4464b96740 4 client.17763 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f4464b96740 1 -- 172.21.15.94:0/1112497204 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fbefeb1c10 con 0x55fbeffbf690 2024-04-21T08:48:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f445b7fe640 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fbeffbf690 0x55fbeffc1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f445b7fe640 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fbeffbf690 0x55fbeffc1ab0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db6b906477fe9c1c server_cookie=90205d75b93083d8 in_seq=0 out_seq=0 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.440+0000 7f443affd640 10 client.17763 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f443affd640 1 -- 172.21.15.94:0/1112497204 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fbefeb1c10 con 0x55fbeffbf690 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f443affd640 10 client.17763 handle_command_reply: tid=0 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 client.17763 shutdown 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 -- 172.21.15.94:0/1112497204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff48a00 msgr2=0x55fbeff5c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff48a00 0x55fbeff5c3f0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f4454036000 tx=0x7f44540095c0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 -- 172.21.15.94:0/1112497204 shutdown_connections 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fbeffbf690 0x55fbeffc1ab0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbeff3fe80 0x55fbeff5beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fbeff48a00 0x55fbeff5c3f0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f443affd640 0 client.17763 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 --2- 172.21.15.94:0/1112497204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fbeff3f4d0 0x55fbeff5b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 -- 172.21.15.94:0/1112497204 >> 172.21.15.94:0/1112497204 conn(0x55fbeff39ba0 msgr2=0x55fbeff44e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 -- 172.21.15.94:0/1112497204 shutdown_connections 2024-04-21T08:48:09.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 1 -- 172.21.15.94:0/1112497204 wait complete. 2024-04-21T08:48:09.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4464b96740 20 client.17763 trim_cache size 0 max 16384 2024-04-21T08:48:09.448 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:48:09.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4461fa4640 1 -- 172.21.15.94:0/3303111089 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44440a3d20 msgr2=0x7f44440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4461fa4640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44440a3d20 0x7f44440a61e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4454004af0 tx=0x7f4454005c50 comp rx=0 tx=0).stop 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4461fa4640 1 -- 172.21.15.94:0/3303111089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f445c101090 msgr2=0x7f445c1973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4461fa4640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f445c101090 0x7f445c1973e0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e9f0 tx=0x7f444c00edf0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4461fa4640 1 -- 172.21.15.94:0/3303111089 shutdown_connections 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4461fa4640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f44440a3d20 0x7f44440a61e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4461fa4640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f445c197e60 0x7f445c1a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4461fa4640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445c106160 0x7f445c197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4461fa4640 1 --2- 172.21.15.94:0/3303111089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f445c101090 0x7f445c1973e0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.441+0000 7f4461fa4640 1 -- 172.21.15.94:0/3303111089 >> 172.21.15.94:0/3303111089 conn(0x7f445c077f50 msgr2=0x7f445c0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.442+0000 7f4461fa4640 1 -- 172.21.15.94:0/3303111089 shutdown_connections 2024-04-21T08:48:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:09.442+0000 7f4461fa4640 1 -- 172.21.15.94:0/3303111089 wait complete. 2024-04-21T08:48:10.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:09 smithi111 ceph-mon[31493]: pgmap v251: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:48:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:09 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/410204714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:09 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:09 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:10.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:09 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:48:10.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:09 smithi094 ceph-mon[27988]: pgmap v251: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:48:10.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:09 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/410204714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:10.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:09 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:10.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:09 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:10.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:09 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:48:10.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:09 smithi173 ceph-mon[31428]: pgmap v251: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:48:10.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:09 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/410204714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:10.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:09 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:10.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:09 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:10.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:09 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:48:10.453 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:48:10.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:10.453 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:48:10.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.561+0000 7f7539985640 1 -- 172.21.15.94:0/716389513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7534100d20 msgr2=0x7f7534101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.561+0000 7f7539985640 1 --2- 172.21.15.94:0/716389513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7534100d20 0x7f7534101100 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f7524009900 tx=0x7f752402bc60 comp rx=0 tx=0).stop 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.561+0000 7f7539985640 1 -- 172.21.15.94:0/716389513 shutdown_connections 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.561+0000 7f7539985640 1 --2- 172.21.15.94:0/716389513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f753410c4f0 0x7f753410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.561+0000 7f7539985640 1 --2- 172.21.15.94:0/716389513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75341016d0 0x7f753410be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.561+0000 7f7539985640 1 --2- 172.21.15.94:0/716389513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7534100d20 0x7f7534101100 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.561+0000 7f7539985640 1 -- 172.21.15.94:0/716389513 >> 172.21.15.94:0/716389513 conn(0x7f7534076150 msgr2=0x7f7534076560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.561+0000 7f7539985640 1 -- 172.21.15.94:0/716389513 shutdown_connections 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.561+0000 7f7539985640 1 -- 172.21.15.94:0/716389513 wait complete. 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f7539985640 1 Processor -- start 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f7539985640 1 -- start start 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f7539985640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7534100d20 0x7f753419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f7539985640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f753410c4f0 0x7f753419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f7539985640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7534198600 0x7f7534198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f7539985640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7534112220 con 0x7f7534198600 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f7539985640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75341120a0 con 0x7f753410c4f0 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f7539985640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f75341123a0 con 0x7f7534100d20 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f75337fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7534198600 0x7f7534198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f75337fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7534198600 0x7f7534198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36808/0 (socket says 172.21.15.94:36808) 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f75337fe640 1 -- 172.21.15.94:0/593343945 learned_addr learned my addr 172.21.15.94:0/593343945 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f752bfff640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f753410c4f0 0x7f753419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f75337fe640 1 -- 172.21.15.94:0/593343945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7534100d20 msgr2=0x7f753419e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f7532ffd640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7534100d20 0x7f753419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f75337fe640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7534100d20 0x7f753419e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f75337fe640 1 -- 172.21.15.94:0/593343945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f753410c4f0 msgr2=0x7f753419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f75337fe640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f753410c4f0 0x7f753419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f75337fe640 1 -- 172.21.15.94:0/593343945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7534199120 con 0x7f7534198600 2024-04-21T08:48:10.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f752bfff640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f753410c4f0 0x7f753419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:10.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.562+0000 7f75337fe640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7534198600 0x7f7534198ab0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f752c00cce0 tx=0x7f752c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.563+0000 7f7538983640 1 -- 172.21.15.94:0/593343945 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c00f450 con 0x7f7534198600 2024-04-21T08:48:10.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.563+0000 7f7539985640 1 -- 172.21.15.94:0/593343945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7534199410 con 0x7f7534198600 2024-04-21T08:48:10.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.563+0000 7f7538983640 1 -- 172.21.15.94:0/593343945 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f752c00faa0 con 0x7f7534198600 2024-04-21T08:48:10.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.563+0000 7f7538983640 1 -- 172.21.15.94:0/593343945 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c018b70 con 0x7f7534198600 2024-04-21T08:48:10.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.563+0000 7f7539985640 1 -- 172.21.15.94:0/593343945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75341a55c0 con 0x7f7534198600 2024-04-21T08:48:10.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f7538983640 1 -- 172.21.15.94:0/593343945 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f752c018d10 con 0x7f7534198600 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f753c572740 1 Processor -- start 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f753c572740 1 -- start start 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f753c572740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd07c0 0x55d7f3dd0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f753c572740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd1170 0x55d7f3dd9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f753c572740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dda6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f753c572740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d7f3d34f00 con 0x55d7f3dda270 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f753c572740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d7f3cd2590 con 0x55d7f3dd1170 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f753c572740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d7f3d30ec0 con 0x55d7f3dd07c0 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f752bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dda6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f752bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dda6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36822/0 (socket says 172.21.15.94:36822) 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f752bfff640 1 -- 172.21.15.94:0/999473269 learned_addr learned my addr 172.21.15.94:0/999473269 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f7532ffd640 1 --2- 172.21.15.94:0/999473269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd1170 0x55d7f3dd9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f7538983640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f75200a3d20 0x7f75200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f752bfff640 1 --2- 172.21.15.94:0/999473269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd07c0 0x55d7f3dd0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.565+0000 7f752bfff640 1 -- 172.21.15.94:0/999473269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd07c0 msgr2=0x55d7f3dd0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f752bfff640 1 --2- 172.21.15.94:0/999473269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd07c0 0x55d7f3dd0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f752bfff640 1 -- 172.21.15.94:0/999473269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd1170 msgr2=0x55d7f3dd9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f752bfff640 1 --2- 172.21.15.94:0/999473269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd1170 0x55d7f3dd9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f752bfff640 1 -- 172.21.15.94:0/999473269 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f3b0f7e0 con 0x55d7f3dda270 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f75337fe640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f75200a3d20 0x7f75200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f7538983640 1 -- 172.21.15.94:0/593343945 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f752c0d1190 con 0x7f7534198600 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f752bfff640 1 --2- 172.21.15.94:0/999473269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dda6d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f751c007b60 tx=0x7f751c007380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c96f2a9e1d17ee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f7529ffb640 1 -- 172.21.15.94:0/999473269 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c01a070 con 0x55d7f3dda270 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f7529ffb640 1 -- 172.21.15.94:0/999473269 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f751c010040 con 0x55d7f3dda270 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f7529ffb640 1 -- 172.21.15.94:0/999473269 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c014610 con 0x55d7f3dda270 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f753c572740 1 -- 172.21.15.94:0/999473269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 msgr2=0x55d7f3dda6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f753c572740 1 --2- 172.21.15.94:0/999473269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dda6d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f751c007b60 tx=0x7f751c007380 comp rx=0 tx=0).stop 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.566+0000 7f75337fe640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f75200a3d20 0x7f75200a61e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f752c0ce500 tx=0x7f752c092130 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 -- 172.21.15.94:0/999473269 shutdown_connections 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 --2- 172.21.15.94:0/999473269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dda6d0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 --2- 172.21.15.94:0/999473269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd1170 0x55d7f3dd9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 --2- 172.21.15.94:0/999473269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd07c0 0x55d7f3dd0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 -- 172.21.15.94:0/999473269 >> 172.21.15.94:0/999473269 conn(0x55d7f3dcafb0 msgr2=0x55d7f3dcb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 -- 172.21.15.94:0/999473269 shutdown_connections 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 -- 172.21.15.94:0/999473269 wait complete. 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 -- start start 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 Processor -- start 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3ddddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3dde310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dde850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d7f3de11b0 con 0x55d7f3dda270 2024-04-21T08:48:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d7f3cee580 con 0x55d7f3dd07c0 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d7f3de1330 con 0x55d7f3dd1170 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f753c572740 10 client.0 _ll_register_callbacks cb 0x55d7f3cf4650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f752bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dde850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f752bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dde850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36828/0 (socket says 172.21.15.94:36828) 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.567+0000 7f752bfff640 1 -- 172.21.15.94:0/1487052538 learned_addr learned my addr 172.21.15.94:0/1487052538 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f7532ffd640 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3dde310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f752bfff640 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3ddddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f752bfff640 1 -- 172.21.15.94:0/1487052538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 msgr2=0x55d7f3dde310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f752bfff640 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3dde310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f752bfff640 1 -- 172.21.15.94:0/1487052538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 msgr2=0x55d7f3ddddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f752bfff640 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3ddddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f752bfff640 1 -- 172.21.15.94:0/1487052538 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f3b6ff70 con 0x55d7f3dda270 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f752bfff640 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dde850 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f751c00f9e0 tx=0x7f751c015d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96ee2fd3837cf1e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f7512ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f7512ffd640 1 -- 172.21.15.94:0/1487052538 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c01a040 con 0x55d7f3dda270 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f7512ffd640 1 -- 172.21.15.94:0/1487052538 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f751c022410 con 0x55d7f3dda270 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f753c572740 1 -- 172.21.15.94:0/1487052538 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f751c010040 con 0x55d7f3dda270 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f7512ffd640 1 -- 172.21.15.94:0/1487052538 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f751c022610 con 0x55d7f3dda270 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f753c572740 10 client.17775 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f753c572740 1 -- 172.21.15.94:0/1487052538 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d7f3a07170 con 0x55d7f3dda270 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f7512ffd640 1 -- 172.21.15.94:0/1487052538 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f751c010040 con 0x55d7f3dda270 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.568+0000 7f7512ffd640 1 -- 172.21.15.94:0/1487052538 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f751c020040 con 0x55d7f3dda270 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.569+0000 7f753c572740 10 client.17775 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.569+0000 7f753c572740 10 client.17775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.569+0000 7f753c572740 20 client.17775 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.569+0000 7f753c572740 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d7f3dee610 0x55d7f3df0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.569+0000 7f753c572740 4 client.17775 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.569+0000 7f753c572740 1 -- 172.21.15.94:0/1487052538 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7f3d42920 con 0x55d7f3dee610 2024-04-21T08:48:10.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.569+0000 7f7532ffd640 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d7f3dee610 0x55d7f3df0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.569+0000 7f7512ffd640 10 client.17775 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.569+0000 7f7532ffd640 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d7f3dee610 0x55d7f3df0a00 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c7d3a72911c7f97 server_cookie=8a29e91b6269511 in_seq=0 out_seq=0 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f7512ffd640 1 -- 172.21.15.94:0/1487052538 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7f3d42920 con 0x55d7f3dee610 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f7512ffd640 10 client.17775 handle_command_reply: tid=0 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f753c572740 1 client.17775 shutdown 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f753c572740 1 -- 172.21.15.94:0/1487052538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 msgr2=0x55d7f3dde850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f753c572740 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dde850 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f751c00f9e0 tx=0x7f751c015d40 comp rx=0 tx=0).stop 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f753c572740 1 -- 172.21.15.94:0/1487052538 shutdown_connections 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f753c572740 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d7f3dee610 0x55d7f3df0a00 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f753c572740 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3dde310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f753c572740 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3dde850 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f753c572740 1 --2- 172.21.15.94:0/1487052538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3ddddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f7512ffd640 0 client.17775 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f753c572740 1 -- 172.21.15.94:0/1487052538 >> 172.21.15.94:0/1487052538 conn(0x55d7f3dcafb0 msgr2=0x55d7f3dccd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.570+0000 7f753c572740 1 -- 172.21.15.94:0/1487052538 shutdown_connections 2024-04-21T08:48:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.571+0000 7f753c572740 1 -- 172.21.15.94:0/1487052538 wait complete. 2024-04-21T08:48:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.571+0000 7f753c572740 20 client.17775 trim_cache size 0 max 16384 2024-04-21T08:48:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f753c572740 1 Processor -- start 2024-04-21T08:48:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f753c572740 1 -- start start 2024-04-21T08:48:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f753c572740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3dea0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f753c572740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3ddb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f753c572740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3ddd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f753c572740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f75341afcf0 con 0x55d7f3dda270 2024-04-21T08:48:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f753c572740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75341ad5c0 con 0x55d7f3dd07c0 2024-04-21T08:48:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f753c572740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f75341ab430 con 0x55d7f3dd1170 2024-04-21T08:48:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f752bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3ddd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f752bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3ddd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36834/0 (socket says 172.21.15.94:36834) 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f752bfff640 1 -- 172.21.15.94:0/211040674 learned_addr learned my addr 172.21.15.94:0/211040674 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.588+0000 7f752bfff640 1 --2- 172.21.15.94:0/211040674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3dea0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.589+0000 7f7532ffd640 1 --2- 172.21.15.94:0/211040674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3ddb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.589+0000 7f752bfff640 1 -- 172.21.15.94:0/211040674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 msgr2=0x55d7f3ddb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.589+0000 7f752bfff640 1 --2- 172.21.15.94:0/211040674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3ddb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.589+0000 7f752bfff640 1 -- 172.21.15.94:0/211040674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 msgr2=0x55d7f3dea0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.589+0000 7f752bfff640 1 --2- 172.21.15.94:0/211040674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3dea0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.589+0000 7f752bfff640 1 -- 172.21.15.94:0/211040674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f3dcc1c0 con 0x55d7f3dda270 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.589+0000 7f752bfff640 1 --2- 172.21.15.94:0/211040674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3ddd350 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f751c004700 tx=0x7f751c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0f8737ecb12ce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.589+0000 7f75297fa640 1 -- 172.21.15.94:0/211040674 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c01a040 con 0x55d7f3dda270 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.589+0000 7f75297fa640 1 -- 172.21.15.94:0/211040674 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f751c029070 con 0x55d7f3dda270 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.589+0000 7f75297fa640 1 -- 172.21.15.94:0/211040674 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c024400 con 0x55d7f3dda270 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 -- 172.21.15.94:0/211040674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 msgr2=0x55d7f3ddd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 --2- 172.21.15.94:0/211040674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3ddd350 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f751c004700 tx=0x7f751c009040 comp rx=0 tx=0).stop 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 -- 172.21.15.94:0/211040674 shutdown_connections 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 --2- 172.21.15.94:0/211040674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3ddb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 --2- 172.21.15.94:0/211040674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3ddd350 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 --2- 172.21.15.94:0/211040674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3dea0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 -- 172.21.15.94:0/211040674 >> 172.21.15.94:0/211040674 conn(0x55d7f3dcafb0 msgr2=0x55d7f3dddd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 -- 172.21.15.94:0/211040674 shutdown_connections 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 -- 172.21.15.94:0/211040674 wait complete. 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 -- start start 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 Processor -- start 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3df5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3df5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3df61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f75341aba80 con 0x55d7f3dda270 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75341acef0 con 0x55d7f3dd07c0 2024-04-21T08:48:10.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f75341ac150 con 0x55d7f3dd1170 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f753c572740 10 client.0 _ll_register_callbacks cb 0x7f751c018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f752bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3df61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f752bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3df61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36838/0 (socket says 172.21.15.94:36838) 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.590+0000 7f752bfff640 1 -- 172.21.15.94:0/1446500049 learned_addr learned my addr 172.21.15.94:0/1446500049 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f7532ffd640 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3df5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f752bfff640 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3df5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f752bfff640 1 -- 172.21.15.94:0/1446500049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 msgr2=0x55d7f3df5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f752bfff640 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3df5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f752bfff640 1 -- 172.21.15.94:0/1446500049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 msgr2=0x55d7f3df5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f752bfff640 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3df5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f752bfff640 1 -- 172.21.15.94:0/1446500049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f3dcd2b0 con 0x55d7f3dda270 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f752bfff640 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3df61a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f751c006b50 tx=0x7f751c00ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d833ba60f778caf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f7512ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f7512ffd640 1 -- 172.21.15.94:0/1446500049 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c01a040 con 0x55d7f3dda270 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f7512ffd640 1 -- 172.21.15.94:0/1446500049 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f751c0072e0 con 0x55d7f3dda270 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f753c572740 1 -- 172.21.15.94:0/1446500049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f751c029070 con 0x55d7f3dda270 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f7512ffd640 1 -- 172.21.15.94:0/1446500049 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f751c024400 con 0x55d7f3dda270 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f753c572740 10 client.17781 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f753c572740 1 -- 172.21.15.94:0/1446500049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d7f3df0f60 con 0x55d7f3dda270 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.591+0000 7f7512ffd640 1 -- 172.21.15.94:0/1446500049 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f751c029070 con 0x55d7f3dda270 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.592+0000 7f7512ffd640 1 -- 172.21.15.94:0/1446500049 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f751c020070 con 0x55d7f3dda270 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.592+0000 7f753c572740 10 client.17781 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.592+0000 7f753c572740 10 client.17781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.592+0000 7f753c572740 20 client.17781 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.592+0000 7f753c572740 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d7f3e54ff0 0x55d7f3e57410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.592+0000 7f753c572740 4 client.17781 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.592+0000 7f753c572740 1 -- 172.21.15.94:0/1446500049 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d7f3d3c790 con 0x55d7f3e54ff0 2024-04-21T08:48:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.592+0000 7f7532ffd640 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d7f3e54ff0 0x55d7f3e57410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.592+0000 7f7512ffd640 10 client.17781 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.592+0000 7f7532ffd640 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d7f3e54ff0 0x55d7f3e57410 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9622731d06df82ed server_cookie=f546dcd0297e11da in_seq=0 out_seq=0 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f7512ffd640 1 -- 172.21.15.94:0/1446500049 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d7f3d3c790 con 0x55d7f3e54ff0 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f7512ffd640 10 client.17781 handle_command_reply: tid=0 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f753c572740 1 client.17781 shutdown 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f753c572740 1 -- 172.21.15.94:0/1446500049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 msgr2=0x55d7f3df61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f753c572740 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3df61a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f751c006b50 tx=0x7f751c00ff50 comp rx=0 tx=0).stop 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f753c572740 1 -- 172.21.15.94:0/1446500049 shutdown_connections 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f753c572740 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d7f3e54ff0 0x55d7f3e57410 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f753c572740 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d7f3dd1170 0x55d7f3df5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f753c572740 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d7f3dda270 0x55d7f3df61a0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f753c572740 1 --2- 172.21.15.94:0/1446500049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7f3dd07c0 0x55d7f3df5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.593+0000 7f753c572740 1 -- 172.21.15.94:0/1446500049 >> 172.21.15.94:0/1446500049 conn(0x55d7f3dcafb0 msgr2=0x55d7f3dde620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7512ffd640 0 client.17781 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:10.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f753c572740 1 -- 172.21.15.94:0/1446500049 shutdown_connections 2024-04-21T08:48:10.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f753c572740 1 -- 172.21.15.94:0/1446500049 wait complete. 2024-04-21T08:48:10.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f753c572740 20 client.17781 trim_cache size 0 max 16384 2024-04-21T08:48:10.600 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:48:10.600 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:48:10.600 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "a46597fd-211b-4d3c-b89f-cfd10e36bdb8", 2024-04-21T08:48:10.600 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:48:10.600 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:48:10.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 -- 172.21.15.94:0/593343945 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f75200a3d20 msgr2=0x7f75200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f75200a3d20 0x7f75200a61e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f752c0ce500 tx=0x7f752c092130 comp rx=0 tx=0).stop 2024-04-21T08:48:10.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 -- 172.21.15.94:0/593343945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7534198600 msgr2=0x7f7534198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7534198600 0x7f7534198ab0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f752c00cce0 tx=0x7f752c00a620 comp rx=0 tx=0).stop 2024-04-21T08:48:10.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 -- 172.21.15.94:0/593343945 shutdown_connections 2024-04-21T08:48:10.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f75200a3d20 0x7f75200a61e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7534198600 0x7f7534198ab0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f753410c4f0 0x7f753419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 --2- 172.21.15.94:0/593343945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7534100d20 0x7f753419e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 -- 172.21.15.94:0/593343945 >> 172.21.15.94:0/593343945 conn(0x7f7534076150 msgr2=0x7f75340fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 -- 172.21.15.94:0/593343945 shutdown_connections 2024-04-21T08:48:10.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.594+0000 7f7539985640 1 -- 172.21.15.94:0/593343945 wait complete. 2024-04-21T08:48:10.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:48:10.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:48:10.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.705+0000 7f796aea5640 1 -- 172.21.15.94:0/2955168258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79641010a0 msgr2=0x7f7964101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.705+0000 7f796aea5640 1 --2- 172.21.15.94:0/2955168258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79641010a0 0x7f7964101500 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f7954009900 tx=0x7f795402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:10.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.705+0000 7f796aea5640 1 -- 172.21.15.94:0/2955168258 shutdown_connections 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.705+0000 7f796aea5640 1 --2- 172.21.15.94:0/2955168258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7964101a40 0x7f796410dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.705+0000 7f796aea5640 1 --2- 172.21.15.94:0/2955168258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79641010a0 0x7f7964101500 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.705+0000 7f796aea5640 1 --2- 172.21.15.94:0/2955168258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7964105dd0 0x7f79641061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.705+0000 7f796aea5640 1 -- 172.21.15.94:0/2955168258 >> 172.21.15.94:0/2955168258 conn(0x7f7964077c40 msgr2=0x7f7964078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.705+0000 7f796aea5640 1 -- 172.21.15.94:0/2955168258 shutdown_connections 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.705+0000 7f796aea5640 1 -- 172.21.15.94:0/2955168258 wait complete. 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f796aea5640 1 Processor -- start 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f796aea5640 1 -- start start 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f796aea5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79641010a0 0x7f7964199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f796aea5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7964101a40 0x7f7964199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f796aea5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7964105dd0 0x7f79641a4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f796aea5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7964111180 con 0x7f7964105dd0 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f796aea5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7964111000 con 0x7f79641010a0 2024-04-21T08:48:10.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f796aea5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7964111300 con 0x7f7964101a40 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f7969442640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7964105dd0 0x7f79641a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f7969442640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7964105dd0 0x7f79641a4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36858/0 (socket says 172.21.15.94:36858) 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f7969442640 1 -- 172.21.15.94:0/871250603 learned_addr learned my addr 172.21.15.94:0/871250603 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f7968c41640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79641010a0 0x7f7964199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f795bfff640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7964101a40 0x7f7964199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f7969442640 1 -- 172.21.15.94:0/871250603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7964101a40 msgr2=0x7f7964199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.706+0000 7f7969442640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7964101a40 0x7f7964199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.707+0000 7f7969442640 1 -- 172.21.15.94:0/871250603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79641010a0 msgr2=0x7f7964199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.707+0000 7f7969442640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79641010a0 0x7f7964199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.707+0000 7f7969442640 1 -- 172.21.15.94:0/871250603 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79641a4c40 con 0x7f7964105dd0 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.707+0000 7f7969442640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7964105dd0 0x7f79641a4700 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f795c00eb10 tx=0x7f795c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.707+0000 7f7959ffb640 1 -- 172.21.15.94:0/871250603 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c004b20 con 0x7f7964105dd0 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.707+0000 7f796aea5640 1 -- 172.21.15.94:0/871250603 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79641a4e10 con 0x7f7964105dd0 2024-04-21T08:48:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.707+0000 7f7959ffb640 1 -- 172.21.15.94:0/871250603 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f795c004cc0 con 0x7f7964105dd0 2024-04-21T08:48:10.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.707+0000 7f7959ffb640 1 -- 172.21.15.94:0/871250603 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c010690 con 0x7f7964105dd0 2024-04-21T08:48:10.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.707+0000 7f796aea5640 1 -- 172.21.15.94:0/871250603 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79641a52d0 con 0x7f7964105dd0 2024-04-21T08:48:10.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f7959ffb640 1 -- 172.21.15.94:0/871250603 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f795c005020 con 0x7f7964105dd0 2024-04-21T08:48:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f796db74740 1 Processor -- start 2024-04-21T08:48:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f796db74740 1 -- start start 2024-04-21T08:48:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f7959ffb640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f794c0a3d20 0x7f794c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f796db74740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937ca960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f796db74740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937caea0 0x557b937d3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f796db74740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 0x557b937d5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f796db74740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x557b9372f1e0 con 0x557b937ca580 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f796db74740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557b936cc390 con 0x557b937caea0 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f796db74740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557b9372b1a0 con 0x557b937d3a80 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f795bfff640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f794c0a3d20 0x7f794c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f7959ffb640 1 -- 172.21.15.94:0/871250603 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f795c0d1280 con 0x7f7964105dd0 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f7968c41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f7968c41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937ca960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36860/0 (socket says 172.21.15.94:36860) 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.710+0000 7f7968c41640 1 -- 172.21.15.94:0/4143812046 learned_addr learned my addr 172.21.15.94:0/4143812046 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f7969442640 1 --2- 172.21.15.94:0/4143812046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 0x557b937d5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f795bfff640 1 --2- 172.21.15.94:0/4143812046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937caea0 0x557b937d3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f7968c41640 1 -- 172.21.15.94:0/4143812046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 msgr2=0x557b937d5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f7968c41640 1 --2- 172.21.15.94:0/4143812046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 0x557b937d5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f7968c41640 1 -- 172.21.15.94:0/4143812046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937caea0 msgr2=0x557b937d3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f7968c41640 1 --2- 172.21.15.94:0/4143812046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937caea0 0x557b937d3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f7968c41640 1 -- 172.21.15.94:0/4143812046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b9350b7d0 con 0x557b937ca580 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f795bfff640 1 --2- 172.21.15.94:0/4143812046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937caea0 0x557b937d3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f795bfff640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f794c0a3d20 0x7f794c0a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7954036040 tx=0x7f79540095f0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f7968c41640 1 --2- 172.21.15.94:0/4143812046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937ca960 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f7960009c70 tx=0x7f7960011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e06124ce920ac1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f794affd640 1 -- 172.21.15.94:0/4143812046 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960004230 con 0x557b937ca580 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f794affd640 1 -- 172.21.15.94:0/4143812046 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79600043d0 con 0x557b937ca580 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.711+0000 7f794affd640 1 -- 172.21.15.94:0/4143812046 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960019700 con 0x557b937ca580 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 -- 172.21.15.94:0/4143812046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 msgr2=0x557b937ca960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 --2- 172.21.15.94:0/4143812046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937ca960 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f7960009c70 tx=0x7f7960011810 comp rx=0 tx=0).stop 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 -- 172.21.15.94:0/4143812046 shutdown_connections 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 --2- 172.21.15.94:0/4143812046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 0x557b937d5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 --2- 172.21.15.94:0/4143812046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937caea0 0x557b937d3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 --2- 172.21.15.94:0/4143812046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937ca960 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 -- 172.21.15.94:0/4143812046 >> 172.21.15.94:0/4143812046 conn(0x557b937c4cb0 msgr2=0x557b937c5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 -- 172.21.15.94:0/4143812046 shutdown_connections 2024-04-21T08:48:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 -- 172.21.15.94:0/4143812046 wait complete. 2024-04-21T08:48:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 -- start start 2024-04-21T08:48:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.712+0000 7f796db74740 1 Processor -- start 2024-04-21T08:48:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f796db74740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937ca580 0x557b937d0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f796db74740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f796db74740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937d3a80 0x557b937d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f796db74740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x557b937d6870 con 0x557b937caea0 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f796db74740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557b936e82f0 con 0x557b937d3a80 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f796db74740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557b937d69f0 con 0x557b937ca580 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f796db74740 10 client.0 _ll_register_callbacks cb 0x557b936f5580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f795bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f795bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36872/0 (socket says 172.21.15.94:36872) 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f795bfff640 1 -- 172.21.15.94:0/3864203597 learned_addr learned my addr 172.21.15.94:0/3864203597 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f7968c41640 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937ca580 0x557b937d0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f795bfff640 1 -- 172.21.15.94:0/3864203597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937ca580 msgr2=0x557b937d0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f795bfff640 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937ca580 0x557b937d0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f795bfff640 1 -- 172.21.15.94:0/3864203597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937d3a80 msgr2=0x557b937d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f795bfff640 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937d3a80 0x557b937d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f795bfff640 1 -- 172.21.15.94:0/3864203597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b934413f0 con 0x557b937caea0 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f795bfff640 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d0fd0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f7954035000 tx=0x7f795402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b846feaa18d9655a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f7943fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f7943fff640 1 -- 172.21.15.94:0/3864203597 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795404f070 con 0x557b937caea0 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f7943fff640 1 -- 172.21.15.94:0/3864203597 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f79540048c0 con 0x557b937caea0 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f796db74740 1 -- 172.21.15.94:0/3864203597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79600043d0 con 0x557b937caea0 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f7943fff640 1 -- 172.21.15.94:0/3864203597 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f795405b050 con 0x557b937caea0 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f796db74740 10 client.17793 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.713+0000 7f796db74740 1 -- 172.21.15.94:0/3864203597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557b934010f0 con 0x557b937caea0 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.714+0000 7f7943fff640 1 -- 172.21.15.94:0/3864203597 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79600043d0 con 0x557b937caea0 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.714+0000 7f7943fff640 1 -- 172.21.15.94:0/3864203597 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f795404a070 con 0x557b937caea0 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.714+0000 7f796db74740 10 client.17793 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.714+0000 7f796db74740 10 client.17793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.714+0000 7f796db74740 20 client.17793 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.714+0000 7f796db74740 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557b937e39d0 0x557b937e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.714+0000 7f796db74740 4 client.17793 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.714+0000 7f796db74740 1 -- 172.21.15.94:0/3864203597 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b9373cc00 con 0x557b937e39d0 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.714+0000 7f7968c41640 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557b937e39d0 0x557b937e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.715+0000 7f7943fff640 10 client.17793 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.715+0000 7f7968c41640 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557b937e39d0 0x557b937e5e50 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4664da6af4ed32bc server_cookie=2b643f20574890b4 in_seq=0 out_seq=0 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.715+0000 7f7943fff640 1 -- 172.21.15.94:0/3864203597 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b9373cc00 con 0x557b937e39d0 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.715+0000 7f7943fff640 10 client.17793 handle_command_reply: tid=0 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.715+0000 7f796db74740 1 client.17793 shutdown 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.715+0000 7f796db74740 1 -- 172.21.15.94:0/3864203597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 msgr2=0x557b937d0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.715+0000 7f796db74740 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d0fd0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f7954035000 tx=0x7f795402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.716+0000 7f796db74740 1 -- 172.21.15.94:0/3864203597 shutdown_connections 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.716+0000 7f796db74740 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557b937e39d0 0x557b937e5e50 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.716+0000 7f796db74740 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937d3a80 0x557b937d1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.716+0000 7f796db74740 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d0fd0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.716+0000 7f7943fff640 0 client.17793 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.716+0000 7f796db74740 1 --2- 172.21.15.94:0/3864203597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937ca580 0x557b937d0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.716+0000 7f796db74740 1 -- 172.21.15.94:0/3864203597 >> 172.21.15.94:0/3864203597 conn(0x557b937c4cb0 msgr2=0x557b937c6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.716+0000 7f796db74740 1 -- 172.21.15.94:0/3864203597 shutdown_connections 2024-04-21T08:48:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.716+0000 7f796db74740 1 -- 172.21.15.94:0/3864203597 wait complete. 2024-04-21T08:48:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.716+0000 7f796db74740 20 client.17793 trim_cache size 0 max 16384 2024-04-21T08:48:10.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f796db74740 1 Processor -- start 2024-04-21T08:48:10.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f796db74740 1 -- start start 2024-04-21T08:48:10.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f796db74740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937d31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f796db74740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937caea0 0x557b937df730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f796db74740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 0x557b937cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f796db74740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f79641b2030 con 0x557b937ca580 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f796db74740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f79641af900 con 0x557b937caea0 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f796db74740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f79641ad770 con 0x557b937d3a80 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f7968c41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937d31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f7968c41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937d31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36878/0 (socket says 172.21.15.94:36878) 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f7968c41640 1 -- 172.21.15.94:0/3445232720 learned_addr learned my addr 172.21.15.94:0/3445232720 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f7968c41640 1 -- 172.21.15.94:0/3445232720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 msgr2=0x557b937cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f7968c41640 1 --2- 172.21.15.94:0/3445232720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 0x557b937cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f7968c41640 1 -- 172.21.15.94:0/3445232720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937caea0 msgr2=0x557b937df730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f7968c41640 1 --2- 172.21.15.94:0/3445232720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937caea0 0x557b937df730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.733+0000 7f7968c41640 1 -- 172.21.15.94:0/3445232720 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b937cc670 con 0x557b937ca580 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f7968c41640 1 --2- 172.21.15.94:0/3445232720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937d31c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f7960009bc0 tx=0x7f7960005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50945ce536a4434 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f794a7fc640 1 -- 172.21.15.94:0/3445232720 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960005070 con 0x557b937ca580 2024-04-21T08:48:10.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f794a7fc640 1 -- 172.21.15.94:0/3445232720 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79600035f0 con 0x557b937ca580 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f794a7fc640 1 -- 172.21.15.94:0/3445232720 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796001a6a0 con 0x557b937ca580 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f796db74740 1 -- 172.21.15.94:0/3445232720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 msgr2=0x557b937d31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f796db74740 1 --2- 172.21.15.94:0/3445232720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937d31c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f7960009bc0 tx=0x7f7960005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f796db74740 1 -- 172.21.15.94:0/3445232720 shutdown_connections 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f796db74740 1 --2- 172.21.15.94:0/3445232720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 0x557b937cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f796db74740 1 --2- 172.21.15.94:0/3445232720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937caea0 0x557b937df730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f796db74740 1 --2- 172.21.15.94:0/3445232720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937ca580 0x557b937d31c0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f796db74740 1 -- 172.21.15.94:0/3445232720 >> 172.21.15.94:0/3445232720 conn(0x557b937c4cb0 msgr2=0x557b937c34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f796db74740 1 -- 172.21.15.94:0/3445232720 shutdown_connections 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.734+0000 7f796db74740 1 -- 172.21.15.94:0/3445232720 wait complete. 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f796db74740 1 -- start start 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f796db74740 1 Processor -- start 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f796db74740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937ca580 0x557b937dc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f796db74740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f796db74740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 0x557b937d7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f796db74740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f79641addc0 con 0x557b937caea0 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f796db74740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f79641af230 con 0x557b937ca580 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f796db74740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f79641ae490 con 0x557b937d3a80 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f796db74740 10 client.0 _ll_register_callbacks cb 0x7f7960005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f795bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f795bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36894/0 (socket says 172.21.15.94:36894) 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f795bfff640 1 -- 172.21.15.94:0/1322793911 learned_addr learned my addr 172.21.15.94:0/1322793911 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f795bfff640 1 -- 172.21.15.94:0/1322793911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 msgr2=0x557b937d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f795bfff640 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 0x557b937d7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f7968c41640 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937ca580 0x557b937dc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f795bfff640 1 -- 172.21.15.94:0/1322793911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937ca580 msgr2=0x557b937dc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f795bfff640 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937ca580 0x557b937dc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f795bfff640 1 -- 172.21.15.94:0/1322793911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b937cacd0 con 0x557b937caea0 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f7943fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.735+0000 7f795bfff640 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d7470 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f795402c170 tx=0x7f7954003c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c489ec93674e99 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f7943fff640 1 -- 172.21.15.94:0/1322793911 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795404f040 con 0x557b937caea0 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f7943fff640 1 -- 172.21.15.94:0/1322793911 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9687+0+0 (secure 0 0 0) 0x7f79540037f0 con 0x557b937caea0 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f796db74740 1 -- 172.21.15.94:0/1322793911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79600035f0 con 0x557b937caea0 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f7943fff640 1 -- 172.21.15.94:0/1322793911 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7954009780 con 0x557b937caea0 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f796db74740 10 client.17799 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f796db74740 1 -- 172.21.15.94:0/1322793911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557b93849f00 con 0x557b937caea0 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f7943fff640 1 -- 172.21.15.94:0/1322793911 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79600035f0 con 0x557b937caea0 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f7943fff640 1 -- 172.21.15.94:0/1322793911 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f795404a070 con 0x557b937caea0 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f796db74740 10 client.17799 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f796db74740 10 client.17799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f796db74740 20 client.17799 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:10.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f796db74740 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557b9384a700 0x557b9384cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f796db74740 4 client.17799 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f796db74740 1 -- 172.21.15.94:0/1322793911 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557b93736a70 con 0x557b9384a700 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.736+0000 7f7968c41640 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557b9384a700 0x557b9384cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f7968c41640 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557b9384a700 0x557b9384cb20 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaef719170570516 server_cookie=84909d9e5277bd31 in_seq=0 out_seq=0 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f7943fff640 10 client.17799 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f7943fff640 1 -- 172.21.15.94:0/1322793911 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557b93736a70 con 0x557b9384a700 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f7943fff640 10 client.17799 handle_command_reply: tid=0 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 client.17799 shutdown 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 -- 172.21.15.94:0/1322793911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 msgr2=0x557b937d7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d7470 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f795402c170 tx=0x7f7954003c70 comp rx=0 tx=0).stop 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 -- 172.21.15.94:0/1322793911 shutdown_connections 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x557b9384a700 0x557b9384cb20 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b937d3a80 0x557b937d7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557b937caea0 0x557b937d7470 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 --2- 172.21.15.94:0/1322793911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b937ca580 0x557b937dc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f7943fff640 0 client.17799 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 -- 172.21.15.94:0/1322793911 >> 172.21.15.94:0/1322793911 conn(0x557b937c4cb0 msgr2=0x557b937c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 -- 172.21.15.94:0/1322793911 shutdown_connections 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 1 -- 172.21.15.94:0/1322793911 wait complete. 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.737+0000 7f796db74740 20 client.17799 trim_cache size 0 max 16384 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stdout: "a46597fd-211b-4d3c-b89f-cfd10e36bdb8": { 2024-04-21T08:48:10.744 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:48:10.745 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "a46597fd-211b-4d3c-b89f-cfd10e36bdb8", 2024-04-21T08:48:10.745 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:48:10.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:48:10.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:48:10.745 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 -- 172.21.15.94:0/871250603 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f794c0a3d20 msgr2=0x7f794c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f794c0a3d20 0x7f794c0a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7954036040 tx=0x7f79540095f0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 -- 172.21.15.94:0/871250603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7964105dd0 msgr2=0x7f79641a4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7964105dd0 0x7f79641a4700 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f795c00eb10 tx=0x7f795c00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 -- 172.21.15.94:0/871250603 shutdown_connections 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f794c0a3d20 0x7f794c0a61e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7964101a40 0x7f7964199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7964105dd0 0x7f79641a4700 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 --2- 172.21.15.94:0/871250603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79641010a0 0x7f7964199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 -- 172.21.15.94:0/871250603 >> 172.21.15.94:0/871250603 conn(0x7f7964077c40 msgr2=0x7f79640fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 -- 172.21.15.94:0/871250603 shutdown_connections 2024-04-21T08:48:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:10.738+0000 7f796aea5640 1 -- 172.21.15.94:0/871250603 wait complete. 2024-04-21T08:48:10.749 INFO:tasks.cephfs.filesystem:scrub status for tag:a46597fd-211b-4d3c-b89f-cfd10e36bdb8 - {'path': '/', 'tag': 'a46597fd-211b-4d3c-b89f-cfd10e36bdb8', 'options': 'recursive,force'} 2024-04-21T08:48:12.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:11 smithi111 ceph-mon[31493]: pgmap v252: 129 pgs: 129 active+clean; 4.6 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:48:12.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:11 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:48:12.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:11 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:12.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:11 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:48:12.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:11 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:12.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:11 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:48:12.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:11 smithi111 ceph-mon[31493]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:48:12.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:11 smithi094 ceph-mon[27988]: pgmap v252: 129 pgs: 129 active+clean; 4.6 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:48:12.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:11 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:48:12.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:11 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:12.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:11 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:48:12.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:11 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:12.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:11 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:48:12.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:11 smithi094 ceph-mon[27988]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:48:12.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:11 smithi173 ceph-mon[31428]: pgmap v252: 129 pgs: 129 active+clean; 4.6 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:48:12.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:11 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:48:12.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:11 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:12.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:11 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:48:12.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:11 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:12.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:11 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:48:12.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:11 smithi173 ceph-mon[31428]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:48:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:13 smithi111 ceph-mon[31493]: pgmap v254: 129 pgs: 129 active+clean; 4.6 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 32 KiB/s wr, 26 op/s 2024-04-21T08:48:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:13 smithi094 ceph-mon[27988]: pgmap v254: 129 pgs: 129 active+clean; 4.6 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 32 KiB/s wr, 26 op/s 2024-04-21T08:48:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:13 smithi173 ceph-mon[31428]: pgmap v254: 129 pgs: 129 active+clean; 4.6 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 32 KiB/s wr, 26 op/s 2024-04-21T08:48:16.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:15 smithi111 ceph-mon[31493]: pgmap v255: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:48:16.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:15 smithi094 ceph-mon[27988]: pgmap v255: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:48:16.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:15 smithi173 ceph-mon[31428]: pgmap v255: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:48:18.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:17 smithi111 ceph-mon[31493]: pgmap v256: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 222 B/s wr, 8 op/s 2024-04-21T08:48:18.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:17 smithi094 ceph-mon[27988]: pgmap v256: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 222 B/s wr, 8 op/s 2024-04-21T08:48:18.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:17 smithi173 ceph-mon[31428]: pgmap v256: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 222 B/s wr, 8 op/s 2024-04-21T08:48:20.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:19 smithi111 ceph-mon[31493]: pgmap v257: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-04-21T08:48:20.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:19 smithi094 ceph-mon[27988]: pgmap v257: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-04-21T08:48:20.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:19 smithi173 ceph-mon[31428]: pgmap v257: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 204 B/s wr, 7 op/s 2024-04-21T08:48:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:21 smithi111 ceph-mon[31493]: pgmap v258: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-21T08:48:22.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:21 smithi094 ceph-mon[27988]: pgmap v258: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-21T08:48:22.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:21 smithi173 ceph-mon[31428]: pgmap v258: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-21T08:48:24.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:23 smithi111 ceph-mon[31493]: pgmap v259: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 99 B/s wr, 7 op/s 2024-04-21T08:48:24.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:23 smithi094 ceph-mon[27988]: pgmap v259: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 99 B/s wr, 7 op/s 2024-04-21T08:48:24.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:23 smithi173 ceph-mon[31428]: pgmap v259: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 99 B/s wr, 7 op/s 2024-04-21T08:48:26.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:25 smithi111 ceph-mon[31493]: pgmap v260: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-21T08:48:26.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:25 smithi094 ceph-mon[27988]: pgmap v260: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-21T08:48:26.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:25 smithi173 ceph-mon[31428]: pgmap v260: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-21T08:48:28.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:27 smithi111 ceph-mon[31493]: pgmap v261: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:48:28.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:27 smithi094 ceph-mon[27988]: pgmap v261: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:48:28.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:27 smithi173 ceph-mon[31428]: pgmap v261: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:48:30.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:29 smithi111 ceph-mon[31493]: pgmap v262: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:30.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:29 smithi094 ceph-mon[27988]: pgmap v262: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:30.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:29 smithi173 ceph-mon[31428]: pgmap v262: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:32.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:31 smithi111 ceph-mon[31493]: pgmap v263: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:48:32.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:31 smithi094 ceph-mon[27988]: pgmap v263: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:48:32.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:31 smithi173 ceph-mon[31428]: pgmap v263: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:48:33.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:32 smithi111 ceph-mon[31493]: pgmap v264: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:33.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:32 smithi094 ceph-mon[27988]: pgmap v264: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:33.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:32 smithi173 ceph-mon[31428]: pgmap v264: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:35.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:35 smithi111 ceph-mon[31493]: pgmap v265: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:48:35.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:35 smithi094 ceph-mon[27988]: pgmap v265: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:48:35.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:35 smithi173 ceph-mon[31428]: pgmap v265: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:48:37.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:37 smithi111 ceph-mon[31493]: pgmap v266: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:48:37.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:37 smithi094 ceph-mon[27988]: pgmap v266: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:48:37.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:37 smithi173 ceph-mon[31428]: pgmap v266: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:48:39.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:39 smithi111 ceph-mon[31493]: pgmap v267: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:39.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:39 smithi094 ceph-mon[27988]: pgmap v267: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:39.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:39 smithi173 ceph-mon[31428]: pgmap v267: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:40.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:48:40.750 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:48:40.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 -- 172.21.15.94:0/2467890732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f786006a820 msgr2=0x7f786006ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:40.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 --2- 172.21.15.94:0/2467890732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f786006a820 0x7f786006ac00 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f7854009900 tx=0x7f785402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:40.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 -- 172.21.15.94:0/2467890732 shutdown_connections 2024-04-21T08:48:40.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 --2- 172.21.15.94:0/2467890732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f786006b1d0 0x7f7860106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 --2- 172.21.15.94:0/2467890732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7860107270 0x7f786010ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 --2- 172.21.15.94:0/2467890732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f786006a820 0x7f786006ac00 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 -- 172.21.15.94:0/2467890732 >> 172.21.15.94:0/2467890732 conn(0x7f7860077c40 msgr2=0x7f7860078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 -- 172.21.15.94:0/2467890732 shutdown_connections 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 -- 172.21.15.94:0/2467890732 wait complete. 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 Processor -- start 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 -- start start 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f786006a820 0x7f7860101170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7860107270 0x7f78601016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.869+0000 7f7867115640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7860101bf0 0x7f78601a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7867115640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f78601121d0 con 0x7f786006a820 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7867115640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7860112050 con 0x7f7860101bf0 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7867115640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7860112350 con 0x7f7860107270 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7864eb1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f786006a820 0x7f7860101170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7864eb1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f786006a820 0x7f7860101170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47196/0 (socket says 172.21.15.94:47196) 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7864eb1640 1 -- 172.21.15.94:0/298378199 learned_addr learned my addr 172.21.15.94:0/298378199 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f78656b2640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7860101bf0 0x7f78601a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f785ffff640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7860107270 0x7f78601016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7864eb1640 1 -- 172.21.15.94:0/298378199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7860107270 msgr2=0x7f78601016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7864eb1640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7860107270 0x7f78601016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7864eb1640 1 -- 172.21.15.94:0/298378199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7860101bf0 msgr2=0x7f78601a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7864eb1640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7860101bf0 0x7f78601a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7864eb1640 1 -- 172.21.15.94:0/298378199 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78601a6fc0 con 0x7f786006a820 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f78656b2640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7860101bf0 0x7f78601a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f785ffff640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7860107270 0x7f78601016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7864eb1640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f786006a820 0x7f7860101170 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f7854005940 tx=0x7f78540045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f785dffb640 1 -- 172.21.15.94:0/298378199 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7854043020 con 0x7f786006a820 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f7867115640 1 -- 172.21.15.94:0/298378199 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78601a7250 con 0x7f786006a820 2024-04-21T08:48:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f785dffb640 1 -- 172.21.15.94:0/298378199 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f785402caf0 con 0x7f786006a820 2024-04-21T08:48:40.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.870+0000 7f785dffb640 1 -- 172.21.15.94:0/298378199 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7854003af0 con 0x7f786006a820 2024-04-21T08:48:40.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.871+0000 7f7867115640 1 -- 172.21.15.94:0/298378199 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78601a77a0 con 0x7f786006a820 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.873+0000 7f785dffb640 1 -- 172.21.15.94:0/298378199 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f7854046020 con 0x7f786006a820 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.873+0000 7f7869ddd740 1 Processor -- start 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.873+0000 7f7869ddd740 1 -- start start 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08580 0x563cf4e08960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f785dffb640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f78500a3d50 0x7f78500a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08ea0 0x563cf4e11540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e11a80 0x563cf4e13e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f7869ddd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563cf4d6d1e0 con 0x563cf4e11a80 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f7869ddd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563cf4d0a390 con 0x563cf4e08ea0 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f7869ddd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563cf4d691a0 con 0x563cf4e08580 2024-04-21T08:48:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f78656b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e11a80 0x563cf4e13e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f7864eb1640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f78500a3d50 0x7f78500a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f78656b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e11a80 0x563cf4e13e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47200/0 (socket says 172.21.15.94:47200) 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f78656b2640 1 -- 172.21.15.94:0/3710852529 learned_addr learned my addr 172.21.15.94:0/3710852529 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f785dffb640 1 -- 172.21.15.94:0/298378199 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f78540f2d40 con 0x7f786006a820 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f78656b2640 1 --2- 172.21.15.94:0/3710852529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08580 0x563cf4e08960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f785ffff640 1 --2- 172.21.15.94:0/3710852529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08ea0 0x563cf4e11540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f78656b2640 1 -- 172.21.15.94:0/3710852529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08580 msgr2=0x563cf4e08960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f78656b2640 1 --2- 172.21.15.94:0/3710852529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08580 0x563cf4e08960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f78656b2640 1 -- 172.21.15.94:0/3710852529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08ea0 msgr2=0x563cf4e11540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f78656b2640 1 --2- 172.21.15.94:0/3710852529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08ea0 0x563cf4e11540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f78656b2640 1 -- 172.21.15.94:0/3710852529 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cf4b497d0 con 0x563cf4e11a80 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f785ffff640 1 --2- 172.21.15.94:0/3710852529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08ea0 0x563cf4e11540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f78656b2640 1 --2- 172.21.15.94:0/3710852529 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e11a80 0x563cf4e13e70 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f78580157f0 tx=0x7f78580098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895932c1d8c408dc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.874+0000 7f7864eb1640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f78500a3d50 0x7f78500a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f78540f0660 tx=0x7f78540bad90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f784affd640 1 -- 172.21.15.94:0/3710852529 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858015e30 con 0x563cf4e11a80 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f784affd640 1 -- 172.21.15.94:0/3710852529 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78580150f0 con 0x563cf4e11a80 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f784affd640 1 -- 172.21.15.94:0/3710852529 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858005020 con 0x563cf4e11a80 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f7869ddd740 1 -- 172.21.15.94:0/3710852529 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e11a80 msgr2=0x563cf4e13e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f7869ddd740 1 --2- 172.21.15.94:0/3710852529 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e11a80 0x563cf4e13e70 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f78580157f0 tx=0x7f78580098a0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f7869ddd740 1 -- 172.21.15.94:0/3710852529 shutdown_connections 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f7869ddd740 1 --2- 172.21.15.94:0/3710852529 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e11a80 0x563cf4e13e70 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f7869ddd740 1 --2- 172.21.15.94:0/3710852529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08ea0 0x563cf4e11540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f7869ddd740 1 --2- 172.21.15.94:0/3710852529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08580 0x563cf4e08960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f7869ddd740 1 -- 172.21.15.94:0/3710852529 >> 172.21.15.94:0/3710852529 conn(0x563cf4e02cb0 msgr2=0x563cf4e03090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f7869ddd740 1 -- 172.21.15.94:0/3710852529 shutdown_connections 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.875+0000 7f7869ddd740 1 -- 172.21.15.94:0/3710852529 wait complete. 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f7869ddd740 1 -- start start 2024-04-21T08:48:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f7869ddd740 1 Processor -- start 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08580 0x563cf4e0eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e11a80 0x563cf4e0f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f7869ddd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563cf4e14870 con 0x563cf4e08ea0 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f7869ddd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563cf4d262f0 con 0x563cf4e11a80 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f7869ddd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563cf4e149f0 con 0x563cf4e08580 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f7869ddd740 10 client.0 _ll_register_callbacks cb 0x563cf4d33580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f785ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f785ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47214/0 (socket says 172.21.15.94:47214) 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f785ffff640 1 -- 172.21.15.94:0/750955713 learned_addr learned my addr 172.21.15.94:0/750955713 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.876+0000 7f78656b2640 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08580 0x563cf4e0eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f785ffff640 1 -- 172.21.15.94:0/750955713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08580 msgr2=0x563cf4e0eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f785ffff640 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08580 0x563cf4e0eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f78656b2640 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e11a80 0x563cf4e0f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f785ffff640 1 -- 172.21.15.94:0/750955713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e11a80 msgr2=0x563cf4e0f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f785ffff640 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e11a80 0x563cf4e0f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f785ffff640 1 -- 172.21.15.94:0/750955713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cf4a7f3f0 con 0x563cf4e08ea0 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f78656b2640 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e11a80 0x563cf4e0f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f785ffff640 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0f020 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f784c011690 tx=0x7f784c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125472ba031eac38 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f783ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f783ffff640 1 -- 172.21.15.94:0/750955713 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c00fb40 con 0x563cf4e08ea0 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f783ffff640 1 -- 172.21.15.94:0/750955713 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f784c00fce0 con 0x563cf4e08ea0 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f7869ddd740 1 -- 172.21.15.94:0/750955713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78580150f0 con 0x563cf4e08ea0 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f783ffff640 1 -- 172.21.15.94:0/750955713 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f784c025050 con 0x563cf4e08ea0 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f7869ddd740 10 client.17811 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f7869ddd740 1 -- 172.21.15.94:0/750955713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563cf4a3f0f0 con 0x563cf4e08ea0 2024-04-21T08:48:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f783ffff640 1 -- 172.21.15.94:0/750955713 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78580150f0 con 0x563cf4e08ea0 2024-04-21T08:48:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f783ffff640 1 -- 172.21.15.94:0/750955713 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f784c01a440 con 0x563cf4e08ea0 2024-04-21T08:48:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f7869ddd740 10 client.17811 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.877+0000 7f7869ddd740 10 client.17811 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.878+0000 7f7869ddd740 20 client.17811 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.878+0000 7f7869ddd740 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563cf4e21b10 0x563cf4e23f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.878+0000 7f7869ddd740 4 client.17811 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.878+0000 7f7869ddd740 1 -- 172.21.15.94:0/750955713 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cf4d7ac00 con 0x563cf4e21b10 2024-04-21T08:48:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.878+0000 7f78656b2640 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563cf4e21b10 0x563cf4e23f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.878+0000 7f783ffff640 10 client.17811 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.878+0000 7f78656b2640 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563cf4e21b10 0x563cf4e23f50 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d53620dc548f8c9 server_cookie=4aa84fa8b56b9362 in_seq=0 out_seq=0 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f783ffff640 1 -- 172.21.15.94:0/750955713 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563cf4d7ac00 con 0x563cf4e21b10 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f783ffff640 10 client.17811 handle_command_reply: tid=0 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f7869ddd740 1 client.17811 shutdown 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f7869ddd740 1 -- 172.21.15.94:0/750955713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 msgr2=0x563cf4e0f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f7869ddd740 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0f020 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f784c011690 tx=0x7f784c011b50 comp rx=0 tx=0).stop 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f7869ddd740 1 -- 172.21.15.94:0/750955713 shutdown_connections 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f7869ddd740 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563cf4e21b10 0x563cf4e23f50 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f7869ddd740 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e11a80 0x563cf4e0f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f7869ddd740 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0f020 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f783ffff640 0 client.17811 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f7869ddd740 1 --2- 172.21.15.94:0/750955713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08580 0x563cf4e0eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.879+0000 7f7869ddd740 1 -- 172.21.15.94:0/750955713 >> 172.21.15.94:0/750955713 conn(0x563cf4e02cb0 msgr2=0x563cf4e04a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:40.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.880+0000 7f7869ddd740 1 -- 172.21.15.94:0/750955713 shutdown_connections 2024-04-21T08:48:40.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.880+0000 7f7869ddd740 1 -- 172.21.15.94:0/750955713 wait complete. 2024-04-21T08:48:40.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.880+0000 7f7869ddd740 20 client.17811 trim_cache size 0 max 16384 2024-04-21T08:48:40.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.897+0000 7f7869ddd740 1 Processor -- start 2024-04-21T08:48:40.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.897+0000 7f7869ddd740 1 -- start start 2024-04-21T08:48:40.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.897+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08580 0x563cf4e11210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.897+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.897+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e11a80 0x563cf4e1d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.897+0000 7f7869ddd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f78601b40a0 con 0x563cf4e08ea0 2024-04-21T08:48:40.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.897+0000 7f7869ddd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f78601b1970 con 0x563cf4e08580 2024-04-21T08:48:40.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.897+0000 7f7869ddd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f78601af7e0 con 0x563cf4e11a80 2024-04-21T08:48:40.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f785ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f785ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47230/0 (socket says 172.21.15.94:47230) 2024-04-21T08:48:40.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f785ffff640 1 -- 172.21.15.94:0/1546103582 learned_addr learned my addr 172.21.15.94:0/1546103582 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:40.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f78656b2640 1 --2- 172.21.15.94:0/1546103582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e11a80 0x563cf4e1d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f78656b2640 1 --2- 172.21.15.94:0/1546103582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08580 0x563cf4e11210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f785ffff640 1 -- 172.21.15.94:0/1546103582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e11a80 msgr2=0x563cf4e1d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:40.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f785ffff640 1 --2- 172.21.15.94:0/1546103582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e11a80 0x563cf4e1d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f785ffff640 1 -- 172.21.15.94:0/1546103582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08580 msgr2=0x563cf4e11210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:40.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f785ffff640 1 --2- 172.21.15.94:0/1546103582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08580 0x563cf4e11210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f785ffff640 1 -- 172.21.15.94:0/1546103582 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cf4e027e0 con 0x563cf4e08ea0 2024-04-21T08:48:40.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f78656b2640 1 --2- 172.21.15.94:0/1546103582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08580 0x563cf4e11210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:40.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f785ffff640 1 --2- 172.21.15.94:0/1546103582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0da20 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f784c001eb0 tx=0x7f784c011fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc186d1c5ee8f476 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:40.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f784a7fc640 1 -- 172.21.15.94:0/1546103582 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c02d070 con 0x563cf4e08ea0 2024-04-21T08:48:40.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f784a7fc640 1 -- 172.21.15.94:0/1546103582 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f784c00fdf0 con 0x563cf4e08ea0 2024-04-21T08:48:40.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.898+0000 7f784a7fc640 1 -- 172.21.15.94:0/1546103582 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c005320 con 0x563cf4e08ea0 2024-04-21T08:48:40.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 -- 172.21.15.94:0/1546103582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 msgr2=0x563cf4e0da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:40.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 --2- 172.21.15.94:0/1546103582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0da20 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f784c001eb0 tx=0x7f784c011fb0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 -- 172.21.15.94:0/1546103582 shutdown_connections 2024-04-21T08:48:40.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 --2- 172.21.15.94:0/1546103582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e11a80 0x563cf4e1d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 --2- 172.21.15.94:0/1546103582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08ea0 0x563cf4e0da20 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 --2- 172.21.15.94:0/1546103582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e08580 0x563cf4e11210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 -- 172.21.15.94:0/1546103582 >> 172.21.15.94:0/1546103582 conn(0x563cf4e02cb0 msgr2=0x563cf4e01d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:40.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 -- 172.21.15.94:0/1546103582 shutdown_connections 2024-04-21T08:48:40.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 -- 172.21.15.94:0/1546103582 wait complete. 2024-04-21T08:48:40.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 -- start start 2024-04-21T08:48:40.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.899+0000 7f7869ddd740 1 Processor -- start 2024-04-21T08:48:40.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08580 0x563cf4e15f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08ea0 0x563cf4e164b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f7869ddd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e11a80 0x563cf4e810d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f7869ddd740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f78601afe30 con 0x563cf4e08580 2024-04-21T08:48:40.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f7869ddd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f78601b12a0 con 0x563cf4e11a80 2024-04-21T08:48:40.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f7869ddd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f78601b0500 con 0x563cf4e08ea0 2024-04-21T08:48:40.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f7869ddd740 10 client.0 _ll_register_callbacks cb 0x7f784c009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:40.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f78656b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08580 0x563cf4e15f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f78656b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08580 0x563cf4e15f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47246/0 (socket says 172.21.15.94:47246) 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f78656b2640 1 -- 172.21.15.94:0/1434415469 learned_addr learned my addr 172.21.15.94:0/1434415469 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f785ffff640 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08ea0 0x563cf4e164b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f78656b2640 1 -- 172.21.15.94:0/1434415469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08ea0 msgr2=0x563cf4e164b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f78656b2640 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08ea0 0x563cf4e164b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f78656b2640 1 -- 172.21.15.94:0/1434415469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e11a80 msgr2=0x563cf4e810d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f78656b2640 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e11a80 0x563cf4e810d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f78656b2640 1 -- 172.21.15.94:0/1434415469 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cf4e81710 con 0x563cf4e08580 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f78656b2640 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08580 0x563cf4e15f70 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f78580157c0 tx=0x7f7858009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4754bc02fb6209 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f783ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f783ffff640 1 -- 172.21.15.94:0/1434415469 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78580150e0 con 0x563cf4e08580 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f783ffff640 1 -- 172.21.15.94:0/1434415469 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f7858015280 con 0x563cf4e08580 2024-04-21T08:48:40.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.900+0000 7f7869ddd740 1 -- 172.21.15.94:0/1434415469 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f784c00fdf0 con 0x563cf4e08580 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f783ffff640 1 -- 172.21.15.94:0/1434415469 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7858010890 con 0x563cf4e08580 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f7869ddd740 10 client.17817 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f7869ddd740 1 -- 172.21.15.94:0/1434415469 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563cf4e88ad0 con 0x563cf4e08580 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f783ffff640 1 -- 172.21.15.94:0/1434415469 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f784c00fdf0 con 0x563cf4e08580 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f783ffff640 1 -- 172.21.15.94:0/1434415469 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f785800c070 con 0x563cf4e08580 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f7869ddd740 10 client.17817 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f7869ddd740 10 client.17817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f7869ddd740 20 client.17817 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f7869ddd740 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563cf4e890c0 0x563cf4e8b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f7869ddd740 4 client.17817 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f7869ddd740 1 -- 172.21.15.94:0/1434415469 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563cf4d74a70 con 0x563cf4e890c0 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.901+0000 7f785ffff640 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563cf4e890c0 0x563cf4e8b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f785ffff640 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563cf4e890c0 0x563cf4e8b4e0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe030e70fd49d65a server_cookie=9b0525a4f1c2bb40 in_seq=0 out_seq=0 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f783ffff640 10 client.17817 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:40.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f783ffff640 1 -- 172.21.15.94:0/1434415469 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563cf4d74a70 con 0x563cf4e890c0 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f783ffff640 10 client.17817 handle_command_reply: tid=0 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 client.17817 shutdown 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 -- 172.21.15.94:0/1434415469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08580 msgr2=0x563cf4e15f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08580 0x563cf4e15f70 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f78580157c0 tx=0x7f7858009b70 comp rx=0 tx=0).stop 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 -- 172.21.15.94:0/1434415469 shutdown_connections 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563cf4e890c0 0x563cf4e8b4e0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563cf4e08ea0 0x563cf4e164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563cf4e11a80 0x563cf4e810d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 --2- 172.21.15.94:0/1434415469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563cf4e08580 0x563cf4e15f70 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f783ffff640 0 client.17817 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 -- 172.21.15.94:0/1434415469 >> 172.21.15.94:0/1434415469 conn(0x563cf4e02cb0 msgr2=0x563cf4e0ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 -- 172.21.15.94:0/1434415469 shutdown_connections 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 1 -- 172.21.15.94:0/1434415469 wait complete. 2024-04-21T08:48:40.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.902+0000 7f7869ddd740 20 client.17817 trim_cache size 0 max 16384 2024-04-21T08:48:40.910 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:48:40.910 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:48:40.910 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:48:40.910 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:48:40.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 -- 172.21.15.94:0/298378199 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f78500a3d50 msgr2=0x7f78500a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:40.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f78500a3d50 0x7f78500a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f78540f0660 tx=0x7f78540bad90 comp rx=0 tx=0).stop 2024-04-21T08:48:40.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 -- 172.21.15.94:0/298378199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f786006a820 msgr2=0x7f7860101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:40.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f786006a820 0x7f7860101170 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f7854005940 tx=0x7f78540045e0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 -- 172.21.15.94:0/298378199 shutdown_connections 2024-04-21T08:48:40.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f78500a3d50 0x7f78500a6210 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7860107270 0x7f78601016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7860101bf0 0x7f78601a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 --2- 172.21.15.94:0/298378199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f786006a820 0x7f7860101170 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:40.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 -- 172.21.15.94:0/298378199 >> 172.21.15.94:0/298378199 conn(0x7f7860077c40 msgr2=0x7f78600ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:40.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 -- 172.21.15.94:0/298378199 shutdown_connections 2024-04-21T08:48:40.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:40.903+0000 7f7867115640 1 -- 172.21.15.94:0/298378199 wait complete. 2024-04-21T08:48:40.913 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:48:40.913 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 -- 172.21.15.94:0/3886440003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53701028f0 msgr2=0x7f537010edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 --2- 172.21.15.94:0/3886440003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53701028f0 0x7f537010edb0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f5368009ca0 tx=0x7f536802c180 comp rx=0 tx=0).stop 2024-04-21T08:48:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 -- 172.21.15.94:0/3886440003 shutdown_connections 2024-04-21T08:48:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 --2- 172.21.15.94:0/3886440003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53701028f0 0x7f537010edb0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 --2- 172.21.15.94:0/3886440003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5370101f50 0x7f53701023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 --2- 172.21.15.94:0/3886440003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5370108010 0x7f53701083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 -- 172.21.15.94:0/3886440003 >> 172.21.15.94:0/3886440003 conn(0x7f53700793e0 msgr2=0x7f53700797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 -- 172.21.15.94:0/3886440003 shutdown_connections 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 -- 172.21.15.94:0/3886440003 wait complete. 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 Processor -- start 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.014+0000 7f5375443640 1 -- start start 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5375443640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5370101f50 0x7f5370198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5375443640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53701028f0 0x7f5370198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5375443640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5370108010 0x7f537019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5375443640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5370112180 con 0x7f5370108010 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5375443640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5370112000 con 0x7f5370101f50 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5375443640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5370112300 con 0x7f53701028f0 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5370108010 0x7f537019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5370108010 0x7f537019cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47270/0 (socket says 172.21.15.94:47270) 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536f7fe640 1 -- 172.21.15.94:0/2988577701 learned_addr learned my addr 172.21.15.94:0/2988577701 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536e7fc640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53701028f0 0x7f5370198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536effd640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5370101f50 0x7f5370198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536f7fe640 1 -- 172.21.15.94:0/2988577701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53701028f0 msgr2=0x7f5370198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536f7fe640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53701028f0 0x7f5370198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536f7fe640 1 -- 172.21.15.94:0/2988577701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5370101f50 msgr2=0x7f5370198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536f7fe640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5370101f50 0x7f5370198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536f7fe640 1 -- 172.21.15.94:0/2988577701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537019d6a0 con 0x7f5370108010 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536e7fc640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53701028f0 0x7f5370198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f536f7fe640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5370108010 0x7f537019cf70 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f536802c790 tx=0x7f5368003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5357fff640 1 -- 172.21.15.94:0/2988577701 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5368039040 con 0x7f5370108010 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5375443640 1 -- 172.21.15.94:0/2988577701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f537019d930 con 0x7f5370108010 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5357fff640 1 -- 172.21.15.94:0/2988577701 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5368044070 con 0x7f5370108010 2024-04-21T08:48:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5357fff640 1 -- 172.21.15.94:0/2988577701 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5368032980 con 0x7f5370108010 2024-04-21T08:48:41.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.015+0000 7f5375443640 1 -- 172.21.15.94:0/2988577701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53701a5220 con 0x7f5370108010 2024-04-21T08:48:41.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.016+0000 7f5375443640 1 -- 172.21.15.94:0/2988577701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5370068de0 con 0x7f5370108010 2024-04-21T08:48:41.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.017+0000 7f5357fff640 1 -- 172.21.15.94:0/2988577701 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5368032b20 con 0x7f5370108010 2024-04-21T08:48:41.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.018+0000 7f5357fff640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f53580a3d20 0x7f53580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.018+0000 7f536effd640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f53580a3d20 0x7f53580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.018+0000 7f5357fff640 1 -- 172.21.15.94:0/2988577701 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f53680f3110 con 0x7f5370108010 2024-04-21T08:48:41.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.018+0000 7f536effd640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f53580a3d20 0x7f53580a61e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f53600099e0 tx=0x7f5360005ed0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.022+0000 7f5357fff640 1 -- 172.21.15.94:0/2988577701 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f53680bf490 con 0x7f5370108010 2024-04-21T08:48:41.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.241+0000 7f5375443640 1 -- 172.21.15.94:0/2988577701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f537010a430 con 0x7f5370108010 2024-04-21T08:48:41.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.242+0000 7f5357fff640 1 -- 172.21.15.94:0/2988577701 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f53680bed90 con 0x7f5370108010 2024-04-21T08:48:41.241 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:48:41.241 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:48:41.241 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:48:41.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 -- 172.21.15.94:0/2988577701 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f53580a3d20 msgr2=0x7f53580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f53580a3d20 0x7f53580a61e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f53600099e0 tx=0x7f5360005ed0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 -- 172.21.15.94:0/2988577701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5370108010 msgr2=0x7f537019cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5370108010 0x7f537019cf70 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f536802c790 tx=0x7f5368003c20 comp rx=0 tx=0).stop 2024-04-21T08:48:41.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 -- 172.21.15.94:0/2988577701 shutdown_connections 2024-04-21T08:48:41.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f53580a3d20 0x7f53580a61e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53701028f0 0x7f5370198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5370108010 0x7f537019cf70 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 --2- 172.21.15.94:0/2988577701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5370101f50 0x7f5370198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 -- 172.21.15.94:0/2988577701 >> 172.21.15.94:0/2988577701 conn(0x7f53700793e0 msgr2=0x7f53701009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 -- 172.21.15.94:0/2988577701 shutdown_connections 2024-04-21T08:48:41.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.244+0000 7f5375443640 1 -- 172.21.15.94:0/2988577701 wait complete. 2024-04-21T08:48:41.254 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:48:41.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.355+0000 7f89c7c62640 1 -- 172.21.15.94:0/1018819428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89c010d2c0 msgr2=0x7f89c01197d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.355+0000 7f89c7c62640 1 --2- 172.21.15.94:0/1018819428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89c010d2c0 0x7f89c01197d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009cd0 tx=0x7f89b802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.355+0000 7f89c7c62640 1 -- 172.21.15.94:0/1018819428 shutdown_connections 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.355+0000 7f89c7c62640 1 --2- 172.21.15.94:0/1018819428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89c010c900 0x7f89c010cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.355+0000 7f89c7c62640 1 --2- 172.21.15.94:0/1018819428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89c010d2c0 0x7f89c01197d0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.355+0000 7f89c7c62640 1 --2- 172.21.15.94:0/1018819428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89c0112910 0x7f89c0112cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.355+0000 7f89c7c62640 1 -- 172.21.15.94:0/1018819428 >> 172.21.15.94:0/1018819428 conn(0x7f89c006cfa0 msgr2=0x7f89c010af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.355+0000 7f89c7c62640 1 -- 172.21.15.94:0/1018819428 shutdown_connections 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.355+0000 7f89c7c62640 1 -- 172.21.15.94:0/1018819428 wait complete. 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c7c62640 1 Processor -- start 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c7c62640 1 -- start start 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c7c62640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89c010c900 0x7f89c01a3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c7c62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89c010d2c0 0x7f89c01a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c7c62640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89c0112910 0x7f89c01a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c7c62640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f89c011cbd0 con 0x7f89c010d2c0 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c7c62640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f89c011ca50 con 0x7f89c010c900 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c7c62640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f89c011cd50 con 0x7f89c0112910 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c51fd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89c010d2c0 0x7f89c01a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c51fd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89c010d2c0 0x7f89c01a36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47278/0 (socket says 172.21.15.94:47278) 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c51fd640 1 -- 172.21.15.94:0/1232709200 learned_addr learned my addr 172.21.15.94:0/1232709200 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c61ff640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89c0112910 0x7f89c01a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c59fe640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89c010c900 0x7f89c01a3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c51fd640 1 -- 172.21.15.94:0/1232709200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89c0112910 msgr2=0x7f89c01a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c51fd640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89c0112910 0x7f89c01a7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c51fd640 1 -- 172.21.15.94:0/1232709200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89c010c900 msgr2=0x7f89c01a3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c51fd640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89c010c900 0x7f89c01a3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c51fd640 1 -- 172.21.15.94:0/1232709200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c01a8160 con 0x7f89c010d2c0 2024-04-21T08:48:41.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.356+0000 7f89c61ff640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89c0112910 0x7f89c01a7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:41.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.357+0000 7f89c51fd640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89c010d2c0 0x7f89c01a36b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f89b000c890 tx=0x7f89b000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.357+0000 7f89b6ffd640 1 -- 172.21.15.94:0/1232709200 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0004220 con 0x7f89c010d2c0 2024-04-21T08:48:41.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.357+0000 7f89b6ffd640 1 -- 172.21.15.94:0/1232709200 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89b00043c0 con 0x7f89c010d2c0 2024-04-21T08:48:41.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.357+0000 7f89c7c62640 1 -- 172.21.15.94:0/1232709200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c01a8450 con 0x7f89c010d2c0 2024-04-21T08:48:41.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.357+0000 7f89b6ffd640 1 -- 172.21.15.94:0/1232709200 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0004a00 con 0x7f89c010d2c0 2024-04-21T08:48:41.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.357+0000 7f89c7c62640 1 -- 172.21.15.94:0/1232709200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c01afce0 con 0x7f89c010d2c0 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.359+0000 7f89b6ffd640 1 -- 172.21.15.94:0/1232709200 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f89b00079b0 con 0x7f89c010d2c0 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.359+0000 7f89ca84c740 1 Processor -- start 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89ca84c740 1 -- start start 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f1c4d0 0x557641f1c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 0x557641f25430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f25a00 0x557641f27e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89ca84c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x557641e811f0 con 0x557641f25a00 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89ca84c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557641e1e4b0 con 0x557641f1ce80 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89ca84c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557641e7d1b0 con 0x557641f1c4d0 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89b6ffd640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89ac0a3d20 0x7f89ac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c51fd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f25a00 0x557641f27e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c61ff640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89ac0a3d20 0x7f89ac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c51fd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f25a00 0x557641f27e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47288/0 (socket says 172.21.15.94:47288) 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c51fd640 1 -- 172.21.15.94:0/1715365916 learned_addr learned my addr 172.21.15.94:0/1715365916 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c59fe640 1 --2- 172.21.15.94:0/1715365916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 0x557641f25430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c61ff640 1 --2- 172.21.15.94:0/1715365916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f1c4d0 0x557641f1c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89b6ffd640 1 -- 172.21.15.94:0/1232709200 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f89b00d0a90 con 0x7f89c010d2c0 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c51fd640 1 -- 172.21.15.94:0/1715365916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f1c4d0 msgr2=0x557641f1c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c51fd640 1 --2- 172.21.15.94:0/1715365916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f1c4d0 0x557641f1c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c51fd640 1 -- 172.21.15.94:0/1715365916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 msgr2=0x557641f25430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c51fd640 1 --2- 172.21.15.94:0/1715365916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 0x557641f25430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.360+0000 7f89c51fd640 1 -- 172.21.15.94:0/1715365916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557641e101b0 con 0x557641f25a00 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.361+0000 7f89c61ff640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89ac0a3d20 0x7f89ac0a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f89b8006010 tx=0x7f89b8032ca0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.361+0000 7f89c51fd640 1 --2- 172.21.15.94:0/1715365916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f25a00 0x557641f27e50 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f89b00cdc90 tx=0x7f89b006f090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f231859659b06b92 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.361+0000 7f89abfff640 1 -- 172.21.15.94:0/1715365916 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b001d070 con 0x557641f25a00 2024-04-21T08:48:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.361+0000 7f89abfff640 1 -- 172.21.15.94:0/1715365916 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89b0085ae0 con 0x557641f25a00 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.361+0000 7f89abfff640 1 -- 172.21.15.94:0/1715365916 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0098c10 con 0x557641f25a00 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.361+0000 7f89ca84c740 1 -- 172.21.15.94:0/1715365916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f25a00 msgr2=0x557641f27e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.361+0000 7f89ca84c740 1 --2- 172.21.15.94:0/1715365916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f25a00 0x557641f27e50 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f89b00cdc90 tx=0x7f89b006f090 comp rx=0 tx=0).stop 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 -- 172.21.15.94:0/1715365916 shutdown_connections 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 --2- 172.21.15.94:0/1715365916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f25a00 0x557641f27e50 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 --2- 172.21.15.94:0/1715365916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 0x557641f25430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 --2- 172.21.15.94:0/1715365916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f1c4d0 0x557641f1c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 -- 172.21.15.94:0/1715365916 >> 172.21.15.94:0/1715365916 conn(0x557641f16ba0 msgr2=0x557641f16f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 -- 172.21.15.94:0/1715365916 shutdown_connections 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 -- 172.21.15.94:0/1715365916 wait complete. 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 -- start start 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 Processor -- start 2024-04-21T08:48:41.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f252d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f1ce80 0x557641f22660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f25a00 0x557641f22bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x557641f288b0 con 0x557641f1c4d0 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557641e3a500 con 0x557641f25a00 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.362+0000 7f89ca84c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557641f28a30 con 0x557641f1ce80 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89ca84c740 10 client.0 _ll_register_callbacks cb 0x557641e08b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c59fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f252d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c59fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f252d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47302/0 (socket says 172.21.15.94:47302) 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c59fe640 1 -- 172.21.15.94:0/2952236695 learned_addr learned my addr 172.21.15.94:0/2952236695 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c61ff640 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f25a00 0x557641f22bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c51fd640 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f1ce80 0x557641f22660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c59fe640 1 -- 172.21.15.94:0/2952236695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f1ce80 msgr2=0x557641f22660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c59fe640 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f1ce80 0x557641f22660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c59fe640 1 -- 172.21.15.94:0/2952236695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f25a00 msgr2=0x557641f22bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c59fe640 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f25a00 0x557641f22bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c59fe640 1 -- 172.21.15.94:0/2952236695 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557641b933f0 con 0x557641f1c4d0 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89c59fe640 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f252d0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0104d0 tx=0x7f89bc010990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4508aba4e4581875 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89a97fa640 1 -- 172.21.15.94:0/2952236695 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc01a020 con 0x557641f1c4d0 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89a97fa640 1 -- 172.21.15.94:0/2952236695 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f89bc016690 con 0x557641f1c4d0 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89ca84c740 1 -- 172.21.15.94:0/2952236695 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89b0085ae0 con 0x557641f1c4d0 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89a97fa640 1 -- 172.21.15.94:0/2952236695 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f89bc025050 con 0x557641f1c4d0 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89ca84c740 10 client.17835 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89ca84c740 1 -- 172.21.15.94:0/2952236695 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557641c5d7d0 con 0x557641f1c4d0 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.363+0000 7f89a97fa640 1 -- 172.21.15.94:0/2952236695 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89b0085ae0 con 0x557641f1c4d0 2024-04-21T08:48:41.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.364+0000 7f89a97fa640 1 -- 172.21.15.94:0/2952236695 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f89bc014a10 con 0x557641f1c4d0 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.364+0000 7f89ca84c740 10 client.17835 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.364+0000 7f89ca84c740 10 client.17835 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.364+0000 7f89ca84c740 20 client.17835 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.364+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557641f35c20 0x557641f38090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.364+0000 7f89ca84c740 4 client.17835 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.364+0000 7f89ca84c740 1 -- 172.21.15.94:0/2952236695 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557641e20930 con 0x557641f35c20 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.364+0000 7f89c51fd640 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557641f35c20 0x557641f38090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.364+0000 7f89a97fa640 10 client.17835 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.364+0000 7f89c51fd640 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557641f35c20 0x557641f38090 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b43bf4bec797d2d server_cookie=507a5f64e2319b58 in_seq=0 out_seq=0 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89a97fa640 1 -- 172.21.15.94:0/2952236695 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557641e20930 con 0x557641f35c20 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89a97fa640 10 client.17835 handle_command_reply: tid=0 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 client.17835 shutdown 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 -- 172.21.15.94:0/2952236695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 msgr2=0x557641f252d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f252d0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0104d0 tx=0x7f89bc010990 comp rx=0 tx=0).stop 2024-04-21T08:48:41.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 -- 172.21.15.94:0/2952236695 shutdown_connections 2024-04-21T08:48:41.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557641f35c20 0x557641f38090 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f1ce80 0x557641f22660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89a97fa640 0 client.17835 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:48:41.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f25a00 0x557641f22bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2952236695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f252d0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 -- 172.21.15.94:0/2952236695 >> 172.21.15.94:0/2952236695 conn(0x557641f16ba0 msgr2=0x557641f188f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 -- 172.21.15.94:0/2952236695 shutdown_connections 2024-04-21T08:48:41.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.365+0000 7f89ca84c740 1 -- 172.21.15.94:0/2952236695 wait complete. 2024-04-21T08:48:41.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.366+0000 7f89ca84c740 20 client.17835 trim_cache size 0 max 16384 2024-04-21T08:48:41.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.378+0000 7f89ca84c740 1 Processor -- start 2024-04-21T08:48:41.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89ca84c740 1 -- start start 2024-04-21T08:48:41.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f317d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 0x557641f24d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 0x557641f252d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89ca84c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f89c01bc930 con 0x557641f1c4d0 2024-04-21T08:48:41.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89ca84c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f89c01ba200 con 0x557641f1ce80 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89ca84c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f89c01b8070 con 0x557641f25a00 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c59fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f317d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c59fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f317d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47312/0 (socket says 172.21.15.94:47312) 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c59fe640 1 -- 172.21.15.94:0/1214234110 learned_addr learned my addr 172.21.15.94:0/1214234110 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c59fe640 1 -- 172.21.15.94:0/1214234110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 msgr2=0x557641f252d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c61ff640 1 --2- 172.21.15.94:0/1214234110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 0x557641f252d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c59fe640 1 --2- 172.21.15.94:0/1214234110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 0x557641f252d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c59fe640 1 -- 172.21.15.94:0/1214234110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 msgr2=0x557641f24d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c59fe640 1 --2- 172.21.15.94:0/1214234110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 0x557641f24d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c59fe640 1 -- 172.21.15.94:0/1214234110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557641f1e270 con 0x557641f1c4d0 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c61ff640 1 --2- 172.21.15.94:0/1214234110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 0x557641f252d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89c59fe640 1 --2- 172.21.15.94:0/1214234110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f317d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f89bc014e30 tx=0x7f89bc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767103ea91ade305 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89ab7fe640 1 -- 172.21.15.94:0/1214234110 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc02e070 con 0x557641f1c4d0 2024-04-21T08:48:41.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89ab7fe640 1 -- 172.21.15.94:0/1214234110 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89bc010e90 con 0x557641f1c4d0 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.379+0000 7f89ab7fe640 1 -- 172.21.15.94:0/1214234110 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc01a040 con 0x557641f1c4d0 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 -- 172.21.15.94:0/1214234110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 msgr2=0x557641f317d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 --2- 172.21.15.94:0/1214234110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f317d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f89bc014e30 tx=0x7f89bc00b480 comp rx=0 tx=0).stop 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 -- 172.21.15.94:0/1214234110 shutdown_connections 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 --2- 172.21.15.94:0/1214234110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 0x557641f252d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 --2- 172.21.15.94:0/1214234110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 0x557641f24d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 --2- 172.21.15.94:0/1214234110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f317d0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 -- 172.21.15.94:0/1214234110 >> 172.21.15.94:0/1214234110 conn(0x557641f16ba0 msgr2=0x557641f15c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 -- 172.21.15.94:0/1214234110 shutdown_connections 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 -- 172.21.15.94:0/1214234110 wait complete. 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 -- start start 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.380+0000 7f89ca84c740 1 Processor -- start 2024-04-21T08:48:41.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f1f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 0x557641f1fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89ca84c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 0x557641f9b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89ca84c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f89c01b86c0 con 0x557641f1c4d0 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89ca84c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f89c01b9b30 con 0x557641f1ce80 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89ca84c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f89c01b8d90 con 0x557641f25a00 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89ca84c740 10 client.0 _ll_register_callbacks cb 0x7f89bc00bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c59fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f1f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c59fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f1f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47318/0 (socket says 172.21.15.94:47318) 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c59fe640 1 -- 172.21.15.94:0/2996896697 learned_addr learned my addr 172.21.15.94:0/2996896697 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c61ff640 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 0x557641f9b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c59fe640 1 -- 172.21.15.94:0/2996896697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 msgr2=0x557641f9b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c59fe640 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 0x557641f9b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c59fe640 1 -- 172.21.15.94:0/2996896697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 msgr2=0x557641f1fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c59fe640 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 0x557641f1fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c59fe640 1 -- 172.21.15.94:0/2996896697 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557641f9b710 con 0x557641f1c4d0 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c61ff640 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 0x557641f9b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89c59fe640 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f1f710 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f89bc02c040 tx=0x7f89bc014920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31ff3bc80f89e81 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89a97fa640 1 -- 172.21.15.94:0/2996896697 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc02e070 con 0x557641f1c4d0 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89a97fa640 1 -- 172.21.15.94:0/2996896697 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f89bc016890 con 0x557641f1c4d0 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.381+0000 7f89ca84c740 1 -- 172.21.15.94:0/2996896697 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89bc010e90 con 0x557641f1c4d0 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89a97fa640 1 -- 172.21.15.94:0/2996896697 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f89bc03a050 con 0x557641f1c4d0 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89ca84c740 10 client.17841 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89ca84c740 1 -- 172.21.15.94:0/2996896697 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557641f9c7c0 con 0x557641f1c4d0 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89a97fa640 1 -- 172.21.15.94:0/2996896697 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89bc010e90 con 0x557641f1c4d0 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89a97fa640 1 -- 172.21.15.94:0/2996896697 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f89bc029be0 con 0x557641f1c4d0 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89ca84c740 10 client.17841 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89ca84c740 10 client.17841 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:48:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89ca84c740 20 client.17841 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557641f9cdc0 0x557641f9f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89ca84c740 4 client.17841 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89ca84c740 1 -- 172.21.15.94:0/2996896697 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557641e8ec10 con 0x557641f9cdc0 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.382+0000 7f89c51fd640 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557641f9cdc0 0x557641f9f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89a97fa640 10 client.17841 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89c51fd640 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557641f9cdc0 0x557641f9f1b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c927075d1e1016b4 server_cookie=a1ee583146094474 in_seq=0 out_seq=0 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89a97fa640 1 -- 172.21.15.94:0/2996896697 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557641e8ec10 con 0x557641f9cdc0 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89a97fa640 10 client.17841 handle_command_reply: tid=0 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 client.17841 shutdown 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 -- 172.21.15.94:0/2996896697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 msgr2=0x557641f1f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f1f710 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f89bc02c040 tx=0x7f89bc014920 comp rx=0 tx=0).stop 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 -- 172.21.15.94:0/2996896697 shutdown_connections 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x557641f9cdc0 0x557641f9f1b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557641f25a00 0x557641f9b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557641f1ce80 0x557641f1fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89a97fa640 0 client.17841 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:48:41.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 --2- 172.21.15.94:0/2996896697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x557641f1c4d0 0x557641f1f710 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 -- 172.21.15.94:0/2996896697 >> 172.21.15.94:0/2996896697 conn(0x557641f16ba0 msgr2=0x557641f1d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 -- 172.21.15.94:0/2996896697 shutdown_connections 2024-04-21T08:48:41.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 1 -- 172.21.15.94:0/2996896697 wait complete. 2024-04-21T08:48:41.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.383+0000 7f89ca84c740 20 client.17841 trim_cache size 0 max 16384 2024-04-21T08:48:41.390 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:48:41.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 -- 172.21.15.94:0/1232709200 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89ac0a3d20 msgr2=0x7f89ac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89ac0a3d20 0x7f89ac0a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f89b8006010 tx=0x7f89b8032ca0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 -- 172.21.15.94:0/1232709200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89c010d2c0 msgr2=0x7f89c01a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89c010d2c0 0x7f89c01a36b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f89b000c890 tx=0x7f89b000cd50 comp rx=0 tx=0).stop 2024-04-21T08:48:41.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 -- 172.21.15.94:0/1232709200 shutdown_connections 2024-04-21T08:48:41.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89ac0a3d20 0x7f89ac0a61e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89c0112910 0x7f89c01a7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89c010d2c0 0x7f89c01a36b0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 --2- 172.21.15.94:0/1232709200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89c010c900 0x7f89c01a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 -- 172.21.15.94:0/1232709200 >> 172.21.15.94:0/1232709200 conn(0x7f89c006cfa0 msgr2=0x7f89c010f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 -- 172.21.15.94:0/1232709200 shutdown_connections 2024-04-21T08:48:41.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.384+0000 7f89c7c62640 1 -- 172.21.15.94:0/1232709200 wait complete. 2024-04-21T08:48:41.395 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:41 smithi094 ceph-mon[27988]: pgmap v268: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:48:41.396 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:48:41.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:41 smithi111 ceph-mon[31493]: pgmap v268: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:48:41.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.497+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/3973949081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfd81031d0 msgr2=0x7fcfd81035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.497+0000 7fcfdcfbd640 1 --2- 172.21.15.94:0/3973949081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfd81031d0 0x7fcfd81035b0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009ca0 tx=0x7fcfc802c230 comp rx=0 tx=0).stop 2024-04-21T08:48:41.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.497+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/3973949081 shutdown_connections 2024-04-21T08:48:41.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.497+0000 7fcfdcfbd640 1 --2- 172.21.15.94:0/3973949081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd810c510 0x7fcfd810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.497+0000 7fcfdcfbd640 1 --2- 172.21.15.94:0/3973949081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfd8103af0 0x7fcfd810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.497+0000 7fcfdcfbd640 1 --2- 172.21.15.94:0/3973949081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfd81031d0 0x7fcfd81035b0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.497+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/3973949081 >> 172.21.15.94:0/3973949081 conn(0x7fcfd8077f50 msgr2=0x7fcfd8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.498+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/3973949081 shutdown_connections 2024-04-21T08:48:41.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.498+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/3973949081 wait complete. 2024-04-21T08:48:41.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.498+0000 7fcfdcfbd640 1 Processor -- start 2024-04-21T08:48:41.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.498+0000 7fcfdcfbd640 1 -- start start 2024-04-21T08:48:41.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.498+0000 7fcfdcfbd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfd81031d0 0x7fcfd8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.498+0000 7fcfdcfbd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfd8103af0 0x7fcfd8198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.498+0000 7fcfdcfbd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd810c510 0x7fcfd819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.498+0000 7fcfdcfbd640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81121a0 con 0x7fcfd8103af0 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.498+0000 7fcfdcfbd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8112020 con 0x7fcfd81031d0 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.498+0000 7fcfdcfbd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8112320 con 0x7fcfd810c510 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfd8103af0 0x7fcfd8198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfd8103af0 0x7fcfd8198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47334/0 (socket says 172.21.15.94:47334) 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/910538775 learned_addr learned my addr 172.21.15.94:0/910538775 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd6d9d640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd810c510 0x7fcfd819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd659c640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfd81031d0 0x7fcfd8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/910538775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd810c510 msgr2=0x7fcfd819d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd810c510 0x7fcfd819d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/910538775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfd81031d0 msgr2=0x7fcfd8198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfd81031d0 0x7fcfd8198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/910538775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd819d730 con 0x7fcfd8103af0 2024-04-21T08:48:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfd8103af0 0x7fcfd8198c80 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c890 tx=0x7fcfcc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfc77fe640 1 -- 172.21.15.94:0/910538775 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc004220 con 0x7fcfd8103af0 2024-04-21T08:48:41.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/910538775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd819da20 con 0x7fcfd8103af0 2024-04-21T08:48:41.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfc77fe640 1 -- 172.21.15.94:0/910538775 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfcc0043c0 con 0x7fcfd8103af0 2024-04-21T08:48:41.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/910538775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd81a52b0 con 0x7fcfd8103af0 2024-04-21T08:48:41.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.499+0000 7fcfc77fe640 1 -- 172.21.15.94:0/910538775 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc01d060 con 0x7fcfd8103af0 2024-04-21T08:48:41.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.501+0000 7fcfc77fe640 1 -- 172.21.15.94:0/910538775 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fcfcc0079b0 con 0x7fcfd8103af0 2024-04-21T08:48:41.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.501+0000 7fcfdfbb2740 1 Processor -- start 2024-04-21T08:48:41.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfdfbb2740 1 -- start start 2024-04-21T08:48:41.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f444790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f44d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f44de60 0x55743f44e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55743f3a91f0 con 0x55743f444d60 2024-04-21T08:48:41.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55743f3464b0 con 0x55743f44de60 2024-04-21T08:48:41.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55743f3a51b0 con 0x55743f4443b0 2024-04-21T08:48:41.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfc77fe640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfc00a3d20 0x7fcfc00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f44d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd6d9d640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfc00a3d20 0x7fcfc00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f44d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47338/0 (socket says 172.21.15.94:47338) 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd659c640 1 -- 172.21.15.94:0/4077005708 learned_addr learned my addr 172.21.15.94:0/4077005708 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd6d9d640 1 --2- 172.21.15.94:0/4077005708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f444790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfc77fe640 1 -- 172.21.15.94:0/910538775 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7fcfcc0d0de0 con 0x7fcfd8103af0 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/4077005708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f44de60 0x55743f44e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd659c640 1 -- 172.21.15.94:0/4077005708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 msgr2=0x55743f444790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd659c640 1 --2- 172.21.15.94:0/4077005708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f444790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd659c640 1 -- 172.21.15.94:0/4077005708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f44de60 msgr2=0x55743f44e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd659c640 1 --2- 172.21.15.94:0/4077005708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f44de60 0x55743f44e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd659c640 1 -- 172.21.15.94:0/4077005708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55743f3381b0 con 0x55743f444d60 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.502+0000 7fcfd6d9d640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfc00a3d20 0x7fcfc00a61e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0005d80 tx=0x7fcfd000ca70 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfd659c640 1 --2- 172.21.15.94:0/4077005708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f44d890 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009ca0 tx=0x7fcfc80323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254a04442794b134 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfc4ff9640 1 -- 172.21.15.94:0/4077005708 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8043020 con 0x55743f444d60 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfc4ff9640 1 -- 172.21.15.94:0/4077005708 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfc8003b70 con 0x55743f444d60 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfc4ff9640 1 -- 172.21.15.94:0/4077005708 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8039040 con 0x55743f444d60 2024-04-21T08:48:41.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/4077005708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 msgr2=0x55743f44d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/4077005708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f44d890 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009ca0 tx=0x7fcfc80323c0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/4077005708 shutdown_connections 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/4077005708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f44de60 0x55743f44e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/4077005708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f44d890 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/4077005708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f444790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/4077005708 >> 172.21.15.94:0/4077005708 conn(0x55743f43eba0 msgr2=0x55743f43ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/4077005708 shutdown_connections 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.503+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/4077005708 wait complete. 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfdfbb2740 1 -- start start 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfdfbb2740 1 Processor -- start 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f454200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f451590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f44de60 0x55743f451ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55743f454d40 con 0x55743f444d60 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55743f362500 con 0x55743f44de60 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55743f454ec0 con 0x55743f4443b0 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfdfbb2740 10 client.0 _ll_register_callbacks cb 0x55743f330b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfd5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f451590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfd5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f451590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47344/0 (socket says 172.21.15.94:47344) 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/1158539474 learned_addr learned my addr 172.21.15.94:0/1158539474 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfd659c640 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f454200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfd6d9d640 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f44de60 0x55743f451ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/1158539474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 msgr2=0x55743f454200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f454200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/1158539474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f44de60 msgr2=0x55743f451ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f44de60 0x55743f451ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.504+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/1158539474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55743f0bb3f0 con 0x55743f444d60 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f451590 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc099320 tx=0x7fcfcc09c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88dfefffa88d764c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfbdffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfbdffb640 1 -- 172.21.15.94:0/1158539474 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0a0d00 con 0x55743f444d60 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfbdffb640 1 -- 172.21.15.94:0/1158539474 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7fcfcc0a5130 con 0x55743f444d60 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/1158539474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfc8003b70 con 0x55743f444d60 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfbdffb640 1 -- 172.21.15.94:0/1158539474 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcfcc0a55c0 con 0x55743f444d60 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfdfbb2740 10 client.17853 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/1158539474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55743f1857d0 con 0x55743f444d60 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfbdffb640 1 -- 172.21.15.94:0/1158539474 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfc8003b70 con 0x55743f444d60 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfbdffb640 1 -- 172.21.15.94:0/1158539474 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fcfcc0bbb10 con 0x55743f444d60 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfdfbb2740 10 client.17853 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfdfbb2740 10 client.17853 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:48:41.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfdfbb2740 20 client.17853 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55743f461fa0 0x55743f464390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfdfbb2740 4 client.17853 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/1158539474 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55743f348930 con 0x55743f461fa0 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.505+0000 7fcfd659c640 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55743f461fa0 0x55743f464390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.506+0000 7fcfd659c640 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55743f461fa0 0x55743f464390 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=573e480916decdb0 server_cookie=b9c2c7125e2ece95 in_seq=0 out_seq=0 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.506+0000 7fcfbdffb640 10 client.17853 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfbdffb640 1 -- 172.21.15.94:0/1158539474 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55743f348930 con 0x55743f461fa0 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfbdffb640 10 client.17853 handle_command_reply: tid=0 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 client.17853 shutdown 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/1158539474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 msgr2=0x55743f451590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f451590 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc099320 tx=0x7fcfcc09c370 comp rx=0 tx=0).stop 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/1158539474 shutdown_connections 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55743f461fa0 0x55743f464390 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f44de60 0x55743f451ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfbdffb640 0 client.17853 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:48:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f451590 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/1158539474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f454200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/1158539474 >> 172.21.15.94:0/1158539474 conn(0x55743f43eba0 msgr2=0x55743f4408f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/1158539474 shutdown_connections 2024-04-21T08:48:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/1158539474 wait complete. 2024-04-21T08:48:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.507+0000 7fcfdfbb2740 20 client.17853 trim_cache size 0 max 16384 2024-04-21T08:48:41.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:41 smithi173 ceph-mon[31428]: pgmap v268: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:48:41.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfdfbb2740 1 Processor -- start 2024-04-21T08:48:41.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfdfbb2740 1 -- start start 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f45dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f444d60 0x55743f44c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f44de60 0x55743f44d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81afbd0 con 0x55743f44de60 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81ad4a0 con 0x55743f444d60 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81ab310 con 0x55743f4443b0 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfd6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f44de60 0x55743f44d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfd6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f44de60 0x55743f44d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47356/0 (socket says 172.21.15.94:47356) 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfd6d9d640 1 -- 172.21.15.94:0/3708675109 learned_addr learned my addr 172.21.15.94:0/3708675109 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/3708675109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f444d60 0x55743f44c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.520+0000 7fcfd659c640 1 --2- 172.21.15.94:0/3708675109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f45dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfd6d9d640 1 -- 172.21.15.94:0/3708675109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 msgr2=0x55743f45dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfd6d9d640 1 --2- 172.21.15.94:0/3708675109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f45dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfd6d9d640 1 -- 172.21.15.94:0/3708675109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f444d60 msgr2=0x55743f44c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfd6d9d640 1 --2- 172.21.15.94:0/3708675109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f444d60 0x55743f44c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfd6d9d640 1 -- 172.21.15.94:0/3708675109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55743f464700 con 0x55743f44de60 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfd659c640 1 --2- 172.21.15.94:0/3708675109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f45dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfd6d9d640 1 --2- 172.21.15.94:0/3708675109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f44de60 0x55743f44d440 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000cde0 tx=0x7fcfd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac6f2cbcbc35fbf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfbffff640 1 -- 172.21.15.94:0/3708675109 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0025070 con 0x55743f44de60 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfbffff640 1 -- 172.21.15.94:0/3708675109 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd000c330 con 0x55743f44de60 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfbffff640 1 -- 172.21.15.94:0/3708675109 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd002a8c0 con 0x55743f44de60 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3708675109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f44de60 msgr2=0x55743f44d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/3708675109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f44de60 0x55743f44d440 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000cde0 tx=0x7fcfd0004730 comp rx=0 tx=0).stop 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3708675109 shutdown_connections 2024-04-21T08:48:41.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/3708675109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f44de60 0x55743f44d440 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/3708675109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f444d60 0x55743f44c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/3708675109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f4443b0 0x55743f45dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3708675109 >> 172.21.15.94:0/3708675109 conn(0x55743f43eba0 msgr2=0x55743f44cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3708675109 shutdown_connections 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.521+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3708675109 wait complete. 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfdfbb2740 1 -- start start 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfdfbb2740 1 Processor -- start 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f4443b0 0x55743f453460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f4539a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfdfbb2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f44de60 0x55743f453f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81ab960 con 0x55743f444d60 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81acdd0 con 0x55743f4443b0 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfdfbb2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81ac030 con 0x55743f44de60 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfdfbb2740 10 client.0 _ll_register_callbacks cb 0x7fcfcc09c7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:41.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f4539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f4539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47362/0 (socket says 172.21.15.94:47362) 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/3404580016 learned_addr learned my addr 172.21.15.94:0/3404580016 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd659c640 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f4443b0 0x55743f453460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd6d9d640 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f44de60 0x55743f453f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/3404580016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f44de60 msgr2=0x55743f453f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f44de60 0x55743f453f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/3404580016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f4443b0 msgr2=0x55743f453460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f4443b0 0x55743f453460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd5d9b640 1 -- 172.21.15.94:0/3404580016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55743f4c77a0 con 0x55743f444d60 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd659c640 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f4443b0 0x55743f453460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.522+0000 7fcfd5d9b640 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f4539a0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc09c650 tx=0x7fcfcc0991f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f6d18afe951472 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfbdffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfbdffb640 1 -- 172.21.15.94:0/3404580016 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc098bf0 con 0x55743f444d60 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfbdffb640 1 -- 172.21.15.94:0/3404580016 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7fcfcc0a5130 con 0x55743f444d60 2024-04-21T08:48:41.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3404580016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfd000c330 con 0x55743f444d60 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfbdffb640 1 -- 172.21.15.94:0/3404580016 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcfcc09c490 con 0x55743f444d60 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfdfbb2740 10 client.17859 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3404580016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55743f4c8a10 con 0x55743f444d60 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfbdffb640 1 -- 172.21.15.94:0/3404580016 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd000c330 con 0x55743f444d60 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfbdffb640 1 -- 172.21.15.94:0/3404580016 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fcfcc0870b0 con 0x55743f444d60 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfdfbb2740 10 client.17859 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfdfbb2740 10 client.17859 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfdfbb2740 20 client.17859 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55743f4c91b0 0x55743f4cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfdfbb2740 4 client.17859 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3404580016 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55743f3b6c10 con 0x55743f4c91b0 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.523+0000 7fcfd659c640 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55743f4c91b0 0x55743f4cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfbdffb640 10 client.17859 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfd659c640 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55743f4c91b0 0x55743f4cb5d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=edfb5c2eead878d5 server_cookie=e15004ae938a5f31 in_seq=0 out_seq=0 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfbdffb640 1 -- 172.21.15.94:0/3404580016 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55743f3b6c10 con 0x55743f4c91b0 2024-04-21T08:48:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfbdffb640 10 client.17859 handle_command_reply: tid=0 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 client.17859 shutdown 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3404580016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 msgr2=0x55743f4539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f4539a0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc09c650 tx=0x7fcfcc0991f0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3404580016 shutdown_connections 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55743f44de60 0x55743f453f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55743f4c91b0 0x55743f4cb5d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55743f444d60 0x55743f4539a0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 --2- 172.21.15.94:0/3404580016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55743f4443b0 0x55743f453460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfbdffb640 0 client.17859 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3404580016 >> 172.21.15.94:0/3404580016 conn(0x55743f43eba0 msgr2=0x55743f43dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3404580016 shutdown_connections 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 1 -- 172.21.15.94:0/3404580016 wait complete. 2024-04-21T08:48:41.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.524+0000 7fcfdfbb2740 20 client.17859 trim_cache size 0 max 16384 2024-04-21T08:48:41.531 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/910538775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfc00a3d20 msgr2=0x7fcfc00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfc00a3d20 0x7fcfc00a61e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0005d80 tx=0x7fcfd000ca70 comp rx=0 tx=0).stop 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/910538775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfd8103af0 msgr2=0x7fcfd8198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfd8103af0 0x7fcfd8198c80 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c890 tx=0x7fcfcc00cd50 comp rx=0 tx=0).stop 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/910538775 shutdown_connections 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcfc00a3d20 0x7fcfc00a61e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd810c510 0x7fcfd819d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfd8103af0 0x7fcfd8198c80 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 --2- 172.21.15.94:0/910538775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfd81031d0 0x7fcfd8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/910538775 >> 172.21.15.94:0/910538775 conn(0x7fcfd8077f50 msgr2=0x7fcfd80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/910538775 shutdown_connections 2024-04-21T08:48:41.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.525+0000 7fcfdcfbd640 1 -- 172.21.15.94:0/910538775 wait complete. 2024-04-21T08:48:41.535 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:48:41.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.636+0000 7fc9efb96640 1 -- 172.21.15.94:0/2307946033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9e81021a0 msgr2=0x7fc9e8102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.636+0000 7fc9efb96640 1 --2- 172.21.15.94:0/2307946033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9e81021a0 0x7fc9e8102580 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009900 tx=0x7fc9e402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:41.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.636+0000 7fc9efb96640 1 -- 172.21.15.94:0/2307946033 shutdown_connections 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.636+0000 7fc9efb96640 1 --2- 172.21.15.94:0/2307946033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e8102b50 0x7fc9e8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.636+0000 7fc9efb96640 1 --2- 172.21.15.94:0/2307946033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e8107270 0x7fc9e810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.636+0000 7fc9efb96640 1 --2- 172.21.15.94:0/2307946033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9e81021a0 0x7fc9e8102580 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.636+0000 7fc9efb96640 1 -- 172.21.15.94:0/2307946033 >> 172.21.15.94:0/2307946033 conn(0x7fc9e80fd9c0 msgr2=0x7fc9e80ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.636+0000 7fc9efb96640 1 -- 172.21.15.94:0/2307946033 shutdown_connections 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.636+0000 7fc9efb96640 1 -- 172.21.15.94:0/2307946033 wait complete. 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9efb96640 1 Processor -- start 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9efb96640 1 -- start start 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9efb96640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9e81021a0 0x7fc9e819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9efb96640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e8107270 0x7fc9e819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9efb96640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e81985d0 0x7fc9e8198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9efb96640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81121d0 con 0x7fc9e81021a0 2024-04-21T08:48:41.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9efb96640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e8112050 con 0x7fc9e81985d0 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9efb96640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e8112350 con 0x7fc9e8107270 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed932640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9e81021a0 0x7fc9e819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed932640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9e81021a0 0x7fc9e819e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47374/0 (socket says 172.21.15.94:47374) 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed932640 1 -- 172.21.15.94:0/3402637448 learned_addr learned my addr 172.21.15.94:0/3402637448 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed131640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e8107270 0x7fc9e819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ee133640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e81985d0 0x7fc9e8198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed932640 1 -- 172.21.15.94:0/3402637448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e8107270 msgr2=0x7fc9e819e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed932640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e8107270 0x7fc9e819e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed932640 1 -- 172.21.15.94:0/3402637448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e81985d0 msgr2=0x7fc9e8198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed932640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e81985d0 0x7fc9e8198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed932640 1 -- 172.21.15.94:0/3402637448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e81990f0 con 0x7fc9e81021a0 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed131640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e8107270 0x7fc9e819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ee133640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e81985d0 0x7fc9e8198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.637+0000 7fc9ed932640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9e81021a0 0x7fc9e819e420 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4005940 tx=0x7fc9e402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.638+0000 7fc9deffd640 1 -- 172.21.15.94:0/3402637448 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4043020 con 0x7fc9e81021a0 2024-04-21T08:48:41.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.638+0000 7fc9deffd640 1 -- 172.21.15.94:0/3402637448 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e402cbe0 con 0x7fc9e81021a0 2024-04-21T08:48:41.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.638+0000 7fc9efb96640 1 -- 172.21.15.94:0/3402637448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e8199380 con 0x7fc9e81021a0 2024-04-21T08:48:41.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.638+0000 7fc9deffd640 1 -- 172.21.15.94:0/3402637448 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4003bf0 con 0x7fc9e81021a0 2024-04-21T08:48:41.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.638+0000 7fc9efb96640 1 -- 172.21.15.94:0/3402637448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e81a55d0 con 0x7fc9e81021a0 2024-04-21T08:48:41.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.640+0000 7fc9deffd640 1 -- 172.21.15.94:0/3402637448 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fc9e4046020 con 0x7fc9e81021a0 2024-04-21T08:48:41.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.640+0000 7fc9f2784740 1 Processor -- start 2024-04-21T08:48:41.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.640+0000 7fc9f2784740 1 -- start start 2024-04-21T08:48:41.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ec4d0 0x5602bf8ec8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9deffd640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc9d40a3d20 0x7fc9d40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ece80 0x5602bf8f5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8f5a00 0x5602bf8f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9f2784740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5602bf8511f0 con 0x5602bf8ece80 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9f2784740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5602bf7ee4b0 con 0x5602bf8ec4d0 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9f2784740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5602bf84d1b0 con 0x5602bf8f5a00 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed932640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc9d40a3d20 0x7fc9d40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed131640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ece80 0x5602bf8f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed131640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ece80 0x5602bf8f5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47390/0 (socket says 172.21.15.94:47390) 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed131640 1 -- 172.21.15.94:0/2123408019 learned_addr learned my addr 172.21.15.94:0/2123408019 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9deffd640 1 -- 172.21.15.94:0/3402637448 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7fc9e40f2e50 con 0x7fc9e81021a0 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ee133640 1 --2- 172.21.15.94:0/2123408019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ec4d0 0x5602bf8ec8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ee133640 1 --2- 172.21.15.94:0/2123408019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8f5a00 0x5602bf8f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed131640 1 -- 172.21.15.94:0/2123408019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8f5a00 msgr2=0x5602bf8f7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed131640 1 --2- 172.21.15.94:0/2123408019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8f5a00 0x5602bf8f7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed131640 1 -- 172.21.15.94:0/2123408019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ec4d0 msgr2=0x5602bf8ec8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed131640 1 --2- 172.21.15.94:0/2123408019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ec4d0 0x5602bf8ec8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed131640 1 -- 172.21.15.94:0/2123408019 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602bf7e01b0 con 0x5602bf8ece80 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed131640 1 --2- 172.21.15.94:0/2123408019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ece80 0x5602bf8f5430 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800bcd0 tx=0x7fc9d800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8d86dba2ec4ffb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9ed932640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc9d40a3d20 0x7fc9d40a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40f0330 tx=0x7fc9e408d480 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9d3fff640 1 -- 172.21.15.94:0/2123408019 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8015070 con 0x5602bf8ece80 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9d3fff640 1 -- 172.21.15.94:0/2123408019 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9d801a910 con 0x5602bf8ece80 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.641+0000 7fc9d3fff640 1 -- 172.21.15.94:0/2123408019 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d80118c0 con 0x5602bf8ece80 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.642+0000 7fc9f2784740 1 -- 172.21.15.94:0/2123408019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ece80 msgr2=0x5602bf8f5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.642+0000 7fc9f2784740 1 --2- 172.21.15.94:0/2123408019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ece80 0x5602bf8f5430 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800bcd0 tx=0x7fc9d800f730 comp rx=0 tx=0).stop 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.642+0000 7fc9f2784740 1 -- 172.21.15.94:0/2123408019 shutdown_connections 2024-04-21T08:48:41.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.642+0000 7fc9f2784740 1 --2- 172.21.15.94:0/2123408019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8f5a00 0x5602bf8f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.642+0000 7fc9f2784740 1 --2- 172.21.15.94:0/2123408019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ece80 0x5602bf8f5430 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.642+0000 7fc9f2784740 1 --2- 172.21.15.94:0/2123408019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ec4d0 0x5602bf8ec8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.642+0000 7fc9f2784740 1 -- 172.21.15.94:0/2123408019 >> 172.21.15.94:0/2123408019 conn(0x5602bf8e6ba0 msgr2=0x5602bf8e6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.642+0000 7fc9f2784740 1 -- 172.21.15.94:0/2123408019 shutdown_connections 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.642+0000 7fc9f2784740 1 -- 172.21.15.94:0/2123408019 wait complete. 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9f2784740 1 -- start start 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9f2784740 1 Processor -- start 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ec4d0 0x5602bf8f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ece80 0x5602bf8f2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8f5a00 0x5602bf8f2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9f2784740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5602bf8f8970 con 0x5602bf8ec4d0 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9f2784740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5602bf80a500 con 0x5602bf8f5a00 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9f2784740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5602bf8f8af0 con 0x5602bf8ece80 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ee133640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ec4d0 0x5602bf8f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9f2784740 10 client.0 _ll_register_callbacks cb 0x5602bf7d8b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:41.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ee133640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ec4d0 0x5602bf8f52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47394/0 (socket says 172.21.15.94:47394) 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ee133640 1 -- 172.21.15.94:0/2273701972 learned_addr learned my addr 172.21.15.94:0/2273701972 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ed131640 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ece80 0x5602bf8f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ee133640 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8f5a00 0x5602bf8f2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ee133640 1 -- 172.21.15.94:0/2273701972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ece80 msgr2=0x5602bf8f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ee133640 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ece80 0x5602bf8f2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ee133640 1 -- 172.21.15.94:0/2273701972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8f5a00 msgr2=0x5602bf8f2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ee133640 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8f5a00 0x5602bf8f2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ee133640 1 -- 172.21.15.94:0/2273701972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602bf5633f0 con 0x5602bf8ec4d0 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9ee133640 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ec4d0 0x5602bf8f52c0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0014b90 tx=0x7fc9e0016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1e0b5afe819a1f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.643+0000 7fc9d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9d17fa640 1 -- 172.21.15.94:0/2273701972 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0011e30 con 0x5602bf8ec4d0 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9d17fa640 1 -- 172.21.15.94:0/2273701972 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7fc9e0004b20 con 0x5602bf8ec4d0 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9f2784740 1 -- 172.21.15.94:0/2273701972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9d801a910 con 0x5602bf8ec4d0 2024-04-21T08:48:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9d17fa640 1 -- 172.21.15.94:0/2273701972 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc9e0026050 con 0x5602bf8ec4d0 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9f2784740 10 client.17871 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9f2784740 1 -- 172.21.15.94:0/2273701972 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5602bf62d7d0 con 0x5602bf8ec4d0 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9d17fa640 1 -- 172.21.15.94:0/2273701972 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9d801a910 con 0x5602bf8ec4d0 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9d17fa640 1 -- 172.21.15.94:0/2273701972 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc9e001f3a0 con 0x5602bf8ec4d0 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9f2784740 10 client.17871 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9f2784740 10 client.17871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9f2784740 20 client.17871 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9f2784740 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5602bf905cb0 0x5602bf9080e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9f2784740 4 client.17871 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9f2784740 1 -- 172.21.15.94:0/2273701972 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602bf7f0930 con 0x5602bf905cb0 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.644+0000 7fc9ed131640 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5602bf905cb0 0x5602bf9080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.645+0000 7fc9d17fa640 10 client.17871 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.645+0000 7fc9ed131640 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5602bf905cb0 0x5602bf9080e0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd4dc867a60c477f server_cookie=ed3ff0bdccb0966b in_seq=0 out_seq=0 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9d17fa640 1 -- 172.21.15.94:0/2273701972 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602bf7f0930 con 0x5602bf905cb0 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9d17fa640 10 client.17871 handle_command_reply: tid=0 2024-04-21T08:48:41.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 client.17871 shutdown 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 -- 172.21.15.94:0/2273701972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ec4d0 msgr2=0x5602bf8f52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ec4d0 0x5602bf8f52c0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0014b90 tx=0x7fc9e0016040 comp rx=0 tx=0).stop 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 -- 172.21.15.94:0/2273701972 shutdown_connections 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5602bf905cb0 0x5602bf9080e0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ece80 0x5602bf8f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8f5a00 0x5602bf8f2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9d17fa640 0 client.17871 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 --2- 172.21.15.94:0/2273701972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8ec4d0 0x5602bf8f52c0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 -- 172.21.15.94:0/2273701972 >> 172.21.15.94:0/2273701972 conn(0x5602bf8e6ba0 msgr2=0x5602bf8e88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 -- 172.21.15.94:0/2273701972 shutdown_connections 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 1 -- 172.21.15.94:0/2273701972 wait complete. 2024-04-21T08:48:41.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.646+0000 7fc9f2784740 20 client.17871 trim_cache size 0 max 16384 2024-04-21T08:48:41.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9f2784740 1 Processor -- start 2024-04-21T08:48:41.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9f2784740 1 -- start start 2024-04-21T08:48:41.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ec4d0 0x5602bf901860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 0x5602bf8f4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9f2784740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81b1d30 con 0x5602bf8f5a00 2024-04-21T08:48:41.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9f2784740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81af600 con 0x5602bf8ece80 2024-04-21T08:48:41.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9f2784740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81ad470 con 0x5602bf8ec4d0 2024-04-21T08:48:41.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9ee133640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9ee133640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8f52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47410/0 (socket says 172.21.15.94:47410) 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.659+0000 7fc9ee133640 1 -- 172.21.15.94:0/3328745495 learned_addr learned my addr 172.21.15.94:0/3328745495 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9ed131640 1 --2- 172.21.15.94:0/3328745495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 0x5602bf8f4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9ee133640 1 -- 172.21.15.94:0/3328745495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ec4d0 msgr2=0x5602bf901860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9ee133640 1 --2- 172.21.15.94:0/3328745495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ec4d0 0x5602bf901860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9ee133640 1 -- 172.21.15.94:0/3328745495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 msgr2=0x5602bf8f4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9ee133640 1 --2- 172.21.15.94:0/3328745495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 0x5602bf8f4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9ee133640 1 -- 172.21.15.94:0/3328745495 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602bf904850 con 0x5602bf8f5a00 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9ed131640 1 --2- 172.21.15.94:0/3328745495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 0x5602bf8f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9ee133640 1 --2- 172.21.15.94:0/3328745495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8f52c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc9e001f7f0 tx=0x7fc9e0009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b9bb66fe99ace9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9d37fe640 1 -- 172.21.15.94:0/3328745495 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e002e070 con 0x5602bf8f5a00 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9d37fe640 1 -- 172.21.15.94:0/3328745495 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e001f3a0 con 0x5602bf8f5a00 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9d37fe640 1 -- 172.21.15.94:0/3328745495 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e001fcb0 con 0x5602bf8f5a00 2024-04-21T08:48:41.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9f2784740 1 -- 172.21.15.94:0/3328745495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 msgr2=0x5602bf8f52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9f2784740 1 --2- 172.21.15.94:0/3328745495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8f52c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc9e001f7f0 tx=0x7fc9e0009fd0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.660+0000 7fc9f2784740 1 -- 172.21.15.94:0/3328745495 shutdown_connections 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 --2- 172.21.15.94:0/3328745495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8f52c0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 --2- 172.21.15.94:0/3328745495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 0x5602bf8f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 --2- 172.21.15.94:0/3328745495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ec4d0 0x5602bf901860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 -- 172.21.15.94:0/3328745495 >> 172.21.15.94:0/3328745495 conn(0x5602bf8e6ba0 msgr2=0x5602bf8e5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 -- 172.21.15.94:0/3328745495 shutdown_connections 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 -- 172.21.15.94:0/3328745495 wait complete. 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 -- start start 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 Processor -- start 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ec4d0 0x5602bf8f9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 0x5602bf8f9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8fa410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81adac0 con 0x5602bf8f5a00 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81aef30 con 0x5602bf8ece80 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81ae190 con 0x5602bf8ec4d0 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9f2784740 10 client.0 _ll_register_callbacks cb 0x7fc9e00147d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9ee133640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8fa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9ee133640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8fa410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47424/0 (socket says 172.21.15.94:47424) 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9ee133640 1 -- 172.21.15.94:0/3599753520 learned_addr learned my addr 172.21.15.94:0/3599753520 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.661+0000 7fc9ee133640 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ec4d0 0x5602bf8f9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9ee133640 1 -- 172.21.15.94:0/3599753520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ec4d0 msgr2=0x5602bf8f9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9ee133640 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ec4d0 0x5602bf8f9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9ed131640 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 0x5602bf8f9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9ee133640 1 -- 172.21.15.94:0/3599753520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 msgr2=0x5602bf8f9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9ee133640 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 0x5602bf8f9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9ee133640 1 -- 172.21.15.94:0/3599753520 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602bf96b470 con 0x5602bf8f5a00 2024-04-21T08:48:41.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9ed131640 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 0x5602bf8f9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9ee133640 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8fa410 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0002230 tx=0x7fc9e0034040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da622d5e5d9be950 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9d17fa640 1 -- 172.21.15.94:0/3599753520 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e002e070 con 0x5602bf8f5a00 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9f2784740 1 -- 172.21.15.94:0/3599753520 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9e001f3a0 con 0x5602bf8f5a00 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9d17fa640 1 -- 172.21.15.94:0/3599753520 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7fc9e0004c40 con 0x5602bf8f5a00 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9d17fa640 1 -- 172.21.15.94:0/3599753520 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc9e0042050 con 0x5602bf8f5a00 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9f2784740 10 client.17877 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9f2784740 1 -- 172.21.15.94:0/3599753520 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5602bf96c680 con 0x5602bf8f5a00 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9d17fa640 1 -- 172.21.15.94:0/3599753520 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e001f3a0 con 0x5602bf8f5a00 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9d17fa640 1 -- 172.21.15.94:0/3599753520 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc9e003d030 con 0x5602bf8f5a00 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9f2784740 10 client.17877 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9f2784740 10 client.17877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9f2784740 20 client.17877 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9f2784740 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5602bf96cce0 0x5602bf96f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9f2784740 4 client.17877 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.662+0000 7fc9f2784740 1 -- 172.21.15.94:0/3599753520 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5602bf85ec10 con 0x5602bf96cce0 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.663+0000 7fc9ed131640 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5602bf96cce0 0x5602bf96f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.663+0000 7fc9d17fa640 10 client.17877 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.663+0000 7fc9ed131640 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5602bf96cce0 0x5602bf96f0d0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a483ab98eef4953 server_cookie=effb560e7bb413fd in_seq=0 out_seq=0 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.663+0000 7fc9d17fa640 1 -- 172.21.15.94:0/3599753520 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5602bf85ec10 con 0x5602bf96cce0 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.663+0000 7fc9d17fa640 10 client.17877 handle_command_reply: tid=0 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.663+0000 7fc9f2784740 1 client.17877 shutdown 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 1 -- 172.21.15.94:0/3599753520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 msgr2=0x5602bf8fa410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8fa410 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0002230 tx=0x7fc9e0034040 comp rx=0 tx=0).stop 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 1 -- 172.21.15.94:0/3599753520 shutdown_connections 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5602bf96cce0 0x5602bf96f0d0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5602bf8f5a00 0x5602bf8fa410 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602bf8ece80 0x5602bf8f9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9d17fa640 0 client.17877 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 1 --2- 172.21.15.94:0/3599753520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602bf8ec4d0 0x5602bf8f9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 1 -- 172.21.15.94:0/3599753520 >> 172.21.15.94:0/3599753520 conn(0x5602bf8e6ba0 msgr2=0x5602bf8ed2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 1 -- 172.21.15.94:0/3599753520 shutdown_connections 2024-04-21T08:48:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 1 -- 172.21.15.94:0/3599753520 wait complete. 2024-04-21T08:48:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9f2784740 20 client.17877 trim_cache size 0 max 16384 2024-04-21T08:48:41.670 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9efb96640 1 -- 172.21.15.94:0/3402637448 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc9d40a3d20 msgr2=0x7fc9d40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9efb96640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc9d40a3d20 0x7fc9d40a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40f0330 tx=0x7fc9e408d480 comp rx=0 tx=0).stop 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9efb96640 1 -- 172.21.15.94:0/3402637448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9e81021a0 msgr2=0x7fc9e819e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.664+0000 7fc9efb96640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9e81021a0 0x7fc9e819e420 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4005940 tx=0x7fc9e402c7e0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.665+0000 7fc9efb96640 1 -- 172.21.15.94:0/3402637448 shutdown_connections 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.665+0000 7fc9efb96640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc9d40a3d20 0x7fc9d40a61e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.665+0000 7fc9efb96640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e8107270 0x7fc9e819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.665+0000 7fc9efb96640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e81985d0 0x7fc9e8198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.665+0000 7fc9efb96640 1 --2- 172.21.15.94:0/3402637448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9e81021a0 0x7fc9e819e420 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.665+0000 7fc9efb96640 1 -- 172.21.15.94:0/3402637448 >> 172.21.15.94:0/3402637448 conn(0x7fc9e80fd9c0 msgr2=0x7fc9e80fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.665+0000 7fc9efb96640 1 -- 172.21.15.94:0/3402637448 shutdown_connections 2024-04-21T08:48:41.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:41.665+0000 7fc9efb96640 1 -- 172.21.15.94:0/3402637448 wait complete. 2024-04-21T08:48:42.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:42 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2988577701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:42.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:42 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2988577701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:42.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:42 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2988577701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:42.675 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:48:42.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:42.676 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:48:42.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.793+0000 7f5581796640 1 -- 172.21.15.94:0/2976367035 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557c10e900 msgr2=0x7f557c110cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.793+0000 7f5581796640 1 --2- 172.21.15.94:0/2976367035 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557c10e900 0x7f557c110cf0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f5574009ca0 tx=0x7f557402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.793+0000 7f5581796640 1 -- 172.21.15.94:0/2976367035 shutdown_connections 2024-04-21T08:48:42.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.793+0000 7f5581796640 1 --2- 172.21.15.94:0/2976367035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f557c101670 0x7f557c10e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.793+0000 7f5581796640 1 --2- 172.21.15.94:0/2976367035 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557c10e900 0x7f557c110cf0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.793+0000 7f5581796640 1 --2- 172.21.15.94:0/2976367035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f557c100d50 0x7f557c101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.793+0000 7f5581796640 1 -- 172.21.15.94:0/2976367035 >> 172.21.15.94:0/2976367035 conn(0x7f557c077f50 msgr2=0x7f557c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.794+0000 7f5581796640 1 -- 172.21.15.94:0/2976367035 shutdown_connections 2024-04-21T08:48:42.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.794+0000 7f5581796640 1 -- 172.21.15.94:0/2976367035 wait complete. 2024-04-21T08:48:42.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.794+0000 7f5581796640 1 Processor -- start 2024-04-21T08:48:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.794+0000 7f5581796640 1 -- start start 2024-04-21T08:48:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5581796640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f557c100d50 0x7f557c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5581796640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f557c101670 0x7f557c1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5581796640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557c10e900 0x7f557c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5581796640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f557c1143c0 con 0x7f557c10e900 2024-04-21T08:48:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5581796640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f557c114240 con 0x7f557c100d50 2024-04-21T08:48:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5581796640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f557c114540 con 0x7f557c101670 2024-04-21T08:48:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557c10e900 0x7f557c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557c10e900 0x7f557c19a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47446/0 (socket says 172.21.15.94:47446) 2024-04-21T08:48:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557b7fe640 1 -- 172.21.15.94:0/301060458 learned_addr learned my addr 172.21.15.94:0/301060458 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557a7fc640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f557c101670 0x7f557c1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557affd640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f557c100d50 0x7f557c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557b7fe640 1 -- 172.21.15.94:0/301060458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f557c101670 msgr2=0x7f557c1a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557b7fe640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f557c101670 0x7f557c1a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557b7fe640 1 -- 172.21.15.94:0/301060458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f557c100d50 msgr2=0x7f557c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557b7fe640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f557c100d50 0x7f557c1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557b7fe640 1 -- 172.21.15.94:0/301060458 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c19afe0 con 0x7f557c10e900 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f557b7fe640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557c10e900 0x7f557c19a720 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f557402c700 tx=0x7f5574034650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5563fff640 1 -- 172.21.15.94:0/301060458 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574041020 con 0x7f557c10e900 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5563fff640 1 -- 172.21.15.94:0/301060458 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5574034ca0 con 0x7f557c10e900 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5581796640 1 -- 172.21.15.94:0/301060458 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557c19b270 con 0x7f557c10e900 2024-04-21T08:48:42.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5563fff640 1 -- 172.21.15.94:0/301060458 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574039040 con 0x7f557c10e900 2024-04-21T08:48:42.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.795+0000 7f5581796640 1 -- 172.21.15.94:0/301060458 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f557c1a7400 con 0x7f557c10e900 2024-04-21T08:48:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.798+0000 7f5563fff640 1 -- 172.21.15.94:0/301060458 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5574044020 con 0x7f557c10e900 2024-04-21T08:48:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.798+0000 7f5584383740 1 Processor -- start 2024-04-21T08:48:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.798+0000 7f5584383740 1 -- start start 2024-04-21T08:48:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.798+0000 7f5584383740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x559431077cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.798+0000 7f5584383740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x559431080840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f5584383740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x559431083260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f5584383740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x559430fdbf00 con 0x5594310778e0 2024-04-21T08:48:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f5584383740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559430f79590 con 0x559431080e10 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f5584383740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559430fd7ec0 con 0x559431078290 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f5563fff640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f55640a3d20 0x7f55640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x559431077cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x559431077cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47462/0 (socket says 172.21.15.94:47462) 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557a7fc640 1 -- 172.21.15.94:0/169739585 learned_addr learned my addr 172.21.15.94:0/169739585 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557affd640 1 --2- 172.21.15.94:0/169739585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x559431080840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557b7fe640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f55640a3d20 0x7f55640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557a7fc640 1 --2- 172.21.15.94:0/169739585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x559431083260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557a7fc640 1 -- 172.21.15.94:0/169739585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 msgr2=0x559431080840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557a7fc640 1 --2- 172.21.15.94:0/169739585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x559431080840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557a7fc640 1 -- 172.21.15.94:0/169739585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 msgr2=0x559431083260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f5563fff640 1 -- 172.21.15.94:0/301060458 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f55740f2840 con 0x7f557c10e900 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557a7fc640 1 --2- 172.21.15.94:0/169739585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x559431083260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557a7fc640 1 -- 172.21.15.94:0/169739585 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559430db67e0 con 0x5594310778e0 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557a7fc640 1 --2- 172.21.15.94:0/169739585 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x559431077cc0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f55700135b0 tx=0x7f5570005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865d93eb39c0584c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f557b7fe640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f55640a3d20 0x7f55640a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f55740be160 tx=0x7f55740095c0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f55617fa640 1 -- 172.21.15.94:0/169739585 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5570004050 con 0x5594310778e0 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f55617fa640 1 -- 172.21.15.94:0/169739585 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55700054d0 con 0x5594310778e0 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.799+0000 7f55617fa640 1 -- 172.21.15.94:0/169739585 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5570004af0 con 0x5594310778e0 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.800+0000 7f5584383740 1 -- 172.21.15.94:0/169739585 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 msgr2=0x559431077cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.800+0000 7f5584383740 1 --2- 172.21.15.94:0/169739585 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x559431077cc0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f55700135b0 tx=0x7f5570005850 comp rx=0 tx=0).stop 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.800+0000 7f5584383740 1 -- 172.21.15.94:0/169739585 shutdown_connections 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.800+0000 7f5584383740 1 --2- 172.21.15.94:0/169739585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x559431080840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.800+0000 7f5584383740 1 --2- 172.21.15.94:0/169739585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x559431083260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.800+0000 7f5584383740 1 --2- 172.21.15.94:0/169739585 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x559431077cc0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.800+0000 7f5584383740 1 -- 172.21.15.94:0/169739585 >> 172.21.15.94:0/169739585 conn(0x559431071fb0 msgr2=0x559431072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.800+0000 7f5584383740 1 -- 172.21.15.94:0/169739585 shutdown_connections 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.800+0000 7f5584383740 1 -- 172.21.15.94:0/169739585 wait complete. 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.800+0000 7f5584383740 1 -- start start 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f5584383740 1 Processor -- start 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f5584383740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f5584383740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x55943107e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f5584383740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x55943107e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f5584383740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x559431083d80 con 0x5594310778e0 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f5584383740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559430f95580 con 0x559431080e10 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f5584383740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559431083f00 con 0x559431078290 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f5584383740 10 client.0 _ll_register_callbacks cb 0x559430f9b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47470/0 (socket says 172.21.15.94:47470) 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557a7fc640 1 -- 172.21.15.94:0/552167463 learned_addr learned my addr 172.21.15.94:0/552167463 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557affd640 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x55943107e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557a7fc640 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x55943107e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557a7fc640 1 -- 172.21.15.94:0/552167463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 msgr2=0x55943107e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557a7fc640 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x55943107e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557a7fc640 1 -- 172.21.15.94:0/552167463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 msgr2=0x55943107e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557a7fc640 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x55943107e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557a7fc640 1 -- 172.21.15.94:0/552167463 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559430e16f70 con 0x5594310778e0 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f557a7fc640 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107ddc0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f55700045a0 tx=0x7f5570005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541755b9bf9e529f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.801+0000 7f555eff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f555eff5640 1 -- 172.21.15.94:0/552167463 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5570004c20 con 0x5594310778e0 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f555eff5640 1 -- 172.21.15.94:0/552167463 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f5570004dc0 con 0x5594310778e0 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f5584383740 1 -- 172.21.15.94:0/552167463 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55700054d0 con 0x5594310778e0 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f555eff5640 1 -- 172.21.15.94:0/552167463 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f557000f500 con 0x5594310778e0 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f5584383740 10 client.17889 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f5584383740 1 -- 172.21.15.94:0/552167463 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559430cae170 con 0x5594310778e0 2024-04-21T08:48:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f555eff5640 1 -- 172.21.15.94:0/552167463 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55700054d0 con 0x5594310778e0 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f555eff5640 1 -- 172.21.15.94:0/552167463 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f557002c020 con 0x5594310778e0 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f5584383740 10 client.17889 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f5584383740 10 client.17889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f5584383740 20 client.17889 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f5584383740 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559431091280 0x559431093670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f5584383740 4 client.17889 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f5584383740 1 -- 172.21.15.94:0/552167463 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559430fe9920 con 0x559431091280 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.802+0000 7f557affd640 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559431091280 0x559431093670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.803+0000 7f555eff5640 10 client.17889 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.803+0000 7f557affd640 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559431091280 0x559431093670 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dc96a45ed9c5381 server_cookie=c78244e3d4ebb165 in_seq=0 out_seq=0 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f555eff5640 1 -- 172.21.15.94:0/552167463 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559430fe9920 con 0x559431091280 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f555eff5640 10 client.17889 handle_command_reply: tid=0 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 client.17889 shutdown 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 -- 172.21.15.94:0/552167463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 msgr2=0x55943107ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107ddc0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f55700045a0 tx=0x7f5570005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 -- 172.21.15.94:0/552167463 shutdown_connections 2024-04-21T08:48:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x559431091280 0x559431093670 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x55943107e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f555eff5640 0 client.17889 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x55943107e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 --2- 172.21.15.94:0/552167463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107ddc0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 -- 172.21.15.94:0/552167463 >> 172.21.15.94:0/552167463 conn(0x559431071fb0 msgr2=0x559431073d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 -- 172.21.15.94:0/552167463 shutdown_connections 2024-04-21T08:48:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 1 -- 172.21.15.94:0/552167463 wait complete. 2024-04-21T08:48:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.804+0000 7f5584383740 20 client.17889 trim_cache size 0 max 16384 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f5584383740 1 Processor -- start 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f5584383740 1 -- start start 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f5584383740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5594310778e0 0x55943108cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f5584383740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559431078290 0x55943107d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f5584383740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431080e10 0x55943107e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f5584383740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f557c1b4080 con 0x559431078290 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f5584383740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f557c1b1950 con 0x5594310778e0 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f5584383740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f557c1af7c0 con 0x559431080e10 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f557affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559431078290 0x55943107d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f557affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559431078290 0x55943107d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47486/0 (socket says 172.21.15.94:47486) 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.821+0000 7f557affd640 1 -- 172.21.15.94:0/493225182 learned_addr learned my addr 172.21.15.94:0/493225182 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:42.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f557affd640 1 -- 172.21.15.94:0/493225182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431080e10 msgr2=0x55943107e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f557a7fc640 1 --2- 172.21.15.94:0/493225182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431080e10 0x55943107e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f557affd640 1 --2- 172.21.15.94:0/493225182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431080e10 0x55943107e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f557affd640 1 -- 172.21.15.94:0/493225182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5594310778e0 msgr2=0x55943108cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f557affd640 1 --2- 172.21.15.94:0/493225182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5594310778e0 0x55943108cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f557affd640 1 -- 172.21.15.94:0/493225182 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559431079560 con 0x559431078290 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f557a7fc640 1 --2- 172.21.15.94:0/493225182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431080e10 0x55943107e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f557affd640 1 --2- 172.21.15.94:0/493225182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559431078290 0x55943107d5b0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f556c009d20 tx=0x7f556c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d2e7fb5bb03d6e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f5560ff9640 1 -- 172.21.15.94:0/493225182 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c01f430 con 0x559431078290 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f5560ff9640 1 -- 172.21.15.94:0/493225182 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f556c012450 con 0x559431078290 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.822+0000 7f5560ff9640 1 -- 172.21.15.94:0/493225182 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0116c0 con 0x559431078290 2024-04-21T08:48:42.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 -- 172.21.15.94:0/493225182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559431078290 msgr2=0x55943107d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 --2- 172.21.15.94:0/493225182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559431078290 0x55943107d5b0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f556c009d20 tx=0x7f556c005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 -- 172.21.15.94:0/493225182 shutdown_connections 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 --2- 172.21.15.94:0/493225182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431080e10 0x55943107e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 --2- 172.21.15.94:0/493225182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x559431078290 0x55943107d5b0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 --2- 172.21.15.94:0/493225182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5594310778e0 0x55943108cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 -- 172.21.15.94:0/493225182 >> 172.21.15.94:0/493225182 conn(0x559431071fb0 msgr2=0x55943107dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 -- 172.21.15.94:0/493225182 shutdown_connections 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 -- 172.21.15.94:0/493225182 wait complete. 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 -- start start 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 Processor -- start 2024-04-21T08:48:42.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x55943107b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x5594310f6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f557c1afe10 con 0x5594310778e0 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f557c1b1280 con 0x559431080e10 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f557c1b04e0 con 0x559431078290 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f5584383740 10 client.0 _ll_register_callbacks cb 0x7f556c009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f557a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f557a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47490/0 (socket says 172.21.15.94:47490) 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f557a7fc640 1 -- 172.21.15.94:0/2632243882 learned_addr learned my addr 172.21.15.94:0/2632243882 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.823+0000 7f557affd640 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x55943107b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f557a7fc640 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x5594310f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f557a7fc640 1 -- 172.21.15.94:0/2632243882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 msgr2=0x55943107b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f557a7fc640 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x55943107b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f557a7fc640 1 -- 172.21.15.94:0/2632243882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 msgr2=0x5594310f6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f557a7fc640 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x5594310f6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f557a7fc640 1 -- 172.21.15.94:0/2632243882 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594310f6b90 con 0x5594310778e0 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f557a7fc640 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107aec0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f5570005440 tx=0x7f5570009270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60eae6d23840109 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f555eff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f555eff5640 1 -- 172.21.15.94:0/2632243882 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557001fd40 con 0x5594310778e0 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f555eff5640 1 -- 172.21.15.94:0/2632243882 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f557001f460 con 0x5594310778e0 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f5584383740 1 -- 172.21.15.94:0/2632243882 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f556c012450 con 0x5594310778e0 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f555eff5640 1 -- 172.21.15.94:0/2632243882 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5570004600 con 0x5594310778e0 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f5584383740 10 client.17895 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f5584383740 1 -- 172.21.15.94:0/2632243882 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5594310f7ce0 con 0x5594310778e0 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f555eff5640 1 -- 172.21.15.94:0/2632243882 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f556c012450 con 0x5594310778e0 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.824+0000 7f555eff5640 1 -- 172.21.15.94:0/2632243882 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f557002c040 con 0x5594310778e0 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.825+0000 7f5584383740 10 client.17895 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.825+0000 7f5584383740 10 client.17895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.825+0000 7f5584383740 20 client.17895 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:42.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.825+0000 7f5584383740 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5594310f8240 0x5594310fa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.825+0000 7f5584383740 4 client.17895 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:48:42.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.825+0000 7f5584383740 1 -- 172.21.15.94:0/2632243882 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559430fe3790 con 0x5594310f8240 2024-04-21T08:48:42.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.825+0000 7f557affd640 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5594310f8240 0x5594310fa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.825+0000 7f555eff5640 10 client.17895 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:42.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.825+0000 7f557affd640 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5594310f8240 0x5594310fa660 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eb25bc5a4874790 server_cookie=6cac1c687c4b676c in_seq=0 out_seq=0 2024-04-21T08:48:42.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.826+0000 7f555eff5640 1 -- 172.21.15.94:0/2632243882 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559430fe3790 con 0x5594310f8240 2024-04-21T08:48:42.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.826+0000 7f555eff5640 10 client.17895 handle_command_reply: tid=0 2024-04-21T08:48:42.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.826+0000 7f5584383740 1 client.17895 shutdown 2024-04-21T08:48:42.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 1 -- 172.21.15.94:0/2632243882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 msgr2=0x55943107aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107aec0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f5570005440 tx=0x7f5570009270 comp rx=0 tx=0).stop 2024-04-21T08:48:42.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 1 -- 172.21.15.94:0/2632243882 shutdown_connections 2024-04-21T08:48:42.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5594310f8240 0x5594310fa660 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559431078290 0x55943107b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559431080e10 0x5594310f6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 1 --2- 172.21.15.94:0/2632243882 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5594310778e0 0x55943107aec0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f555eff5640 0 client.17895 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:42.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 1 -- 172.21.15.94:0/2632243882 >> 172.21.15.94:0/2632243882 conn(0x559431071fb0 msgr2=0x559431073b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 1 -- 172.21.15.94:0/2632243882 shutdown_connections 2024-04-21T08:48:42.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 1 -- 172.21.15.94:0/2632243882 wait complete. 2024-04-21T08:48:42.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5584383740 20 client.17895 trim_cache size 0 max 16384 2024-04-21T08:48:42.832 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "fd9f2d04-b2d9-4956-bb02-9143ff9924ae", 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5581796640 1 -- 172.21.15.94:0/301060458 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f55640a3d20 msgr2=0x7f55640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5581796640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f55640a3d20 0x7f55640a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f55740be160 tx=0x7f55740095c0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5581796640 1 -- 172.21.15.94:0/301060458 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557c10e900 msgr2=0x7f557c19a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5581796640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557c10e900 0x7f557c19a720 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f557402c700 tx=0x7f5574034650 comp rx=0 tx=0).stop 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5581796640 1 -- 172.21.15.94:0/301060458 shutdown_connections 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5581796640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f55640a3d20 0x7f55640a61e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5581796640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f557c101670 0x7f557c1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5581796640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557c10e900 0x7f557c19a720 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5581796640 1 --2- 172.21.15.94:0/301060458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f557c100d50 0x7f557c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.827+0000 7f5581796640 1 -- 172.21.15.94:0/301060458 >> 172.21.15.94:0/301060458 conn(0x7f557c077f50 msgr2=0x7f557c10ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.828+0000 7f5581796640 1 -- 172.21.15.94:0/301060458 shutdown_connections 2024-04-21T08:48:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.828+0000 7f5581796640 1 -- 172.21.15.94:0/301060458 wait complete. 2024-04-21T08:48:42.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:48:42.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:48:42.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.939+0000 7f5f3739f640 1 -- 172.21.15.94:0/3079962014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f301038a0 msgr2=0x7f5f3010bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.939+0000 7f5f3739f640 1 --2- 172.21.15.94:0/3079962014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f301038a0 0x7f5f3010bd80 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009900 tx=0x7f5f2002bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:42.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.939+0000 7f5f3739f640 1 -- 172.21.15.94:0/3079962014 shutdown_connections 2024-04-21T08:48:42.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.939+0000 7f5f3739f640 1 --2- 172.21.15.94:0/3079962014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f3010c350 0x7f5f3010e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.939+0000 7f5f3739f640 1 --2- 172.21.15.94:0/3079962014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f301038a0 0x7f5f3010bd80 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.939+0000 7f5f3739f640 1 --2- 172.21.15.94:0/3079962014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30102f80 0x7f5f30103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.939+0000 7f5f3739f640 1 -- 172.21.15.94:0/3079962014 >> 172.21.15.94:0/3079962014 conn(0x7f5f300fc6e0 msgr2=0x7f5f300feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.939+0000 7f5f3739f640 1 -- 172.21.15.94:0/3079962014 shutdown_connections 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.939+0000 7f5f3739f640 1 -- 172.21.15.94:0/3079962014 wait complete. 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.939+0000 7f5f3739f640 1 Processor -- start 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f3739f640 1 -- start start 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f3739f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30102f80 0x7f5f30198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f3739f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f301038a0 0x7f5f30198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f3739f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f3010c350 0x7f5f3019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f3739f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5f30111f70 con 0x7f5f3010c350 2024-04-21T08:48:42.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f3739f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f30111df0 con 0x7f5f30102f80 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f3739f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301120f0 con 0x7f5f301038a0 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f36b9e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f3010c350 0x7f5f3019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f36b9e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f3010c350 0x7f5f3019cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47518/0 (socket says 172.21.15.94:47518) 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f36b9e640 1 -- 172.21.15.94:0/4015084836 learned_addr learned my addr 172.21.15.94:0/4015084836 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f3639d640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30102f80 0x7f5f30198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f301038a0 0x7f5f30198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f36b9e640 1 -- 172.21.15.94:0/4015084836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f301038a0 msgr2=0x7f5f30198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f36b9e640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f301038a0 0x7f5f30198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f36b9e640 1 -- 172.21.15.94:0/4015084836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30102f80 msgr2=0x7f5f30198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f36b9e640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30102f80 0x7f5f30198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f36b9e640 1 -- 172.21.15.94:0/4015084836 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3019d460 con 0x7f5f3010c350 2024-04-21T08:48:42.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f36b9e640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f3010c350 0x7f5f3019cd30 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00c970 tx=0x7f5f2c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f277fe640 1 -- 172.21.15.94:0/4015084836 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c007bb0 con 0x7f5f3010c350 2024-04-21T08:48:42.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f3739f640 1 -- 172.21.15.94:0/4015084836 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3019d750 con 0x7f5f3010c350 2024-04-21T08:48:42.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f277fe640 1 -- 172.21.15.94:0/4015084836 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f2c007d50 con 0x7f5f3010c350 2024-04-21T08:48:42.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.940+0000 7f5f277fe640 1 -- 172.21.15.94:0/4015084836 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c018770 con 0x7f5f3010c350 2024-04-21T08:48:42.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.941+0000 7f5f3739f640 1 -- 172.21.15.94:0/4015084836 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f301a4fe0 con 0x7f5f3010c350 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.943+0000 7f5f277fe640 1 -- 172.21.15.94:0/4015084836 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5f2c00f990 con 0x7f5f3010c350 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.943+0000 7f5f3b1f1740 1 Processor -- start 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.943+0000 7f5f3b1f1740 1 -- start start 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.943+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd6b20 0x5581d1fd6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.943+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd7440 0x5581d1fdfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.943+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.943+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5581d1f371e0 con 0x5581d1fe0020 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.943+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5581d1ed4390 con 0x5581d1fd7440 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.943+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5581d1f331a0 con 0x5581d1fd6b20 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f35b9c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f277fe640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f1c0a3d20 0x7f5f1c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f35b9c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47528/0 (socket says 172.21.15.94:47528) 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f35b9c640 1 -- 172.21.15.94:0/2198644759 learned_addr learned my addr 172.21.15.94:0/2198644759 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f3639d640 1 --2- 172.21.15.94:0/2198644759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd7440 0x5581d1fdfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/2198644759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd6b20 0x5581d1fd6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f36b9e640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f1c0a3d20 0x7f5f1c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f277fe640 1 -- 172.21.15.94:0/4015084836 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f5f2c0d0cc0 con 0x7f5f3010c350 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f3639d640 1 -- 172.21.15.94:0/2198644759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd6b20 msgr2=0x5581d1fd6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f3639d640 1 --2- 172.21.15.94:0/2198644759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd6b20 0x5581d1fd6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f3639d640 1 -- 172.21.15.94:0/2198644759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 msgr2=0x5581d1fe2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f3639d640 1 --2- 172.21.15.94:0/2198644759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f3639d640 1 -- 172.21.15.94:0/2198644759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581d1d137d0 con 0x5581d1fd7440 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f36b9e640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f1c0a3d20 0x7f5f1c0a61e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0ce090 tx=0x7f5f2c06c110 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f3639d640 1 --2- 172.21.15.94:0/2198644759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd7440 0x5581d1fdfae0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5f28009fb0 tx=0x7f5f28003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4db46bbdc971afd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f24bfe640 1 -- 172.21.15.94:0/2198644759 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28004860 con 0x5581d1fd7440 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f24bfe640 1 -- 172.21.15.94:0/2198644759 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f28005020 con 0x5581d1fd7440 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.944+0000 7f5f24bfe640 1 -- 172.21.15.94:0/2198644759 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f280126b0 con 0x5581d1fd7440 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.945+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/2198644759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd7440 msgr2=0x5581d1fdfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.945+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/2198644759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd7440 0x5581d1fdfae0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5f28009fb0 tx=0x7f5f28003e00 comp rx=0 tx=0).stop 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.945+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/2198644759 shutdown_connections 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.945+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/2198644759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.945+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/2198644759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd7440 0x5581d1fdfae0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.945+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/2198644759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd6b20 0x5581d1fd6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.945+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/2198644759 >> 172.21.15.94:0/2198644759 conn(0x5581d1fd1250 msgr2=0x5581d1fd1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.945+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/2198644759 shutdown_connections 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.945+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/2198644759 wait complete. 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3b1f1740 1 -- start start 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3b1f1740 1 Processor -- start 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd6b20 0x5581d1fdcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fd7440 0x5581d1fdd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fe0020 0x5581d1fdda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5581d1fe2e10 con 0x5581d1fd7440 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5581d1ef02f0 con 0x5581d1fe0020 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5581d1fe2f90 con 0x5581d1fd6b20 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3b1f1740 10 client.0 _ll_register_callbacks cb 0x5581d1efd580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3639d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fd7440 0x5581d1fdd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3639d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fd7440 0x5581d1fdd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47534/0 (socket says 172.21.15.94:47534) 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3639d640 1 -- 172.21.15.94:0/3064311053 learned_addr learned my addr 172.21.15.94:0/3064311053 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd6b20 0x5581d1fdcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fe0020 0x5581d1fdda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3639d640 1 -- 172.21.15.94:0/3064311053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd6b20 msgr2=0x5581d1fdcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3639d640 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd6b20 0x5581d1fdcfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3639d640 1 -- 172.21.15.94:0/3064311053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fe0020 msgr2=0x5581d1fdda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3639d640 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fe0020 0x5581d1fdda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3639d640 1 -- 172.21.15.94:0/3064311053 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581d1c493f0 con 0x5581d1fd7440 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fe0020 0x5581d1fdda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f3639d640 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fd7440 0x5581d1fdd510 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f5f28004910 tx=0x7f5f28004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9544be30d33471d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.946+0000 7f5f19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3064311053 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f280051d0 con 0x5581d1fd7440 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3064311053 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f5f28005370 con 0x5581d1fd7440 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3064311053 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f28005020 con 0x5581d1fd7440 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3064311053 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5f28005570 con 0x5581d1fd7440 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f3b1f1740 10 client.17907 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3064311053 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5581d1c090f0 con 0x5581d1fd7440 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3064311053 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f28005020 con 0x5581d1fd7440 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3064311053 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5f2800f040 con 0x5581d1fd7440 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f3b1f1740 10 client.17907 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f3b1f1740 10 client.17907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f3b1f1740 20 client.17907 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581d1fefee0 0x5581d1ff22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f3b1f1740 4 client.17907 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3064311053 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581d1f44c00 con 0x5581d1fefee0 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.947+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581d1fefee0 0x5581d1ff22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.948+0000 7f5f19ffb640 10 client.17907 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.948+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581d1fefee0 0x5581d1ff22d0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=572ab1cd0b54ac44 server_cookie=39600d3cbc8bcf7 in_seq=0 out_seq=0 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.948+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3064311053 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581d1f44c00 con 0x5581d1fefee0 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.948+0000 7f5f19ffb640 10 client.17907 handle_command_reply: tid=0 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.948+0000 7f5f3b1f1740 1 client.17907 shutdown 2024-04-21T08:48:42.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3064311053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fd7440 msgr2=0x5581d1fdd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fd7440 0x5581d1fdd510 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f5f28004910 tx=0x7f5f28004940 comp rx=0 tx=0).stop 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3064311053 shutdown_connections 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581d1fefee0 0x5581d1ff22d0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fe0020 0x5581d1fdda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f19ffb640 0 client.17907 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fd7440 0x5581d1fdd510 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3064311053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd6b20 0x5581d1fdcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3064311053 >> 172.21.15.94:0/3064311053 conn(0x5581d1fd1250 msgr2=0x5581d1fd2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3064311053 shutdown_connections 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3064311053 wait complete. 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.949+0000 7f5f3b1f1740 20 client.17907 trim_cache size 0 max 16384 2024-04-21T08:48:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f3b1f1740 1 Processor -- start 2024-04-21T08:48:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f3b1f1740 1 -- start start 2024-04-21T08:48:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd6b20 0x5581d1fdf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd7440 0x5581d1febd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1ff7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301af9e0 con 0x5581d1fe0020 2024-04-21T08:48:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301ad2b0 con 0x5581d1fd6b20 2024-04-21T08:48:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301ab120 con 0x5581d1fd7440 2024-04-21T08:48:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f35b9c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1ff7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f35b9c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1ff7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47544/0 (socket says 172.21.15.94:47544) 2024-04-21T08:48:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f35b9c640 1 -- 172.21.15.94:0/4191487190 learned_addr learned my addr 172.21.15.94:0/4191487190 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/4191487190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd6b20 0x5581d1fdf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f3639d640 1 --2- 172.21.15.94:0/4191487190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd7440 0x5581d1febd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f35b9c640 1 -- 172.21.15.94:0/4191487190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd7440 msgr2=0x5581d1febd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/4191487190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd7440 0x5581d1febd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f35b9c640 1 -- 172.21.15.94:0/4191487190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd6b20 msgr2=0x5581d1fdf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/4191487190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd6b20 0x5581d1fdf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.966+0000 7f5f35b9c640 1 -- 172.21.15.94:0/4191487190 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581d1fec830 con 0x5581d1fe0020 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/4191487190 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1ff7b30 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009230 tx=0x7f5f20045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff45b0727383da2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f1bfff640 1 -- 172.21.15.94:0/4191487190 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2002cc40 con 0x5581d1fe0020 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f1bfff640 1 -- 172.21.15.94:0/4191487190 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f200054f0 con 0x5581d1fe0020 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f1bfff640 1 -- 172.21.15.94:0/4191487190 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f200046c0 con 0x5581d1fe0020 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/4191487190 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 msgr2=0x5581d1ff7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/4191487190 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1ff7b30 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009230 tx=0x7f5f20045040 comp rx=0 tx=0).stop 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/4191487190 shutdown_connections 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/4191487190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd7440 0x5581d1febd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/4191487190 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1ff7b30 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/4191487190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd6b20 0x5581d1fdf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/4191487190 >> 172.21.15.94:0/4191487190 conn(0x5581d1fd1250 msgr2=0x5581d1fd1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/4191487190 shutdown_connections 2024-04-21T08:48:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.967+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/4191487190 wait complete. 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f3b1f1740 1 -- start start 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f3b1f1740 1 Processor -- start 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd6b20 0x5581d1fe8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd7440 0x5581d1fe3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f3b1f1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301ab770 con 0x5581d1fe0020 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301acbe0 con 0x5581d1fd6b20 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f3b1f1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301abe40 con 0x5581d1fd7440 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f3b1f1740 10 client.0 _ll_register_callbacks cb 0x7f5f20009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f35b9c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f35b9c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47546/0 (socket says 172.21.15.94:47546) 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f35b9c640 1 -- 172.21.15.94:0/3503537706 learned_addr learned my addr 172.21.15.94:0/3503537706 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:48:42.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd6b20 0x5581d1fe8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f3639d640 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd7440 0x5581d1fe3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f35b9c640 1 -- 172.21.15.94:0/3503537706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd7440 msgr2=0x5581d1fe3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd7440 0x5581d1fe3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f35b9c640 1 -- 172.21.15.94:0/3503537706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd6b20 msgr2=0x5581d1fe8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd6b20 0x5581d1fe8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.968+0000 7f5f35b9c640 1 -- 172.21.15.94:0/3503537706 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581d20553a0 con 0x5581d1fe0020 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f35b9c640 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe4200 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f5f200096c0 tx=0x7f5f20052040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f858eec94c01080 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3503537706 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2004e470 con 0x5581d1fe0020 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3503537706 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9599+0+0 (secure 0 0 0) 0x7f5f2004e6e0 con 0x5581d1fe0020 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3503537706 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f200054f0 con 0x5581d1fe0020 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3503537706 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5f2004e8e0 con 0x5581d1fe0020 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f3b1f1740 10 client.17913 fetch_fsmap learned FSMap version 24 2024-04-21T08:48:42.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3503537706 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5581d2056530 con 0x5581d1fe0020 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3503537706 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f200054f0 con 0x5581d1fe0020 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3503537706 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5f20040040 con 0x5581d1fe0020 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f3b1f1740 10 client.17913 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f3b1f1740 10 client.17913 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f3b1f1740 20 client.17913 populate_metadata read hostname 'smithi094' 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581d1fefee0 0x5581d2058c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f3b1f1740 4 client.17913 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.969+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3503537706 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5581d1f3ea70 con 0x5581d1fefee0 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3639d640 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581d1fefee0 0x5581d2058c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f19ffb640 10 client.17913 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3639d640 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581d1fefee0 0x5581d2058c60 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32305904bf88013a server_cookie=6cf6c99cc88314c0 in_seq=0 out_seq=0 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f19ffb640 1 -- 172.21.15.94:0/3503537706 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5581d1f3ea70 con 0x5581d1fefee0 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f19ffb640 10 client.17913 handle_command_reply: tid=0 2024-04-21T08:48:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 client.17913 shutdown 2024-04-21T08:48:42.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3503537706 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 msgr2=0x5581d1fe4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe4200 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f5f200096c0 tx=0x7f5f20052040 comp rx=0 tx=0).stop 2024-04-21T08:48:42.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3503537706 shutdown_connections 2024-04-21T08:48:42.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581d1fefee0 0x5581d2058c60 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581d1fd7440 0x5581d1fe3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5581d1fe0020 0x5581d1fe4200 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f19ffb640 0 client.17913 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:48:42.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 --2- 172.21.15.94:0/3503537706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5581d1fd6b20 0x5581d1fe8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3503537706 >> 172.21.15.94:0/3503537706 conn(0x5581d1fd1250 msgr2=0x5581d1fcb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3503537706 shutdown_connections 2024-04-21T08:48:42.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 1 -- 172.21.15.94:0/3503537706 wait complete. 2024-04-21T08:48:42.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.970+0000 7f5f3b1f1740 20 client.17913 trim_cache size 0 max 16384 2024-04-21T08:48:42.978 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:48:42.978 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:48:42.978 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:48:42.978 INFO:teuthology.orchestra.run.smithi094.stdout: "fd9f2d04-b2d9-4956-bb02-9143ff9924ae": { 2024-04-21T08:48:42.978 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:48:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "fd9f2d04-b2d9-4956-bb02-9143ff9924ae", 2024-04-21T08:48:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:48:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:48:42.979 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:48:42.979 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 -- 172.21.15.94:0/4015084836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f1c0a3d20 msgr2=0x7f5f1c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f1c0a3d20 0x7f5f1c0a61e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0ce090 tx=0x7f5f2c06c110 comp rx=0 tx=0).stop 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 -- 172.21.15.94:0/4015084836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f3010c350 msgr2=0x7f5f3019cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f3010c350 0x7f5f3019cd30 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00c970 tx=0x7f5f2c00ce30 comp rx=0 tx=0).stop 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 -- 172.21.15.94:0/4015084836 shutdown_connections 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f5f1c0a3d20 0x7f5f1c0a61e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f301038a0 0x7f5f30198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f3010c350 0x7f5f3019cd30 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 --2- 172.21.15.94:0/4015084836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30102f80 0x7f5f30198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 -- 172.21.15.94:0/4015084836 >> 172.21.15.94:0/4015084836 conn(0x7f5f300fc6e0 msgr2=0x7f5f300fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 -- 172.21.15.94:0/4015084836 shutdown_connections 2024-04-21T08:48:42.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:48:42.971+0000 7f5f3739f640 1 -- 172.21.15.94:0/4015084836 wait complete. 2024-04-21T08:48:42.981 INFO:tasks.cephfs.filesystem:scrub status for tag:fd9f2d04-b2d9-4956-bb02-9143ff9924ae - {'path': '/', 'tag': 'fd9f2d04-b2d9-4956-bb02-9143ff9924ae', 'options': 'recursive,force'} 2024-04-21T08:48:43.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:43 smithi094 ceph-mon[27988]: pgmap v269: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:43.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:43 smithi111 ceph-mon[31493]: pgmap v269: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:43.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:43 smithi173 ceph-mon[31428]: pgmap v269: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:44.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:44 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:48:44.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:44 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:44.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:44 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:48:44.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:44 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:44.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:44 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:48:44.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:44 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:44.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:44 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:48:44.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:44 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:44.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:44 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:48:44.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:44 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:44.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:44 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:48:44.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:44 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:45.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:45 smithi111 ceph-mon[31493]: pgmap v270: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:48:45.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:45 smithi094 ceph-mon[27988]: pgmap v270: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:48:45.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:45 smithi173 ceph-mon[31428]: pgmap v270: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-04-21T08:48:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:47 smithi111 ceph-mon[31493]: pgmap v271: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:48:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:47 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:48:47.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:47 smithi094 ceph-mon[27988]: pgmap v271: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:48:47.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:47 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:48:47.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:47 smithi173 ceph-mon[31428]: pgmap v271: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:48:47.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:47 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:48:49.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:49 smithi111 ceph-mon[31493]: pgmap v272: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:48:49.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:49 smithi094 ceph-mon[27988]: pgmap v272: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:48:49.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:49 smithi173 ceph-mon[31428]: pgmap v272: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:48:51.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:51 smithi111 ceph-mon[31493]: pgmap v273: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:48:51.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:51 smithi094 ceph-mon[27988]: pgmap v273: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:48:51.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:51 smithi173 ceph-mon[31428]: pgmap v273: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:48:53.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:53 smithi111 ceph-mon[31493]: pgmap v274: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:48:53.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:53 smithi094 ceph-mon[27988]: pgmap v274: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:48:53.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:53 smithi173 ceph-mon[31428]: pgmap v274: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:48:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:55 smithi111 ceph-mon[31493]: pgmap v275: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-04-21T08:48:55.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:55 smithi094 ceph-mon[27988]: pgmap v275: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-04-21T08:48:55.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:55 smithi173 ceph-mon[31428]: pgmap v275: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-04-21T08:48:57.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:57 smithi111 ceph-mon[31493]: pgmap v276: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:48:57.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:57 smithi094 ceph-mon[27988]: pgmap v276: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:48:57.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:57 smithi173 ceph-mon[31428]: pgmap v276: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-21T08:48:59.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:48:59 smithi111 ceph-mon[31493]: pgmap v277: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:59.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:48:59 smithi094 ceph-mon[27988]: pgmap v277: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:48:59.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:48:59 smithi173 ceph-mon[31428]: pgmap v277: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:01.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:01 smithi111 ceph-mon[31493]: pgmap v278: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:49:01.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:01 smithi094 ceph-mon[27988]: pgmap v278: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:49:01.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:01 smithi173 ceph-mon[31428]: pgmap v278: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:49:03.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:03 smithi111 ceph-mon[31493]: pgmap v279: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:03.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:03 smithi094 ceph-mon[27988]: pgmap v279: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:03.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:03 smithi173 ceph-mon[31428]: pgmap v279: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:05.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:05 smithi111 ceph-mon[31493]: pgmap v280: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:49:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:05 smithi094 ceph-mon[27988]: pgmap v280: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:49:05.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:05 smithi173 ceph-mon[31428]: pgmap v280: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:49:07.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:07 smithi111 ceph-mon[31493]: pgmap v281: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:49:07.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:07 smithi094 ceph-mon[27988]: pgmap v281: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:49:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:07 smithi173 ceph-mon[31428]: pgmap v281: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:49:08.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:08 smithi094 ceph-mon[27988]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T08:49:08.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:08 smithi173 ceph-mon[31428]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T08:49:08.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:08 smithi111 ceph-mon[31493]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T08:49:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:09 smithi094 ceph-mon[27988]: pgmap v283: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:49:09.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:09 smithi173 ceph-mon[31428]: pgmap v283: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:49:09.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:09 smithi111 ceph-mon[31493]: pgmap v283: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:49:10.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:10 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:10.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:10 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:49:10.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:10 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:49:10.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:10 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:49:10.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:10 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:10.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:10 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:49:10.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:10 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:49:10.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:10 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:49:10.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:10 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:10.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:10 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:49:10.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:10 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:49:10.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:10 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:49:11.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:11 smithi094 ceph-mon[27988]: pgmap v284: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:49:11.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:11 smithi173 ceph-mon[31428]: pgmap v284: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:49:11.589 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:11 smithi111 ceph-mon[31493]: pgmap v284: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:49:12.982 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:49:12.982 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.101+0000 7f6fec870640 1 -- 172.21.15.94:0/3365951415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe8103ae0 msgr2=0x7f6fe810bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.101+0000 7f6fec870640 1 --2- 172.21.15.94:0/3365951415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe8103ae0 0x7f6fe810bfc0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009900 tx=0x7f6fdc02bd60 comp rx=0 tx=0).stop 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.101+0000 7f6fec870640 1 -- 172.21.15.94:0/3365951415 shutdown_connections 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.101+0000 7f6fec870640 1 --2- 172.21.15.94:0/3365951415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fe810c500 0x7f6fe810e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.101+0000 7f6fec870640 1 --2- 172.21.15.94:0/3365951415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe8103ae0 0x7f6fe810bfc0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.101+0000 7f6fec870640 1 --2- 172.21.15.94:0/3365951415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fe81031c0 0x7f6fe81035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.101+0000 7f6fec870640 1 -- 172.21.15.94:0/3365951415 >> 172.21.15.94:0/3365951415 conn(0x7f6fe8078000 msgr2=0x7f6fe8078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.101+0000 7f6fec870640 1 -- 172.21.15.94:0/3365951415 shutdown_connections 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.101+0000 7f6fec870640 1 -- 172.21.15.94:0/3365951415 wait complete. 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fec870640 1 Processor -- start 2024-04-21T08:49:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fec870640 1 -- start start 2024-04-21T08:49:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fec870640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fe81031c0 0x7f6fe8198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fec870640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe8103ae0 0x7f6fe8198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fec870640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fe810c500 0x7f6fe819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fec870640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8112170 con 0x7f6fe8103ae0 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fec870640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8111ff0 con 0x7f6fe81031c0 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fec870640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81122f0 con 0x7f6fe810c500 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fe5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe8103ae0 0x7f6fe8198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fe5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe8103ae0 0x7f6fe8198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49118/0 (socket says 172.21.15.94:49118) 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fe5d9b640 1 -- 172.21.15.94:0/2546849478 learned_addr learned my addr 172.21.15.94:0/2546849478 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.102+0000 7f6fe659c640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fe81031c0 0x7f6fe8198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fe6d9d640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fe810c500 0x7f6fe819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fe5d9b640 1 -- 172.21.15.94:0/2546849478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fe810c500 msgr2=0x7f6fe819d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fe810c500 0x7f6fe819d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fe5d9b640 1 -- 172.21.15.94:0/2546849478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fe81031c0 msgr2=0x7f6fe8198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fe81031c0 0x7f6fe8198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fe5d9b640 1 -- 172.21.15.94:0/2546849478 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe819d740 con 0x7f6fe8103ae0 2024-04-21T08:49:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fe659c640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fe81031c0 0x7f6fe8198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe8103ae0 0x7f6fe8198c90 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc005940 tx=0x7f6fdc02d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fd77fe640 1 -- 172.21.15.94:0/2546849478 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc043020 con 0x7f6fe8103ae0 2024-04-21T08:49:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fd77fe640 1 -- 172.21.15.94:0/2546849478 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fdc02dda0 con 0x7f6fe8103ae0 2024-04-21T08:49:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fec870640 1 -- 172.21.15.94:0/2546849478 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe819d9d0 con 0x7f6fe8103ae0 2024-04-21T08:49:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fd77fe640 1 -- 172.21.15.94:0/2546849478 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc003bf0 con 0x7f6fe8103ae0 2024-04-21T08:49:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.103+0000 7f6fec870640 1 -- 172.21.15.94:0/2546849478 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe81a52c0 con 0x7f6fe8103ae0 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.105+0000 7f6fd77fe640 1 -- 172.21.15.94:0/2546849478 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f6fdc046020 con 0x7f6fe8103ae0 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fef45c740 1 Processor -- start 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fd77fe640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6fd00a3d20 0x7f6fd00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fef45c740 1 -- start start 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a01960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fe6d9d640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6fd00a3d20 0x7f6fd00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a0a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a0aa80 0x562149a0ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fef45c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5621499661e0 con 0x562149a01580 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fef45c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562149903390 con 0x562149a01ea0 2024-04-21T08:49:13.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fef45c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5621499621a0 con 0x562149a0aa80 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fd77fe640 1 -- 172.21.15.94:0/2546849478 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f6fdc0f2fa0 con 0x7f6fe8103ae0 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fe659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a01960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fe659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a01960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49128/0 (socket says 172.21.15.94:49128) 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.106+0000 7f6fe659c640 1 -- 172.21.15.94:0/1165269139 learned_addr learned my addr 172.21.15.94:0/1165269139 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fe6d9d640 1 --2- 172.21.15.94:0/1165269139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a0aa80 0x562149a0ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/1165269139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a0a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fe659c640 1 -- 172.21.15.94:0/1165269139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a0aa80 msgr2=0x562149a0ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fe659c640 1 --2- 172.21.15.94:0/1165269139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a0aa80 0x562149a0ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fe659c640 1 -- 172.21.15.94:0/1165269139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 msgr2=0x562149a0a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fe659c640 1 --2- 172.21.15.94:0/1165269139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a0a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fe659c640 1 -- 172.21.15.94:0/1165269139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621497427d0 con 0x562149a01580 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fe6d9d640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6fd00a3d20 0x7f6fd00a61e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00125e0 tx=0x7f6fe00129e0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fe659c640 1 --2- 172.21.15.94:0/1165269139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a01960 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800cca0 tx=0x7f6fd800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54d1c31ddbb8a35 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fd4ff9640 1 -- 172.21.15.94:0/1165269139 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8005220 con 0x562149a01580 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fd4ff9640 1 -- 172.21.15.94:0/1165269139 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fd8004c50 con 0x562149a01580 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.107+0000 7f6fd4ff9640 1 -- 172.21.15.94:0/1165269139 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8010690 con 0x562149a01580 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 -- 172.21.15.94:0/1165269139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 msgr2=0x562149a01960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 --2- 172.21.15.94:0/1165269139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a01960 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800cca0 tx=0x7f6fd800e720 comp rx=0 tx=0).stop 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 -- 172.21.15.94:0/1165269139 shutdown_connections 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 --2- 172.21.15.94:0/1165269139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a0aa80 0x562149a0ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 --2- 172.21.15.94:0/1165269139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a0a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 --2- 172.21.15.94:0/1165269139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a01960 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 -- 172.21.15.94:0/1165269139 >> 172.21.15.94:0/1165269139 conn(0x5621499fbcb0 msgr2=0x5621499fc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 -- 172.21.15.94:0/1165269139 shutdown_connections 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 -- 172.21.15.94:0/1165269139 wait complete. 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 -- start start 2024-04-21T08:49:13.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.108+0000 7f6fef45c740 1 Processor -- start 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a01580 0x562149a07a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a07fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a0aa80 0x562149a08500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fef45c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x562149a0d870 con 0x562149a0aa80 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fef45c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56214991f2f0 con 0x562149a01ea0 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fef45c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562149a0d9f0 con 0x562149a01580 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fef45c740 10 client.0 _ll_register_callbacks cb 0x56214992c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a0aa80 0x562149a08500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a0aa80 0x562149a08500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49136/0 (socket says 172.21.15.94:49136) 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe6d9d640 1 -- 172.21.15.94:0/3617204333 learned_addr learned my addr 172.21.15.94:0/3617204333 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe659c640 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a01580 0x562149a07a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a07fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe6d9d640 1 -- 172.21.15.94:0/3617204333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a01580 msgr2=0x562149a07a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe6d9d640 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a01580 0x562149a07a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe6d9d640 1 -- 172.21.15.94:0/3617204333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 msgr2=0x562149a07fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe6d9d640 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a07fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe6d9d640 1 -- 172.21.15.94:0/3617204333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621496783f0 con 0x562149a0aa80 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fe6d9d640 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a0aa80 0x562149a08500 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000ee80 tx=0x7f6fe0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9470918c832e568 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.109+0000 7f6fcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fcdffb640 1 -- 172.21.15.94:0/3617204333 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0004d20 con 0x562149a0aa80 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fcdffb640 1 -- 172.21.15.94:0/3617204333 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f6fe0003970 con 0x562149a0aa80 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fef45c740 1 -- 172.21.15.94:0/3617204333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fd8004c50 con 0x562149a0aa80 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fcdffb640 1 -- 172.21.15.94:0/3617204333 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6fe002e050 con 0x562149a0aa80 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fef45c740 10 client.17925 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fef45c740 1 -- 172.21.15.94:0/3617204333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5621496380f0 con 0x562149a0aa80 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fcdffb640 1 -- 172.21.15.94:0/3617204333 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fd8004c50 con 0x562149a0aa80 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fcdffb640 1 -- 172.21.15.94:0/3617204333 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6fe00044d0 con 0x562149a0aa80 2024-04-21T08:49:13.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fef45c740 10 client.17925 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fef45c740 10 client.17925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fef45c740 20 client.17925 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fef45c740 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562149a1aa10 0x562149a1ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fef45c740 4 client.17925 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.110+0000 7f6fef45c740 1 -- 172.21.15.94:0/3617204333 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562149973c00 con 0x562149a1aa10 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.111+0000 7f6fe659c640 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562149a1aa10 0x562149a1ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.111+0000 7f6fe659c640 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562149a1aa10 0x562149a1ce30 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f484f68cb8b7a28 server_cookie=3fa62ef5f01239b2 in_seq=0 out_seq=0 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.111+0000 7f6fcdffb640 10 client.17925 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fcdffb640 1 -- 172.21.15.94:0/3617204333 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562149973c00 con 0x562149a1aa10 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fcdffb640 10 client.17925 handle_command_reply: tid=0 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 client.17925 shutdown 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 -- 172.21.15.94:0/3617204333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a0aa80 msgr2=0x562149a08500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a0aa80 0x562149a08500 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000ee80 tx=0x7f6fe0004ab0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 -- 172.21.15.94:0/3617204333 shutdown_connections 2024-04-21T08:49:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562149a1aa10 0x562149a1ce30 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a0aa80 0x562149a08500 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a07fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 --2- 172.21.15.94:0/3617204333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a01580 0x562149a07a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fcdffb640 0 client.17925 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 -- 172.21.15.94:0/3617204333 >> 172.21.15.94:0/3617204333 conn(0x5621499fbcb0 msgr2=0x5621499fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 -- 172.21.15.94:0/3617204333 shutdown_connections 2024-04-21T08:49:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 1 -- 172.21.15.94:0/3617204333 wait complete. 2024-04-21T08:49:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.112+0000 7f6fef45c740 20 client.17925 trim_cache size 0 max 16384 2024-04-21T08:49:13.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.129+0000 7f6fef45c740 1 Processor -- start 2024-04-21T08:49:13.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.129+0000 7f6fef45c740 1 -- start start 2024-04-21T08:49:13.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a01580 0x562149a069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01ea0 0x562149a16760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a0aa80 0x562149a22700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fef45c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81b1e40 con 0x562149a01ea0 2024-04-21T08:49:13.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fef45c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81af710 con 0x562149a0aa80 2024-04-21T08:49:13.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fef45c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81ad580 con 0x562149a01580 2024-04-21T08:49:13.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01ea0 0x562149a16760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01ea0 0x562149a16760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49148/0 (socket says 172.21.15.94:49148) 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe5d9b640 1 -- 172.21.15.94:0/2253647724 learned_addr learned my addr 172.21.15.94:0/2253647724 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe659c640 1 --2- 172.21.15.94:0/2253647724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a01580 0x562149a069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe6d9d640 1 --2- 172.21.15.94:0/2253647724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a0aa80 0x562149a22700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe5d9b640 1 -- 172.21.15.94:0/2253647724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a01580 msgr2=0x562149a069c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/2253647724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a01580 0x562149a069c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe5d9b640 1 -- 172.21.15.94:0/2253647724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a0aa80 msgr2=0x562149a22700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/2253647724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a0aa80 0x562149a22700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe5d9b640 1 -- 172.21.15.94:0/2253647724 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562149a0ce70 con 0x562149a01ea0 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/2253647724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01ea0 0x562149a16760 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0bb5f0 tx=0x7f6fdc02bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354be1c3bc7748a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fcffff640 1 -- 172.21.15.94:0/2253647724 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0bec20 con 0x562149a01ea0 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fcffff640 1 -- 172.21.15.94:0/2253647724 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fdc0c2fd0 con 0x562149a01ea0 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.130+0000 7f6fcffff640 1 -- 172.21.15.94:0/2253647724 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0bb290 con 0x562149a01ea0 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 -- 172.21.15.94:0/2253647724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01ea0 msgr2=0x562149a16760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 --2- 172.21.15.94:0/2253647724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01ea0 0x562149a16760 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0bb5f0 tx=0x7f6fdc02bd60 comp rx=0 tx=0).stop 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 -- 172.21.15.94:0/2253647724 shutdown_connections 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 --2- 172.21.15.94:0/2253647724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a0aa80 0x562149a22700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 --2- 172.21.15.94:0/2253647724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01ea0 0x562149a16760 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 --2- 172.21.15.94:0/2253647724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a01580 0x562149a069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 -- 172.21.15.94:0/2253647724 >> 172.21.15.94:0/2253647724 conn(0x5621499fbcb0 msgr2=0x562149a09a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 -- 172.21.15.94:0/2253647724 shutdown_connections 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 -- 172.21.15.94:0/2253647724 wait complete. 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 -- start start 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.131+0000 7f6fef45c740 1 Processor -- start 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a12f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a0f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fef45c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a0aa80 0x562149a0f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fef45c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81adbd0 con 0x562149a01580 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fef45c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81af040 con 0x562149a01ea0 2024-04-21T08:49:13.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fef45c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81ae2a0 con 0x562149a0aa80 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fef45c740 10 client.0 _ll_register_callbacks cb 0x7f6fe000e690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fe659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a12f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fe659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a12f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49154/0 (socket says 172.21.15.94:49154) 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fe659c640 1 -- 172.21.15.94:0/2398946050 learned_addr learned my addr 172.21.15.94:0/2398946050 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fe659c640 1 -- 172.21.15.94:0/2398946050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a0aa80 msgr2=0x562149a0f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fe659c640 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a0aa80 0x562149a0f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fe659c640 1 -- 172.21.15.94:0/2398946050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 msgr2=0x562149a0f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a0f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fe659c640 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a0f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fe659c640 1 -- 172.21.15.94:0/2398946050 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562149a0ff60 con 0x562149a01580 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fe659c640 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a12f60 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80055e0 tx=0x7f6fd8005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d026d42409e1323 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:13.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fcdffb640 1 -- 172.21.15.94:0/2398946050 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8028460 con 0x562149a01580 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fcdffb640 1 -- 172.21.15.94:0/2398946050 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f6fd8028600 con 0x562149a01580 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fef45c740 1 -- 172.21.15.94:0/2398946050 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fdc0c2fd0 con 0x562149a01580 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.132+0000 7f6fcdffb640 1 -- 172.21.15.94:0/2398946050 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6fd8036050 con 0x562149a01580 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fef45c740 10 client.17931 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fef45c740 1 -- 172.21.15.94:0/2398946050 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562149a8a6d0 con 0x562149a01580 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fcdffb640 1 -- 172.21.15.94:0/2398946050 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fdc0c2fd0 con 0x562149a01580 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fcdffb640 1 -- 172.21.15.94:0/2398946050 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6fd8010ac0 con 0x562149a01580 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fef45c740 10 client.17931 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fef45c740 10 client.17931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fef45c740 20 client.17931 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fef45c740 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562149a8ab70 0x562149a8cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fef45c740 4 client.17931 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fef45c740 1 -- 172.21.15.94:0/2398946050 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56214996da70 con 0x562149a8ab70 2024-04-21T08:49:13.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.133+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562149a8ab70 0x562149a8cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fcdffb640 10 client.17931 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fe5d9b640 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562149a8ab70 0x562149a8cfd0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d51b1fbf7b1996f0 server_cookie=7e14bc1f0e44492e in_seq=0 out_seq=0 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fcdffb640 1 -- 172.21.15.94:0/2398946050 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56214996da70 con 0x562149a8ab70 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fcdffb640 10 client.17931 handle_command_reply: tid=0 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 client.17931 shutdown 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 -- 172.21.15.94:0/2398946050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 msgr2=0x562149a12f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a12f60 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80055e0 tx=0x7f6fd8005570 comp rx=0 tx=0).stop 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 -- 172.21.15.94:0/2398946050 shutdown_connections 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x562149a8ab70 0x562149a8cfd0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562149a0aa80 0x562149a0f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562149a01ea0 0x562149a0f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fcdffb640 0 client.17931 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 --2- 172.21.15.94:0/2398946050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562149a01580 0x562149a12f60 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 -- 172.21.15.94:0/2398946050 >> 172.21.15.94:0/2398946050 conn(0x5621499fbcb0 msgr2=0x562149a09130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 -- 172.21.15.94:0/2398946050 shutdown_connections 2024-04-21T08:49:13.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 1 -- 172.21.15.94:0/2398946050 wait complete. 2024-04-21T08:49:13.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.134+0000 7f6fef45c740 20 client.17931 trim_cache size 0 max 16384 2024-04-21T08:49:13.141 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:49:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:49:13.141 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:49:13.141 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:49:13.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 -- 172.21.15.94:0/2546849478 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6fd00a3d20 msgr2=0x7f6fd00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6fd00a3d20 0x7f6fd00a61e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00125e0 tx=0x7f6fe00129e0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 -- 172.21.15.94:0/2546849478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe8103ae0 msgr2=0x7f6fe8198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe8103ae0 0x7f6fe8198c90 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc005940 tx=0x7f6fdc02d9a0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 -- 172.21.15.94:0/2546849478 shutdown_connections 2024-04-21T08:49:13.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f6fd00a3d20 0x7f6fd00a61e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fe810c500 0x7f6fe819d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe8103ae0 0x7f6fe8198c90 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 --2- 172.21.15.94:0/2546849478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fe81031c0 0x7f6fe8198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 -- 172.21.15.94:0/2546849478 >> 172.21.15.94:0/2546849478 conn(0x7f6fe8078000 msgr2=0x7f6fe80fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 -- 172.21.15.94:0/2546849478 shutdown_connections 2024-04-21T08:49:13.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.135+0000 7f6fec870640 1 -- 172.21.15.94:0/2546849478 wait complete. 2024-04-21T08:49:13.145 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:12 smithi094 ceph-mon[27988]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T08:49:13.146 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:49:13.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:49:13.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:12 smithi111 ceph-mon[31493]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T08:49:13.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.245+0000 7fad06f1e640 1 -- 172.21.15.94:0/1153006872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad00108010 msgr2=0x7fad001083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.245+0000 7fad06f1e640 1 --2- 172.21.15.94:0/1153006872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad00108010 0x7fad001083f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7facf0009900 tx=0x7facf002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 -- 172.21.15.94:0/1153006872 shutdown_connections 2024-04-21T08:49:13.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 --2- 172.21.15.94:0/1153006872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad001028f0 0x7fad0010edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 --2- 172.21.15.94:0/1153006872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad00101f50 0x7fad001023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 --2- 172.21.15.94:0/1153006872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad00108010 0x7fad001083f0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 -- 172.21.15.94:0/1153006872 >> 172.21.15.94:0/1153006872 conn(0x7fad000793e0 msgr2=0x7fad000797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 -- 172.21.15.94:0/1153006872 shutdown_connections 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 -- 172.21.15.94:0/1153006872 wait complete. 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 Processor -- start 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 -- start start 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad00101f50 0x7fad00198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad001028f0 0x7fad00198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad00108010 0x7fad0019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fad001121f0 con 0x7fad00108010 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fad00112070 con 0x7fad00101f50 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad06f1e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fad00112370 con 0x7fad001028f0 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad054bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad00108010 0x7fad0019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad054bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad00108010 0x7fad0019cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49178/0 (socket says 172.21.15.94:49178) 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.246+0000 7fad054bb640 1 -- 172.21.15.94:0/3744849573 learned_addr learned my addr 172.21.15.94:0/3744849573 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7facfffff640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad001028f0 0x7fad00198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7fad04cba640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad00101f50 0x7fad00198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7fad054bb640 1 -- 172.21.15.94:0/3744849573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad001028f0 msgr2=0x7fad00198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7fad054bb640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad001028f0 0x7fad00198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7fad054bb640 1 -- 172.21.15.94:0/3744849573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad00101f50 msgr2=0x7fad00198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7fad054bb640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad00101f50 0x7fad00198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7fad054bb640 1 -- 172.21.15.94:0/3744849573 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0019d6f0 con 0x7fad00108010 2024-04-21T08:49:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7fad04cba640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad00101f50 0x7fad00198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7fad054bb640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad00108010 0x7fad0019cfc0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7facf800eaa0 tx=0x7facf800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7facfdffb640 1 -- 172.21.15.94:0/3744849573 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf800cdb0 con 0x7fad00108010 2024-04-21T08:49:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7facfdffb640 1 -- 172.21.15.94:0/3744849573 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facf8005020 con 0x7fad00108010 2024-04-21T08:49:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7facfdffb640 1 -- 172.21.15.94:0/3744849573 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf80116d0 con 0x7fad00108010 2024-04-21T08:49:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7fad06f1e640 1 -- 172.21.15.94:0/3744849573 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0019d9e0 con 0x7fad00108010 2024-04-21T08:49:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.247+0000 7fad06f1e640 1 -- 172.21.15.94:0/3744849573 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad001a5270 con 0x7fad00108010 2024-04-21T08:49:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.248+0000 7fad06f1e640 1 -- 172.21.15.94:0/3744849573 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad00068de0 con 0x7fad00108010 2024-04-21T08:49:13.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.249+0000 7facfdffb640 1 -- 172.21.15.94:0/3744849573 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7facf80051d0 con 0x7fad00108010 2024-04-21T08:49:13.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.250+0000 7facfdffb640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7face80a3d20 0x7face80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.250+0000 7fad04cba640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7face80a3d20 0x7face80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.250+0000 7facfdffb640 1 -- 172.21.15.94:0/3744849573 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7facf80d0d90 con 0x7fad00108010 2024-04-21T08:49:13.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.250+0000 7fad04cba640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7face80a3d20 0x7face80a61e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7facf0006010 tx=0x7facf0005a20 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.254+0000 7facfdffb640 1 -- 172.21.15.94:0/3744849573 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7facf8016030 con 0x7fad00108010 2024-04-21T08:49:13.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:12 smithi173 ceph-mon[31428]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T08:49:13.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.469+0000 7fad06f1e640 1 -- 172.21.15.94:0/3744849573 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad0002d5c0 con 0x7fad00108010 2024-04-21T08:49:13.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.470+0000 7facfdffb640 1 -- 172.21.15.94:0/3744849573 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7facf809d000 con 0x7fad00108010 2024-04-21T08:49:13.470 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:49:13.470 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:49:13.470 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:49:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 -- 172.21.15.94:0/3744849573 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7face80a3d20 msgr2=0x7face80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7face80a3d20 0x7face80a61e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7facf0006010 tx=0x7facf0005a20 comp rx=0 tx=0).stop 2024-04-21T08:49:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 -- 172.21.15.94:0/3744849573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad00108010 msgr2=0x7fad0019cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad00108010 0x7fad0019cfc0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7facf800eaa0 tx=0x7facf800ef60 comp rx=0 tx=0).stop 2024-04-21T08:49:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 -- 172.21.15.94:0/3744849573 shutdown_connections 2024-04-21T08:49:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7face80a3d20 0x7face80a61e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad001028f0 0x7fad00198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad00108010 0x7fad0019cfc0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 --2- 172.21.15.94:0/3744849573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad00101f50 0x7fad00198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 -- 172.21.15.94:0/3744849573 >> 172.21.15.94:0/3744849573 conn(0x7fad000793e0 msgr2=0x7fad001009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 -- 172.21.15.94:0/3744849573 shutdown_connections 2024-04-21T08:49:13.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.472+0000 7fad06f1e640 1 -- 172.21.15.94:0/3744849573 wait complete. 2024-04-21T08:49:13.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:49:13.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.583+0000 7fa589750640 1 -- 172.21.15.94:0/3450562020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa584107260 msgr2=0x7fa58410ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.583+0000 7fa589750640 1 --2- 172.21.15.94:0/3450562020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa584107260 0x7fa58410ee30 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fa574009900 tx=0x7fa57402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:13.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.583+0000 7fa589750640 1 -- 172.21.15.94:0/3450562020 shutdown_connections 2024-04-21T08:49:13.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.583+0000 7fa589750640 1 --2- 172.21.15.94:0/3450562020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa584102b40 0x7fa584106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.583+0000 7fa589750640 1 --2- 172.21.15.94:0/3450562020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa584107260 0x7fa58410ee30 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.583+0000 7fa589750640 1 --2- 172.21.15.94:0/3450562020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa584102190 0x7fa584102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.583+0000 7fa589750640 1 -- 172.21.15.94:0/3450562020 >> 172.21.15.94:0/3450562020 conn(0x7fa5840793a0 msgr2=0x7fa5840797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.583+0000 7fa589750640 1 -- 172.21.15.94:0/3450562020 shutdown_connections 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.583+0000 7fa589750640 1 -- 172.21.15.94:0/3450562020 wait complete. 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.583+0000 7fa589750640 1 Processor -- start 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa589750640 1 -- start start 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa589750640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa584102190 0x7fa5841986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa589750640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa584107260 0x7fa584198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa589750640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa58419cfb0 0x7fa58419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa589750640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa5841121b0 con 0x7fa58419cfb0 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa589750640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa584112030 con 0x7fa584107260 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa589750640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa584112330 con 0x7fa584102190 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa5837fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa58419cfb0 0x7fa58419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa5837fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa58419cfb0 0x7fa58419d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56944/0 (socket says 172.21.15.94:56944) 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa5837fe640 1 -- 172.21.15.94:0/3455792367 learned_addr learned my addr 172.21.15.94:0/3455792367 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa582ffd640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa584102190 0x7fa5841986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa5837fe640 1 -- 172.21.15.94:0/3455792367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa584102190 msgr2=0x7fa5841986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa5837fe640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa584102190 0x7fa5841986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa5837fe640 1 -- 172.21.15.94:0/3455792367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa584107260 msgr2=0x7fa584198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa5837fe640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa584107260 0x7fa584198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa5837fe640 1 -- 172.21.15.94:0/3455792367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa58419daa0 con 0x7fa58419cfb0 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa5837fe640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa58419cfb0 0x7fa58419d460 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fa574009230 tx=0x7fa5740045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.584+0000 7fa56bfff640 1 -- 172.21.15.94:0/3455792367 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574043020 con 0x7fa58419cfb0 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.585+0000 7fa589750640 1 -- 172.21.15.94:0/3455792367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa58419dd30 con 0x7fa58419cfb0 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.585+0000 7fa56bfff640 1 -- 172.21.15.94:0/3455792367 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa57402cb70 con 0x7fa58419cfb0 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.585+0000 7fa56bfff640 1 -- 172.21.15.94:0/3455792367 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574003ac0 con 0x7fa58419cfb0 2024-04-21T08:49:13.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.585+0000 7fa589750640 1 -- 172.21.15.94:0/3455792367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5841a5550 con 0x7fa58419cfb0 2024-04-21T08:49:13.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa56bfff640 1 -- 172.21.15.94:0/3455792367 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa574046020 con 0x7fa58419cfb0 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa58c41b740 1 Processor -- start 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa58c41b740 1 -- start start 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e34d0 0x55e1c40e38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40ec430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40eca00 0x55e1c40eee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa58c41b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55e1c40481f0 con 0x55e1c40e3e80 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa58c41b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e1c3fe54b0 con 0x55e1c40eca00 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa58c41b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e1c40441b0 con 0x55e1c40e34d0 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa56bfff640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa56c0a3d20 0x7fa56c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa582ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40ec430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa582ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40ec430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56960/0 (socket says 172.21.15.94:56960) 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.587+0000 7fa582ffd640 1 -- 172.21.15.94:0/536921593 learned_addr learned my addr 172.21.15.94:0/536921593 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa5827fc640 1 --2- 172.21.15.94:0/536921593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e34d0 0x55e1c40e38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa5837fe640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa56c0a3d20 0x7fa56c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa5827fc640 1 --2- 172.21.15.94:0/536921593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40eca00 0x55e1c40eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa56bfff640 1 -- 172.21.15.94:0/3455792367 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fa5740f2e10 con 0x7fa58419cfb0 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa582ffd640 1 -- 172.21.15.94:0/536921593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e34d0 msgr2=0x55e1c40e38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa582ffd640 1 --2- 172.21.15.94:0/536921593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e34d0 0x55e1c40e38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa582ffd640 1 -- 172.21.15.94:0/536921593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40eca00 msgr2=0x55e1c40eee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa582ffd640 1 --2- 172.21.15.94:0/536921593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40eca00 0x55e1c40eee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa582ffd640 1 -- 172.21.15.94:0/536921593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1c3fd71b0 con 0x55e1c40e3e80 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa5827fc640 1 --2- 172.21.15.94:0/536921593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40eca00 0x55e1c40eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa582ffd640 1 --2- 172.21.15.94:0/536921593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40ec430 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fa57c0039f0 tx=0x7fa57c003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e36080501bfbac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa5837fe640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa56c0a3d20 0x7fa56c0a61e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa574049000 tx=0x7fa5740894d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa5697fa640 1 -- 172.21.15.94:0/536921593 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57c004b20 con 0x55e1c40e3e80 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa5697fa640 1 -- 172.21.15.94:0/536921593 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa57c004cc0 con 0x55e1c40e3e80 2024-04-21T08:49:13.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.588+0000 7fa5697fa640 1 -- 172.21.15.94:0/536921593 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57c0126b0 con 0x55e1c40e3e80 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 -- 172.21.15.94:0/536921593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 msgr2=0x55e1c40ec430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 --2- 172.21.15.94:0/536921593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40ec430 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fa57c0039f0 tx=0x7fa57c003eb0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 -- 172.21.15.94:0/536921593 shutdown_connections 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 --2- 172.21.15.94:0/536921593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40eca00 0x55e1c40eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 --2- 172.21.15.94:0/536921593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40ec430 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 --2- 172.21.15.94:0/536921593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e34d0 0x55e1c40e38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 -- 172.21.15.94:0/536921593 >> 172.21.15.94:0/536921593 conn(0x55e1c40ddba0 msgr2=0x55e1c40ddf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 -- 172.21.15.94:0/536921593 shutdown_connections 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 -- 172.21.15.94:0/536921593 wait complete. 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 -- start start 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 Processor -- start 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40ec300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e3e80 0x55e1c40e9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40eca00 0x55e1c40e9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55e1c40ef970 con 0x55e1c40eca00 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e1c4001500 con 0x55e1c40e34d0 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.589+0000 7fa58c41b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e1c40efaf0 con 0x55e1c40e3e80 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa58c41b740 10 client.0 _ll_register_callbacks cb 0x55e1c3fcfb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa5827fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40eca00 0x55e1c40e9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa5827fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40eca00 0x55e1c40e9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56974/0 (socket says 172.21.15.94:56974) 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa5827fc640 1 -- 172.21.15.94:0/312920384 learned_addr learned my addr 172.21.15.94:0/312920384 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa5827fc640 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40ec300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa582ffd640 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e3e80 0x55e1c40e9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa5827fc640 1 -- 172.21.15.94:0/312920384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e3e80 msgr2=0x55e1c40e9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa5827fc640 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e3e80 0x55e1c40e9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa5827fc640 1 -- 172.21.15.94:0/312920384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 msgr2=0x55e1c40ec300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa5827fc640 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40ec300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa5827fc640 1 -- 172.21.15.94:0/312920384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1c3d5a3f0 con 0x55e1c40eca00 2024-04-21T08:49:13.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa582ffd640 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e3e80 0x55e1c40e9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:13.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa5827fc640 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40eca00 0x55e1c40e9c10 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fa57800df80 tx=0x7fa57800d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e7241be84ddbe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa566ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:13.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa566ff5640 1 -- 172.21.15.94:0/312920384 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57800fcd0 con 0x55e1c40eca00 2024-04-21T08:49:13.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa566ff5640 1 -- 172.21.15.94:0/312920384 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fa578005020 con 0x55e1c40eca00 2024-04-21T08:49:13.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa58c41b740 1 -- 172.21.15.94:0/312920384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa57c004cc0 con 0x55e1c40eca00 2024-04-21T08:49:13.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa566ff5640 1 -- 172.21.15.94:0/312920384 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa578025050 con 0x55e1c40eca00 2024-04-21T08:49:13.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa58c41b740 10 client.17949 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:13.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.590+0000 7fa58c41b740 1 -- 172.21.15.94:0/312920384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e1c3e247d0 con 0x55e1c40eca00 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa566ff5640 1 -- 172.21.15.94:0/312920384 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa57c004cc0 con 0x55e1c40eca00 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa566ff5640 1 -- 172.21.15.94:0/312920384 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa5780103d0 con 0x55e1c40eca00 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa58c41b740 10 client.17949 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa58c41b740 10 client.17949 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa58c41b740 20 client.17949 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa58c41b740 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55e1c40fccd0 0x55e1c40ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa58c41b740 4 client.17949 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa58c41b740 1 -- 172.21.15.94:0/312920384 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1c3fe7930 con 0x55e1c40fccd0 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa582ffd640 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55e1c40fccd0 0x55e1c40ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa582ffd640 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55e1c40fccd0 0x55e1c40ff0c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=32101c4e3724cf32 server_cookie=4e8a808013cbfc29 in_seq=0 out_seq=0 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.591+0000 7fa566ff5640 10 client.17949 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa566ff5640 1 -- 172.21.15.94:0/312920384 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e1c3fe7930 con 0x55e1c40fccd0 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa566ff5640 10 client.17949 handle_command_reply: tid=0 2024-04-21T08:49:13.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 client.17949 shutdown 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 -- 172.21.15.94:0/312920384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40eca00 msgr2=0x55e1c40e9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40eca00 0x55e1c40e9c10 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fa57800df80 tx=0x7fa57800d9e0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 -- 172.21.15.94:0/312920384 shutdown_connections 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55e1c40fccd0 0x55e1c40ff0c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40e3e80 0x55e1c40e9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40eca00 0x55e1c40e9c10 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa566ff5640 0 client.17949 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 --2- 172.21.15.94:0/312920384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40ec300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 -- 172.21.15.94:0/312920384 >> 172.21.15.94:0/312920384 conn(0x55e1c40ddba0 msgr2=0x55e1c40df8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 -- 172.21.15.94:0/312920384 shutdown_connections 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 1 -- 172.21.15.94:0/312920384 wait complete. 2024-04-21T08:49:13.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.592+0000 7fa58c41b740 20 client.17949 trim_cache size 0 max 16384 2024-04-21T08:49:13.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.605+0000 7fa58c41b740 1 Processor -- start 2024-04-21T08:49:13.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.605+0000 7fa58c41b740 1 -- start start 2024-04-21T08:49:13.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.605+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40f8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.605+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40e85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.605+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40eca00 0x55e1c40ebdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.605+0000 7fa58c41b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa5841b1dd0 con 0x55e1c40e3e80 2024-04-21T08:49:13.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.605+0000 7fa58c41b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa5841af6a0 con 0x55e1c40e34d0 2024-04-21T08:49:13.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.605+0000 7fa58c41b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa5841ad510 con 0x55e1c40eca00 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa582ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40e85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa582ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40e85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56976/0 (socket says 172.21.15.94:56976) 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa582ffd640 1 -- 172.21.15.94:0/3904357639 learned_addr learned my addr 172.21.15.94:0/3904357639 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa5827fc640 1 --2- 172.21.15.94:0/3904357639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa582ffd640 1 -- 172.21.15.94:0/3904357639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40eca00 msgr2=0x55e1c40ebdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa582ffd640 1 --2- 172.21.15.94:0/3904357639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40eca00 0x55e1c40ebdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa582ffd640 1 -- 172.21.15.94:0/3904357639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 msgr2=0x55e1c40f8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa582ffd640 1 --2- 172.21.15.94:0/3904357639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40f8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa582ffd640 1 -- 172.21.15.94:0/3904357639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1c40e27c0 con 0x55e1c40e3e80 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa582ffd640 1 --2- 172.21.15.94:0/3904357639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40e85d0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x55e1c40e2fa0 tx=0x7fa57c0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed819834f85ce414 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa568ff9640 1 -- 172.21.15.94:0/3904357639 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57c02f070 con 0x55e1c40e3e80 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa568ff9640 1 -- 172.21.15.94:0/3904357639 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa57c0047c0 con 0x55e1c40e3e80 2024-04-21T08:49:13.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.606+0000 7fa568ff9640 1 -- 172.21.15.94:0/3904357639 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57c012450 con 0x55e1c40e3e80 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 -- 172.21.15.94:0/3904357639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 msgr2=0x55e1c40e85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 --2- 172.21.15.94:0/3904357639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40e85d0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x55e1c40e2fa0 tx=0x7fa57c0096c0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 -- 172.21.15.94:0/3904357639 shutdown_connections 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 --2- 172.21.15.94:0/3904357639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40eca00 0x55e1c40ebdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 --2- 172.21.15.94:0/3904357639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40e85d0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 --2- 172.21.15.94:0/3904357639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40f8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 -- 172.21.15.94:0/3904357639 >> 172.21.15.94:0/3904357639 conn(0x55e1c40ddba0 msgr2=0x55e1c40dcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 -- 172.21.15.94:0/3904357639 shutdown_connections 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 -- 172.21.15.94:0/3904357639 wait complete. 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 -- start start 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 Processor -- start 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40f5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40eca00 0x55e1c40f0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa5841adb60 con 0x55e1c40e3e80 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa5841aefd0 con 0x55e1c40e34d0 2024-04-21T08:49:13.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa5841ae230 con 0x55e1c40eca00 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa58c41b740 10 client.0 _ll_register_callbacks cb 0x7fa578012550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa582ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40f5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa582ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40f5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56978/0 (socket says 172.21.15.94:56978) 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.607+0000 7fa582ffd640 1 -- 172.21.15.94:0/3107740414 learned_addr learned my addr 172.21.15.94:0/3107740414 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa5827fc640 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40eca00 0x55e1c40f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa5827fc640 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa582ffd640 1 -- 172.21.15.94:0/3107740414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40eca00 msgr2=0x55e1c40f0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa582ffd640 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40eca00 0x55e1c40f0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa582ffd640 1 -- 172.21.15.94:0/3107740414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 msgr2=0x55e1c40f4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa582ffd640 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40f4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa582ffd640 1 -- 172.21.15.94:0/3107740414 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1c40f0d30 con 0x55e1c40e3e80 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa5827fc640 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa582ffd640 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40f5300 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fa57c000f80 tx=0x7fa57c004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97fa731517828cf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa566ff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa566ff5640 1 -- 172.21.15.94:0/3107740414 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57c038060 con 0x55e1c40e3e80 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa566ff5640 1 -- 172.21.15.94:0/3107740414 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fa57c029050 con 0x55e1c40e3e80 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa58c41b740 1 -- 172.21.15.94:0/3107740414 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa57c0047c0 con 0x55e1c40e3e80 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa566ff5640 1 -- 172.21.15.94:0/3107740414 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa57c03f050 con 0x55e1c40e3e80 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa58c41b740 10 client.17955 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa58c41b740 1 -- 172.21.15.94:0/3107740414 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e1c4162f30 con 0x55e1c40e3e80 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.608+0000 7fa566ff5640 1 -- 172.21.15.94:0/3107740414 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa57c0047c0 con 0x55e1c40e3e80 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa566ff5640 1 -- 172.21.15.94:0/3107740414 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa57c00f040 con 0x55e1c40e3e80 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa58c41b740 10 client.17955 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa58c41b740 10 client.17955 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:49:13.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa58c41b740 20 client.17955 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa58c41b740 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55e1c41634b0 0x55e1c41658d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa58c41b740 4 client.17955 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa58c41b740 1 -- 172.21.15.94:0/3107740414 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e1c4055c10 con 0x55e1c41634b0 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa5827fc640 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55e1c41634b0 0x55e1c41658d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa566ff5640 10 client.17955 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa5827fc640 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55e1c41634b0 0x55e1c41658d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f2fff3f137d4e3b4 server_cookie=fa04da9f414aa333 in_seq=0 out_seq=0 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa566ff5640 1 -- 172.21.15.94:0/3107740414 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e1c4055c10 con 0x55e1c41634b0 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa566ff5640 10 client.17955 handle_command_reply: tid=0 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.609+0000 7fa58c41b740 1 client.17955 shutdown 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 1 -- 172.21.15.94:0/3107740414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 msgr2=0x55e1c40f5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40f5300 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fa57c000f80 tx=0x7fa57c004970 comp rx=0 tx=0).stop 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 1 -- 172.21.15.94:0/3107740414 shutdown_connections 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55e1c41634b0 0x55e1c41658d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e1c40eca00 0x55e1c40f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa566ff5640 0 client.17955 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:49:13.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e1c40e3e80 0x55e1c40f5300 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 1 --2- 172.21.15.94:0/3107740414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1c40e34d0 0x55e1c40f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 1 -- 172.21.15.94:0/3107740414 >> 172.21.15.94:0/3107740414 conn(0x55e1c40ddba0 msgr2=0x55e1c40e4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 1 -- 172.21.15.94:0/3107740414 shutdown_connections 2024-04-21T08:49:13.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 1 -- 172.21.15.94:0/3107740414 wait complete. 2024-04-21T08:49:13.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa58c41b740 20 client.17955 trim_cache size 0 max 16384 2024-04-21T08:49:13.617 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:49:13.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 -- 172.21.15.94:0/3455792367 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa56c0a3d20 msgr2=0x7fa56c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa56c0a3d20 0x7fa56c0a61e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa574049000 tx=0x7fa5740894d0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 -- 172.21.15.94:0/3455792367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa58419cfb0 msgr2=0x7fa58419d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa58419cfb0 0x7fa58419d460 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fa574009230 tx=0x7fa5740045e0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 -- 172.21.15.94:0/3455792367 shutdown_connections 2024-04-21T08:49:13.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa56c0a3d20 0x7fa56c0a61e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa58419cfb0 0x7fa58419d460 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa584107260 0x7fa584198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 --2- 172.21.15.94:0/3455792367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa584102190 0x7fa5841986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 -- 172.21.15.94:0/3455792367 >> 172.21.15.94:0/3455792367 conn(0x7fa5840793a0 msgr2=0x7fa584100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 -- 172.21.15.94:0/3455792367 shutdown_connections 2024-04-21T08:49:13.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.610+0000 7fa589750640 1 -- 172.21.15.94:0/3455792367 wait complete. 2024-04-21T08:49:13.621 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:49:13.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.721+0000 7ff1e5f15640 1 -- 172.21.15.94:0/4140795841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1e01010a0 msgr2=0x7ff1e0101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.721+0000 7ff1e5f15640 1 --2- 172.21.15.94:0/4140795841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1e01010a0 0x7ff1e0101480 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009900 tx=0x7ff1d002bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:13.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.721+0000 7ff1e5f15640 1 -- 172.21.15.94:0/4140795841 shutdown_connections 2024-04-21T08:49:13.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.721+0000 7ff1e5f15640 1 --2- 172.21.15.94:0/4140795841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1e0106170 0x7ff1e010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.721+0000 7ff1e5f15640 1 --2- 172.21.15.94:0/4140795841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e0101a50 0x7ff1e0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.721+0000 7ff1e5f15640 1 --2- 172.21.15.94:0/4140795841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1e01010a0 0x7ff1e0101480 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.721+0000 7ff1e5f15640 1 -- 172.21.15.94:0/4140795841 >> 172.21.15.94:0/4140795841 conn(0x7ff1e0077eb0 msgr2=0x7ff1e00782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.721+0000 7ff1e5f15640 1 -- 172.21.15.94:0/4140795841 shutdown_connections 2024-04-21T08:49:13.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.721+0000 7ff1e5f15640 1 -- 172.21.15.94:0/4140795841 wait complete. 2024-04-21T08:49:13.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.722+0000 7ff1e5f15640 1 Processor -- start 2024-04-21T08:49:13.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.722+0000 7ff1e5f15640 1 -- start start 2024-04-21T08:49:13.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.722+0000 7ff1e5f15640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1e01010a0 0x7ff1e01975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.722+0000 7ff1e5f15640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e0106170 0x7ff1e0197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.722+0000 7ff1e5f15640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1e019bea0 0x7ff1e019c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.722+0000 7ff1e5f15640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01110e0 con 0x7ff1e019bea0 2024-04-21T08:49:13.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.722+0000 7ff1e5f15640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e0110f60 con 0x7ff1e0106170 2024-04-21T08:49:13.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.722+0000 7ff1e5f15640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e0111260 con 0x7ff1e01010a0 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.722+0000 7ff1dffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1e019bea0 0x7ff1e019c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1e019bea0 0x7ff1e019c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57006/0 (socket says 172.21.15.94:57006) 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dffff640 1 -- 172.21.15.94:0/2729451799 learned_addr learned my addr 172.21.15.94:0/2729451799 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1df7fe640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1e01010a0 0x7ff1e01975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1deffd640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e0106170 0x7ff1e0197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dffff640 1 -- 172.21.15.94:0/2729451799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1e01010a0 msgr2=0x7ff1e01975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dffff640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1e01010a0 0x7ff1e01975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dffff640 1 -- 172.21.15.94:0/2729451799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e0106170 msgr2=0x7ff1e0197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dffff640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e0106170 0x7ff1e0197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dffff640 1 -- 172.21.15.94:0/2729451799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e019c990 con 0x7ff1e019bea0 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1deffd640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e0106170 0x7ff1e0197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dffff640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1e019bea0 0x7ff1e019c350 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800eaa0 tx=0x7ff1d800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dcff9640 1 -- 172.21.15.94:0/2729451799 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d800cdb0 con 0x7ff1e019bea0 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dcff9640 1 -- 172.21.15.94:0/2729451799 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d8005020 con 0x7ff1e019bea0 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1e5f15640 1 -- 172.21.15.94:0/2729451799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e019cc80 con 0x7ff1e019bea0 2024-04-21T08:49:13.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1dcff9640 1 -- 172.21.15.94:0/2729451799 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80116d0 con 0x7ff1e019bea0 2024-04-21T08:49:13.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.723+0000 7ff1e5f15640 1 -- 172.21.15.94:0/2729451799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e01a4470 con 0x7ff1e019bea0 2024-04-21T08:49:13.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1dcff9640 1 -- 172.21.15.94:0/2729451799 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ff1d80051d0 con 0x7ff1e019bea0 2024-04-21T08:49:13.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1e8be9740 1 Processor -- start 2024-04-21T08:49:13.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1e8be9740 1 -- start start 2024-04-21T08:49:13.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d068b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d06e80 0x55def9d0f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d0fa00 0x55def9d11e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55def9c6b1f0 con 0x55def9d064d0 2024-04-21T08:49:13.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55def9c084b0 con 0x55def9d0fa00 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55def9c671b0 con 0x55def9d06e80 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1deffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d068b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1deffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d068b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57016/0 (socket says 172.21.15.94:57016) 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1deffd640 1 -- 172.21.15.94:0/3128336801 learned_addr learned my addr 172.21.15.94:0/3128336801 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.726+0000 7ff1dcff9640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff1c80a3d20 0x7ff1c80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1df7fe640 1 --2- 172.21.15.94:0/3128336801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d06e80 0x55def9d0f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1deffd640 1 --2- 172.21.15.94:0/3128336801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d0fa00 0x55def9d11e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1dffff640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff1c80a3d20 0x7ff1c80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1deffd640 1 -- 172.21.15.94:0/3128336801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d06e80 msgr2=0x55def9d0f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1deffd640 1 --2- 172.21.15.94:0/3128336801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d06e80 0x55def9d0f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1deffd640 1 -- 172.21.15.94:0/3128336801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d0fa00 msgr2=0x55def9d11e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1deffd640 1 --2- 172.21.15.94:0/3128336801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d0fa00 0x55def9d11e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1deffd640 1 -- 172.21.15.94:0/3128336801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55def9bfa1b0 con 0x55def9d064d0 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1dcff9640 1 -- 172.21.15.94:0/2729451799 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7ff1d80d0d70 con 0x7ff1e019bea0 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1deffd640 1 --2- 172.21.15.94:0/3128336801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d068b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4014750 tx=0x7ff1d400c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5535b3e99c97dbff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1c5ffb640 1 -- 172.21.15.94:0/3128336801 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d401a030 con 0x55def9d064d0 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1dffff640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff1c80a3d20 0x7ff1c80a61e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8098d40 tx=0x7ff1d8098b80 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1c5ffb640 1 -- 172.21.15.94:0/3128336801 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d40054d0 con 0x55def9d064d0 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.727+0000 7ff1c5ffb640 1 -- 172.21.15.94:0/3128336801 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d40049a0 con 0x55def9d064d0 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 -- 172.21.15.94:0/3128336801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 msgr2=0x55def9d068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/3128336801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d068b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4014750 tx=0x7ff1d400c640 comp rx=0 tx=0).stop 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 -- 172.21.15.94:0/3128336801 shutdown_connections 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/3128336801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d06e80 0x55def9d0f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/3128336801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d0fa00 0x55def9d11e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/3128336801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d068b0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 -- 172.21.15.94:0/3128336801 >> 172.21.15.94:0/3128336801 conn(0x55def9d00ba0 msgr2=0x55def9d00f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 -- 172.21.15.94:0/3128336801 shutdown_connections 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 -- 172.21.15.94:0/3128336801 wait complete. 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 -- start start 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.728+0000 7ff1e8be9740 1 Processor -- start 2024-04-21T08:49:13.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d0f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 0x55def9d0c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d0fa00 0x55def9d0cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55def9d12970 con 0x55def9d064d0 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55def9c24500 con 0x55def9d06e80 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55def9d12af0 con 0x55def9d0fa00 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1e8be9740 10 client.0 _ll_register_callbacks cb 0x55def9bf2b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1deffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d0f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1deffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d0f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57020/0 (socket says 172.21.15.94:57020) 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1deffd640 1 -- 172.21.15.94:0/830905839 learned_addr learned my addr 172.21.15.94:0/830905839 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1deffd640 1 -- 172.21.15.94:0/830905839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d0fa00 msgr2=0x55def9d0cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1deffd640 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d0fa00 0x55def9d0cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1df7fe640 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 0x55def9d0c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1deffd640 1 -- 172.21.15.94:0/830905839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 msgr2=0x55def9d0c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1deffd640 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 0x55def9d0c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1deffd640 1 -- 172.21.15.94:0/830905839 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55def997d3f0 con 0x55def9d064d0 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1deffd640 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d0f2c0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ff1d40058c0 tx=0x7ff1d4010bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59586a7e120ea4aa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1beffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1beffd640 1 -- 172.21.15.94:0/830905839 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d401a050 con 0x55def9d064d0 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1beffd640 1 -- 172.21.15.94:0/830905839 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7ff1d40048c0 con 0x55def9d064d0 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1e8be9740 1 -- 172.21.15.94:0/830905839 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d40054d0 con 0x55def9d064d0 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1beffd640 1 -- 172.21.15.94:0/830905839 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff1d4033050 con 0x55def9d064d0 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1e8be9740 10 client.17967 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.729+0000 7ff1e8be9740 1 -- 172.21.15.94:0/830905839 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55def9a477d0 con 0x55def9d064d0 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.730+0000 7ff1beffd640 1 -- 172.21.15.94:0/830905839 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d40054d0 con 0x55def9d064d0 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.730+0000 7ff1beffd640 1 -- 172.21.15.94:0/830905839 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff1d401fa60 con 0x55def9d064d0 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.730+0000 7ff1e8be9740 10 client.17967 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.730+0000 7ff1e8be9740 10 client.17967 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.730+0000 7ff1e8be9740 20 client.17967 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.730+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55def9d1fa60 0x55def9d21e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.730+0000 7ff1e8be9740 4 client.17967 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.730+0000 7ff1e8be9740 1 -- 172.21.15.94:0/830905839 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55def9c0a930 con 0x55def9d1fa60 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.730+0000 7ff1df7fe640 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55def9d1fa60 0x55def9d21e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1df7fe640 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55def9d1fa60 0x55def9d21e50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ebd0f697989a9251 server_cookie=6e840c458474d64 in_seq=0 out_seq=0 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1beffd640 10 client.17967 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1beffd640 1 -- 172.21.15.94:0/830905839 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55def9c0a930 con 0x55def9d1fa60 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1beffd640 10 client.17967 handle_command_reply: tid=0 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1e8be9740 1 client.17967 shutdown 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1e8be9740 1 -- 172.21.15.94:0/830905839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 msgr2=0x55def9d0f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d0f2c0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ff1d40058c0 tx=0x7ff1d4010bf0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1e8be9740 1 -- 172.21.15.94:0/830905839 shutdown_connections 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d0fa00 0x55def9d0cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55def9d1fa60 0x55def9d21e50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 0x55def9d0c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.731+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/830905839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d0f2c0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.732+0000 7ff1e8be9740 1 -- 172.21.15.94:0/830905839 >> 172.21.15.94:0/830905839 conn(0x55def9d00ba0 msgr2=0x55def9d028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.732+0000 7ff1beffd640 0 client.17967 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:49:13.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.732+0000 7ff1e8be9740 1 -- 172.21.15.94:0/830905839 shutdown_connections 2024-04-21T08:49:13.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.732+0000 7ff1e8be9740 1 -- 172.21.15.94:0/830905839 wait complete. 2024-04-21T08:49:13.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.732+0000 7ff1e8be9740 20 client.17967 trim_cache size 0 max 16384 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1e8be9740 1 Processor -- start 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1e8be9740 1 -- start start 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d064d0 0x55def9d0ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 0x55def9d0f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d0fa00 0x55def9d27480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01afbf0 con 0x55def9d0fa00 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01ad4c0 con 0x55def9d06e80 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01ab330 con 0x55def9d064d0 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1deffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d0fa00 0x55def9d27480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1deffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d0fa00 0x55def9d27480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57032/0 (socket says 172.21.15.94:57032) 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1deffd640 1 -- 172.21.15.94:0/4036562443 learned_addr learned my addr 172.21.15.94:0/4036562443 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1deffd640 1 --2- 172.21.15.94:0/4036562443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d064d0 0x55def9d0ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1deffd640 1 -- 172.21.15.94:0/4036562443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d064d0 msgr2=0x55def9d0ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1deffd640 1 --2- 172.21.15.94:0/4036562443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d064d0 0x55def9d0ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1deffd640 1 -- 172.21.15.94:0/4036562443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 msgr2=0x55def9d0f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1deffd640 1 --2- 172.21.15.94:0/4036562443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 0x55def9d0f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1deffd640 1 -- 172.21.15.94:0/4036562443 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55def9d003d0 con 0x55def9d0fa00 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1deffd640 1 --2- 172.21.15.94:0/4036562443 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d0fa00 0x55def9d27480 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7ff1d401fe60 tx=0x7ff1d4024040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f71b6594744f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.745+0000 7ff1c57fa640 1 -- 172.21.15.94:0/4036562443 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d401a070 con 0x55def9d0fa00 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1c57fa640 1 -- 172.21.15.94:0/4036562443 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d4023070 con 0x55def9d0fa00 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1c57fa640 1 -- 172.21.15.94:0/4036562443 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d402dbc0 con 0x55def9d0fa00 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1e8be9740 1 -- 172.21.15.94:0/4036562443 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d0fa00 msgr2=0x55def9d27480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/4036562443 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d0fa00 0x55def9d27480 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7ff1d401fe60 tx=0x7ff1d4024040 comp rx=0 tx=0).stop 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1e8be9740 1 -- 172.21.15.94:0/4036562443 shutdown_connections 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/4036562443 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d0fa00 0x55def9d27480 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/4036562443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 0x55def9d0f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/4036562443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d064d0 0x55def9d0ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1e8be9740 1 -- 172.21.15.94:0/4036562443 >> 172.21.15.94:0/4036562443 conn(0x55def9d00ba0 msgr2=0x55def9d02860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1e8be9740 1 -- 172.21.15.94:0/4036562443 shutdown_connections 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1e8be9740 1 -- 172.21.15.94:0/4036562443 wait complete. 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.746+0000 7ff1e8be9740 1 -- start start 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1e8be9740 1 Processor -- start 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d17d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 0x55def9d18280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1e8be9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d0fa00 0x55def9d08c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01ab980 con 0x55def9d064d0 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01acdf0 con 0x55def9d06e80 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1e8be9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01ac050 con 0x55def9d0fa00 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1e8be9740 10 client.0 _ll_register_callbacks cb 0x7ff1d400c340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1deffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d17d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1deffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d17d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57048/0 (socket says 172.21.15.94:57048) 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1deffd640 1 -- 172.21.15.94:0/2177866666 learned_addr learned my addr 172.21.15.94:0/2177866666 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1deffd640 1 -- 172.21.15.94:0/2177866666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d0fa00 msgr2=0x55def9d08c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1deffd640 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d0fa00 0x55def9d08c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1deffd640 1 -- 172.21.15.94:0/2177866666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 msgr2=0x55def9d18280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1deffd640 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 0x55def9d18280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1deffd640 1 -- 172.21.15.94:0/2177866666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55def9d094a0 con 0x55def9d064d0 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1deffd640 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d17d40 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4000c00 tx=0x7ff1d4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744044657d957554 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1beffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.747+0000 7ff1beffd640 1 -- 172.21.15.94:0/2177866666 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d401a040 con 0x55def9d064d0 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1beffd640 1 -- 172.21.15.94:0/2177866666 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7ff1d4010d10 con 0x55def9d064d0 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1e8be9740 1 -- 172.21.15.94:0/2177866666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d4023070 con 0x55def9d064d0 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1beffd640 1 -- 172.21.15.94:0/2177866666 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff1d402d040 con 0x55def9d064d0 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1e8be9740 10 client.17973 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1e8be9740 1 -- 172.21.15.94:0/2177866666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55def9d85830 con 0x55def9d064d0 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1beffd640 1 -- 172.21.15.94:0/2177866666 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d4023070 con 0x55def9d064d0 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1beffd640 1 -- 172.21.15.94:0/2177866666 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff1d4004660 con 0x55def9d064d0 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1e8be9740 10 client.17973 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1e8be9740 10 client.17973 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1e8be9740 20 client.17973 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55def9d85eb0 0x55def9d882d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1e8be9740 4 client.17973 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1e8be9740 1 -- 172.21.15.94:0/2177866666 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55def9c78c10 con 0x55def9d85eb0 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.748+0000 7ff1df7fe640 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55def9d85eb0 0x55def9d882d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1beffd640 10 client.17973 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1df7fe640 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55def9d85eb0 0x55def9d882d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e63ca336150f8de9 server_cookie=2812868310309745 in_seq=0 out_seq=0 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1beffd640 1 -- 172.21.15.94:0/2177866666 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55def9c78c10 con 0x55def9d85eb0 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1beffd640 10 client.17973 handle_command_reply: tid=0 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 client.17973 shutdown 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 -- 172.21.15.94:0/2177866666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 msgr2=0x55def9d17d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d17d40 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4000c00 tx=0x7ff1d4010740 comp rx=0 tx=0).stop 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 -- 172.21.15.94:0/2177866666 shutdown_connections 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55def9d0fa00 0x55def9d08c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55def9d85eb0 0x55def9d882d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55def9d06e80 0x55def9d18280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 --2- 172.21.15.94:0/2177866666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55def9d064d0 0x55def9d17d40 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1beffd640 0 client.17973 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:49:13.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 -- 172.21.15.94:0/2177866666 >> 172.21.15.94:0/2177866666 conn(0x55def9d00ba0 msgr2=0x55def9d07260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 -- 172.21.15.94:0/2177866666 shutdown_connections 2024-04-21T08:49:13.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 1 -- 172.21.15.94:0/2177866666 wait complete. 2024-04-21T08:49:13.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.749+0000 7ff1e8be9740 20 client.17973 trim_cache size 0 max 16384 2024-04-21T08:49:13.755 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 -- 172.21.15.94:0/2729451799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff1c80a3d20 msgr2=0x7ff1c80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff1c80a3d20 0x7ff1c80a61e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8098d40 tx=0x7ff1d8098b80 comp rx=0 tx=0).stop 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 -- 172.21.15.94:0/2729451799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1e019bea0 msgr2=0x7ff1e019c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1e019bea0 0x7ff1e019c350 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800eaa0 tx=0x7ff1d800ef60 comp rx=0 tx=0).stop 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 -- 172.21.15.94:0/2729451799 shutdown_connections 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ff1c80a3d20 0x7ff1c80a61e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1e019bea0 0x7ff1e019c350 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e0106170 0x7ff1e0197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 --2- 172.21.15.94:0/2729451799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1e01010a0 0x7ff1e01975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 -- 172.21.15.94:0/2729451799 >> 172.21.15.94:0/2729451799 conn(0x7ff1e0077eb0 msgr2=0x7ff1e00fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 -- 172.21.15.94:0/2729451799 shutdown_connections 2024-04-21T08:49:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.750+0000 7ff1e5f15640 1 -- 172.21.15.94:0/2729451799 wait complete. 2024-04-21T08:49:13.760 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:49:13.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 -- 172.21.15.94:0/2063306359 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113c10c510 msgr2=0x7f113c10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 --2- 172.21.15.94:0/2063306359 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113c10c510 0x7f113c10e9b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f1134009ca0 tx=0x7f113402c230 comp rx=0 tx=0).stop 2024-04-21T08:49:13.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 -- 172.21.15.94:0/2063306359 shutdown_connections 2024-04-21T08:49:13.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 --2- 172.21.15.94:0/2063306359 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113c10c510 0x7f113c10e9b0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 --2- 172.21.15.94:0/2063306359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113c103af0 0x7f113c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 --2- 172.21.15.94:0/2063306359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f113c1031d0 0x7f113c1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 -- 172.21.15.94:0/2063306359 >> 172.21.15.94:0/2063306359 conn(0x7f113c077f50 msgr2=0x7f113c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 -- 172.21.15.94:0/2063306359 shutdown_connections 2024-04-21T08:49:13.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 -- 172.21.15.94:0/2063306359 wait complete. 2024-04-21T08:49:13.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 Processor -- start 2024-04-21T08:49:13.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.863+0000 7f1140afd640 1 -- start start 2024-04-21T08:49:13.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f1140afd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113c1031d0 0x7f113c100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f1140afd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f113c103af0 0x7f113c101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f1140afd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113c10c510 0x7f113c1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f1140afd640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f113c1121a0 con 0x7f113c10c510 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f1140afd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f113c112020 con 0x7f113c1031d0 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f1140afd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f113c112320 con 0x7f113c103af0 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f113ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113c10c510 0x7f113c1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f113ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113c10c510 0x7f113c1018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57066/0 (socket says 172.21.15.94:57066) 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f113ad9d640 1 -- 172.21.15.94:0/4010634775 learned_addr learned my addr 172.21.15.94:0/4010634775 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f113ad9d640 1 -- 172.21.15.94:0/4010634775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f113c103af0 msgr2=0x7f113c101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f1139d9b640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f113c103af0 0x7f113c101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f113ad9d640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f113c103af0 0x7f113c101360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f113ad9d640 1 -- 172.21.15.94:0/4010634775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113c1031d0 msgr2=0x7f113c100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f113ad9d640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113c1031d0 0x7f113c100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f113ad9d640 1 -- 172.21.15.94:0/4010634775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113c1a6c00 con 0x7f113c10c510 2024-04-21T08:49:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f113ad9d640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113c10c510 0x7f113c1018a0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f113402c840 tx=0x7f1134003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f112b7fe640 1 -- 172.21.15.94:0/4010634775 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134039040 con 0x7f113c10c510 2024-04-21T08:49:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f1140afd640 1 -- 172.21.15.94:0/4010634775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f113c1a6e90 con 0x7f113c10c510 2024-04-21T08:49:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f112b7fe640 1 -- 172.21.15.94:0/4010634775 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1134044070 con 0x7f113c10c510 2024-04-21T08:49:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f112b7fe640 1 -- 172.21.15.94:0/4010634775 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11340329c0 con 0x7f113c10c510 2024-04-21T08:49:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.864+0000 7f1140afd640 1 -- 172.21.15.94:0/4010634775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f113c1a73a0 con 0x7f113c10c510 2024-04-21T08:49:13.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.866+0000 7f112b7fe640 1 -- 172.21.15.94:0/4010634775 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1134032b60 con 0x7f113c10c510 2024-04-21T08:49:13.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f11437c7740 1 Processor -- start 2024-04-21T08:49:13.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f11437c7740 1 -- start start 2024-04-21T08:49:13.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f112b7fe640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f11240a3d50 0x7f11240a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f11437c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301e4d0 0x55f72301e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f11437c7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301ee80 0x55f723027430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f11437c7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723029e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f1139d9b640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f11240a3d50 0x7f11240a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f11437c7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55f722f831f0 con 0x55f72301ee80 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f11437c7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f722f204b0 con 0x55f723027a00 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f11437c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f722f7f1b0 con 0x55f72301e4d0 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f112b7fe640 1 -- 172.21.15.94:0/4010634775 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f11340f30b0 con 0x7f113c10c510 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f1139d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301ee80 0x55f723027430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f1139d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301ee80 0x55f723027430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57082/0 (socket says 172.21.15.94:57082) 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.867+0000 7f1139d9b640 1 -- 172.21.15.94:0/251617453 learned_addr learned my addr 172.21.15.94:0/251617453 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f113a59c640 1 --2- 172.21.15.94:0/251617453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301e4d0 0x55f72301e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f113ad9d640 1 --2- 172.21.15.94:0/251617453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723029e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f1139d9b640 1 -- 172.21.15.94:0/251617453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301e4d0 msgr2=0x55f72301e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f1139d9b640 1 --2- 172.21.15.94:0/251617453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301e4d0 0x55f72301e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f1139d9b640 1 -- 172.21.15.94:0/251617453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 msgr2=0x55f723029e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f1139d9b640 1 --2- 172.21.15.94:0/251617453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723029e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f1139d9b640 1 -- 172.21.15.94:0/251617453 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f722f121b0 con 0x55f72301ee80 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f113ad9d640 1 --2- 172.21.15.94:0/251617453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723029e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f1139d9b640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f11240a3d50 0x7f11240a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1130015890 tx=0x7f1130009a30 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f1139d9b640 1 --2- 172.21.15.94:0/251617453 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301ee80 0x55f723027430 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f11300136a0 tx=0x7f113001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa972853c957b29 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f1128ff9640 1 -- 172.21.15.94:0/251617453 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11300092a0 con 0x55f72301ee80 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f1128ff9640 1 -- 172.21.15.94:0/251617453 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11300054d0 con 0x55f72301ee80 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.868+0000 7f1128ff9640 1 -- 172.21.15.94:0/251617453 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1130004d80 con 0x55f72301ee80 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 -- 172.21.15.94:0/251617453 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301ee80 msgr2=0x55f723027430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 --2- 172.21.15.94:0/251617453 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301ee80 0x55f723027430 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f11300136a0 tx=0x7f113001a000 comp rx=0 tx=0).stop 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 -- 172.21.15.94:0/251617453 shutdown_connections 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 --2- 172.21.15.94:0/251617453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723029e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 --2- 172.21.15.94:0/251617453 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301ee80 0x55f723027430 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 --2- 172.21.15.94:0/251617453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301e4d0 0x55f72301e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 -- 172.21.15.94:0/251617453 >> 172.21.15.94:0/251617453 conn(0x55f723018ba0 msgr2=0x55f723018f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 -- 172.21.15.94:0/251617453 shutdown_connections 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 -- 172.21.15.94:0/251617453 wait complete. 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 -- start start 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 Processor -- start 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.869+0000 7f11437c7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723027340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f11437c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 0x55f7230246d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f11437c7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723024ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f11437c7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55f72302a910 con 0x55f72301e4d0 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f11437c7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f722f3c500 con 0x55f723027a00 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f11437c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f72302aa90 con 0x55f72301ee80 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f11437c7740 10 client.0 _ll_register_callbacks cb 0x55f722f0ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f113a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723027340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f113a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723027340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57086/0 (socket says 172.21.15.94:57086) 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f113a59c640 1 -- 172.21.15.94:0/3910759811 learned_addr learned my addr 172.21.15.94:0/3910759811 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f1139d9b640 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 0x55f7230246d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f113a59c640 1 -- 172.21.15.94:0/3910759811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 msgr2=0x55f7230246d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f113a59c640 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 0x55f7230246d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f113a59c640 1 -- 172.21.15.94:0/3910759811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 msgr2=0x55f723024ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f113a59c640 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723024ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f113a59c640 1 -- 172.21.15.94:0/3910759811 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f722c953f0 con 0x55f72301e4d0 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f113a59c640 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723027340 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f112c0039b0 tx=0x7f112c003e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336637e72a2815cb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f1121ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f1121ffb640 1 -- 172.21.15.94:0/3910759811 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c01a470 con 0x55f72301e4d0 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f1121ffb640 1 -- 172.21.15.94:0/3910759811 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f112c01a610 con 0x55f72301e4d0 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f11437c7740 1 -- 172.21.15.94:0/3910759811 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11300054d0 con 0x55f72301e4d0 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f1121ffb640 1 -- 172.21.15.94:0/3910759811 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f112c023050 con 0x55f72301e4d0 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f11437c7740 10 client.17985 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.870+0000 7f11437c7740 1 -- 172.21.15.94:0/3910759811 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f722d5f7d0 con 0x55f72301e4d0 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f1121ffb640 1 -- 172.21.15.94:0/3910759811 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11300054d0 con 0x55f72301e4d0 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f1121ffb640 1 -- 172.21.15.94:0/3910759811 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f112c011a10 con 0x55f72301e4d0 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f11437c7740 10 client.17985 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:13.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f11437c7740 10 client.17985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f11437c7740 20 client.17985 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f11437c7740 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f723037bf0 0x55f72303a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f11437c7740 4 client.17985 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f11437c7740 1 -- 172.21.15.94:0/3910759811 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f722f22930 con 0x55f723037bf0 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f1139d9b640 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f723037bf0 0x55f72303a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f1121ffb640 10 client.17985 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.871+0000 7f1139d9b640 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f723037bf0 0x55f72303a040 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ecf30722daf5c53 server_cookie=698cc9cf8df938fe in_seq=0 out_seq=0 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f1121ffb640 1 -- 172.21.15.94:0/3910759811 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f722f22930 con 0x55f723037bf0 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f1121ffb640 10 client.17985 handle_command_reply: tid=0 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 client.17985 shutdown 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 -- 172.21.15.94:0/3910759811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 msgr2=0x55f723027340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723027340 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f112c0039b0 tx=0x7f112c003e70 comp rx=0 tx=0).stop 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 -- 172.21.15.94:0/3910759811 shutdown_connections 2024-04-21T08:49:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f723037bf0 0x55f72303a040 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 0x55f7230246d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723024ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f1121ffb640 0 client.17985 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 --2- 172.21.15.94:0/3910759811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723027340 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 -- 172.21.15.94:0/3910759811 >> 172.21.15.94:0/3910759811 conn(0x55f723018ba0 msgr2=0x55f72301a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 -- 172.21.15.94:0/3910759811 shutdown_connections 2024-04-21T08:49:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 1 -- 172.21.15.94:0/3910759811 wait complete. 2024-04-21T08:49:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.872+0000 7f11437c7740 20 client.17985 trim_cache size 0 max 16384 2024-04-21T08:49:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.885+0000 7f11437c7740 1 Processor -- start 2024-04-21T08:49:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f11437c7740 1 -- start start 2024-04-21T08:49:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f11437c7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723033820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f11437c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 0x55f723026e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f11437c7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723027340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f11437c7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f113c1b3fe0 con 0x55f72301e4d0 2024-04-21T08:49:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f11437c7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f113c1b18b0 con 0x55f723027a00 2024-04-21T08:49:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f11437c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f113c1af720 con 0x55f72301ee80 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f113a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723033820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f113a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723033820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57090/0 (socket says 172.21.15.94:57090) 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f113a59c640 1 -- 172.21.15.94:0/1993631692 learned_addr learned my addr 172.21.15.94:0/1993631692 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f113a59c640 1 -- 172.21.15.94:0/1993631692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 msgr2=0x55f723026e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f113a59c640 1 --2- 172.21.15.94:0/1993631692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 0x55f723026e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f113a59c640 1 -- 172.21.15.94:0/1993631692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 msgr2=0x55f723027340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f113a59c640 1 --2- 172.21.15.94:0/1993631692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723027340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f113a59c640 1 -- 172.21.15.94:0/1993631692 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7230252e0 con 0x55f72301e4d0 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f113a59c640 1 --2- 172.21.15.94:0/1993631692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723033820 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f112c000c00 tx=0x7f112c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae285a1a7dc85afe server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f1123fff640 1 -- 172.21.15.94:0/1993631692 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c00b960 con 0x55f72301e4d0 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f1123fff640 1 -- 172.21.15.94:0/1993631692 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f112c015070 con 0x55f72301e4d0 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.886+0000 7f1123fff640 1 -- 172.21.15.94:0/1993631692 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c011680 con 0x55f72301e4d0 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 -- 172.21.15.94:0/1993631692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 msgr2=0x55f723033820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 --2- 172.21.15.94:0/1993631692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723033820 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f112c000c00 tx=0x7f112c0046c0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 -- 172.21.15.94:0/1993631692 shutdown_connections 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 --2- 172.21.15.94:0/1993631692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 0x55f723026e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 --2- 172.21.15.94:0/1993631692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f723027a00 0x55f723027340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 --2- 172.21.15.94:0/1993631692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f72301e4d0 0x55f723033820 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 -- 172.21.15.94:0/1993631692 >> 172.21.15.94:0/1993631692 conn(0x55f723018ba0 msgr2=0x55f723017c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 -- 172.21.15.94:0/1993631692 shutdown_connections 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 -- 172.21.15.94:0/1993631692 wait complete. 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 -- start start 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 Processor -- start 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.887+0000 7f11437c7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f72301e4d0 0x55f723030260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f11437c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 0x55f72302b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f11437c7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f723027a00 0x55f72302bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f11437c7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f113c1afd70 con 0x55f723027a00 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f11437c7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f113c1b11e0 con 0x55f72301e4d0 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f11437c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f113c1b0440 con 0x55f72301ee80 2024-04-21T08:49:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f11437c7740 10 client.0 _ll_register_callbacks cb 0x7f112c0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f113ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f723027a00 0x55f72302bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f113ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f723027a00 0x55f72302bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57104/0 (socket says 172.21.15.94:57104) 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f113ad9d640 1 -- 172.21.15.94:0/1369685295 learned_addr learned my addr 172.21.15.94:0/1369685295 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f113ad9d640 1 -- 172.21.15.94:0/1369685295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 msgr2=0x55f72302b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f113ad9d640 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 0x55f72302b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f113ad9d640 1 -- 172.21.15.94:0/1369685295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f72301e4d0 msgr2=0x55f723030260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f113ad9d640 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f72301e4d0 0x55f723030260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f113ad9d640 1 -- 172.21.15.94:0/1369685295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f72309d180 con 0x55f723027a00 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f113ad9d640 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f723027a00 0x55f72302bcf0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f1134009dc0 tx=0x7f11340bb0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0899dde9393d3cd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f1121ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f1121ffb640 1 -- 172.21.15.94:0/1369685295 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11340bf190 con 0x55f723027a00 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f1121ffb640 1 -- 172.21.15.94:0/1369685295 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f11340c7510 con 0x55f723027a00 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f11437c7740 1 -- 172.21.15.94:0/1369685295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f112c015070 con 0x55f723027a00 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f1121ffb640 1 -- 172.21.15.94:0/1369685295 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f11340333a0 con 0x55f723027a00 2024-04-21T08:49:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f11437c7740 10 client.17991 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f11437c7740 1 -- 172.21.15.94:0/1369685295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f72309e300 con 0x55f723027a00 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.888+0000 7f1121ffb640 1 -- 172.21.15.94:0/1369685295 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f112c015070 con 0x55f723027a00 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f1121ffb640 1 -- 172.21.15.94:0/1369685295 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f113403c030 con 0x55f723027a00 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f11437c7740 10 client.17991 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f11437c7740 10 client.17991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f11437c7740 20 client.17991 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f11437c7740 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f72309ea20 0x55f7230a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f11437c7740 4 client.17991 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f11437c7740 1 -- 172.21.15.94:0/1369685295 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f722f90c10 con 0x55f72309ea20 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f113a59c640 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f72309ea20 0x55f7230a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f113a59c640 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f72309ea20 0x55f7230a0e40 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc268e49ddf871b3 server_cookie=e4cf3829ada73555 in_seq=0 out_seq=0 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f1121ffb640 10 client.17991 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f1121ffb640 1 -- 172.21.15.94:0/1369685295 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f722f90c10 con 0x55f72309ea20 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f1121ffb640 10 client.17991 handle_command_reply: tid=0 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.889+0000 7f11437c7740 1 client.17991 shutdown 2024-04-21T08:49:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 1 -- 172.21.15.94:0/1369685295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f723027a00 msgr2=0x55f72302bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f723027a00 0x55f72302bcf0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f1134009dc0 tx=0x7f11340bb0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 1 -- 172.21.15.94:0/1369685295 shutdown_connections 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55f72309ea20 0x55f7230a0e40 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f72301ee80 0x55f72302b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1121ffb640 0 client.17991 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f723027a00 0x55f72302bcf0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 1 --2- 172.21.15.94:0/1369685295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f72301e4d0 0x55f723030260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 1 -- 172.21.15.94:0/1369685295 >> 172.21.15.94:0/1369685295 conn(0x55f723018ba0 msgr2=0x55f72301f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 1 -- 172.21.15.94:0/1369685295 shutdown_connections 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 1 -- 172.21.15.94:0/1369685295 wait complete. 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f11437c7740 20 client.17991 trim_cache size 0 max 16384 2024-04-21T08:49:13.896 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:49:13.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 -- 172.21.15.94:0/4010634775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f11240a3d50 msgr2=0x7f11240a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f11240a3d50 0x7f11240a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1130015890 tx=0x7f1130009a30 comp rx=0 tx=0).stop 2024-04-21T08:49:13.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 -- 172.21.15.94:0/4010634775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113c10c510 msgr2=0x7f113c1018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113c10c510 0x7f113c1018a0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f113402c840 tx=0x7f1134003c20 comp rx=0 tx=0).stop 2024-04-21T08:49:13.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 -- 172.21.15.94:0/4010634775 shutdown_connections 2024-04-21T08:49:13.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f11240a3d50 0x7f11240a6210 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f113c103af0 0x7f113c101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113c10c510 0x7f113c1018a0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 --2- 172.21.15.94:0/4010634775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113c1031d0 0x7f113c100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 -- 172.21.15.94:0/4010634775 >> 172.21.15.94:0/4010634775 conn(0x7f113c077f50 msgr2=0x7f113c0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 -- 172.21.15.94:0/4010634775 shutdown_connections 2024-04-21T08:49:13.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:13.890+0000 7f1140afd640 1 -- 172.21.15.94:0/4010634775 wait complete. 2024-04-21T08:49:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:13 smithi094 ceph-mon[27988]: pgmap v286: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 32 KiB/s wr, 26 op/s 2024-04-21T08:49:14.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:13 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3744849573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:13 smithi111 ceph-mon[31493]: pgmap v286: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 32 KiB/s wr, 26 op/s 2024-04-21T08:49:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:13 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3744849573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:14.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:13 smithi173 ceph-mon[31428]: pgmap v286: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 32 KiB/s wr, 26 op/s 2024-04-21T08:49:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:13 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3744849573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:14.901 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:49:14.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:49:14.902 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:49:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.009+0000 7fe67d4d2640 1 -- 172.21.15.94:0/3524720384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6781021c0 msgr2=0x7fe6781025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.009+0000 7fe67d4d2640 1 --2- 172.21.15.94:0/3524720384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6781021c0 0x7fe6781025a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fe668009ca0 tx=0x7fe66802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.009+0000 7fe67d4d2640 1 -- 172.21.15.94:0/3524720384 shutdown_connections 2024-04-21T08:49:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.009+0000 7fe67d4d2640 1 --2- 172.21.15.94:0/3524720384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe678102b70 0x7fe678106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.009+0000 7fe67d4d2640 1 --2- 172.21.15.94:0/3524720384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe678107290 0x7fe67810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.009+0000 7fe67d4d2640 1 --2- 172.21.15.94:0/3524720384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6781021c0 0x7fe6781025a0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.009+0000 7fe67d4d2640 1 -- 172.21.15.94:0/3524720384 >> 172.21.15.94:0/3524720384 conn(0x7fe6780793e0 msgr2=0x7fe6780797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.010+0000 7fe67d4d2640 1 -- 172.21.15.94:0/3524720384 shutdown_connections 2024-04-21T08:49:15.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.010+0000 7fe67d4d2640 1 -- 172.21.15.94:0/3524720384 wait complete. 2024-04-21T08:49:15.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.010+0000 7fe67d4d2640 1 Processor -- start 2024-04-21T08:49:15.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.010+0000 7fe67d4d2640 1 -- start start 2024-04-21T08:49:15.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.010+0000 7fe67d4d2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781021c0 0x7fe6781987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.010+0000 7fe67d4d2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe678107290 0x7fe678198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.010+0000 7fe67d4d2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67819d020 0x7fe67819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe67d4d2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781121f0 con 0x7fe678107290 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe67d4d2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe678112070 con 0x7fe67819d020 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe67d4d2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe678112370 con 0x7fe6781021c0 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe6767fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe678107290 0x7fe678198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe6767fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe678107290 0x7fe678198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57118/0 (socket says 172.21.15.94:57118) 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe6767fc640 1 -- 172.21.15.94:0/285411910 learned_addr learned my addr 172.21.15.94:0/285411910 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe676ffd640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781021c0 0x7fe6781987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe6777fe640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67819d020 0x7fe67819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe6767fc640 1 -- 172.21.15.94:0/285411910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781021c0 msgr2=0x7fe6781987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe6767fc640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781021c0 0x7fe6781987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe6767fc640 1 -- 172.21.15.94:0/285411910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67819d020 msgr2=0x7fe67819d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe6767fc640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67819d020 0x7fe67819d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe6767fc640 1 -- 172.21.15.94:0/285411910 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe67819db10 con 0x7fe678107290 2024-04-21T08:49:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe6767fc640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe678107290 0x7fe678198cf0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00d740 tx=0x7fe66c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe65ffff640 1 -- 172.21.15.94:0/285411910 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c017070 con 0x7fe678107290 2024-04-21T08:49:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe65ffff640 1 -- 172.21.15.94:0/285411910 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe66c00de50 con 0x7fe678107290 2024-04-21T08:49:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe67d4d2640 1 -- 172.21.15.94:0/285411910 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe67819de00 con 0x7fe678107290 2024-04-21T08:49:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe65ffff640 1 -- 172.21.15.94:0/285411910 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c004ce0 con 0x7fe678107290 2024-04-21T08:49:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.011+0000 7fe67d4d2640 1 -- 172.21.15.94:0/285411910 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6781a55a0 con 0x7fe678107290 2024-04-21T08:49:15.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.013+0000 7fe65ffff640 1 -- 172.21.15.94:0/285411910 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fe66c004e80 con 0x7fe678107290 2024-04-21T08:49:15.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6801a0740 1 Processor -- start 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6801a0740 1 -- start start 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4bdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4be290 0x55fe6c4c6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6801a0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55fe6c421f00 con 0x55fe6c4bd8e0 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6801a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fe6c3bf590 con 0x55fe6c4be290 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6801a0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fe6c41dec0 con 0x55fe6c4c6e10 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe65ffff640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe6600a3d20 0x7fe6600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6777fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4bdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6777fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4bdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57122/0 (socket says 172.21.15.94:57122) 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6777fe640 1 -- 172.21.15.94:0/4041778810 learned_addr learned my addr 172.21.15.94:0/4041778810 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6777fe640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe6600a3d20 0x7fe6600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe676ffd640 1 --2- 172.21.15.94:0/4041778810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4be290 0x55fe6c4c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe65ffff640 1 -- 172.21.15.94:0/285411910 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fe66c0d0540 con 0x7fe678107290 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6767fc640 1 --2- 172.21.15.94:0/4041778810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6777fe640 1 -- 172.21.15.94:0/4041778810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 msgr2=0x55fe6c4c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6777fe640 1 --2- 172.21.15.94:0/4041778810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6777fe640 1 -- 172.21.15.94:0/4041778810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4be290 msgr2=0x55fe6c4c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6777fe640 1 --2- 172.21.15.94:0/4041778810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4be290 0x55fe6c4c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.014+0000 7fe6777fe640 1 -- 172.21.15.94:0/4041778810 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe6c1fc7e0 con 0x55fe6c4bd8e0 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe6777fe640 1 --2- 172.21.15.94:0/4041778810 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4bdcc0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fe670005940 tx=0x7fe670012e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57a7cab033747f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe6777fe640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe6600a3d20 0x7fe6600a61e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe670005fd0 tx=0x7fe67001a000 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe65d7fa640 1 -- 172.21.15.94:0/4041778810 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670020070 con 0x55fe6c4bd8e0 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe65d7fa640 1 -- 172.21.15.94:0/4041778810 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe67001c430 con 0x55fe6c4bd8e0 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe65d7fa640 1 -- 172.21.15.94:0/4041778810 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670011640 con 0x55fe6c4bd8e0 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe6801a0740 1 -- 172.21.15.94:0/4041778810 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4bd8e0 msgr2=0x55fe6c4bdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe6801a0740 1 --2- 172.21.15.94:0/4041778810 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4bdcc0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fe670005940 tx=0x7fe670012e80 comp rx=0 tx=0).stop 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe6801a0740 1 -- 172.21.15.94:0/4041778810 shutdown_connections 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe6801a0740 1 --2- 172.21.15.94:0/4041778810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe6801a0740 1 --2- 172.21.15.94:0/4041778810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4be290 0x55fe6c4c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe6801a0740 1 --2- 172.21.15.94:0/4041778810 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4bdcc0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe6801a0740 1 -- 172.21.15.94:0/4041778810 >> 172.21.15.94:0/4041778810 conn(0x55fe6c4b7fb0 msgr2=0x55fe6c4b8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.015+0000 7fe6801a0740 1 -- 172.21.15.94:0/4041778810 shutdown_connections 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6801a0740 1 -- 172.21.15.94:0/4041778810 wait complete. 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6801a0740 1 -- start start 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6801a0740 1 Processor -- start 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6801a0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55fe6c4c9d20 con 0x55fe6c4be290 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6801a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fe6c3db580 con 0x55fe6c4c6e10 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6801a0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fe6c4c9ea0 con 0x55fe6c4bd8e0 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6801a0740 10 client.0 _ll_register_callbacks cb 0x55fe6c3e1650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6767fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6767fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57138/0 (socket says 172.21.15.94:57138) 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6767fc640 1 -- 172.21.15.94:0/1892841124 learned_addr learned my addr 172.21.15.94:0/1892841124 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe6777fe640 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.016+0000 7fe676ffd640 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6767fc640 1 -- 172.21.15.94:0/1892841124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 msgr2=0x55fe6c4c3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6767fc640 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4c3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6767fc640 1 -- 172.21.15.94:0/1892841124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4c6e10 msgr2=0x55fe6c4c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6767fc640 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6767fc640 1 -- 172.21.15.94:0/1892841124 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe6c25cf70 con 0x55fe6c4be290 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6767fc640 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c43c0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fe66c0985b0 tx=0x7fe66c09c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca3fca9f1b3cf3c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe65aff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe65aff5640 1 -- 172.21.15.94:0/1892841124 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0b5060 con 0x55fe6c4be290 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe65aff5640 1 -- 172.21.15.94:0/1892841124 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fe66c0987c0 con 0x55fe6c4be290 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6801a0740 1 -- 172.21.15.94:0/1892841124 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe67001c430 con 0x55fe6c4be290 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe65aff5640 1 -- 172.21.15.94:0/1892841124 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe66c020040 con 0x55fe6c4be290 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6801a0740 10 client.18003 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6801a0740 1 -- 172.21.15.94:0/1892841124 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fe6c0f4170 con 0x55fe6c4be290 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe65aff5640 1 -- 172.21.15.94:0/1892841124 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe66c062020 con 0x55fe6c4be290 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe65aff5640 1 -- 172.21.15.94:0/1892841124 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe66c0d0160 con 0x55fe6c4be290 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6801a0740 10 client.18003 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6801a0740 10 client.18003 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6801a0740 20 client.18003 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6801a0740 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fe6c4db760 0x55fe6c4ddb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6801a0740 4 client.18003 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.017+0000 7fe6801a0740 1 -- 172.21.15.94:0/1892841124 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe6c42f920 con 0x55fe6c4db760 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.018+0000 7fe676ffd640 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fe6c4db760 0x55fe6c4ddb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.018+0000 7fe65aff5640 10 client.18003 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.018+0000 7fe676ffd640 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fe6c4db760 0x55fe6c4ddb50 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e234219d1048c88c server_cookie=a3e2f33302101b0a in_seq=0 out_seq=0 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe65aff5640 1 -- 172.21.15.94:0/1892841124 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe6c42f920 con 0x55fe6c4db760 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe65aff5640 10 client.18003 handle_command_reply: tid=0 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 client.18003 shutdown 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 -- 172.21.15.94:0/1892841124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 msgr2=0x55fe6c4c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c43c0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fe66c0985b0 tx=0x7fe66c09c040 comp rx=0 tx=0).stop 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 -- 172.21.15.94:0/1892841124 shutdown_connections 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fe6c4db760 0x55fe6c4ddb50 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe65aff5640 0 client.18003 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:15.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c43c0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 --2- 172.21.15.94:0/1892841124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4c3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 -- 172.21.15.94:0/1892841124 >> 172.21.15.94:0/1892841124 conn(0x55fe6c4b7fb0 msgr2=0x55fe6c4b9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 -- 172.21.15.94:0/1892841124 shutdown_connections 2024-04-21T08:49:15.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 1 -- 172.21.15.94:0/1892841124 wait complete. 2024-04-21T08:49:15.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.019+0000 7fe6801a0740 20 client.18003 trim_cache size 0 max 16384 2024-04-21T08:49:15.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.036+0000 7fe6801a0740 1 Processor -- start 2024-04-21T08:49:15.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.036+0000 7fe6801a0740 1 -- start start 2024-04-21T08:49:15.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6801a0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781afb00 con 0x55fe6c4be290 2024-04-21T08:49:15.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6801a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781ad3d0 con 0x55fe6c4bd8e0 2024-04-21T08:49:15.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6801a0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781ab240 con 0x55fe6c4c6e10 2024-04-21T08:49:15.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6767fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6767fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57152/0 (socket says 172.21.15.94:57152) 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6767fc640 1 -- 172.21.15.94:0/112464362 learned_addr learned my addr 172.21.15.94:0/112464362 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe676ffd640 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6777fe640 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6767fc640 1 -- 172.21.15.94:0/112464362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 msgr2=0x55fe6c4c4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6767fc640 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6767fc640 1 -- 172.21.15.94:0/112464362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4bd8e0 msgr2=0x55fe6c4c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6767fc640 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4c65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6767fc640 1 -- 172.21.15.94:0/112464362 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe6c4bec40 con 0x55fe6c4be290 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe676ffd640 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6777fe640 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe6767fc640 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c2dc0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fe66c09be50 tx=0x7fe66c09bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc827882c164ad6c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe65cff9640 1 -- 172.21.15.94:0/112464362 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0b5040 con 0x55fe6c4be290 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe65cff9640 1 -- 172.21.15.94:0/112464362 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe66c062070 con 0x55fe6c4be290 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.037+0000 7fe65cff9640 1 -- 172.21.15.94:0/112464362 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0987c0 con 0x55fe6c4be290 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 -- 172.21.15.94:0/112464362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 msgr2=0x55fe6c4c2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c2dc0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fe66c09be50 tx=0x7fe66c09bf20 comp rx=0 tx=0).stop 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 -- 172.21.15.94:0/112464362 shutdown_connections 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4be290 0x55fe6c4c2dc0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 --2- 172.21.15.94:0/112464362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 -- 172.21.15.94:0/112464362 >> 172.21.15.94:0/112464362 conn(0x55fe6c4b7fb0 msgr2=0x55fe6c4b9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 -- 172.21.15.94:0/112464362 shutdown_connections 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 -- 172.21.15.94:0/112464362 wait complete. 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 -- start start 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 Processor -- start 2024-04-21T08:49:15.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4cf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4be290 0x55fe6c4cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781ab890 con 0x55fe6c4c6e10 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781acd00 con 0x55fe6c4be290 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781abf60 con 0x55fe6c4bd8e0 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6801a0740 10 client.0 _ll_register_callbacks cb 0x7fe66c097d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.038+0000 7fe6777fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6777fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4cfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57154/0 (socket says 172.21.15.94:57154) 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6777fe640 1 -- 172.21.15.94:0/2041906747 learned_addr learned my addr 172.21.15.94:0/2041906747 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe676ffd640 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6777fe640 1 -- 172.21.15.94:0/2041906747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 msgr2=0x55fe6c4cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6777fe640 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4cf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6777fe640 1 -- 172.21.15.94:0/2041906747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4be290 msgr2=0x55fe6c4cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6777fe640 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4be290 0x55fe6c4cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6777fe640 1 -- 172.21.15.94:0/2041906747 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe6c540e40 con 0x55fe6c4c6e10 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe676ffd640 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6777fe640 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4cfbc0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fe670005bf0 tx=0x7fe670012690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5ed0b6ac391c6d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe65aff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe65aff5640 1 -- 172.21.15.94:0/2041906747 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670020040 con 0x55fe6c4c6e10 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe65aff5640 1 -- 172.21.15.94:0/2041906747 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fe670005450 con 0x55fe6c4c6e10 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6801a0740 1 -- 172.21.15.94:0/2041906747 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe66c062070 con 0x55fe6c4c6e10 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe65aff5640 1 -- 172.21.15.94:0/2041906747 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe670037050 con 0x55fe6c4c6e10 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6801a0740 10 client.18009 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe6801a0740 1 -- 172.21.15.94:0/2041906747 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fe6c5420b0 con 0x55fe6c4c6e10 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe65aff5640 1 -- 172.21.15.94:0/2041906747 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe66c062070 con 0x55fe6c4c6e10 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.039+0000 7fe65aff5640 1 -- 172.21.15.94:0/2041906747 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe67002b030 con 0x55fe6c4c6e10 2024-04-21T08:49:15.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.040+0000 7fe6801a0740 10 client.18009 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.040+0000 7fe6801a0740 10 client.18009 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.040+0000 7fe6801a0740 20 client.18009 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.040+0000 7fe6801a0740 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fe6c542650 0x55fe6c544a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.040+0000 7fe6801a0740 4 client.18009 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.040+0000 7fe6801a0740 1 -- 172.21.15.94:0/2041906747 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fe6c429790 con 0x55fe6c542650 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.040+0000 7fe676ffd640 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fe6c542650 0x55fe6c544a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.040+0000 7fe65aff5640 10 client.18009 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.040+0000 7fe676ffd640 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fe6c542650 0x55fe6c544a70 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aadee1f1f20ee26a server_cookie=1eef7fceffa24038 in_seq=0 out_seq=0 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe65aff5640 1 -- 172.21.15.94:0/2041906747 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fe6c429790 con 0x55fe6c542650 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe65aff5640 10 client.18009 handle_command_reply: tid=0 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe6801a0740 1 client.18009 shutdown 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe6801a0740 1 -- 172.21.15.94:0/2041906747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4c6e10 msgr2=0x55fe6c4cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe6801a0740 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4cfbc0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fe670005bf0 tx=0x7fe670012690 comp rx=0 tx=0).stop 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe6801a0740 1 -- 172.21.15.94:0/2041906747 shutdown_connections 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe6801a0740 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55fe6c542650 0x55fe6c544a70 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe6801a0740 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55fe6c4c6e10 0x55fe6c4cfbc0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe6801a0740 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fe6c4be290 0x55fe6c4cf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe6801a0740 1 --2- 172.21.15.94:0/2041906747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe6c4bd8e0 0x55fe6c4cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.041+0000 7fe6801a0740 1 -- 172.21.15.94:0/2041906747 >> 172.21.15.94:0/2041906747 conn(0x55fe6c4b7fb0 msgr2=0x55fe6c4b9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe65aff5640 0 client.18009 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:15.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe6801a0740 1 -- 172.21.15.94:0/2041906747 shutdown_connections 2024-04-21T08:49:15.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe6801a0740 1 -- 172.21.15.94:0/2041906747 wait complete. 2024-04-21T08:49:15.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe6801a0740 20 client.18009 trim_cache size 0 max 16384 2024-04-21T08:49:15.049 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:49:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:49:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "343f5485-f5f2-465c-939b-f510e4f02468", 2024-04-21T08:49:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:49:15.049 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 -- 172.21.15.94:0/285411910 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe6600a3d20 msgr2=0x7fe6600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe6600a3d20 0x7fe6600a61e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe670005fd0 tx=0x7fe67001a000 comp rx=0 tx=0).stop 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 -- 172.21.15.94:0/285411910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe678107290 msgr2=0x7fe678198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe678107290 0x7fe678198cf0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00d740 tx=0x7fe66c00dc00 comp rx=0 tx=0).stop 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 -- 172.21.15.94:0/285411910 shutdown_connections 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe6600a3d20 0x7fe6600a61e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67819d020 0x7fe67819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe678107290 0x7fe678198cf0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 --2- 172.21.15.94:0/285411910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6781021c0 0x7fe6781987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 -- 172.21.15.94:0/285411910 >> 172.21.15.94:0/285411910 conn(0x7fe6780793e0 msgr2=0x7fe678108460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 -- 172.21.15.94:0/285411910 shutdown_connections 2024-04-21T08:49:15.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.042+0000 7fe67d4d2640 1 -- 172.21.15.94:0/285411910 wait complete. 2024-04-21T08:49:15.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:49:15.053 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:49:15.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.153+0000 7fc136b7e640 1 -- 172.21.15.94:0/1581207057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1301021a0 msgr2=0x7fc130102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.153+0000 7fc136b7e640 1 --2- 172.21.15.94:0/1581207057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1301021a0 0x7fc130102580 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fc128009900 tx=0x7fc12802bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.153+0000 7fc136b7e640 1 -- 172.21.15.94:0/1581207057 shutdown_connections 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.153+0000 7fc136b7e640 1 --2- 172.21.15.94:0/1581207057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc130102b50 0x7fc130106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.153+0000 7fc136b7e640 1 --2- 172.21.15.94:0/1581207057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc130107270 0x7fc13010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.153+0000 7fc136b7e640 1 --2- 172.21.15.94:0/1581207057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1301021a0 0x7fc130102580 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.153+0000 7fc136b7e640 1 -- 172.21.15.94:0/1581207057 >> 172.21.15.94:0/1581207057 conn(0x7fc1300fda40 msgr2=0x7fc1300ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.153+0000 7fc136b7e640 1 -- 172.21.15.94:0/1581207057 shutdown_connections 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.153+0000 7fc136b7e640 1 -- 172.21.15.94:0/1581207057 wait complete. 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc136b7e640 1 Processor -- start 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc136b7e640 1 -- start start 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc136b7e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1301021a0 0x7fc1301987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc136b7e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc130107270 0x7fc130198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc136b7e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13019d010 0x7fc13019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc136b7e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc130112220 con 0x7fc130107270 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc136b7e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc1301120a0 con 0x7fc13019d010 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc136b7e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc1301123a0 con 0x7fc1301021a0 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc12ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc130107270 0x7fc130198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc12ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc130107270 0x7fc130198ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57160/0 (socket says 172.21.15.94:57160) 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc12ffff640 1 -- 172.21.15.94:0/489440176 learned_addr learned my addr 172.21.15.94:0/489440176 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc13491a640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1301021a0 0x7fc1301987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.154+0000 7fc13511b640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13019d010 0x7fc13019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc12ffff640 1 -- 172.21.15.94:0/489440176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1301021a0 msgr2=0x7fc1301987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc12ffff640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1301021a0 0x7fc1301987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc12ffff640 1 -- 172.21.15.94:0/489440176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13019d010 msgr2=0x7fc13019d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc12ffff640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13019d010 0x7fc13019d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc12ffff640 1 -- 172.21.15.94:0/489440176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc13019db00 con 0x7fc130107270 2024-04-21T08:49:15.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc13511b640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13019d010 0x7fc13019d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc12ffff640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc130107270 0x7fc130198ce0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fc12000ebc0 tx=0x7fc12000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc12dffb640 1 -- 172.21.15.94:0/489440176 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120004d20 con 0x7fc130107270 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc136b7e640 1 -- 172.21.15.94:0/489440176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc13019ddf0 con 0x7fc130107270 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc12dffb640 1 -- 172.21.15.94:0/489440176 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1200054d0 con 0x7fc130107270 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc12dffb640 1 -- 172.21.15.94:0/489440176 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120010720 con 0x7fc130107270 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.155+0000 7fc136b7e640 1 -- 172.21.15.94:0/489440176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1301a55e0 con 0x7fc130107270 2024-04-21T08:49:15.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.157+0000 7fc12dffb640 1 -- 172.21.15.94:0/489440176 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fc120005020 con 0x7fc130107270 2024-04-21T08:49:15.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13976e740 1 Processor -- start 2024-04-21T08:49:15.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13976e740 1 -- start start 2024-04-21T08:49:15.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12dffb640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc11c0a3d20 0x7fc11c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13976e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946230 0x558186946610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13976e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x558186953950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13511b640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc11c0a3d20 0x7fc11c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13976e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186956280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13976e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5581868ab1e0 con 0x558186946b50 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13976e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558186848390 con 0x558186953e90 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13976e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5581868a71a0 con 0x558186946230 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12dffb640 1 -- 172.21.15.94:0/489440176 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fc1200d1530 con 0x7fc130107270 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x558186953950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x558186953950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57164/0 (socket says 172.21.15.94:57164) 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12ffff640 1 -- 172.21.15.94:0/817307159 learned_addr learned my addr 172.21.15.94:0/817307159 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13491a640 1 --2- 172.21.15.94:0/817307159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946230 0x558186946610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12ffff640 1 -- 172.21.15.94:0/817307159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946230 msgr2=0x558186946610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13511b640 1 --2- 172.21.15.94:0/817307159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186956280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12ffff640 1 --2- 172.21.15.94:0/817307159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946230 0x558186946610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12ffff640 1 -- 172.21.15.94:0/817307159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 msgr2=0x558186956280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12ffff640 1 --2- 172.21.15.94:0/817307159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186956280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12ffff640 1 -- 172.21.15.94:0/817307159 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581866877d0 con 0x558186946b50 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13511b640 1 --2- 172.21.15.94:0/817307159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186956280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc13511b640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc11c0a3d20 0x7fc11c0a61e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc118005fd0 tx=0x7fc118009e50 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.158+0000 7fc12ffff640 1 --2- 172.21.15.94:0/817307159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x558186953950 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fc12009ce40 tx=0x7fc120005ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64efe848f1382fe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc126ffd640 1 -- 172.21.15.94:0/817307159 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1200a5db0 con 0x558186946b50 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc126ffd640 1 -- 172.21.15.94:0/817307159 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc120005680 con 0x558186946b50 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc126ffd640 1 -- 172.21.15.94:0/817307159 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1200cecc0 con 0x558186946b50 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc13976e740 1 -- 172.21.15.94:0/817307159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 msgr2=0x558186953950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc13976e740 1 --2- 172.21.15.94:0/817307159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x558186953950 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fc12009ce40 tx=0x7fc120005ac0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc13976e740 1 -- 172.21.15.94:0/817307159 shutdown_connections 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc13976e740 1 --2- 172.21.15.94:0/817307159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186956280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc13976e740 1 --2- 172.21.15.94:0/817307159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x558186953950 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc13976e740 1 --2- 172.21.15.94:0/817307159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946230 0x558186946610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc13976e740 1 -- 172.21.15.94:0/817307159 >> 172.21.15.94:0/817307159 conn(0x558186940cb0 msgr2=0x558186941090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc13976e740 1 -- 172.21.15.94:0/817307159 shutdown_connections 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.159+0000 7fc13976e740 1 -- 172.21.15.94:0/817307159 wait complete. 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13976e740 1 -- start start 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13976e740 1 Processor -- start 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13976e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x558186950eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13976e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946b50 0x5581869513f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13976e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186951930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13976e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x558186956cd0 con 0x558186946230 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13976e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5581868642f0 con 0x558186953e90 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13976e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558186956e50 con 0x558186946b50 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13976e740 10 client.0 _ll_register_callbacks cb 0x558186871580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13491a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x558186950eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13491a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x558186950eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57168/0 (socket says 172.21.15.94:57168) 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13491a640 1 -- 172.21.15.94:0/1352974279 learned_addr learned my addr 172.21.15.94:0/1352974279 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc12ffff640 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946b50 0x5581869513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13511b640 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186951930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13491a640 1 -- 172.21.15.94:0/1352974279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946b50 msgr2=0x5581869513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13491a640 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946b50 0x5581869513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13491a640 1 -- 172.21.15.94:0/1352974279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 msgr2=0x558186951930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13491a640 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186951930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.160+0000 7fc13491a640 1 -- 172.21.15.94:0/1352974279 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581865bd3f0 con 0x558186946230 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc13491a640 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x558186950eb0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fc128006010 tx=0x7fc12802c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac21959c7147127e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc10ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc10ffff640 1 -- 172.21.15.94:0/1352974279 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12802c5d0 con 0x558186946230 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc10ffff640 1 -- 172.21.15.94:0/1352974279 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fc128005030 con 0x558186946230 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc13976e740 1 -- 172.21.15.94:0/1352974279 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc120005680 con 0x558186946230 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc10ffff640 1 -- 172.21.15.94:0/1352974279 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc128052050 con 0x558186946230 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc13976e740 10 client.18021 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc13976e740 1 -- 172.21.15.94:0/1352974279 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55818657d0f0 con 0x558186946230 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc10ffff640 1 -- 172.21.15.94:0/1352974279 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc120005680 con 0x558186946230 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc10ffff640 1 -- 172.21.15.94:0/1352974279 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc1280035f0 con 0x558186946230 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc13976e740 10 client.18021 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc13976e740 10 client.18021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc13976e740 20 client.18021 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc13976e740 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558186963eb0 0x5581869662e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc13976e740 4 client.18021 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.161+0000 7fc13976e740 1 -- 172.21.15.94:0/1352974279 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581868b8c00 con 0x558186963eb0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.162+0000 7fc12ffff640 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558186963eb0 0x5581869662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.162+0000 7fc10ffff640 10 client.18021 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.162+0000 7fc12ffff640 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558186963eb0 0x5581869662e0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42426f90a64c7bd5 server_cookie=507b01f0358db886 in_seq=0 out_seq=0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc10ffff640 1 -- 172.21.15.94:0/1352974279 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581868b8c00 con 0x558186963eb0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc10ffff640 10 client.18021 handle_command_reply: tid=0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 client.18021 shutdown 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 -- 172.21.15.94:0/1352974279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 msgr2=0x558186950eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x558186950eb0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fc128006010 tx=0x7fc12802c9f0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 -- 172.21.15.94:0/1352974279 shutdown_connections 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x558186963eb0 0x5581869662e0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946b50 0x5581869513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc10ffff640 0 client.18021 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186951930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 --2- 172.21.15.94:0/1352974279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x558186950eb0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 -- 172.21.15.94:0/1352974279 >> 172.21.15.94:0/1352974279 conn(0x558186940cb0 msgr2=0x558186942a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 -- 172.21.15.94:0/1352974279 shutdown_connections 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 1 -- 172.21.15.94:0/1352974279 wait complete. 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.163+0000 7fc13976e740 20 client.18021 trim_cache size 0 max 16384 2024-04-21T08:49:15.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc13976e740 1 Processor -- start 2024-04-21T08:49:15.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc13976e740 1 -- start start 2024-04-21T08:49:15.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc13976e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946230 0x5581869535e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc13976e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x55818695fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc13976e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186941090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc13976e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc1301afbb0 con 0x558186946b50 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc13976e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc1301ad480 con 0x558186953e90 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc13976e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc1301ab2f0 con 0x558186946230 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc12ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x55818695fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc12ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x55818695fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57170/0 (socket says 172.21.15.94:57170) 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc12ffff640 1 -- 172.21.15.94:0/964938614 learned_addr learned my addr 172.21.15.94:0/964938614 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc12ffff640 1 -- 172.21.15.94:0/964938614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946230 msgr2=0x5581869535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc12ffff640 1 --2- 172.21.15.94:0/964938614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946230 0x5581869535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc12ffff640 1 -- 172.21.15.94:0/964938614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 msgr2=0x558186941090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc12ffff640 1 --2- 172.21.15.94:0/964938614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186941090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.180+0000 7fc12ffff640 1 -- 172.21.15.94:0/964938614 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558186946980 con 0x558186946b50 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc12ffff640 1 --2- 172.21.15.94:0/964938614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x55818695fbd0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fc12000cba0 tx=0x7fc12000eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db0039a06bc7db server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc1267fc640 1 -- 172.21.15.94:0/964938614 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1200a5db0 con 0x558186946b50 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc1267fc640 1 -- 172.21.15.94:0/964938614 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc12009b440 con 0x558186946b50 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc1267fc640 1 -- 172.21.15.94:0/964938614 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1200d1160 con 0x558186946b50 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc13976e740 1 -- 172.21.15.94:0/964938614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 msgr2=0x55818695fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc13976e740 1 --2- 172.21.15.94:0/964938614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x55818695fbd0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fc12000cba0 tx=0x7fc12000eec0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc13976e740 1 -- 172.21.15.94:0/964938614 shutdown_connections 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc13976e740 1 --2- 172.21.15.94:0/964938614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186953e90 0x558186941090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc13976e740 1 --2- 172.21.15.94:0/964938614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946b50 0x55818695fbd0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc13976e740 1 --2- 172.21.15.94:0/964938614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186946230 0x5581869535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc13976e740 1 -- 172.21.15.94:0/964938614 >> 172.21.15.94:0/964938614 conn(0x558186940cb0 msgr2=0x558186942100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc13976e740 1 -- 172.21.15.94:0/964938614 shutdown_connections 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.181+0000 7fc13976e740 1 -- 172.21.15.94:0/964938614 wait complete. 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13976e740 1 -- start start 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13976e740 1 Processor -- start 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13976e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x55818695c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13976e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186946b50 0x5581869579b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13976e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186953e90 0x5581869580a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13976e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc1301ab940 con 0x558186946230 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13976e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc1301acdb0 con 0x558186946b50 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13976e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc1301ac010 con 0x558186953e90 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13976e740 10 client.0 _ll_register_callbacks cb 0x7fc128005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13491a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x55818695c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13491a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x55818695c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57176/0 (socket says 172.21.15.94:57176) 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13491a640 1 -- 172.21.15.94:0/135469356 learned_addr learned my addr 172.21.15.94:0/135469356 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc12ffff640 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186946b50 0x5581869579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13491a640 1 -- 172.21.15.94:0/135469356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186953e90 msgr2=0x5581869580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13491a640 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186953e90 0x5581869580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13491a640 1 -- 172.21.15.94:0/135469356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186946b50 msgr2=0x5581869579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13491a640 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186946b50 0x5581869579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc13491a640 1 -- 172.21.15.94:0/135469356 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55818694bc60 con 0x558186946230 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.182+0000 7fc12ffff640 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186946b50 0x5581869579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc13491a640 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x55818695c640 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fc12802c170 tx=0x7fc1280098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b22e804b51a0e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc10ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc10ffff640 1 -- 172.21.15.94:0/135469356 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12805b060 con 0x558186946230 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc10ffff640 1 -- 172.21.15.94:0/135469356 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fc128005070 con 0x558186946230 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc13976e740 1 -- 172.21.15.94:0/135469356 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc12009b440 con 0x558186946230 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc10ffff640 1 -- 172.21.15.94:0/135469356 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc128062050 con 0x558186946230 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc13976e740 10 client.18027 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc13976e740 1 -- 172.21.15.94:0/135469356 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5581869ca430 con 0x558186946230 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc10ffff640 1 -- 172.21.15.94:0/135469356 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc12009b440 con 0x558186946230 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc10ffff640 1 -- 172.21.15.94:0/135469356 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc1280046c0 con 0x558186946230 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc13976e740 10 client.18027 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc13976e740 10 client.18027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc13976e740 20 client.18027 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc13976e740 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581869cabe0 0x5581869cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc13976e740 4 client.18027 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc13976e740 1 -- 172.21.15.94:0/135469356 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5581868b2a70 con 0x5581869cabe0 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.183+0000 7fc12ffff640 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581869cabe0 0x5581869cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc10ffff640 10 client.18027 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc12ffff640 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581869cabe0 0x5581869cd000 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15f11e643a408eb6 server_cookie=fe60fc1aa90e87df in_seq=0 out_seq=0 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc10ffff640 1 -- 172.21.15.94:0/135469356 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5581868b2a70 con 0x5581869cabe0 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc10ffff640 10 client.18027 handle_command_reply: tid=0 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 client.18027 shutdown 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 -- 172.21.15.94:0/135469356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 msgr2=0x55818695c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x55818695c640 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fc12802c170 tx=0x7fc1280098a0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 -- 172.21.15.94:0/135469356 shutdown_connections 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5581869cabe0 0x5581869cd000 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558186953e90 0x5581869580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc10ffff640 0 client.18027 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558186946b50 0x5581869579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 --2- 172.21.15.94:0/135469356 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558186946230 0x55818695c640 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 -- 172.21.15.94:0/135469356 >> 172.21.15.94:0/135469356 conn(0x558186940cb0 msgr2=0x558186942990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 -- 172.21.15.94:0/135469356 shutdown_connections 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 1 -- 172.21.15.94:0/135469356 wait complete. 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.184+0000 7fc13976e740 20 client.18027 trim_cache size 0 max 16384 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi094.stdout: "343f5485-f5f2-465c-939b-f510e4f02468": { 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "343f5485-f5f2-465c-939b-f510e4f02468", 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 -- 172.21.15.94:0/489440176 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc11c0a3d20 msgr2=0x7fc11c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc11c0a3d20 0x7fc11c0a61e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc118005fd0 tx=0x7fc118009e50 comp rx=0 tx=0).stop 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 -- 172.21.15.94:0/489440176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc130107270 msgr2=0x7fc130198ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc130107270 0x7fc130198ce0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fc12000ebc0 tx=0x7fc12000c6a0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 -- 172.21.15.94:0/489440176 shutdown_connections 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fc11c0a3d20 0x7fc11c0a61e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13019d010 0x7fc13019d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc130107270 0x7fc130198ce0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 --2- 172.21.15.94:0/489440176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1301021a0 0x7fc1301987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 -- 172.21.15.94:0/489440176 >> 172.21.15.94:0/489440176 conn(0x7fc1300fda40 msgr2=0x7fc1300fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 -- 172.21.15.94:0/489440176 shutdown_connections 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:15.185+0000 7fc136b7e640 1 -- 172.21.15.94:0/489440176 wait complete. 2024-04-21T08:49:15.196 INFO:tasks.cephfs.filesystem:scrub status for tag:343f5485-f5f2-465c-939b-f510e4f02468 - {'path': '/', 'tag': '343f5485-f5f2-465c-939b-f510e4f02468', 'options': 'recursive,force'} 2024-04-21T08:49:16.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:15 smithi111 ceph-mon[31493]: pgmap v287: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T08:49:16.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:15 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:49:16.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:15 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:16.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:15 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:49:16.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:15 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:16.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:15 smithi094 ceph-mon[27988]: pgmap v287: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T08:49:16.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:15 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:49:16.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:15 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:16.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:15 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:49:16.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:15 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:16.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:15 smithi173 ceph-mon[31428]: pgmap v287: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T08:49:16.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:15 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:49:16.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:15 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:16.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:15 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:49:16.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:15 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:17.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:16 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:49:17.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:16 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:49:17.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:16 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:49:18.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:17 smithi111 ceph-mon[31493]: pgmap v288: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 230 B/s wr, 7 op/s 2024-04-21T08:49:18.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:17 smithi094 ceph-mon[27988]: pgmap v288: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 230 B/s wr, 7 op/s 2024-04-21T08:49:18.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:17 smithi173 ceph-mon[31428]: pgmap v288: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 230 B/s wr, 7 op/s 2024-04-21T08:49:20.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:19 smithi111 ceph-mon[31493]: pgmap v289: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T08:49:20.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:19 smithi094 ceph-mon[27988]: pgmap v289: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T08:49:20.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:19 smithi173 ceph-mon[31428]: pgmap v289: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T08:49:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:21 smithi111 ceph-mon[31493]: pgmap v290: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-21T08:49:22.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:21 smithi094 ceph-mon[27988]: pgmap v290: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-21T08:49:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:21 smithi173 ceph-mon[31428]: pgmap v290: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-21T08:49:24.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:23 smithi111 ceph-mon[31493]: pgmap v291: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 99 B/s wr, 7 op/s 2024-04-21T08:49:24.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:23 smithi094 ceph-mon[27988]: pgmap v291: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 99 B/s wr, 7 op/s 2024-04-21T08:49:24.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:23 smithi173 ceph-mon[31428]: pgmap v291: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 99 B/s wr, 7 op/s 2024-04-21T08:49:26.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:25 smithi111 ceph-mon[31493]: pgmap v292: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-21T08:49:26.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:25 smithi094 ceph-mon[27988]: pgmap v292: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-21T08:49:26.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:25 smithi173 ceph-mon[31428]: pgmap v292: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-21T08:49:28.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:27 smithi111 ceph-mon[31493]: pgmap v293: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:49:28.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:27 smithi094 ceph-mon[27988]: pgmap v293: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:49:28.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:27 smithi173 ceph-mon[31428]: pgmap v293: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:49:30.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:29 smithi111 ceph-mon[31493]: pgmap v294: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:49:30.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:29 smithi094 ceph-mon[27988]: pgmap v294: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:49:30.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:29 smithi173 ceph-mon[31428]: pgmap v294: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:49:32.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:31 smithi111 ceph-mon[31493]: pgmap v295: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:49:32.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:31 smithi094 ceph-mon[27988]: pgmap v295: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:49:32.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:31 smithi173 ceph-mon[31428]: pgmap v295: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:49:34.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:33 smithi111 ceph-mon[31493]: pgmap v296: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:34.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:33 smithi094 ceph-mon[27988]: pgmap v296: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:34.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:33 smithi173 ceph-mon[31428]: pgmap v296: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:36.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:35 smithi111 ceph-mon[31493]: pgmap v297: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:49:36.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:35 smithi094 ceph-mon[27988]: pgmap v297: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:49:36.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:35 smithi173 ceph-mon[31428]: pgmap v297: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:49:38.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:37 smithi111 ceph-mon[31493]: pgmap v298: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:49:38.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:37 smithi094 ceph-mon[27988]: pgmap v298: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:49:38.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:37 smithi173 ceph-mon[31428]: pgmap v298: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:49:40.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:39 smithi111 ceph-mon[31493]: pgmap v299: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:40.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:39 smithi094 ceph-mon[27988]: pgmap v299: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:40.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:39 smithi173 ceph-mon[31428]: pgmap v299: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:42.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:41 smithi111 ceph-mon[31493]: pgmap v300: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:49:42.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:41 smithi094 ceph-mon[27988]: pgmap v300: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:49:42.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:41 smithi173 ceph-mon[31428]: pgmap v300: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:49:43.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:42 smithi111 ceph-mon[31493]: pgmap v301: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:43.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:42 smithi094 ceph-mon[27988]: pgmap v301: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:43.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:42 smithi173 ceph-mon[31428]: pgmap v301: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:45.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:49:45.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:49:45.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.314+0000 7f40066c8640 1 -- 172.21.15.94:0/3679996506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4000108920 msgr2=0x7f400006a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.314+0000 7f40066c8640 1 --2- 172.21.15.94:0/3679996506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4000108920 0x7f400006a5d0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009900 tx=0x7f3ff402bcd0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.314+0000 7f40066c8640 1 -- 172.21.15.94:0/3679996506 shutdown_connections 2024-04-21T08:49:45.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.314+0000 7f40066c8640 1 --2- 172.21.15.94:0/3679996506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f400006ace0 0x7f400010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.314+0000 7f40066c8640 1 --2- 172.21.15.94:0/3679996506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4000108920 0x7f400006a5d0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.314+0000 7f40066c8640 1 --2- 172.21.15.94:0/3679996506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4000108000 0x7f40001083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.314+0000 7f40066c8640 1 -- 172.21.15.94:0/3679996506 >> 172.21.15.94:0/3679996506 conn(0x7f4000076c50 msgr2=0x7f4000077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.314+0000 7f40066c8640 1 -- 172.21.15.94:0/3679996506 shutdown_connections 2024-04-21T08:49:45.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.314+0000 7f40066c8640 1 -- 172.21.15.94:0/3679996506 wait complete. 2024-04-21T08:49:45.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f40066c8640 1 Processor -- start 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f40066c8640 1 -- start start 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f40066c8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f400006ace0 0x7f4000198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f40066c8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4000108000 0x7f4000198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f40066c8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4000108920 0x7f400019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f40066c8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4000112160 con 0x7f4000108920 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f40066c8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4000111fe0 con 0x7f400006ace0 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f40066c8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f40001122e0 con 0x7f4000108000 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f4004c65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4000108920 0x7f400019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f4004c65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4000108920 0x7f400019cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59544/0 (socket says 172.21.15.94:59544) 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.315+0000 7f4004c65640 1 -- 172.21.15.94:0/2226238979 learned_addr learned my addr 172.21.15.94:0/2226238979 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f3ffffff640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f400006ace0 0x7f4000198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4000108000 0x7f4000198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f4004c65640 1 -- 172.21.15.94:0/2226238979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4000108000 msgr2=0x7f4000198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f4004c65640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4000108000 0x7f4000198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f4004c65640 1 -- 172.21.15.94:0/2226238979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f400006ace0 msgr2=0x7f4000198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f4004c65640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f400006ace0 0x7f4000198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f4004c65640 1 -- 172.21.15.94:0/2226238979 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400019d6f0 con 0x7f4000108920 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f4004c65640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4000108920 0x7f400019cfc0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800cab0 tx=0x7f3ff800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f3ffd7fa640 1 -- 172.21.15.94:0/2226238979 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff800adc0 con 0x7f4000108920 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f3ffd7fa640 1 -- 172.21.15.94:0/2226238979 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff800f900 con 0x7f4000108920 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f40066c8640 1 -- 172.21.15.94:0/2226238979 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f400019d9e0 con 0x7f4000108920 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f3ffd7fa640 1 -- 172.21.15.94:0/2226238979 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff80189a0 con 0x7f4000108920 2024-04-21T08:49:45.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.316+0000 7f40066c8640 1 -- 172.21.15.94:0/2226238979 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40001a5270 con 0x7f4000108920 2024-04-21T08:49:45.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f3ffd7fa640 1 -- 172.21.15.94:0/2226238979 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f3ff800f450 con 0x7f4000108920 2024-04-21T08:49:45.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f4009398740 1 Processor -- start 2024-04-21T08:49:45.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f4009398740 1 -- start start 2024-04-21T08:49:45.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f4009398740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c956960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f4009398740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f4009398740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 0x55bf5c961e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f4009398740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c8bb1e0 con 0x55bf5c956ea0 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f4009398740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c858390 con 0x55bf5c95fa80 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f4009398740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c8b71a0 con 0x55bf5c956580 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59558/0 (socket says 172.21.15.94:59558) 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.319+0000 7f3ffffff640 1 -- 172.21.15.94:0/4266904115 learned_addr learned my addr 172.21.15.94:0/4266904115 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/4266904115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c956960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3ffd7fa640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3fe80a3d20 0x7f3fe80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/4266904115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 0x55bf5c961e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f4004c65640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3fe80a3d20 0x7f3fe80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3ffffff640 1 -- 172.21.15.94:0/4266904115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 msgr2=0x55bf5c956960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3ffffff640 1 --2- 172.21.15.94:0/4266904115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c956960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3ffffff640 1 -- 172.21.15.94:0/4266904115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 msgr2=0x55bf5c961e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3ffffff640 1 --2- 172.21.15.94:0/4266904115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 0x55bf5c961e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3ffffff640 1 -- 172.21.15.94:0/4266904115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf5c6977d0 con 0x55bf5c956ea0 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/4266904115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 0x55bf5c961e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3ffd7fa640 1 -- 172.21.15.94:0/2226238979 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f3ff80d0e90 con 0x7f4000108920 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3ffffff640 1 --2- 172.21.15.94:0/4266904115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95f540 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000aba0 tx=0x7f3ff000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3529b8e5b87c8f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3fe67fc640 1 -- 172.21.15.94:0/4266904115 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff001d070 con 0x55bf5c956ea0 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f4004c65640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3fe80a3d20 0x7f3fe80a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80ce260 tx=0x7f3ff80bd040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3fe67fc640 1 -- 172.21.15.94:0/4266904115 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff0018af0 con 0x55bf5c956ea0 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.320+0000 7f3fe67fc640 1 -- 172.21.15.94:0/4266904115 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0010a70 con 0x55bf5c956ea0 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.321+0000 7f4009398740 1 -- 172.21.15.94:0/4266904115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 msgr2=0x55bf5c95f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.321+0000 7f4009398740 1 --2- 172.21.15.94:0/4266904115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95f540 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000aba0 tx=0x7f3ff000f790 comp rx=0 tx=0).stop 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.321+0000 7f4009398740 1 -- 172.21.15.94:0/4266904115 shutdown_connections 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.321+0000 7f4009398740 1 --2- 172.21.15.94:0/4266904115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 0x55bf5c961e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.321+0000 7f4009398740 1 --2- 172.21.15.94:0/4266904115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95f540 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.321+0000 7f4009398740 1 --2- 172.21.15.94:0/4266904115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c956960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.321+0000 7f4009398740 1 -- 172.21.15.94:0/4266904115 >> 172.21.15.94:0/4266904115 conn(0x55bf5c950cb0 msgr2=0x55bf5c951090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.321+0000 7f4009398740 1 -- 172.21.15.94:0/4266904115 shutdown_connections 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.321+0000 7f4009398740 1 -- 172.21.15.94:0/4266904115 wait complete. 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.321+0000 7f4009398740 1 -- start start 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f4009398740 1 Processor -- start 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f4009398740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c95cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f4009398740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f4009398740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 0x55bf5c95d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f4009398740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c962870 con 0x55bf5c956ea0 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f4009398740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c8742f0 con 0x55bf5c95fa80 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f4009398740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55bf5c9629f0 con 0x55bf5c956580 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f4009398740 10 client.0 _ll_register_callbacks cb 0x55bf5c881580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59570/0 (socket says 172.21.15.94:59570) 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3ffffff640 1 -- 172.21.15.94:0/2841275794 learned_addr learned my addr 172.21.15.94:0/2841275794 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 0x55bf5c95d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3ffffff640 1 -- 172.21.15.94:0/2841275794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 msgr2=0x55bf5c95cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3ffffff640 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c95cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3ffffff640 1 -- 172.21.15.94:0/2841275794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 msgr2=0x55bf5c95d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3ffffff640 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 0x55bf5c95d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3ffffff640 1 -- 172.21.15.94:0/2841275794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf5c5cd3f0 con 0x55bf5c956ea0 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 0x55bf5c95d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3ffffff640 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95d010 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00186f0 tx=0x7f3ff000af10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52262e7648d16c9e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.322+0000 7f3fdf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/2841275794 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff001d050 con 0x55bf5c956ea0 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/2841275794 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f3ff0005240 con 0x55bf5c956ea0 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f4009398740 1 -- 172.21.15.94:0/2841275794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ff0018af0 con 0x55bf5c956ea0 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/2841275794 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3ff0032050 con 0x55bf5c956ea0 2024-04-21T08:49:45.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f4009398740 10 client.18039 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f4009398740 1 -- 172.21.15.94:0/2841275794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bf5c58d0f0 con 0x55bf5c956ea0 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/2841275794 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff0018af0 con 0x55bf5c956ea0 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/2841275794 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3ff0004750 con 0x55bf5c956ea0 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f4009398740 10 client.18039 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f4009398740 10 client.18039 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f4009398740 20 client.18039 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f4009398740 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf5c96fa30 0x55bf5c971e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f4009398740 4 client.18039 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f4009398740 1 -- 172.21.15.94:0/2841275794 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf5c8c8c00 con 0x55bf5c96fa30 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.323+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf5c96fa30 0x55bf5c971e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.324+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf5c96fa30 0x55bf5c971e20 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d30aea8c1730d96f server_cookie=b50cd0cae48f0295 in_seq=0 out_seq=0 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.324+0000 7f3fdf7fe640 10 client.18039 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/2841275794 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf5c8c8c00 con 0x55bf5c96fa30 2024-04-21T08:49:45.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f3fdf7fe640 10 client.18039 handle_command_reply: tid=0 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 client.18039 shutdown 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 -- 172.21.15.94:0/2841275794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 msgr2=0x55bf5c95d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95d010 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00186f0 tx=0x7f3ff000af10 comp rx=0 tx=0).stop 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 -- 172.21.15.94:0/2841275794 shutdown_connections 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf5c96fa30 0x55bf5c971e20 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c95fa80 0x55bf5c95d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c95d010 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f3fdf7fe640 0 client.18039 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 --2- 172.21.15.94:0/2841275794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c95cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 -- 172.21.15.94:0/2841275794 >> 172.21.15.94:0/2841275794 conn(0x55bf5c950cb0 msgr2=0x55bf5c952a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 -- 172.21.15.94:0/2841275794 shutdown_connections 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 1 -- 172.21.15.94:0/2841275794 wait complete. 2024-04-21T08:49:45.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.325+0000 7f4009398740 20 client.18039 trim_cache size 0 max 16384 2024-04-21T08:49:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.342+0000 7f4009398740 1 Processor -- start 2024-04-21T08:49:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.342+0000 7f4009398740 1 -- start start 2024-04-21T08:49:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.342+0000 7f4009398740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c956580 0x55bf5c95f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.342+0000 7f4009398740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c96b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.342+0000 7f4009398740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c95fa80 0x55bf5c95bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.342+0000 7f4009398740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f40001b1da0 con 0x55bf5c956ea0 2024-04-21T08:49:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.342+0000 7f4009398740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f40001af670 con 0x55bf5c956580 2024-04-21T08:49:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.342+0000 7f4009398740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f40001ad4e0 con 0x55bf5c95fa80 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c96b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c96b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59574/0 (socket says 172.21.15.94:59574) 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3ffffff640 1 -- 172.21.15.94:0/1081264226 learned_addr learned my addr 172.21.15.94:0/1081264226 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3ffffff640 1 -- 172.21.15.94:0/1081264226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c95fa80 msgr2=0x55bf5c95bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3ffffff640 1 --2- 172.21.15.94:0/1081264226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c95fa80 0x55bf5c95bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3ffffff640 1 -- 172.21.15.94:0/1081264226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c956580 msgr2=0x55bf5c95f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3ffffff640 1 --2- 172.21.15.94:0/1081264226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c956580 0x55bf5c95f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3ffffff640 1 -- 172.21.15.94:0/1081264226 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf5c9623d0 con 0x55bf5c956ea0 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3ffffff640 1 --2- 172.21.15.94:0/1081264226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c96b730 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0000c00 tx=0x7f3ff000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4836ab7fd9912206 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3fe5ffb640 1 -- 172.21.15.94:0/1081264226 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff001d040 con 0x55bf5c956ea0 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3fe5ffb640 1 -- 172.21.15.94:0/1081264226 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff0029070 con 0x55bf5c956ea0 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.343+0000 7f3fe5ffb640 1 -- 172.21.15.94:0/1081264226 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff00052a0 con 0x55bf5c956ea0 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 -- 172.21.15.94:0/1081264226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 msgr2=0x55bf5c96b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 --2- 172.21.15.94:0/1081264226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c96b730 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0000c00 tx=0x7f3ff000f790 comp rx=0 tx=0).stop 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 -- 172.21.15.94:0/1081264226 shutdown_connections 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 --2- 172.21.15.94:0/1081264226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c95fa80 0x55bf5c95bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 --2- 172.21.15.94:0/1081264226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c956ea0 0x55bf5c96b730 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 --2- 172.21.15.94:0/1081264226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c956580 0x55bf5c95f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 -- 172.21.15.94:0/1081264226 >> 172.21.15.94:0/1081264226 conn(0x55bf5c950cb0 msgr2=0x55bf5c94f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 -- 172.21.15.94:0/1081264226 shutdown_connections 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 -- 172.21.15.94:0/1081264226 wait complete. 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 -- start start 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.344+0000 7f4009398740 1 Processor -- start 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f4009398740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c967e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f4009398740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c956ea0 0x55bf5c959340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f4009398740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c95fa80 0x55bf5c959880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f4009398740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f40001adb30 con 0x55bf5c95fa80 2024-04-21T08:49:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f4009398740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f40001aefa0 con 0x55bf5c956ea0 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f4009398740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f40001ae200 con 0x55bf5c956580 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f4009398740 10 client.0 _ll_register_callbacks cb 0x7f3ff001b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fff7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c95fa80 0x55bf5c959880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fff7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c95fa80 0x55bf5c959880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59584/0 (socket says 172.21.15.94:59584) 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fff7fe640 1 -- 172.21.15.94:0/863339913 learned_addr learned my addr 172.21.15.94:0/863339913 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c967e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fff7fe640 1 -- 172.21.15.94:0/863339913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 msgr2=0x55bf5c967e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c967e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fff7fe640 1 -- 172.21.15.94:0/863339913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c956ea0 msgr2=0x55bf5c959340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c956ea0 0x55bf5c959340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fff7fe640 1 -- 172.21.15.94:0/863339913 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf5c959ec0 con 0x55bf5c95fa80 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fff7fe640 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c95fa80 0x55bf5c959880 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009230 tx=0x7f3ff402bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e26bcff0a05b4a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fdf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/863339913 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff402cb20 con 0x55bf5c95fa80 2024-04-21T08:49:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/863339913 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f3ff403a040 con 0x55bf5c95fa80 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f4009398740 1 -- 172.21.15.94:0/863339913 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ff0029070 con 0x55bf5c95fa80 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/863339913 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3ff402ccc0 con 0x55bf5c95fa80 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f4009398740 10 client.18045 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.345+0000 7f4009398740 1 -- 172.21.15.94:0/863339913 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bf5c9d5c60 con 0x55bf5c95fa80 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.346+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/863339913 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff0029070 con 0x55bf5c95fa80 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.346+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/863339913 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3ff4004b10 con 0x55bf5c95fa80 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.346+0000 7f4009398740 10 client.18045 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.346+0000 7f4009398740 10 client.18045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.346+0000 7f4009398740 20 client.18045 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.346+0000 7f4009398740 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf5c9d63c0 0x55bf5c9d8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.346+0000 7f4009398740 4 client.18045 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.346+0000 7f4009398740 1 -- 172.21.15.94:0/863339913 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bf5c8c2a70 con 0x55bf5c9d63c0 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.346+0000 7f3ffffff640 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf5c9d63c0 0x55bf5c9d8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f3ffffff640 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf5c9d63c0 0x55bf5c9d8820 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91b5830f808b31fc server_cookie=c0ee7cdbced7c056 in_seq=0 out_seq=0 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f3fdf7fe640 10 client.18045 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:45.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f3fdf7fe640 1 -- 172.21.15.94:0/863339913 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bf5c8c2a70 con 0x55bf5c9d63c0 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f3fdf7fe640 10 client.18045 handle_command_reply: tid=0 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 client.18045 shutdown 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 -- 172.21.15.94:0/863339913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c95fa80 msgr2=0x55bf5c959880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c95fa80 0x55bf5c959880 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009230 tx=0x7f3ff402bcd0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 -- 172.21.15.94:0/863339913 shutdown_connections 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55bf5c9d63c0 0x55bf5c9d8820 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bf5c95fa80 0x55bf5c959880 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf5c956ea0 0x55bf5c959340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f3fdf7fe640 0 client.18045 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 --2- 172.21.15.94:0/863339913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bf5c956580 0x55bf5c967e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 -- 172.21.15.94:0/863339913 >> 172.21.15.94:0/863339913 conn(0x55bf5c950cb0 msgr2=0x55bf5c95ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 -- 172.21.15.94:0/863339913 shutdown_connections 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 1 -- 172.21.15.94:0/863339913 wait complete. 2024-04-21T08:49:45.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.347+0000 7f4009398740 20 client.18045 trim_cache size 0 max 16384 2024-04-21T08:49:45.353 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:49:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:49:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:49:45.353 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 -- 172.21.15.94:0/2226238979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3fe80a3d20 msgr2=0x7f3fe80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3fe80a3d20 0x7f3fe80a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80ce260 tx=0x7f3ff80bd040 comp rx=0 tx=0).stop 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 -- 172.21.15.94:0/2226238979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4000108920 msgr2=0x7f400019cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4000108920 0x7f400019cfc0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800cab0 tx=0x7f3ff800cf70 comp rx=0 tx=0).stop 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 -- 172.21.15.94:0/2226238979 shutdown_connections 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3fe80a3d20 0x7f3fe80a61e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4000108000 0x7f4000198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4000108920 0x7f400019cfc0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 --2- 172.21.15.94:0/2226238979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f400006ace0 0x7f4000198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 -- 172.21.15.94:0/2226238979 >> 172.21.15.94:0/2226238979 conn(0x7f4000076c50 msgr2=0x7f4000100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 -- 172.21.15.94:0/2226238979 shutdown_connections 2024-04-21T08:49:45.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.348+0000 7f40066c8640 1 -- 172.21.15.94:0/2226238979 wait complete. 2024-04-21T08:49:45.358 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:49:45.358 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:49:45.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:45 smithi111 ceph-mon[31493]: pgmap v302: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:49:45.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.457+0000 7fdd8b575640 1 -- 172.21.15.94:0/1765931175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd84106f50 msgr2=0x7fdd84107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.457+0000 7fdd8b575640 1 --2- 172.21.15.94:0/1765931175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd84106f50 0x7fdd84107330 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009930 tx=0x7fdd7802bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:45.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.457+0000 7fdd8b575640 1 -- 172.21.15.94:0/1765931175 shutdown_connections 2024-04-21T08:49:45.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.457+0000 7fdd8b575640 1 --2- 172.21.15.94:0/1765931175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd841004c0 0x7fdd84100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.457+0000 7fdd8b575640 1 --2- 172.21.15.94:0/1765931175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd84100e80 0x7fdd8410ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.457+0000 7fdd8b575640 1 --2- 172.21.15.94:0/1765931175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd84106f50 0x7fdd84107330 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.457+0000 7fdd8b575640 1 -- 172.21.15.94:0/1765931175 >> 172.21.15.94:0/1765931175 conn(0x7fdd84077450 msgr2=0x7fdd840fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.457+0000 7fdd8b575640 1 -- 172.21.15.94:0/1765931175 shutdown_connections 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.457+0000 7fdd8b575640 1 -- 172.21.15.94:0/1765931175 wait complete. 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd8b575640 1 Processor -- start 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd8b575640 1 -- start start 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd8b575640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd841004c0 0x7fdd8419e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd8b575640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd84100e80 0x7fdd8419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd8b575640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd84106f50 0x7fdd841986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd8b575640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdd84112280 con 0x7fdd841004c0 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd8b575640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdd84112100 con 0x7fdd84106f50 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd8b575640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd84112400 con 0x7fdd84100e80 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd89311640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd841004c0 0x7fdd8419e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd89311640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd841004c0 0x7fdd8419e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59604/0 (socket says 172.21.15.94:59604) 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd89311640 1 -- 172.21.15.94:0/3941502860 learned_addr learned my addr 172.21.15.94:0/3941502860 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd89b12640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd84106f50 0x7fdd841986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd88b10640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd84100e80 0x7fdd8419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd89311640 1 -- 172.21.15.94:0/3941502860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd84100e80 msgr2=0x7fdd8419ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd89311640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd84100e80 0x7fdd8419ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd89311640 1 -- 172.21.15.94:0/3941502860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd84106f50 msgr2=0x7fdd841986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd89311640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd84106f50 0x7fdd841986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.458+0000 7fdd89311640 1 -- 172.21.15.94:0/3941502860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd84198f60 con 0x7fdd841004c0 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.459+0000 7fdd89311640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd841004c0 0x7fdd8419e540 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fdd78005940 tx=0x7fdd7802cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.459+0000 7fdd767fc640 1 -- 172.21.15.94:0/3941502860 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78043020 con 0x7fdd841004c0 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.459+0000 7fdd767fc640 1 -- 172.21.15.94:0/3941502860 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd780043d0 con 0x7fdd841004c0 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.459+0000 7fdd8b575640 1 -- 172.21.15.94:0/3941502860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd841991f0 con 0x7fdd841004c0 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.459+0000 7fdd767fc640 1 -- 172.21.15.94:0/3941502860 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78003af0 con 0x7fdd841004c0 2024-04-21T08:49:45.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.459+0000 7fdd8b575640 1 -- 172.21.15.94:0/3941502860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd841a54c0 con 0x7fdd841004c0 2024-04-21T08:49:45.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.460+0000 7fdd8b575640 1 -- 172.21.15.94:0/3941502860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd84199560 con 0x7fdd841004c0 2024-04-21T08:49:45.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.461+0000 7fdd767fc640 1 -- 172.21.15.94:0/3941502860 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fdd78046020 con 0x7fdd841004c0 2024-04-21T08:49:45.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.461+0000 7fdd767fc640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd700a3d20 0x7fdd700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.462+0000 7fdd88b10640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd700a3d20 0x7fdd700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.462+0000 7fdd767fc640 1 -- 172.21.15.94:0/3941502860 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fdd780f2bd0 con 0x7fdd841004c0 2024-04-21T08:49:45.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.462+0000 7fdd88b10640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd700a3d20 0x7fdd700a61e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdd84199ef0 tx=0x7fdd80005f90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.465+0000 7fdd767fc640 1 -- 172.21.15.94:0/3941502860 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd7804d070 con 0x7fdd841004c0 2024-04-21T08:49:45.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:45 smithi094 ceph-mon[27988]: pgmap v302: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:49:45.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:45 smithi173 ceph-mon[31428]: pgmap v302: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:49:45.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.678+0000 7fdd8b575640 1 -- 172.21.15.94:0/3941502860 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd841037e0 con 0x7fdd841004c0 2024-04-21T08:49:45.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.679+0000 7fdd767fc640 1 -- 172.21.15.94:0/3941502860 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7fdd780bef50 con 0x7fdd841004c0 2024-04-21T08:49:45.678 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:49:45.678 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:49:45.678 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:49:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 -- 172.21.15.94:0/3941502860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd700a3d20 msgr2=0x7fdd700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd700a3d20 0x7fdd700a61e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdd84199ef0 tx=0x7fdd80005f90 comp rx=0 tx=0).stop 2024-04-21T08:49:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 -- 172.21.15.94:0/3941502860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd841004c0 msgr2=0x7fdd8419e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd841004c0 0x7fdd8419e540 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fdd78005940 tx=0x7fdd7802cd20 comp rx=0 tx=0).stop 2024-04-21T08:49:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 -- 172.21.15.94:0/3941502860 shutdown_connections 2024-04-21T08:49:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fdd700a3d20 0x7fdd700a61e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd84100e80 0x7fdd8419ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd84106f50 0x7fdd841986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 --2- 172.21.15.94:0/3941502860 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd841004c0 0x7fdd8419e540 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 -- 172.21.15.94:0/3941502860 >> 172.21.15.94:0/3941502860 conn(0x7fdd84077450 msgr2=0x7fdd84105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 -- 172.21.15.94:0/3941502860 shutdown_connections 2024-04-21T08:49:45.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.681+0000 7fdd8b575640 1 -- 172.21.15.94:0/3941502860 wait complete. 2024-04-21T08:49:45.691 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:49:45.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.791+0000 7fe2a7088640 1 -- 172.21.15.94:0/2344351645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a0103af0 msgr2=0x7fe2a010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.791+0000 7fe2a7088640 1 --2- 172.21.15.94:0/2344351645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a0103af0 0x7fe2a010bfd0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009900 tx=0x7fe28c02bd60 comp rx=0 tx=0).stop 2024-04-21T08:49:45.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.791+0000 7fe2a7088640 1 -- 172.21.15.94:0/2344351645 shutdown_connections 2024-04-21T08:49:45.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.791+0000 7fe2a7088640 1 --2- 172.21.15.94:0/2344351645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2a010c510 0x7fe2a010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.791+0000 7fe2a7088640 1 --2- 172.21.15.94:0/2344351645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a0103af0 0x7fe2a010bfd0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.791+0000 7fe2a7088640 1 --2- 172.21.15.94:0/2344351645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2a01031d0 0x7fe2a01035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.791+0000 7fe2a7088640 1 -- 172.21.15.94:0/2344351645 >> 172.21.15.94:0/2344351645 conn(0x7fe2a0077f50 msgr2=0x7fe2a0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a7088640 1 -- 172.21.15.94:0/2344351645 shutdown_connections 2024-04-21T08:49:45.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a7088640 1 -- 172.21.15.94:0/2344351645 wait complete. 2024-04-21T08:49:45.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a7088640 1 Processor -- start 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a7088640 1 -- start start 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a7088640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a01031d0 0x7fe2a019e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a7088640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2a0103af0 0x7fe2a019e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a7088640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2a010c510 0x7fe2a01985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a7088640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a01121a0 con 0x7fe2a01031d0 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a7088640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a0112020 con 0x7fe2a0103af0 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a7088640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a0112320 con 0x7fe2a010c510 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a4e24640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a01031d0 0x7fe2a019e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a4e24640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a01031d0 0x7fe2a019e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59634/0 (socket says 172.21.15.94:59634) 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a4e24640 1 -- 172.21.15.94:0/1127534624 learned_addr learned my addr 172.21.15.94:0/1127534624 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.792+0000 7fe2a5625640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2a010c510 0x7fe2a01985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe29ffff640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2a0103af0 0x7fe2a019e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe2a4e24640 1 -- 172.21.15.94:0/1127534624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2a010c510 msgr2=0x7fe2a01985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2a010c510 0x7fe2a01985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe2a4e24640 1 -- 172.21.15.94:0/1127534624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2a0103af0 msgr2=0x7fe2a019e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2a0103af0 0x7fe2a019e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe2a4e24640 1 -- 172.21.15.94:0/1127534624 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a0198d70 con 0x7fe2a01031d0 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe2a5625640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2a010c510 0x7fe2a01985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a01031d0 0x7fe2a019e440 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fe29400c970 tx=0x7fe29400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe29dffb640 1 -- 172.21.15.94:0/1127534624 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294007bb0 con 0x7fe2a01031d0 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe29dffb640 1 -- 172.21.15.94:0/1127534624 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe294007d50 con 0x7fe2a01031d0 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe2a7088640 1 -- 172.21.15.94:0/1127534624 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a0199060 con 0x7fe2a01031d0 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe29dffb640 1 -- 172.21.15.94:0/1127534624 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29400f450 con 0x7fe2a01031d0 2024-04-21T08:49:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.793+0000 7fe2a7088640 1 -- 172.21.15.94:0/1127534624 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a01a5300 con 0x7fe2a01031d0 2024-04-21T08:49:45.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.795+0000 7fe29dffb640 1 -- 172.21.15.94:0/1127534624 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fe294004f20 con 0x7fe2a01031d0 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a9c7f740 1 Processor -- start 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a9c7f740 1 -- start start 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe29dffb640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2900a3d20 0x7fe2900a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a2417f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417fe80 0x560a24188430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 0x560a2418ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x560a240e41f0 con 0x560a2417f4d0 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560a240814b0 con 0x560a2417fe80 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560a240e01b0 con 0x560a24188a00 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a5625640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2900a3d20 0x7fe2900a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe29dffb640 1 -- 172.21.15.94:0/1127534624 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fe2940d0a40 con 0x7fe2a01031d0 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a4e24640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a2417f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a4e24640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a2417f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59640/0 (socket says 172.21.15.94:59640) 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a4e24640 1 -- 172.21.15.94:0/57795699 learned_addr learned my addr 172.21.15.94:0/57795699 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a5625640 1 --2- 172.21.15.94:0/57795699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 0x560a2418ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe29ffff640 1 --2- 172.21.15.94:0/57795699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417fe80 0x560a24188430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a4e24640 1 -- 172.21.15.94:0/57795699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 msgr2=0x560a2418ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/57795699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 0x560a2418ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a4e24640 1 -- 172.21.15.94:0/57795699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417fe80 msgr2=0x560a24188430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/57795699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417fe80 0x560a24188430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a4e24640 1 -- 172.21.15.94:0/57795699 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a240731b0 con 0x560a2417f4d0 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a5625640 1 --2- 172.21.15.94:0/57795699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 0x560a2418ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a5625640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2900a3d20 0x7fe2900a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe29800ad90 tx=0x7fe29800e720 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.796+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/57795699 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a2417f8b0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fe294098f80 tx=0x7fe2940056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a10d65f269bf7c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe28affd640 1 -- 172.21.15.94:0/57795699 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29401d040 con 0x560a2417f4d0 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe28affd640 1 -- 172.21.15.94:0/57795699 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2940a0960 con 0x560a2417f4d0 2024-04-21T08:49:45.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe28affd640 1 -- 172.21.15.94:0/57795699 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294059b40 con 0x560a2417f4d0 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/57795699 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 msgr2=0x560a2417f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/57795699 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a2417f8b0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fe294098f80 tx=0x7fe2940056b0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/57795699 shutdown_connections 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/57795699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 0x560a2418ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/57795699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417fe80 0x560a24188430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/57795699 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a2417f8b0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/57795699 >> 172.21.15.94:0/57795699 conn(0x560a24179ba0 msgr2=0x560a24179f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/57795699 shutdown_connections 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.797+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/57795699 wait complete. 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a9c7f740 1 -- start start 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a9c7f740 1 Processor -- start 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417f4d0 0x560a24188300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24185690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 0x560a24185c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x560a2418fd80 con 0x560a2417fe80 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560a2409d500 con 0x560a2417f4d0 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560a2418ff00 con 0x560a24188a00 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a9c7f740 10 client.0 _ll_register_callbacks cb 0x560a2406bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe29ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24185690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe29ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24185690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59644/0 (socket says 172.21.15.94:59644) 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe29ffff640 1 -- 172.21.15.94:0/2071379372 learned_addr learned my addr 172.21.15.94:0/2071379372 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417f4d0 0x560a24188300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a5625640 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 0x560a24185c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe29ffff640 1 -- 172.21.15.94:0/2071379372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 msgr2=0x560a24185c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe29ffff640 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 0x560a24185c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe29ffff640 1 -- 172.21.15.94:0/2071379372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417f4d0 msgr2=0x560a24188300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe29ffff640 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417f4d0 0x560a24188300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe29ffff640 1 -- 172.21.15.94:0/2071379372 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a23df63f0 con 0x560a2417fe80 2024-04-21T08:49:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a5625640 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 0x560a24185c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.798+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417f4d0 0x560a24188300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe29ffff640 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24185690 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe28c005ab0 tx=0x7fe28c02cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc79b05b609b091 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe27ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe27ffff640 1 -- 172.21.15.94:0/2071379372 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c005d50 con 0x560a2417fe80 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe27ffff640 1 -- 172.21.15.94:0/2071379372 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fe28c005220 con 0x560a2417fe80 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/2071379372 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2940a0960 con 0x560a2417fe80 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe27ffff640 1 -- 172.21.15.94:0/2071379372 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe28c04c050 con 0x560a2417fe80 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe2a9c7f740 10 client.18063 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/2071379372 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560a23ec07d0 con 0x560a2417fe80 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe27ffff640 1 -- 172.21.15.94:0/2071379372 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2940a0960 con 0x560a2417fe80 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe27ffff640 1 -- 172.21.15.94:0/2071379372 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe28c0035f0 con 0x560a2417fe80 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe2a9c7f740 10 client.18063 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe2a9c7f740 10 client.18063 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:49:45.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe2a9c7f740 20 client.18063 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x560a2419cfc0 0x560a2419f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe2a9c7f740 4 client.18063 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.799+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/2071379372 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a24083930 con 0x560a2419cfc0 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.800+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x560a2419cfc0 0x560a2419f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.800+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x560a2419cfc0 0x560a2419f3f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=529b16d1bd836a09 server_cookie=d9e19a626d3a77f4 in_seq=0 out_seq=0 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.800+0000 7fe27ffff640 10 client.18063 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe27ffff640 1 -- 172.21.15.94:0/2071379372 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a24083930 con 0x560a2419cfc0 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe27ffff640 10 client.18063 handle_command_reply: tid=0 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 client.18063 shutdown 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/2071379372 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 msgr2=0x560a24185690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24185690 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe28c005ab0 tx=0x7fe28c02cbc0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/2071379372 shutdown_connections 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x560a2419cfc0 0x560a2419f3f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a24188a00 0x560a24185c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24185690 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe27ffff640 0 client.18063 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:49:45.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/2071379372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a2417f4d0 0x560a24188300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/2071379372 >> 172.21.15.94:0/2071379372 conn(0x560a24179ba0 msgr2=0x560a2417b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/2071379372 shutdown_connections 2024-04-21T08:49:45.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/2071379372 wait complete. 2024-04-21T08:49:45.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.801+0000 7fe2a9c7f740 20 client.18063 trim_cache size 0 max 16384 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a9c7f740 1 Processor -- start 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a9c7f740 1 -- start start 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a24198c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a2417fe80 0x560a24187dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 0x560a24188300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a01afd80 con 0x560a2417f4d0 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a01ad650 con 0x560a24188a00 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a01ab4c0 con 0x560a2417fe80 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a4e24640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a24198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a4e24640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a24198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59648/0 (socket says 172.21.15.94:59648) 2024-04-21T08:49:45.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a4e24640 1 -- 172.21.15.94:0/27058870 learned_addr learned my addr 172.21.15.94:0/27058870 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a4e24640 1 -- 172.21.15.94:0/27058870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a2417fe80 msgr2=0x560a24187dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a5625640 1 --2- 172.21.15.94:0/27058870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 0x560a24188300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe29ffff640 1 --2- 172.21.15.94:0/27058870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a2417fe80 0x560a24187dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/27058870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a2417fe80 0x560a24187dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a4e24640 1 -- 172.21.15.94:0/27058870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 msgr2=0x560a24188300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.814+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/27058870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 0x560a24188300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a4e24640 1 -- 172.21.15.94:0/27058870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a241755e0 con 0x560a2417f4d0 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a5625640 1 --2- 172.21.15.94:0/27058870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 0x560a24188300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/27058870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a24198c40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fe29400cfd0 tx=0x7fe294004980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a8b70db874901c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe28a7fc640 1 -- 172.21.15.94:0/27058870 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29401d040 con 0x560a2417f4d0 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe28a7fc640 1 -- 172.21.15.94:0/27058870 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe294098c20 con 0x560a2417f4d0 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe28a7fc640 1 -- 172.21.15.94:0/27058870 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294061310 con 0x560a2417f4d0 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/27058870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 msgr2=0x560a24198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/27058870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a24198c40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fe29400cfd0 tx=0x7fe294004980 comp rx=0 tx=0).stop 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/27058870 shutdown_connections 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/27058870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a2417fe80 0x560a24187dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/27058870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 0x560a24188300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/27058870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417f4d0 0x560a24198c40 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/27058870 >> 172.21.15.94:0/27058870 conn(0x560a24179ba0 msgr2=0x560a24178c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/27058870 shutdown_connections 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.815+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/27058870 wait complete. 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a9c7f740 1 -- start start 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a9c7f740 1 Processor -- start 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a2417f4d0 0x560a24195410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24182000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a9c7f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 0x560a24182540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a01abb10 con 0x560a2417fe80 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a01acf80 con 0x560a24188a00 2024-04-21T08:49:45.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a9c7f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a01ac1e0 con 0x560a2417f4d0 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a9c7f740 10 client.0 _ll_register_callbacks cb 0x7fe28c004490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe29ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24182000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe29ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24182000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59652/0 (socket says 172.21.15.94:59652) 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe29ffff640 1 -- 172.21.15.94:0/1397907886 learned_addr learned my addr 172.21.15.94:0/1397907886 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a5625640 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 0x560a24182540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.816+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a2417f4d0 0x560a24195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe29ffff640 1 -- 172.21.15.94:0/1397907886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a2417f4d0 msgr2=0x560a24195410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe29ffff640 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a2417f4d0 0x560a24195410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe29ffff640 1 -- 172.21.15.94:0/1397907886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 msgr2=0x560a24182540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe29ffff640 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 0x560a24182540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe29ffff640 1 -- 172.21.15.94:0/1397907886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a24182c80 con 0x560a2417fe80 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe2a5625640 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 0x560a24182540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:45.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe29ffff640 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24182000 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fe28c02dfd0 tx=0x7fe28c003de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198969bfb2d62d59 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe27ffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe27ffff640 1 -- 172.21.15.94:0/1397907886 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c005d50 con 0x560a2417fe80 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe27ffff640 1 -- 172.21.15.94:0/1397907886 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fe28c005230 con 0x560a2417fe80 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/1397907886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe294098c20 con 0x560a2417fe80 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe27ffff640 1 -- 172.21.15.94:0/1397907886 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe28c02c3e0 con 0x560a2417fe80 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe2a9c7f740 10 client.18069 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/1397907886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560a24203410 con 0x560a2417fe80 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe27ffff640 1 -- 172.21.15.94:0/1397907886 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe294098c20 con 0x560a2417fe80 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe27ffff640 1 -- 172.21.15.94:0/1397907886 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe28c004c90 con 0x560a2417fe80 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe2a9c7f740 10 client.18069 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe2a9c7f740 10 client.18069 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe2a9c7f740 20 client.18069 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x560a24203be0 0x560a24206000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe2a9c7f740 4 client.18069 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.817+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/1397907886 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a240f1c10 con 0x560a24203be0 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.818+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x560a24203be0 0x560a24206000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.818+0000 7fe27ffff640 10 client.18069 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.818+0000 7fe2a4e24640 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x560a24203be0 0x560a24206000 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=332109bf2f8a45f4 server_cookie=9897c648c659142b in_seq=0 out_seq=0 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.818+0000 7fe27ffff640 1 -- 172.21.15.94:0/1397907886 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a240f1c10 con 0x560a24203be0 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.818+0000 7fe27ffff640 10 client.18069 handle_command_reply: tid=0 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.818+0000 7fe2a9c7f740 1 client.18069 shutdown 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.818+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/1397907886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 msgr2=0x560a24182000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.818+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24182000 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fe28c02dfd0 tx=0x7fe28c003de0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/1397907886 shutdown_connections 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x560a24203be0 0x560a24206000 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a24188a00 0x560a24182540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560a2417fe80 0x560a24182000 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe27ffff640 0 client.18069 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:49:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a9c7f740 1 --2- 172.21.15.94:0/1397907886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560a2417f4d0 0x560a24195410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/1397907886 >> 172.21.15.94:0/1397907886 conn(0x560a24179ba0 msgr2=0x560a24180260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/1397907886 shutdown_connections 2024-04-21T08:49:45.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a9c7f740 1 -- 172.21.15.94:0/1397907886 wait complete. 2024-04-21T08:49:45.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a9c7f740 20 client.18069 trim_cache size 0 max 16384 2024-04-21T08:49:45.825 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:49:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 -- 172.21.15.94:0/1127534624 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2900a3d20 msgr2=0x7fe2900a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2900a3d20 0x7fe2900a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe29800ad90 tx=0x7fe29800e720 comp rx=0 tx=0).stop 2024-04-21T08:49:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 -- 172.21.15.94:0/1127534624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a01031d0 msgr2=0x7fe2a019e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a01031d0 0x7fe2a019e440 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fe29400c970 tx=0x7fe29400ce30 comp rx=0 tx=0).stop 2024-04-21T08:49:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 -- 172.21.15.94:0/1127534624 shutdown_connections 2024-04-21T08:49:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe2900a3d20 0x7fe2900a61e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2a010c510 0x7fe2a01985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2a0103af0 0x7fe2a019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 --2- 172.21.15.94:0/1127534624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a01031d0 0x7fe2a019e440 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 -- 172.21.15.94:0/1127534624 >> 172.21.15.94:0/1127534624 conn(0x7fe2a0077f50 msgr2=0x7fe2a00fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 -- 172.21.15.94:0/1127534624 shutdown_connections 2024-04-21T08:49:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.819+0000 7fe2a7088640 1 -- 172.21.15.94:0/1127534624 wait complete. 2024-04-21T08:49:45.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 -- 172.21.15.94:0/502846316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd040101a40 msgr2=0x7fd040105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 --2- 172.21.15.94:0/502846316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd040101a40 0x7fd040105a80 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009900 tx=0x7fd03c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 -- 172.21.15.94:0/502846316 shutdown_connections 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 --2- 172.21.15.94:0/502846316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd040106160 0x7fd04010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 --2- 172.21.15.94:0/502846316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd040101a40 0x7fd040105a80 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 --2- 172.21.15.94:0/502846316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd040101090 0x7fd040101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 -- 172.21.15.94:0/502846316 >> 172.21.15.94:0/502846316 conn(0x7fd040077f50 msgr2=0x7fd040078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 -- 172.21.15.94:0/502846316 shutdown_connections 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 -- 172.21.15.94:0/502846316 wait complete. 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 Processor -- start 2024-04-21T08:49:45.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.931+0000 7fd048048640 1 -- start start 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd048048640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd040101090 0x7fd04010e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd048048640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd040106160 0x7fd04010ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd048048640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04010f140 0x7fd0401a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd048048640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd0401121a0 con 0x7fd04010f140 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd048048640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd040112020 con 0x7fd040101090 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd048048640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd040112320 con 0x7fd040106160 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0465e5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04010f140 0x7fd0401a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0465e5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04010f140 0x7fd0401a68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59666/0 (socket says 172.21.15.94:59666) 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0465e5640 1 -- 172.21.15.94:0/2275714777 learned_addr learned my addr 172.21.15.94:0/2275714777 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd045de4640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd040101090 0x7fd04010e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0455e3640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd040106160 0x7fd04010ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0465e5640 1 -- 172.21.15.94:0/2275714777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd040106160 msgr2=0x7fd04010ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0465e5640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd040106160 0x7fd04010ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0465e5640 1 -- 172.21.15.94:0/2275714777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd040101090 msgr2=0x7fd04010e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0465e5640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd040101090 0x7fd04010e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0465e5640 1 -- 172.21.15.94:0/2275714777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0401a6ff0 con 0x7fd04010f140 2024-04-21T08:49:45.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0455e3640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd040106160 0x7fd04010ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:45.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd0465e5640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04010f140 0x7fd0401a68f0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fd03800eaa0 tx=0x7fd03800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd032ffd640 1 -- 172.21.15.94:0/2275714777 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03800cdb0 con 0x7fd04010f140 2024-04-21T08:49:45.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd032ffd640 1 -- 172.21.15.94:0/2275714777 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd038005020 con 0x7fd04010f140 2024-04-21T08:49:45.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd048048640 1 -- 172.21.15.94:0/2275714777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0401a72e0 con 0x7fd04010f140 2024-04-21T08:49:45.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd032ffd640 1 -- 172.21.15.94:0/2275714777 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0380116d0 con 0x7fd04010f140 2024-04-21T08:49:45.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.932+0000 7fd048048640 1 -- 172.21.15.94:0/2275714777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0401a77b0 con 0x7fd04010f140 2024-04-21T08:49:45.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd032ffd640 1 -- 172.21.15.94:0/2275714777 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fd0380051d0 con 0x7fd04010f140 2024-04-21T08:49:45.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd04ac30740 1 Processor -- start 2024-04-21T08:49:45.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd04ac30740 1 -- start start 2024-04-21T08:49:45.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d1b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d24430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d26e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd04ac30740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55dcc5c801f0 con 0x55dcc5d1be80 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd04ac30740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dcc5c1d4b0 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd04ac30740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dcc5c7c1b0 con 0x55dcc5d24a00 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd045de4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d24430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd045de4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d24430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59682/0 (socket says 172.21.15.94:59682) 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd045de4640 1 -- 172.21.15.94:0/3917427871 learned_addr learned my addr 172.21.15.94:0/3917427871 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.935+0000 7fd032ffd640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd02c0a3d50 0x7fd02c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd0455e3640 1 --2- 172.21.15.94:0/3917427871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d26e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd0455e3640 1 --2- 172.21.15.94:0/3917427871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d1b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd045de4640 1 -- 172.21.15.94:0/3917427871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 msgr2=0x55dcc5d26e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd045de4640 1 --2- 172.21.15.94:0/3917427871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d26e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd045de4640 1 -- 172.21.15.94:0/3917427871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1b4d0 msgr2=0x55dcc5d1b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd045de4640 1 --2- 172.21.15.94:0/3917427871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d1b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd045de4640 1 -- 172.21.15.94:0/3917427871 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcc5c0f1b0 con 0x55dcc5d1be80 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd0455e3640 1 --2- 172.21.15.94:0/3917427871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d1b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd0465e5640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd02c0a3d50 0x7fd02c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd032ffd640 1 -- 172.21.15.94:0/2275714777 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fd0380d0d80 con 0x7fd04010f140 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd045de4640 1 --2- 172.21.15.94:0/3917427871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d24430 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd03400bd10 tx=0x7fd0340107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfc39a12868b8bb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd02bfff640 1 -- 172.21.15.94:0/3917427871 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd034004030 con 0x55dcc5d1be80 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd02bfff640 1 -- 172.21.15.94:0/3917427871 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0340035f0 con 0x55dcc5d1be80 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.936+0000 7fd02bfff640 1 -- 172.21.15.94:0/3917427871 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd034011770 con 0x55dcc5d1be80 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 -- 172.21.15.94:0/3917427871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 msgr2=0x55dcc5d24430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd0465e5640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd02c0a3d50 0x7fd02c0a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd0380a5030 tx=0x7fd03806e330 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3917427871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d24430 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd03400bd10 tx=0x7fd0340107a0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 -- 172.21.15.94:0/3917427871 shutdown_connections 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3917427871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d26e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3917427871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d24430 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3917427871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d1b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 -- 172.21.15.94:0/3917427871 >> 172.21.15.94:0/3917427871 conn(0x55dcc5d15ba0 msgr2=0x55dcc5d15f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 -- 172.21.15.94:0/3917427871 shutdown_connections 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 -- 172.21.15.94:0/3917427871 wait complete. 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 -- start start 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.937+0000 7fd04ac30740 1 Processor -- start 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d242f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 0x55dcc5d21680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d21c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd04ac30740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55dcc5d27970 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd04ac30740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dcc5c39500 con 0x55dcc5d1be80 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd04ac30740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dcc5d27af0 con 0x55dcc5d24a00 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd04ac30740 10 client.0 _ll_register_callbacks cb 0x55dcc5c07b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0455e3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d242f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0455e3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d242f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59694/0 (socket says 172.21.15.94:59694) 2024-04-21T08:49:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0455e3640 1 -- 172.21.15.94:0/3383348028 learned_addr learned my addr 172.21.15.94:0/3383348028 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0455e3640 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d21c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd045de4640 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 0x55dcc5d21680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0455e3640 1 -- 172.21.15.94:0/3383348028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 msgr2=0x55dcc5d21c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0455e3640 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d21c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0455e3640 1 -- 172.21.15.94:0/3383348028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 msgr2=0x55dcc5d21680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0455e3640 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 0x55dcc5d21680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0455e3640 1 -- 172.21.15.94:0/3383348028 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcc59923f0 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0455e3640 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d242f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fd04006b6e0 tx=0x7fd03c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46e647225532bd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0297fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0297fa640 1 -- 172.21.15.94:0/3383348028 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c005230 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0297fa640 1 -- 172.21.15.94:0/3383348028 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fd03c0053d0 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd04ac30740 1 -- 172.21.15.94:0/3383348028 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0340035f0 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd0297fa640 1 -- 172.21.15.94:0/3383348028 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd03c053050 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd04ac30740 10 client.18081 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.938+0000 7fd04ac30740 1 -- 172.21.15.94:0/3383348028 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dcc5a5c7d0 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.939+0000 7fd0297fa640 1 -- 172.21.15.94:0/3383348028 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0340035f0 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.939+0000 7fd0297fa640 1 -- 172.21.15.94:0/3383348028 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd03c040040 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.939+0000 7fd04ac30740 10 client.18081 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.939+0000 7fd04ac30740 10 client.18081 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.939+0000 7fd04ac30740 20 client.18081 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.939+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55dcc5d34b60 0x55dcc5d36fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.939+0000 7fd04ac30740 4 client.18081 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.939+0000 7fd04ac30740 1 -- 172.21.15.94:0/3383348028 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcc5c1f930 con 0x55dcc5d34b60 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.939+0000 7fd045de4640 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55dcc5d34b60 0x55dcc5d36fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.940+0000 7fd0297fa640 10 client.18081 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.940+0000 7fd045de4640 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55dcc5d34b60 0x55dcc5d36fb0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7a57b953c1335405 server_cookie=b0b81b5acb3a3bc4 in_seq=0 out_seq=0 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.940+0000 7fd0297fa640 1 -- 172.21.15.94:0/3383348028 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dcc5c1f930 con 0x55dcc5d34b60 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.940+0000 7fd0297fa640 10 client.18081 handle_command_reply: tid=0 2024-04-21T08:49:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.940+0000 7fd04ac30740 1 client.18081 shutdown 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.940+0000 7fd04ac30740 1 -- 172.21.15.94:0/3383348028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 msgr2=0x55dcc5d242f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.940+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d242f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fd04006b6e0 tx=0x7fd03c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.941+0000 7fd04ac30740 1 -- 172.21.15.94:0/3383348028 shutdown_connections 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.941+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d21c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.941+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55dcc5d34b60 0x55dcc5d36fb0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.941+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 0x55dcc5d21680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.941+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3383348028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d242f0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.941+0000 7fd0297fa640 0 client.18081 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.941+0000 7fd04ac30740 1 -- 172.21.15.94:0/3383348028 >> 172.21.15.94:0/3383348028 conn(0x55dcc5d15ba0 msgr2=0x55dcc5d178f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.941+0000 7fd04ac30740 1 -- 172.21.15.94:0/3383348028 shutdown_connections 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.941+0000 7fd04ac30740 1 -- 172.21.15.94:0/3383348028 wait complete. 2024-04-21T08:49:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.941+0000 7fd04ac30740 20 client.18081 trim_cache size 0 max 16384 2024-04-21T08:49:45.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd04ac30740 1 Processor -- start 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd04ac30740 1 -- start start 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d30830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d14a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d24a00 0x55dcc5d23db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd04ac30740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd0401b4030 con 0x55dcc5d1be80 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd04ac30740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd0401b1900 con 0x55dcc5d24a00 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd04ac30740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd0401af770 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd045de4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d14a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd045de4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d14a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59698/0 (socket says 172.21.15.94:59698) 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd045de4640 1 -- 172.21.15.94:0/3233322351 learned_addr learned my addr 172.21.15.94:0/3233322351 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd0455e3640 1 --2- 172.21.15.94:0/3233322351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d24a00 0x55dcc5d23db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd045de4640 1 -- 172.21.15.94:0/3233322351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d1b4d0 msgr2=0x55dcc5d30830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd045de4640 1 --2- 172.21.15.94:0/3233322351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d30830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd045de4640 1 -- 172.21.15.94:0/3233322351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d24a00 msgr2=0x55dcc5d23db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd045de4640 1 --2- 172.21.15.94:0/3233322351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d24a00 0x55dcc5d23db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd045de4640 1 -- 172.21.15.94:0/3233322351 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcc5d27180 con 0x55dcc5d1be80 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd045de4640 1 --2- 172.21.15.94:0/3233322351 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d14a80 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x55dcc5d1afa0 tx=0x7fd034003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312d3256d7d3c7e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.954+0000 7fd02b7fe640 1 -- 172.21.15.94:0/3233322351 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03402e070 con 0x55dcc5d1be80 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd02b7fe640 1 -- 172.21.15.94:0/3233322351 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd034005040 con 0x55dcc5d1be80 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd02b7fe640 1 -- 172.21.15.94:0/3233322351 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0340293d0 con 0x55dcc5d1be80 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 -- 172.21.15.94:0/3233322351 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 msgr2=0x55dcc5d14a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3233322351 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d14a80 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x55dcc5d1afa0 tx=0x7fd034003d60 comp rx=0 tx=0).stop 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 -- 172.21.15.94:0/3233322351 shutdown_connections 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3233322351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d24a00 0x55dcc5d23db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3233322351 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1be80 0x55dcc5d14a80 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 --2- 172.21.15.94:0/3233322351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d30830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 -- 172.21.15.94:0/3233322351 >> 172.21.15.94:0/3233322351 conn(0x55dcc5d15ba0 msgr2=0x55dcc5d14370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 -- 172.21.15.94:0/3233322351 shutdown_connections 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 -- 172.21.15.94:0/3233322351 wait complete. 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 -- start start 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.955+0000 7fd04ac30740 1 Processor -- start 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d2cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 0x55dcc5d2d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd04ac30740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d1de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd04ac30740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd0401afdc0 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd04ac30740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd0401b1230 con 0x55dcc5d1be80 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd04ac30740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd0401b0490 con 0x55dcc5d24a00 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd04ac30740 10 client.0 _ll_register_callbacks cb 0x7fd03c02c940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0455e3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d2cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0455e3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d2cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59702/0 (socket says 172.21.15.94:59702) 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0455e3640 1 -- 172.21.15.94:0/1595397251 learned_addr learned my addr 172.21.15.94:0/1595397251 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd045de4640 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 0x55dcc5d2d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0455e3640 1 -- 172.21.15.94:0/1595397251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 msgr2=0x55dcc5d1de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0455e3640 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d1de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0455e3640 1 -- 172.21.15.94:0/1595397251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 msgr2=0x55dcc5d2d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0455e3640 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 0x55dcc5d2d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0455e3640 1 -- 172.21.15.94:0/1595397251 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcc5d1e670 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd045de4640 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 0x55dcc5d2d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0455e3640 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d2cd40 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fd03c02bc70 tx=0x7fd03c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1e7d4c26eeba0f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0297fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0297fa640 1 -- 172.21.15.94:0/1595397251 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c05b020 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd0297fa640 1 -- 172.21.15.94:0/1595397251 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7fd03c043480 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.956+0000 7fd04ac30740 1 -- 172.21.15.94:0/1595397251 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd034005040 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd0297fa640 1 -- 172.21.15.94:0/1595397251 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd03c047020 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd04ac30740 10 client.18087 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd04ac30740 1 -- 172.21.15.94:0/1595397251 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dcc5d9ac90 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd0297fa640 1 -- 172.21.15.94:0/1595397251 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd034005040 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd0297fa640 1 -- 172.21.15.94:0/1595397251 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd03c034440 con 0x55dcc5d1b4d0 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd04ac30740 10 client.18087 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd04ac30740 10 client.18087 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd04ac30740 20 client.18087 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd04ac30740 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55dcc5d9b3b0 0x55dcc5d9d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd04ac30740 4 client.18087 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd04ac30740 1 -- 172.21.15.94:0/1595397251 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dcc5c8dc10 con 0x55dcc5d9b3b0 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.957+0000 7fd045de4640 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55dcc5d9b3b0 0x55dcc5d9d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd0297fa640 10 client.18087 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:49:45.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd045de4640 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55dcc5d9b3b0 0x55dcc5d9d7d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=af33435ae04bca35 server_cookie=7064419aca8f0356 in_seq=0 out_seq=0 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd0297fa640 1 -- 172.21.15.94:0/1595397251 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dcc5c8dc10 con 0x55dcc5d9b3b0 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd0297fa640 10 client.18087 handle_command_reply: tid=0 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 client.18087 shutdown 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 -- 172.21.15.94:0/1595397251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 msgr2=0x55dcc5d2cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d2cd40 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fd03c02bc70 tx=0x7fd03c009560 comp rx=0 tx=0).stop 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 -- 172.21.15.94:0/1595397251 shutdown_connections 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dcc5d24a00 0x55dcc5d1de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55dcc5d9b3b0 0x55dcc5d9d7d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dcc5d1be80 0x55dcc5d2d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd0297fa640 0 client.18087 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 --2- 172.21.15.94:0/1595397251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dcc5d1b4d0 0x55dcc5d2cd40 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 -- 172.21.15.94:0/1595397251 >> 172.21.15.94:0/1595397251 conn(0x55dcc5d15ba0 msgr2=0x55dcc5d14c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 -- 172.21.15.94:0/1595397251 shutdown_connections 2024-04-21T08:49:45.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 1 -- 172.21.15.94:0/1595397251 wait complete. 2024-04-21T08:49:45.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.958+0000 7fd04ac30740 20 client.18087 trim_cache size 0 max 16384 2024-04-21T08:49:45.967 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:49:45.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 -- 172.21.15.94:0/2275714777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd02c0a3d50 msgr2=0x7fd02c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd02c0a3d50 0x7fd02c0a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd0380a5030 tx=0x7fd03806e330 comp rx=0 tx=0).stop 2024-04-21T08:49:45.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 -- 172.21.15.94:0/2275714777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04010f140 msgr2=0x7fd0401a68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04010f140 0x7fd0401a68f0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fd03800eaa0 tx=0x7fd03800ef60 comp rx=0 tx=0).stop 2024-04-21T08:49:45.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 -- 172.21.15.94:0/2275714777 shutdown_connections 2024-04-21T08:49:45.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fd02c0a3d50 0x7fd02c0a6210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd040106160 0x7fd04010ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd04010f140 0x7fd0401a68f0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 --2- 172.21.15.94:0/2275714777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd040101090 0x7fd04010e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 -- 172.21.15.94:0/2275714777 >> 172.21.15.94:0/2275714777 conn(0x7fd040077f50 msgr2=0x7fd0400ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 -- 172.21.15.94:0/2275714777 shutdown_connections 2024-04-21T08:49:45.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:45.959+0000 7fd048048640 1 -- 172.21.15.94:0/2275714777 wait complete. 2024-04-21T08:49:45.970 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.070+0000 7faa3d575640 1 -- 172.21.15.94:0/3360805053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa381015c0 msgr2=0x7faa38101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.070+0000 7faa3d575640 1 --2- 172.21.15.94:0/3360805053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa381015c0 0x7faa38101a20 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009900 tx=0x7faa2c02bc80 comp rx=0 tx=0).stop 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.070+0000 7faa3d575640 1 -- 172.21.15.94:0/3360805053 shutdown_connections 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.070+0000 7faa3d575640 1 --2- 172.21.15.94:0/3360805053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa38101f60 0x7faa3810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.070+0000 7faa3d575640 1 --2- 172.21.15.94:0/3360805053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa381015c0 0x7faa38101a20 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.070+0000 7faa3d575640 1 --2- 172.21.15.94:0/3360805053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa38105e20 0x7faa38106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.070+0000 7faa3d575640 1 -- 172.21.15.94:0/3360805053 >> 172.21.15.94:0/3360805053 conn(0x7faa380773e0 msgr2=0x7faa380fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.070+0000 7faa3d575640 1 -- 172.21.15.94:0/3360805053 shutdown_connections 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.070+0000 7faa3d575640 1 -- 172.21.15.94:0/3360805053 wait complete. 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa3d575640 1 Processor -- start 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa3d575640 1 -- start start 2024-04-21T08:49:46.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa3d575640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa381015c0 0x7faa3819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa3d575640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa38101f60 0x7faa3819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa3d575640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa38105e20 0x7faa38198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa3d575640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7faa381122f0 con 0x7faa38105e20 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa3d575640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa38112170 con 0x7faa38101f60 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa3d575640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faa38112470 con 0x7faa381015c0 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa377fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa38105e20 0x7faa38198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa377fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa38105e20 0x7faa38198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59714/0 (socket says 172.21.15.94:59714) 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.071+0000 7faa377fe640 1 -- 172.21.15.94:0/753138631 learned_addr learned my addr 172.21.15.94:0/753138631 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa367fc640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa38101f60 0x7faa3819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa36ffd640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa381015c0 0x7faa3819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa377fe640 1 -- 172.21.15.94:0/753138631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa381015c0 msgr2=0x7faa3819e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa377fe640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa381015c0 0x7faa3819e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa377fe640 1 -- 172.21.15.94:0/753138631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa38101f60 msgr2=0x7faa3819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa377fe640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa38101f60 0x7faa3819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa377fe640 1 -- 172.21.15.94:0/753138631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa38198d90 con 0x7faa38105e20 2024-04-21T08:49:46.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa36ffd640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa381015c0 0x7faa3819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:46.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa377fe640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa38105e20 0x7faa38198610 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7faa3000eaa0 tx=0x7faa3000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa1ffff640 1 -- 172.21.15.94:0/753138631 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3000cdb0 con 0x7faa38105e20 2024-04-21T08:49:46.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa3d575640 1 -- 172.21.15.94:0/753138631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa38199080 con 0x7faa38105e20 2024-04-21T08:49:46.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa1ffff640 1 -- 172.21.15.94:0/753138631 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa30005020 con 0x7faa38105e20 2024-04-21T08:49:46.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa1ffff640 1 -- 172.21.15.94:0/753138631 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa300116d0 con 0x7faa38105e20 2024-04-21T08:49:46.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.072+0000 7faa3d575640 1 -- 172.21.15.94:0/753138631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa381a5320 con 0x7faa38105e20 2024-04-21T08:49:46.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.074+0000 7faa1ffff640 1 -- 172.21.15.94:0/753138631 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7faa300051d0 con 0x7faa38105e20 2024-04-21T08:49:46.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa40165740 1 Processor -- start 2024-04-21T08:49:46.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa40165740 1 -- start start 2024-04-21T08:49:46.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa1ffff640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7faa200a3d20 0x7faa200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa40165740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cda8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa40165740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa40165740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa40165740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d6c7c3f1f0 con 0x55d6c7cda4d0 2024-04-21T08:49:46.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa40165740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d6c7bdc4b0 con 0x55d6c7ce3a00 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa40165740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d6c7c3b1b0 con 0x55d6c7cdae80 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa367fc640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7faa200a3d20 0x7faa200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa36ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cda8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa1ffff640 1 -- 172.21.15.94:0/753138631 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7faa300d0ed0 con 0x7faa38105e20 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa36ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cda8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59716/0 (socket says 172.21.15.94:59716) 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa36ffd640 1 -- 172.21.15.94:0/420188117 learned_addr learned my addr 172.21.15.94:0/420188117 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa377fe640 1 --2- 172.21.15.94:0/420188117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa367fc640 1 --2- 172.21.15.94:0/420188117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa36ffd640 1 -- 172.21.15.94:0/420188117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cdae80 msgr2=0x55d6c7ce3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa36ffd640 1 --2- 172.21.15.94:0/420188117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa36ffd640 1 -- 172.21.15.94:0/420188117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7ce3a00 msgr2=0x55d6c7ce5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa36ffd640 1 --2- 172.21.15.94:0/420188117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa36ffd640 1 -- 172.21.15.94:0/420188117 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6c7bce1b0 con 0x55d6c7cda4d0 2024-04-21T08:49:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa367fc640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7faa200a3d20 0x7faa200a61e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faa38199750 tx=0x7faa2c02ca00 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.075+0000 7faa36ffd640 1 --2- 172.21.15.94:0/420188117 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cda8b0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7faa28009c30 tx=0x7faa28010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba62572c33379ea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa1d7fa640 1 -- 172.21.15.94:0/420188117 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa28004030 con 0x55d6c7cda4d0 2024-04-21T08:49:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa1d7fa640 1 -- 172.21.15.94:0/420188117 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa280041d0 con 0x55d6c7cda4d0 2024-04-21T08:49:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa1d7fa640 1 -- 172.21.15.94:0/420188117 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa280126b0 con 0x55d6c7cda4d0 2024-04-21T08:49:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa40165740 1 -- 172.21.15.94:0/420188117 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cda4d0 msgr2=0x55d6c7cda8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa40165740 1 --2- 172.21.15.94:0/420188117 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cda8b0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7faa28009c30 tx=0x7faa28010740 comp rx=0 tx=0).stop 2024-04-21T08:49:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa40165740 1 -- 172.21.15.94:0/420188117 shutdown_connections 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa40165740 1 --2- 172.21.15.94:0/420188117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa40165740 1 --2- 172.21.15.94:0/420188117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa40165740 1 --2- 172.21.15.94:0/420188117 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cda8b0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa40165740 1 -- 172.21.15.94:0/420188117 >> 172.21.15.94:0/420188117 conn(0x55d6c7cd4ba0 msgr2=0x55d6c7cd4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa40165740 1 -- 172.21.15.94:0/420188117 shutdown_connections 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.076+0000 7faa40165740 1 -- 172.21.15.94:0/420188117 wait complete. 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa40165740 1 -- start start 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa40165740 1 Processor -- start 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa40165740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cda4d0 0x55d6c7ce3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa40165740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa40165740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa40165740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55d6c7ce6970 con 0x55d6c7ce3a00 2024-04-21T08:49:46.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa40165740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d6c7bf8500 con 0x55d6c7cda4d0 2024-04-21T08:49:46.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa40165740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d6c7ce6af0 con 0x55d6c7cdae80 2024-04-21T08:49:46.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa40165740 10 client.0 _ll_register_callbacks cb 0x55d6c7bc6b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:46.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa377fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa377fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59718/0 (socket says 172.21.15.94:59718) 2024-04-21T08:49:46.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa377fe640 1 -- 172.21.15.94:0/417104657 learned_addr learned my addr 172.21.15.94:0/417104657 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:46.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa367fc640 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa36ffd640 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cda4d0 0x55d6c7ce3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa377fe640 1 -- 172.21.15.94:0/417104657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cdae80 msgr2=0x55d6c7ce0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa377fe640 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa377fe640 1 -- 172.21.15.94:0/417104657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cda4d0 msgr2=0x55d6c7ce3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa377fe640 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cda4d0 0x55d6c7ce3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa377fe640 1 -- 172.21.15.94:0/417104657 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6c79513f0 con 0x55d6c7ce3a00 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa36ffd640 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cda4d0 0x55d6c7ce3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa377fe640 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce0c10 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7faa300ce250 tx=0x7faa3009d140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ca7cec5277eda1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.077+0000 7faa1aff5640 1 -- 172.21.15.94:0/417104657 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3001f040 con 0x55d6c7ce3a00 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa1aff5640 1 -- 172.21.15.94:0/417104657 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7faa3000ad50 con 0x55d6c7ce3a00 2024-04-21T08:49:46.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa40165740 1 -- 172.21.15.94:0/417104657 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa280041d0 con 0x55d6c7ce3a00 2024-04-21T08:49:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa1aff5640 1 -- 172.21.15.94:0/417104657 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faa300a56b0 con 0x55d6c7ce3a00 2024-04-21T08:49:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa40165740 10 client.18099 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa40165740 1 -- 172.21.15.94:0/417104657 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d6c7a1b7d0 con 0x55d6c7ce3a00 2024-04-21T08:49:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa1aff5640 1 -- 172.21.15.94:0/417104657 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa280041d0 con 0x55d6c7ce3a00 2024-04-21T08:49:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa1aff5640 1 -- 172.21.15.94:0/417104657 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7faa3000f040 con 0x55d6c7ce3a00 2024-04-21T08:49:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa40165740 10 client.18099 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa40165740 10 client.18099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa40165740 20 client.18099 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:46.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa40165740 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d6c7cf3b40 0x55d6c7cf5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa40165740 4 client.18099 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:46.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa40165740 1 -- 172.21.15.94:0/417104657 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6c7bde930 con 0x55d6c7cf3b40 2024-04-21T08:49:46.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.078+0000 7faa36ffd640 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d6c7cf3b40 0x55d6c7cf5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.079+0000 7faa1aff5640 10 client.18099 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:46.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.079+0000 7faa36ffd640 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d6c7cf3b40 0x55d6c7cf5fd0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6dc96e31cfc7ef7 server_cookie=cc51e76d791d5039 in_seq=0 out_seq=0 2024-04-21T08:49:46.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.079+0000 7faa1aff5640 1 -- 172.21.15.94:0/417104657 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d6c7bde930 con 0x55d6c7cf3b40 2024-04-21T08:49:46.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.079+0000 7faa1aff5640 10 client.18099 handle_command_reply: tid=0 2024-04-21T08:49:46.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.079+0000 7faa40165740 1 client.18099 shutdown 2024-04-21T08:49:46.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 1 -- 172.21.15.94:0/417104657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 msgr2=0x55d6c7ce0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce0c10 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7faa300ce250 tx=0x7faa3009d140 comp rx=0 tx=0).stop 2024-04-21T08:49:46.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 1 -- 172.21.15.94:0/417104657 shutdown_connections 2024-04-21T08:49:46.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d6c7cf3b40 0x55d6c7cf5fd0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce0c10 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 1 --2- 172.21.15.94:0/417104657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cda4d0 0x55d6c7ce3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa1aff5640 0 client.18099 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:46.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 1 -- 172.21.15.94:0/417104657 >> 172.21.15.94:0/417104657 conn(0x55d6c7cd4ba0 msgr2=0x55d6c7cd68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 1 -- 172.21.15.94:0/417104657 shutdown_connections 2024-04-21T08:49:46.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 1 -- 172.21.15.94:0/417104657 wait complete. 2024-04-21T08:49:46.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.080+0000 7faa40165740 20 client.18099 trim_cache size 0 max 16384 2024-04-21T08:49:46.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa40165740 1 Processor -- start 2024-04-21T08:49:46.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa40165740 1 -- start start 2024-04-21T08:49:46.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa40165740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cef830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa40165740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa40165740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa40165740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7faa381b1ec0 con 0x55d6c7ce3a00 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa40165740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa381af790 con 0x55d6c7cdae80 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa40165740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faa381ad600 con 0x55d6c7cda4d0 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa377fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa377fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59726/0 (socket says 172.21.15.94:59726) 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa377fe640 1 -- 172.21.15.94:0/1923428786 learned_addr learned my addr 172.21.15.94:0/1923428786 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa377fe640 1 -- 172.21.15.94:0/1923428786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 msgr2=0x55d6c7cef830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa36ffd640 1 --2- 172.21.15.94:0/1923428786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cef830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa367fc640 1 --2- 172.21.15.94:0/1923428786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa377fe640 1 --2- 172.21.15.94:0/1923428786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cef830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa377fe640 1 -- 172.21.15.94:0/1923428786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cdae80 msgr2=0x55d6c7ce2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa377fe640 1 --2- 172.21.15.94:0/1923428786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa377fe640 1 -- 172.21.15.94:0/1923428786 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6c7cdac20 con 0x55d6c7ce3a00 2024-04-21T08:49:46.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.093+0000 7faa367fc640 1 --2- 172.21.15.94:0/1923428786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa377fe640 1 --2- 172.21.15.94:0/1923428786 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce3300 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7faa300ce2a0 tx=0x7faa30005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e7a7447d8f24f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa1cff9640 1 -- 172.21.15.94:0/1923428786 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3001f070 con 0x55d6c7ce3a00 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa1cff9640 1 -- 172.21.15.94:0/1923428786 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa30016210 con 0x55d6c7ce3a00 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa1cff9640 1 -- 172.21.15.94:0/1923428786 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3009cca0 con 0x55d6c7ce3a00 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa40165740 1 -- 172.21.15.94:0/1923428786 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 msgr2=0x55d6c7ce3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa40165740 1 --2- 172.21.15.94:0/1923428786 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce3300 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7faa300ce2a0 tx=0x7faa30005890 comp rx=0 tx=0).stop 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa40165740 1 -- 172.21.15.94:0/1923428786 shutdown_connections 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa40165740 1 --2- 172.21.15.94:0/1923428786 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce3300 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa40165740 1 --2- 172.21.15.94:0/1923428786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa40165740 1 --2- 172.21.15.94:0/1923428786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cef830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa40165740 1 -- 172.21.15.94:0/1923428786 >> 172.21.15.94:0/1923428786 conn(0x55d6c7cd4ba0 msgr2=0x55d6c7cdb260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa40165740 1 -- 172.21.15.94:0/1923428786 shutdown_connections 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.094+0000 7faa40165740 1 -- 172.21.15.94:0/1923428786 wait complete. 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa40165740 1 -- start start 2024-04-21T08:49:46.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa40165740 1 Processor -- start 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa40165740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cec110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa40165740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa40165740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa40165740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7faa381adc50 con 0x55d6c7cdae80 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa40165740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa381af0c0 con 0x55d6c7ce3a00 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa40165740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faa381ae320 con 0x55d6c7cda4d0 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa40165740 10 client.0 _ll_register_callbacks cb 0x7faa3009c3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa367fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa367fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59734/0 (socket says 172.21.15.94:59734) 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa367fc640 1 -- 172.21.15.94:0/114935423 learned_addr learned my addr 172.21.15.94:0/114935423 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa36ffd640 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cec110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa367fc640 1 -- 172.21.15.94:0/114935423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 msgr2=0x55d6c7cec110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa367fc640 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cec110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa367fc640 1 -- 172.21.15.94:0/114935423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7ce3a00 msgr2=0x55d6c7ce7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa367fc640 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa367fc640 1 -- 172.21.15.94:0/114935423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6c7ce81a0 con 0x55d6c7cdae80 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa36ffd640 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cec110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa367fc640 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce7430 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7faa2c02c1c0 tx=0x7faa2c02bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719d7a6c6d2d290 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.095+0000 7faa1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa1aff5640 1 -- 172.21.15.94:0/114935423 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c005510 con 0x55d6c7cdae80 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa1aff5640 1 -- 172.21.15.94:0/114935423 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7faa2c0046c0 con 0x55d6c7cdae80 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa40165740 1 -- 172.21.15.94:0/114935423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa30016210 con 0x55d6c7cdae80 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa1aff5640 1 -- 172.21.15.94:0/114935423 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faa2c05d050 con 0x55d6c7cdae80 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa40165740 10 client.18105 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa40165740 1 -- 172.21.15.94:0/114935423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d6c7d5a0f0 con 0x55d6c7cdae80 2024-04-21T08:49:46.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa1aff5640 1 -- 172.21.15.94:0/114935423 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa30016210 con 0x55d6c7cdae80 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa1aff5640 1 -- 172.21.15.94:0/114935423 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7faa2c003ae0 con 0x55d6c7cdae80 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa40165740 10 client.18105 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa40165740 10 client.18105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa40165740 20 client.18105 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa40165740 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d6c7d5a8f0 0x55d6c7d5cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa40165740 4 client.18105 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.096+0000 7faa40165740 1 -- 172.21.15.94:0/114935423 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d6c7c4cc10 con 0x55d6c7d5a8f0 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa36ffd640 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d6c7d5a8f0 0x55d6c7d5cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa1aff5640 10 client.18105 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa36ffd640 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d6c7d5a8f0 0x55d6c7d5cd30 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=300c223c1908d7e server_cookie=b62a3ca1453ef33b in_seq=0 out_seq=0 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa1aff5640 1 -- 172.21.15.94:0/114935423 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d6c7c4cc10 con 0x55d6c7d5a8f0 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa1aff5640 10 client.18105 handle_command_reply: tid=0 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 client.18105 shutdown 2024-04-21T08:49:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 -- 172.21.15.94:0/114935423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cdae80 msgr2=0x55d6c7ce7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce7430 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7faa2c02c1c0 tx=0x7faa2c02bc80 comp rx=0 tx=0).stop 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 -- 172.21.15.94:0/114935423 shutdown_connections 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55d6c7d5a8f0 0x55d6c7d5cd30 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c7ce3a00 0x55d6c7ce7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55d6c7cdae80 0x55d6c7ce7430 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa1aff5640 0 client.18105 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 --2- 172.21.15.94:0/114935423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d6c7cda4d0 0x55d6c7cec110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 -- 172.21.15.94:0/114935423 >> 172.21.15.94:0/114935423 conn(0x55d6c7cd4ba0 msgr2=0x55d6c7cdbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 -- 172.21.15.94:0/114935423 shutdown_connections 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 1 -- 172.21.15.94:0/114935423 wait complete. 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.097+0000 7faa40165740 20 client.18105 trim_cache size 0 max 16384 2024-04-21T08:49:46.103 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:49:46.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 -- 172.21.15.94:0/753138631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7faa200a3d20 msgr2=0x7faa200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7faa200a3d20 0x7faa200a61e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faa38199750 tx=0x7faa2c02ca00 comp rx=0 tx=0).stop 2024-04-21T08:49:46.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 -- 172.21.15.94:0/753138631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa38105e20 msgr2=0x7faa38198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa38105e20 0x7faa38198610 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7faa3000eaa0 tx=0x7faa3000ef60 comp rx=0 tx=0).stop 2024-04-21T08:49:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 -- 172.21.15.94:0/753138631 shutdown_connections 2024-04-21T08:49:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7faa200a3d20 0x7faa200a61e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa38105e20 0x7faa38198610 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa38101f60 0x7faa3819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 --2- 172.21.15.94:0/753138631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa381015c0 0x7faa3819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 -- 172.21.15.94:0/753138631 >> 172.21.15.94:0/753138631 conn(0x7faa380773e0 msgr2=0x7faa38104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 -- 172.21.15.94:0/753138631 shutdown_connections 2024-04-21T08:49:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:46.098+0000 7faa3d575640 1 -- 172.21.15.94:0/753138631 wait complete. 2024-04-21T08:49:46.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:46 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3941502860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:46.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:46 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3941502860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:46.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:46 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3941502860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:47.108 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:49:47.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:49:47.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:49:47.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.230+0000 7f658f05c640 1 -- 172.21.15.94:0/3629109130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6588102b40 msgr2=0x7f6588106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.230+0000 7f658f05c640 1 --2- 172.21.15.94:0/3629109130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6588102b40 0x7f6588106b80 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f657c009900 tx=0x7f657c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.230+0000 7f658f05c640 1 -- 172.21.15.94:0/3629109130 shutdown_connections 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.230+0000 7f658f05c640 1 --2- 172.21.15.94:0/3629109130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6588107260 0x7f658810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.230+0000 7f658f05c640 1 --2- 172.21.15.94:0/3629109130 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6588102b40 0x7f6588106b80 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.230+0000 7f658f05c640 1 --2- 172.21.15.94:0/3629109130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6588102190 0x7f6588102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.230+0000 7f658f05c640 1 -- 172.21.15.94:0/3629109130 >> 172.21.15.94:0/3629109130 conn(0x7f65880793a0 msgr2=0x7f65880797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.230+0000 7f658f05c640 1 -- 172.21.15.94:0/3629109130 shutdown_connections 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.230+0000 7f658f05c640 1 -- 172.21.15.94:0/3629109130 wait complete. 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658f05c640 1 Processor -- start 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658f05c640 1 -- start start 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658f05c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6588102190 0x7f6588074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658f05c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6588107260 0x7f6588074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658f05c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6588074cd0 0x7f65881a6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658f05c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f65881121a0 con 0x7f6588074cd0 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658f05c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6588112020 con 0x7f6588102190 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658f05c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6588112320 con 0x7f6588107260 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658d5f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6588074cd0 0x7f65881a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658d5f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6588074cd0 0x7f65881a6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59744/0 (socket says 172.21.15.94:59744) 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658d5f9640 1 -- 172.21.15.94:0/4151311590 learned_addr learned my addr 172.21.15.94:0/4151311590 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f6587fff640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6588107260 0x7f6588074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658cdf8640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6588102190 0x7f6588074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658d5f9640 1 -- 172.21.15.94:0/4151311590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6588107260 msgr2=0x7f6588074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658d5f9640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6588107260 0x7f6588074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658d5f9640 1 -- 172.21.15.94:0/4151311590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6588102190 msgr2=0x7f6588074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.231+0000 7f658d5f9640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6588102190 0x7f6588074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.232+0000 7f658d5f9640 1 -- 172.21.15.94:0/4151311590 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65881a7050 con 0x7f6588074cd0 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.232+0000 7f658d5f9640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6588074cd0 0x7f65881a6950 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f658000eaa0 tx=0x7f658000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.232+0000 7f6585ffb640 1 -- 172.21.15.94:0/4151311590 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658000cdb0 con 0x7f6588074cd0 2024-04-21T08:49:47.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.232+0000 7f658f05c640 1 -- 172.21.15.94:0/4151311590 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65881a7340 con 0x7f6588074cd0 2024-04-21T08:49:47.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.232+0000 7f6585ffb640 1 -- 172.21.15.94:0/4151311590 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6580005020 con 0x7f6588074cd0 2024-04-21T08:49:47.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.232+0000 7f6585ffb640 1 -- 172.21.15.94:0/4151311590 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65800116d0 con 0x7f6588074cd0 2024-04-21T08:49:47.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.232+0000 7f658f05c640 1 -- 172.21.15.94:0/4151311590 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65881a7860 con 0x7f6588074cd0 2024-04-21T08:49:47.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.234+0000 7f6585ffb640 1 -- 172.21.15.94:0/4151311590 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f65800051d0 con 0x7f6588074cd0 2024-04-21T08:49:47.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6591c50740 1 Processor -- start 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6591c50740 1 -- start start 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6585ffb640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d50 0x7f65700a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6591c50740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa47cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6591c50740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa50840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6591c50740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa53260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6591c50740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55dd1f9abf00 con 0x55dd1fa50e10 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6591c50740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd1f949590 con 0x55dd1fa478e0 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6591c50740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dd1f9a7ec0 con 0x55dd1fa48290 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6587fff640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d50 0x7f65700a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f6585ffb640 1 -- 172.21.15.94:0/4151311590 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f65800d0cc0 con 0x7f6588074cd0 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.235+0000 7f658d5f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa53260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f658d5f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa53260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59760/0 (socket says 172.21.15.94:59760) 2024-04-21T08:49:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f658d5f9640 1 -- 172.21.15.94:0/4037975379 learned_addr learned my addr 172.21.15.94:0/4037975379 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f658cdf8640 1 --2- 172.21.15.94:0/4037975379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa47cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f6587fff640 1 --2- 172.21.15.94:0/4037975379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa50840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f658d5f9640 1 -- 172.21.15.94:0/4037975379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 msgr2=0x55dd1fa50840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f658d5f9640 1 --2- 172.21.15.94:0/4037975379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa50840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f658d5f9640 1 -- 172.21.15.94:0/4037975379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 msgr2=0x55dd1fa47cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f658d5f9640 1 --2- 172.21.15.94:0/4037975379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa47cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f658d5f9640 1 -- 172.21.15.94:0/4037975379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd1f7867e0 con 0x55dd1fa50e10 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f6587fff640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d50 0x7f65700a6210 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f657c009230 tx=0x7f657c009560 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f658d5f9640 1 --2- 172.21.15.94:0/4037975379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa53260 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f65800992b0 tx=0x7f658009c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ca0d9b595fe426 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f656effd640 1 -- 172.21.15.94:0/4037975379 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658001f040 con 0x55dd1fa50e10 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f656effd640 1 -- 172.21.15.94:0/4037975379 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6580098f50 con 0x55dd1fa50e10 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.236+0000 7f656effd640 1 -- 172.21.15.94:0/4037975379 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65800a5270 con 0x55dd1fa50e10 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 -- 172.21.15.94:0/4037975379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 msgr2=0x55dd1fa53260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 --2- 172.21.15.94:0/4037975379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa53260 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f65800992b0 tx=0x7f658009c530 comp rx=0 tx=0).stop 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 -- 172.21.15.94:0/4037975379 shutdown_connections 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 --2- 172.21.15.94:0/4037975379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa50840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 --2- 172.21.15.94:0/4037975379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa53260 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 --2- 172.21.15.94:0/4037975379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa47cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 -- 172.21.15.94:0/4037975379 >> 172.21.15.94:0/4037975379 conn(0x55dd1fa41fb0 msgr2=0x55dd1fa42390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 -- 172.21.15.94:0/4037975379 shutdown_connections 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 -- 172.21.15.94:0/4037975379 wait complete. 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 -- start start 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.237+0000 7f6591c50740 1 Processor -- start 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f6591c50740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa4de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f6591c50740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa4e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f6591c50740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa4e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f6591c50740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55dd1fa53d20 con 0x55dd1fa50e10 2024-04-21T08:49:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f6591c50740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd1f965580 con 0x55dd1fa478e0 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f6591c50740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dd1fa53ea0 con 0x55dd1fa48290 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f6591c50740 10 client.0 _ll_register_callbacks cb 0x55dd1f96b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f658d5f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa4e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f658d5f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa4e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59776/0 (socket says 172.21.15.94:59776) 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f658d5f9640 1 -- 172.21.15.94:0/2523768428 learned_addr learned my addr 172.21.15.94:0/2523768428 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f6587fff640 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa4e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f658cdf8640 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa4de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f658d5f9640 1 -- 172.21.15.94:0/2523768428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 msgr2=0x55dd1fa4e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f658d5f9640 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa4e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f658d5f9640 1 -- 172.21.15.94:0/2523768428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 msgr2=0x55dd1fa4de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f658d5f9640 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa4de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f658d5f9640 1 -- 172.21.15.94:0/2523768428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd1f7e6f70 con 0x55dd1fa50e10 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f6587fff640 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa4e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f658d5f9640 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa4e880 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f658000efd0 tx=0x7f65800a0d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ccc3f0cb78616f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.238+0000 7f6567fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6567fff640 1 -- 172.21.15.94:0/2523768428 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65800b9030 con 0x55dd1fa50e10 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6567fff640 1 -- 172.21.15.94:0/2523768428 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f65800a5220 con 0x55dd1fa50e10 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6591c50740 1 -- 172.21.15.94:0/2523768428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6580098f50 con 0x55dd1fa50e10 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6567fff640 1 -- 172.21.15.94:0/2523768428 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f658001f070 con 0x55dd1fa50e10 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6591c50740 10 client.18117 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6591c50740 1 -- 172.21.15.94:0/2523768428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dd1f67e170 con 0x55dd1fa50e10 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6567fff640 1 -- 172.21.15.94:0/2523768428 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6580098f50 con 0x55dd1fa50e10 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6567fff640 1 -- 172.21.15.94:0/2523768428 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f658000f040 con 0x55dd1fa50e10 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6591c50740 10 client.18117 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6591c50740 10 client.18117 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6591c50740 20 client.18117 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:47.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6591c50740 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd1fa61190 0x55dd1fa63580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6591c50740 4 client.18117 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f6591c50740 1 -- 172.21.15.94:0/2523768428 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd1f9b9920 con 0x55dd1fa61190 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.239+0000 7f658cdf8640 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd1fa61190 0x55dd1fa63580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.240+0000 7f6567fff640 10 client.18117 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.240+0000 7f658cdf8640 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd1fa61190 0x55dd1fa63580 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19f0d631bd39dcb7 server_cookie=fe802e1f28ab4acc in_seq=0 out_seq=0 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6567fff640 1 -- 172.21.15.94:0/2523768428 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd1f9b9920 con 0x55dd1fa61190 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6567fff640 10 client.18117 handle_command_reply: tid=0 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 client.18117 shutdown 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 -- 172.21.15.94:0/2523768428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 msgr2=0x55dd1fa4e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa4e880 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f658000efd0 tx=0x7f65800a0d60 comp rx=0 tx=0).stop 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 -- 172.21.15.94:0/2523768428 shutdown_connections 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd1fa61190 0x55dd1fa63580 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa4e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa4e880 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6567fff640 0 client.18117 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:47.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 --2- 172.21.15.94:0/2523768428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa4de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 -- 172.21.15.94:0/2523768428 >> 172.21.15.94:0/2523768428 conn(0x55dd1fa41fb0 msgr2=0x55dd1fa43d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 -- 172.21.15.94:0/2523768428 shutdown_connections 2024-04-21T08:49:47.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 1 -- 172.21.15.94:0/2523768428 wait complete. 2024-04-21T08:49:47.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.241+0000 7f6591c50740 20 client.18117 trim_cache size 0 max 16384 2024-04-21T08:49:47.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f6591c50740 1 Processor -- start 2024-04-21T08:49:47.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f6591c50740 1 -- start start 2024-04-21T08:49:47.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f6591c50740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa50530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f6591c50740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa40e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f6591c50740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f6591c50740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f65881b4120 con 0x55dd1fa50e10 2024-04-21T08:49:47.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f6591c50740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f65881b19f0 con 0x55dd1fa478e0 2024-04-21T08:49:47.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f6591c50740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65881af860 con 0x55dd1fa48290 2024-04-21T08:49:47.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658d5f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658d5f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa665a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59782/0 (socket says 172.21.15.94:59782) 2024-04-21T08:49:47.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658d5f9640 1 -- 172.21.15.94:0/1922760790 learned_addr learned my addr 172.21.15.94:0/1922760790 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:47.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658cdf8640 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa50530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658d5f9640 1 -- 172.21.15.94:0/1922760790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 msgr2=0x55dd1fa40e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f6587fff640 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa40e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658d5f9640 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa40e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658d5f9640 1 -- 172.21.15.94:0/1922760790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 msgr2=0x55dd1fa50530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658d5f9640 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa50530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658d5f9640 1 -- 172.21.15.94:0/1922760790 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd1fa693c0 con 0x55dd1fa50e10 2024-04-21T08:49:47.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f6587fff640 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa40e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:47.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658cdf8640 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa50530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:47.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.259+0000 7f658d5f9640 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa665a0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f658009c5e0 tx=0x7f6580005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15926eb0aaf8e9f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f656e7fc640 1 -- 172.21.15.94:0/1922760790 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65800a54a0 con 0x55dd1fa50e10 2024-04-21T08:49:47.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f656e7fc640 1 -- 172.21.15.94:0/1922760790 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6580098630 con 0x55dd1fa50e10 2024-04-21T08:49:47.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f656e7fc640 1 -- 172.21.15.94:0/1922760790 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65800a1f90 con 0x55dd1fa50e10 2024-04-21T08:49:47.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f6591c50740 1 -- 172.21.15.94:0/1922760790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 msgr2=0x55dd1fa665a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f6591c50740 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa665a0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f658009c5e0 tx=0x7f6580005890 comp rx=0 tx=0).stop 2024-04-21T08:49:47.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f6591c50740 1 -- 172.21.15.94:0/1922760790 shutdown_connections 2024-04-21T08:49:47.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f6591c50740 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa48290 0x55dd1fa40e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f6591c50740 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa50e10 0x55dd1fa665a0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f6591c50740 1 --2- 172.21.15.94:0/1922760790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa478e0 0x55dd1fa50530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f6591c50740 1 -- 172.21.15.94:0/1922760790 >> 172.21.15.94:0/1922760790 conn(0x55dd1fa41fb0 msgr2=0x55dd1fa40780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f6591c50740 1 -- 172.21.15.94:0/1922760790 shutdown_connections 2024-04-21T08:49:47.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.260+0000 7f6591c50740 1 -- 172.21.15.94:0/1922760790 wait complete. 2024-04-21T08:49:47.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6591c50740 1 -- start start 2024-04-21T08:49:47.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6591c50740 1 Processor -- start 2024-04-21T08:49:47.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6591c50740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa478e0 0x55dd1fa64230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6591c50740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa48290 0x55dd1fa64770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6591c50740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa50e10 0x55dd1fa64cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6591c50740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f65881afeb0 con 0x55dd1fa478e0 2024-04-21T08:49:47.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6591c50740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f65881b1320 con 0x55dd1fa48290 2024-04-21T08:49:47.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6591c50740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65881b0580 con 0x55dd1fa50e10 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6591c50740 10 client.0 _ll_register_callbacks cb 0x7f657800b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f658cdf8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa478e0 0x55dd1fa64230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f658cdf8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa478e0 0x55dd1fa64230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59798/0 (socket says 172.21.15.94:59798) 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f658cdf8640 1 -- 172.21.15.94:0/2024410005 learned_addr learned my addr 172.21.15.94:0/2024410005 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6587fff640 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa48290 0x55dd1fa64770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f658cdf8640 1 -- 172.21.15.94:0/2024410005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa50e10 msgr2=0x55dd1fa64cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f658d5f9640 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa50e10 0x55dd1fa64cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f658cdf8640 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa50e10 0x55dd1fa64cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f658cdf8640 1 -- 172.21.15.94:0/2024410005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa48290 msgr2=0x55dd1fa64770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f658cdf8640 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa48290 0x55dd1fa64770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f658cdf8640 1 -- 172.21.15.94:0/2024410005 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd1fac6570 con 0x55dd1fa478e0 2024-04-21T08:49:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.261+0000 7f6587fff640 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa48290 0x55dd1fa64770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:47.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f658cdf8640 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa478e0 0x55dd1fa64230 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x55dd1fa53eb0 tx=0x7f657800bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae25bc01a647190 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6567fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:47.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6567fff640 1 -- 172.21.15.94:0/2024410005 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6578005220 con 0x55dd1fa478e0 2024-04-21T08:49:47.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6567fff640 1 -- 172.21.15.94:0/2024410005 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f65780053c0 con 0x55dd1fa478e0 2024-04-21T08:49:47.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6591c50740 1 -- 172.21.15.94:0/2024410005 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6580098630 con 0x55dd1fa478e0 2024-04-21T08:49:47.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6567fff640 1 -- 172.21.15.94:0/2024410005 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f657802f050 con 0x55dd1fa478e0 2024-04-21T08:49:47.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6591c50740 10 client.18123 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6591c50740 1 -- 172.21.15.94:0/2024410005 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dd1fac7860 con 0x55dd1fa478e0 2024-04-21T08:49:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6567fff640 1 -- 172.21.15.94:0/2024410005 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6580098630 con 0x55dd1fa478e0 2024-04-21T08:49:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6567fff640 1 -- 172.21.15.94:0/2024410005 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f657801f1e0 con 0x55dd1fa478e0 2024-04-21T08:49:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6591c50740 10 client.18123 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6591c50740 10 client.18123 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6591c50740 20 client.18123 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6591c50740 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd1fac7f90 0x55dd1faca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6591c50740 4 client.18123 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:49:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.262+0000 7f6591c50740 1 -- 172.21.15.94:0/2024410005 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dd1f9b3790 con 0x55dd1fac7f90 2024-04-21T08:49:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.263+0000 7f6587fff640 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd1fac7f90 0x55dd1faca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.263+0000 7f6587fff640 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd1fac7f90 0x55dd1faca3b0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb9eb6b67364087d server_cookie=b6684ee29a877d44 in_seq=0 out_seq=0 2024-04-21T08:49:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.263+0000 7f6567fff640 10 client.18123 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.264+0000 7f6567fff640 1 -- 172.21.15.94:0/2024410005 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dd1f9b3790 con 0x55dd1fac7f90 2024-04-21T08:49:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.264+0000 7f6567fff640 10 client.18123 handle_command_reply: tid=0 2024-04-21T08:49:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.264+0000 7f6591c50740 1 client.18123 shutdown 2024-04-21T08:49:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 1 -- 172.21.15.94:0/2024410005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa478e0 msgr2=0x55dd1fa64230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa478e0 0x55dd1fa64230 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x55dd1fa53eb0 tx=0x7f657800bec0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 1 -- 172.21.15.94:0/2024410005 shutdown_connections 2024-04-21T08:49:47.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x55dd1fac7f90 0x55dd1faca3b0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dd1fa50e10 0x55dd1fa64cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd1fa48290 0x55dd1fa64770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 1 --2- 172.21.15.94:0/2024410005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55dd1fa478e0 0x55dd1fa64230 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 1 -- 172.21.15.94:0/2024410005 >> 172.21.15.94:0/2024410005 conn(0x55dd1fa41fb0 msgr2=0x55dd1fa41010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6567fff640 0 client.18123 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:47.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 1 -- 172.21.15.94:0/2024410005 shutdown_connections 2024-04-21T08:49:47.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 1 -- 172.21.15.94:0/2024410005 wait complete. 2024-04-21T08:49:47.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f6591c50740 20 client.18123 trim_cache size 0 max 16384 2024-04-21T08:49:47.276 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:49:47.276 INFO:teuthology.orchestra.run.smithi094.stdout: "return_code": 0, 2024-04-21T08:49:47.276 INFO:teuthology.orchestra.run.smithi094.stdout: "scrub_tag": "6532b6c7-1e24-4fe5-8740-183271cd00c8", 2024-04-21T08:49:47.276 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": "asynchronous" 2024-04-21T08:49:47.276 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:49:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.265+0000 7f658f05c640 1 -- 172.21.15.94:0/4151311590 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d50 msgr2=0x7f65700a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d50 0x7f65700a6210 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f657c009230 tx=0x7f657c009560 comp rx=0 tx=0).stop 2024-04-21T08:49:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 -- 172.21.15.94:0/4151311590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6588074cd0 msgr2=0x7f65881a6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6588074cd0 0x7f65881a6950 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f658000eaa0 tx=0x7f658000ef60 comp rx=0 tx=0).stop 2024-04-21T08:49:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 -- 172.21.15.94:0/4151311590 shutdown_connections 2024-04-21T08:49:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d50 0x7f65700a6210 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6588107260 0x7f6588074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6588074cd0 0x7f65881a6950 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 --2- 172.21.15.94:0/4151311590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6588102190 0x7f6588074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 -- 172.21.15.94:0/4151311590 >> 172.21.15.94:0/4151311590 conn(0x7f65880793a0 msgr2=0x7f6588100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 -- 172.21.15.94:0/4151311590 shutdown_connections 2024-04-21T08:49:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.266+0000 7f658f05c640 1 -- 172.21.15.94:0/4151311590 wait complete. 2024-04-21T08:49:47.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:49:47.280 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:49:47.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.382+0000 7f3b3264e640 1 -- 172.21.15.94:0/3137749822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b2c10c580 msgr2=0x7f3b2c10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.382+0000 7f3b3264e640 1 --2- 172.21.15.94:0/3137749822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b2c10c580 0x7f3b2c10e9b0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009ca0 tx=0x7f3b2402c180 comp rx=0 tx=0).stop 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.382+0000 7f3b3264e640 1 -- 172.21.15.94:0/3137749822 shutdown_connections 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.382+0000 7f3b3264e640 1 --2- 172.21.15.94:0/3137749822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b2c10c580 0x7f3b2c10e9b0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.382+0000 7f3b3264e640 1 --2- 172.21.15.94:0/3137749822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b2c103ad0 0x7f3b2c10bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.382+0000 7f3b3264e640 1 --2- 172.21.15.94:0/3137749822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b2c1031b0 0x7f3b2c103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.382+0000 7f3b3264e640 1 -- 172.21.15.94:0/3137749822 >> 172.21.15.94:0/3137749822 conn(0x7f3b2c0771d0 msgr2=0x7f3b2c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.382+0000 7f3b3264e640 1 -- 172.21.15.94:0/3137749822 shutdown_connections 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.382+0000 7f3b3264e640 1 -- 172.21.15.94:0/3137749822 wait complete. 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b3264e640 1 Processor -- start 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b3264e640 1 -- start start 2024-04-21T08:49:47.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b3264e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b2c1031b0 0x7f3b2c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b3264e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b2c103ad0 0x7f3b2c198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b3264e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b2c10c580 0x7f3b2c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b3264e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1121a0 con 0x7f3b2c103ad0 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b3264e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c112020 con 0x7f3b2c1031b0 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b3264e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c112320 con 0x7f3b2c10c580 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b2b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b2c103ad0 0x7f3b2c198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b2b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b2c103ad0 0x7f3b2c198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59822/0 (socket says 172.21.15.94:59822) 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b2b7fe640 1 -- 172.21.15.94:0/4145676172 learned_addr learned my addr 172.21.15.94:0/4145676172 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b30beb640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b2c10c580 0x7f3b2c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b2bfff640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b2c1031b0 0x7f3b2c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b2b7fe640 1 -- 172.21.15.94:0/4145676172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b2c10c580 msgr2=0x7f3b2c19cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b2c10c580 0x7f3b2c19cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b2b7fe640 1 -- 172.21.15.94:0/4145676172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b2c1031b0 msgr2=0x7f3b2c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b2c1031b0 0x7f3b2c198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b2b7fe640 1 -- 172.21.15.94:0/4145676172 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2c19d650 con 0x7f3b2c103ad0 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.383+0000 7f3b30beb640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b2c10c580 0x7f3b2c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.384+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b2c103ad0 0x7f3b2c198ba0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000c920 tx=0x7f3b2000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.384+0000 7f3b297fa640 1 -- 172.21.15.94:0/4145676172 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20007c70 con 0x7f3b2c103ad0 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.384+0000 7f3b3264e640 1 -- 172.21.15.94:0/4145676172 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2c19d940 con 0x7f3b2c103ad0 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.384+0000 7f3b297fa640 1 -- 172.21.15.94:0/4145676172 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b20007e10 con 0x7f3b2c103ad0 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.384+0000 7f3b297fa640 1 -- 172.21.15.94:0/4145676172 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20018820 con 0x7f3b2c103ad0 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.384+0000 7f3b3264e640 1 -- 172.21.15.94:0/4145676172 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b2c1a51d0 con 0x7f3b2c103ad0 2024-04-21T08:49:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.386+0000 7f3b297fa640 1 -- 172.21.15.94:0/4145676172 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f3b2000f450 con 0x7f3b2c103ad0 2024-04-21T08:49:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b35241740 1 Processor -- start 2024-04-21T08:49:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b35241740 1 -- start start 2024-04-21T08:49:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b35241740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723c2b0 0x563bd723c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b35241740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7249950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b35241740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd7249e90 0x563bd724c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b35241740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563bd71a11e0 con 0x563bd723cbd0 2024-04-21T08:49:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b35241740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563bd713e390 con 0x563bd7249e90 2024-04-21T08:49:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b35241740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563bd719d1a0 con 0x563bd723c2b0 2024-04-21T08:49:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b2bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7249950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b2bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7249950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59830/0 (socket says 172.21.15.94:59830) 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b2bfff640 1 -- 172.21.15.94:0/1101519411 learned_addr learned my addr 172.21.15.94:0/1101519411 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b297fa640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b140a3d20 0x7f3b140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b30beb640 1 --2- 172.21.15.94:0/1101519411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723c2b0 0x563bd723c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/1101519411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd7249e90 0x563bd724c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b30beb640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b140a3d20 0x7f3b140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b2bfff640 1 -- 172.21.15.94:0/1101519411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723c2b0 msgr2=0x563bd723c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b2bfff640 1 --2- 172.21.15.94:0/1101519411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723c2b0 0x563bd723c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b2bfff640 1 -- 172.21.15.94:0/1101519411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd7249e90 msgr2=0x563bd724c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b297fa640 1 -- 172.21.15.94:0/4145676172 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f3b200d0d60 con 0x7f3b2c103ad0 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b2bfff640 1 --2- 172.21.15.94:0/1101519411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd7249e90 0x563bd724c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b2bfff640 1 -- 172.21.15.94:0/1101519411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bd6f7d7d0 con 0x563bd723cbd0 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.387+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/1101519411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd7249e90 0x563bd724c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b2bfff640 1 --2- 172.21.15.94:0/1101519411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7249950 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009bc0 tx=0x7f3b1c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd26f9ca3f51dcb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b30beb640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b140a3d20 0x7f3b140a61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3b24004af0 tx=0x7f3b2402c180 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b127fc640 1 -- 172.21.15.94:0/1101519411 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c016070 con 0x563bd723cbd0 2024-04-21T08:49:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b127fc640 1 -- 172.21.15.94:0/1101519411 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1c011920 con 0x563bd723cbd0 2024-04-21T08:49:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b127fc640 1 -- 172.21.15.94:0/1101519411 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0107b0 con 0x563bd723cbd0 2024-04-21T08:49:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b35241740 1 -- 172.21.15.94:0/1101519411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 msgr2=0x563bd7249950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b35241740 1 --2- 172.21.15.94:0/1101519411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7249950 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009bc0 tx=0x7f3b1c00e6c0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b35241740 1 -- 172.21.15.94:0/1101519411 shutdown_connections 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b35241740 1 --2- 172.21.15.94:0/1101519411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd7249e90 0x563bd724c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b35241740 1 --2- 172.21.15.94:0/1101519411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7249950 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b35241740 1 --2- 172.21.15.94:0/1101519411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723c2b0 0x563bd723c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b35241740 1 -- 172.21.15.94:0/1101519411 >> 172.21.15.94:0/1101519411 conn(0x563bd7236cb0 msgr2=0x563bd7237090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b35241740 1 -- 172.21.15.94:0/1101519411 shutdown_connections 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.388+0000 7f3b35241740 1 -- 172.21.15.94:0/1101519411 wait complete. 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b35241740 1 -- start start 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b35241740 1 Processor -- start 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b35241740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723c2b0 0x563bd7246ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b35241740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723cbd0 0x563bd7247400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b35241740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7247940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b35241740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563bd724cc50 con 0x563bd7249e90 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b35241740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563bd715a2f0 con 0x563bd723cbd0 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b35241740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563bd724cdd0 con 0x563bd723c2b0 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b35241740 10 client.0 _ll_register_callbacks cb 0x563bd7167580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b30beb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7247940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b30beb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7247940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59832/0 (socket says 172.21.15.94:59832) 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b30beb640 1 -- 172.21.15.94:0/1924697667 learned_addr learned my addr 172.21.15.94:0/1924697667 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723cbd0 0x563bd7247400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.389+0000 7f3b2bfff640 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723c2b0 0x563bd7246ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b30beb640 1 -- 172.21.15.94:0/1924697667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723c2b0 msgr2=0x563bd7246ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b30beb640 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723c2b0 0x563bd7246ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b30beb640 1 -- 172.21.15.94:0/1924697667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723cbd0 msgr2=0x563bd7247400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b30beb640 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723cbd0 0x563bd7247400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b30beb640 1 -- 172.21.15.94:0/1924697667 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bd6eb33f0 con 0x563bd7249e90 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b30beb640 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7247940 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f3b24036000 tx=0x7f3b240095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a77983139a76d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/1924697667 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24039030 con 0x563bd7249e90 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b35241740 1 -- 172.21.15.94:0/1924697667 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b1c011920 con 0x563bd7249e90 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/1924697667 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f3b24004800 con 0x563bd7249e90 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/1924697667 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3b2405b050 con 0x563bd7249e90 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b35241740 10 client.18135 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b35241740 1 -- 172.21.15.94:0/1924697667 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563bd6e730f0 con 0x563bd7249e90 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/1924697667 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1c011920 con 0x563bd7249e90 2024-04-21T08:49:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/1924697667 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3b24034ba0 con 0x563bd7249e90 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b35241740 10 client.18135 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b35241740 10 client.18135 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b35241740 20 client.18135 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.390+0000 7f3b35241740 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563bd7259f40 0x563bd725c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.391+0000 7f3b35241740 4 client.18135 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.391+0000 7f3b35241740 1 -- 172.21.15.94:0/1924697667 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bd71aec00 con 0x563bd7259f40 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.391+0000 7f3b2bfff640 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563bd7259f40 0x563bd725c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.391+0000 7f3b2bfff640 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563bd7259f40 0x563bd725c3a0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=411de19f1d18de5a server_cookie=540739fffcc620ca in_seq=0 out_seq=0 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.391+0000 7f3b0b7fe640 10 client.18135 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/1924697667 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563bd71aec00 con 0x563bd7259f40 2024-04-21T08:49:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b0b7fe640 10 client.18135 handle_command_reply: tid=0 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 client.18135 shutdown 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 -- 172.21.15.94:0/1924697667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 msgr2=0x563bd7247940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7247940 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f3b24036000 tx=0x7f3b240095c0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 -- 172.21.15.94:0/1924697667 shutdown_connections 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563bd7259f40 0x563bd725c3a0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7247940 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723cbd0 0x563bd7247400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b0b7fe640 0 client.18135 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 --2- 172.21.15.94:0/1924697667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723c2b0 0x563bd7246ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 -- 172.21.15.94:0/1924697667 >> 172.21.15.94:0/1924697667 conn(0x563bd7236cb0 msgr2=0x563bd7238a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 -- 172.21.15.94:0/1924697667 shutdown_connections 2024-04-21T08:49:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 1 -- 172.21.15.94:0/1924697667 wait complete. 2024-04-21T08:49:47.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.392+0000 7f3b35241740 20 client.18135 trim_cache size 0 max 16384 2024-04-21T08:49:47.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b35241740 1 Processor -- start 2024-04-21T08:49:47.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b35241740 1 -- start start 2024-04-21T08:49:47.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b35241740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723c2b0 0x563bd7255b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b35241740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723cbd0 0x563bd72467c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b35241740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7235bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b35241740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1afb50 con 0x563bd7249e90 2024-04-21T08:49:47.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b35241740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1ad420 con 0x563bd723c2b0 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b35241740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1ab290 con 0x563bd723cbd0 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b30beb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7235bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b30beb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7235bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59848/0 (socket says 172.21.15.94:59848) 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.409+0000 7f3b30beb640 1 -- 172.21.15.94:0/3091161413 learned_addr learned my addr 172.21.15.94:0/3091161413 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/3091161413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723cbd0 0x563bd72467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b30beb640 1 -- 172.21.15.94:0/3091161413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723cbd0 msgr2=0x563bd72467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b30beb640 1 --2- 172.21.15.94:0/3091161413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723cbd0 0x563bd72467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b30beb640 1 -- 172.21.15.94:0/3091161413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723c2b0 msgr2=0x563bd7255b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b30beb640 1 --2- 172.21.15.94:0/3091161413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723c2b0 0x563bd7255b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b30beb640 1 -- 172.21.15.94:0/3091161413 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bd724c280 con 0x563bd7249e90 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/3091161413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723cbd0 0x563bd72467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:47.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b30beb640 1 --2- 172.21.15.94:0/3091161413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7235bf0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009430 tx=0x7f3b2402c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dde0cdf22c72bee server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b11ffb640 1 -- 172.21.15.94:0/3091161413 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24063070 con 0x563bd7249e90 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b11ffb640 1 -- 172.21.15.94:0/3091161413 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b24052070 con 0x563bd7249e90 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.410+0000 7f3b11ffb640 1 -- 172.21.15.94:0/3091161413 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24043d80 con 0x563bd7249e90 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 -- 172.21.15.94:0/3091161413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 msgr2=0x563bd7235bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 --2- 172.21.15.94:0/3091161413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7235bf0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009430 tx=0x7f3b2402c800 comp rx=0 tx=0).stop 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 -- 172.21.15.94:0/3091161413 shutdown_connections 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 --2- 172.21.15.94:0/3091161413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd723cbd0 0x563bd72467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 --2- 172.21.15.94:0/3091161413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd7249e90 0x563bd7235bf0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 --2- 172.21.15.94:0/3091161413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723c2b0 0x563bd7255b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 -- 172.21.15.94:0/3091161413 >> 172.21.15.94:0/3091161413 conn(0x563bd7236cb0 msgr2=0x563bd72354e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 -- 172.21.15.94:0/3091161413 shutdown_connections 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 -- 172.21.15.94:0/3091161413 wait complete. 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 -- start start 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 Processor -- start 2024-04-21T08:49:47.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723c2b0 0x563bd7252280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7243750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd7249e90 0x563bd7243c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1ab8e0 con 0x563bd723cbd0 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1acd50 con 0x563bd723c2b0 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1abfb0 con 0x563bd7249e90 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b35241740 10 client.0 _ll_register_callbacks cb 0x7f3b24009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b2b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7243750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b2b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7243750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59854/0 (socket says 172.21.15.94:59854) 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.411+0000 7f3b2b7fe640 1 -- 172.21.15.94:0/3015706746 learned_addr learned my addr 172.21.15.94:0/3015706746 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b30beb640 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd7249e90 0x563bd7243c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b2b7fe640 1 -- 172.21.15.94:0/3015706746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd7249e90 msgr2=0x563bd7243c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd7249e90 0x563bd7243c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b2b7fe640 1 -- 172.21.15.94:0/3015706746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723c2b0 msgr2=0x563bd7252280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723c2b0 0x563bd7252280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b2b7fe640 1 -- 172.21.15.94:0/3015706746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bd7244250 con 0x563bd723cbd0 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b2b7fe640 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7243750 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f3b200992a0 tx=0x7f3b2001d840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc286de9d3f1f5f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/3015706746 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b200a0c80 con 0x563bd723cbd0 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/3015706746 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9599+0+0 (secure 0 0 0) 0x7f3b2000fa90 con 0x563bd723cbd0 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b35241740 1 -- 172.21.15.94:0/3015706746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b24052070 con 0x563bd723cbd0 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/3015706746 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3b200a5540 con 0x563bd723cbd0 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b35241740 10 client.18141 fetch_fsmap learned FSMap version 24 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b35241740 1 -- 172.21.15.94:0/3015706746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563bd72c0300 con 0x563bd723cbd0 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/3015706746 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b24052070 con 0x563bd723cbd0 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/3015706746 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3b2009cb30 con 0x563bd723cbd0 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b35241740 10 client.18141 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:49:47.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b35241740 10 client.18141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b35241740 20 client.18141 populate_metadata read hostname 'smithi094' 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b35241740 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563bd72c09e0 0x563bd72c2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b35241740 4 client.18141 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.412+0000 7f3b35241740 1 -- 172.21.15.94:0/3015706746 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563bd71a8a70 con 0x563bd72c09e0 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.413+0000 7f3b2bfff640 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563bd72c09e0 0x563bd72c2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.413+0000 7f3b0b7fe640 10 client.18141 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.413+0000 7f3b2bfff640 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563bd72c09e0 0x563bd72c2e00 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21374758637d4cce server_cookie=cc0582facba16dac in_seq=0 out_seq=0 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.413+0000 7f3b0b7fe640 1 -- 172.21.15.94:0/3015706746 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563bd71a8a70 con 0x563bd72c09e0 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.413+0000 7f3b0b7fe640 10 client.18141 handle_command_reply: tid=0 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.413+0000 7f3b35241740 1 client.18141 shutdown 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 1 -- 172.21.15.94:0/3015706746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 msgr2=0x563bd7243750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7243750 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f3b200992a0 tx=0x7f3b2001d840 comp rx=0 tx=0).stop 2024-04-21T08:49:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 1 -- 172.21.15.94:0/3015706746 shutdown_connections 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x563bd72c09e0 0x563bd72c2e00 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563bd7249e90 0x563bd7243c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563bd723cbd0 0x563bd7243750 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b0b7fe640 0 client.18141 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 1 --2- 172.21.15.94:0/3015706746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563bd723c2b0 0x563bd7252280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 1 -- 172.21.15.94:0/3015706746 >> 172.21.15.94:0/3015706746 conn(0x563bd7236cb0 msgr2=0x563bd7247710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 1 -- 172.21.15.94:0/3015706746 shutdown_connections 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 1 -- 172.21.15.94:0/3015706746 wait complete. 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b35241740 20 client.18141 trim_cache size 0 max 16384 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:49:47.420 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:49:47.421 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": { 2024-04-21T08:49:47.421 INFO:teuthology.orchestra.run.smithi094.stdout: "6532b6c7-1e24-4fe5-8740-183271cd00c8": { 2024-04-21T08:49:47.421 INFO:teuthology.orchestra.run.smithi094.stdout: "path": "/", 2024-04-21T08:49:47.421 INFO:teuthology.orchestra.run.smithi094.stdout: "tag": "6532b6c7-1e24-4fe5-8740-183271cd00c8", 2024-04-21T08:49:47.421 INFO:teuthology.orchestra.run.smithi094.stdout: "options": "recursive,force" 2024-04-21T08:49:47.421 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:49:47.421 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:49:47.421 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:49:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 -- 172.21.15.94:0/4145676172 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b140a3d20 msgr2=0x7f3b140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b140a3d20 0x7f3b140a61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3b24004af0 tx=0x7f3b2402c180 comp rx=0 tx=0).stop 2024-04-21T08:49:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 -- 172.21.15.94:0/4145676172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b2c103ad0 msgr2=0x7f3b2c198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b2c103ad0 0x7f3b2c198ba0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000c920 tx=0x7f3b2000cde0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 -- 172.21.15.94:0/4145676172 shutdown_connections 2024-04-21T08:49:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f3b140a3d20 0x7f3b140a61e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b2c10c580 0x7f3b2c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b2c103ad0 0x7f3b2c198ba0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 --2- 172.21.15.94:0/4145676172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b2c1031b0 0x7f3b2c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 -- 172.21.15.94:0/4145676172 >> 172.21.15.94:0/4145676172 conn(0x7f3b2c0771d0 msgr2=0x7f3b2c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 -- 172.21.15.94:0/4145676172 shutdown_connections 2024-04-21T08:49:47.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:49:47.414+0000 7f3b3264e640 1 -- 172.21.15.94:0/4145676172 wait complete. 2024-04-21T08:49:47.423 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:47 smithi094 ceph-mon[27988]: pgmap v303: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:49:47.424 INFO:tasks.cephfs.filesystem:scrub status for tag:6532b6c7-1e24-4fe5-8740-183271cd00c8 - {'path': '/', 'tag': '6532b6c7-1e24-4fe5-8740-183271cd00c8', 'options': 'recursive,force'} 2024-04-21T08:49:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:47 smithi111 ceph-mon[31493]: pgmap v303: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:49:47.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:47 smithi173 ceph-mon[31428]: pgmap v303: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:49:48.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:48 smithi111 ceph-mon[31493]: scrub queued for path: / 2024-04-21T08:49:48.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:48 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:48.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:48 smithi111 ceph-mon[31493]: scrub summary: active paths [/] 2024-04-21T08:49:48.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:48 smithi111 ceph-mon[31493]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:48.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:48 smithi094 ceph-mon[27988]: scrub queued for path: / 2024-04-21T08:49:48.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:48 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:48.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:48 smithi094 ceph-mon[27988]: scrub summary: active paths [/] 2024-04-21T08:49:48.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:48 smithi094 ceph-mon[27988]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:48.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:48 smithi173 ceph-mon[31428]: scrub queued for path: / 2024-04-21T08:49:48.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:48 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:48.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:48 smithi173 ceph-mon[31428]: scrub summary: active paths [/] 2024-04-21T08:49:48.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:48 smithi173 ceph-mon[31428]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:49.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:49 smithi111 ceph-mon[31493]: pgmap v304: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:49.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:49 smithi094 ceph-mon[27988]: pgmap v304: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:49.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:49 smithi173 ceph-mon[31428]: pgmap v304: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:49:51.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:51 smithi111 ceph-mon[31493]: pgmap v305: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:49:51.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:51 smithi094 ceph-mon[27988]: pgmap v305: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:49:51.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:51 smithi173 ceph-mon[31428]: pgmap v305: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-21T08:49:52.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:52 smithi111 ceph-mon[31493]: scrub summary: idle 2024-04-21T08:49:52.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:52 smithi094 ceph-mon[27988]: scrub summary: idle 2024-04-21T08:49:52.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:52 smithi173 ceph-mon[31428]: scrub summary: idle 2024-04-21T08:49:53.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:53 smithi111 ceph-mon[31493]: pgmap v306: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:49:53.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:53 smithi094 ceph-mon[27988]: pgmap v306: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:49:53.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:53 smithi173 ceph-mon[31428]: pgmap v306: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:49:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:55 smithi111 ceph-mon[31493]: pgmap v307: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:49:55.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:55 smithi094 ceph-mon[27988]: pgmap v307: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:49:55.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:55 smithi173 ceph-mon[31428]: pgmap v307: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-21T08:49:57.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:57 smithi111 ceph-mon[31493]: pgmap v308: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:49:57.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:57 smithi094 ceph-mon[27988]: pgmap v308: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:49:57.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:57 smithi173 ceph-mon[31428]: pgmap v308: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-04-21T08:49:59.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:49:59 smithi111 ceph-mon[31493]: pgmap v309: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:49:59.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:49:59 smithi094 ceph-mon[27988]: pgmap v309: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:49:59.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:49:59 smithi173 ceph-mon[31428]: pgmap v309: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-04-21T08:50:00.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:00 smithi094 ceph-mon[27988]: overall HEALTH_OK 2024-04-21T08:50:00.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:00 smithi173 ceph-mon[31428]: overall HEALTH_OK 2024-04-21T08:50:00.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:00 smithi111 ceph-mon[31493]: overall HEALTH_OK 2024-04-21T08:50:01.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:01 smithi094 ceph-mon[27988]: pgmap v310: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-21T08:50:01.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:01 smithi173 ceph-mon[31428]: pgmap v310: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-21T08:50:01.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:01 smithi111 ceph-mon[31493]: pgmap v310: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-21T08:50:03.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:03 smithi094 ceph-mon[27988]: pgmap v311: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T08:50:03.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:03 smithi173 ceph-mon[31428]: pgmap v311: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T08:50:03.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:03 smithi111 ceph-mon[31493]: pgmap v311: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T08:50:05.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:05 smithi094 ceph-mon[27988]: pgmap v312: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:50:05.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:05 smithi173 ceph-mon[31428]: pgmap v312: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:50:05.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:05 smithi111 ceph-mon[31493]: pgmap v312: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:50:07.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:07 smithi094 ceph-mon[27988]: pgmap v313: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:50:07.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:07 smithi173 ceph-mon[31428]: pgmap v313: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:50:07.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:07 smithi111 ceph-mon[31493]: pgmap v313: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T08:50:07.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T08:50:08.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.028+0000 7f1e96754640 1 -- 172.21.15.94:0/3455862879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e90101aa0 msgr2=0x7f1e90105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.028+0000 7f1e96754640 1 --2- 172.21.15.94:0/3455862879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e90101aa0 0x7f1e90105a90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009860 tx=0x7f1e8402bb70 comp rx=0 tx=0).stop 2024-04-21T08:50:08.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.028+0000 7f1e96754640 1 -- 172.21.15.94:0/3455862879 shutdown_connections 2024-04-21T08:50:08.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.028+0000 7f1e96754640 1 --2- 172.21.15.94:0/3455862879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e90106170 0x7f1e9010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.028+0000 7f1e96754640 1 --2- 172.21.15.94:0/3455862879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e90101aa0 0x7f1e90105a90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.028+0000 7f1e96754640 1 --2- 172.21.15.94:0/3455862879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e901010f0 0x7f1e901014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.028+0000 7f1e96754640 1 -- 172.21.15.94:0/3455862879 >> 172.21.15.94:0/3455862879 conn(0x7f1e90077fd0 msgr2=0x7f1e900783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:08.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.028+0000 7f1e96754640 1 -- 172.21.15.94:0/3455862879 shutdown_connections 2024-04-21T08:50:08.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.028+0000 7f1e96754640 1 -- 172.21.15.94:0/3455862879 wait complete. 2024-04-21T08:50:08.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.029+0000 7f1e96754640 1 Processor -- start 2024-04-21T08:50:08.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.029+0000 7f1e96754640 1 -- start start 2024-04-21T08:50:08.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e96754640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e901010f0 0x7f1e90199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e96754640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e90106170 0x7f1e90199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e96754640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e90199fd0 0x7f1e901a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e96754640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1e90111040 con 0x7f1e901010f0 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e96754640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e90110ec0 con 0x7f1e90199fd0 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e96754640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1e901111c0 con 0x7f1e90106170 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e8ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e901010f0 0x7f1e90199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e8ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e901010f0 0x7f1e90199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34056/0 (socket says 172.21.15.94:34056) 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e8ffff640 1 -- 172.21.15.94:0/1749276391 learned_addr learned my addr 172.21.15.94:0/1749276391 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e8f7fe640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e90106170 0x7f1e90199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.030+0000 7f1e94cf1640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e90199fd0 0x7f1e901a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e8ffff640 1 -- 172.21.15.94:0/1749276391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e90106170 msgr2=0x7f1e90199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e8ffff640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e90106170 0x7f1e90199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e8ffff640 1 -- 172.21.15.94:0/1749276391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e90199fd0 msgr2=0x7f1e901a4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e8ffff640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e90199fd0 0x7f1e901a4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e8ffff640 1 -- 172.21.15.94:0/1749276391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e901a4d80 con 0x7f1e901010f0 2024-04-21T08:50:08.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e8ffff640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e901010f0 0x7f1e90199500 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000e9c0 tx=0x7f1e8000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:08.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e8d7fa640 1 -- 172.21.15.94:0/1749276391 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8000ccc0 con 0x7f1e901010f0 2024-04-21T08:50:08.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e96754640 1 -- 172.21.15.94:0/1749276391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e901a5070 con 0x7f1e901010f0 2024-04-21T08:50:08.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e8d7fa640 1 -- 172.21.15.94:0/1749276391 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e800054d0 con 0x7f1e901010f0 2024-04-21T08:50:08.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e8d7fa640 1 -- 172.21.15.94:0/1749276391 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80010430 con 0x7f1e901010f0 2024-04-21T08:50:08.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.031+0000 7f1e96754640 1 -- 172.21.15.94:0/1749276391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e901a5590 con 0x7f1e901010f0 2024-04-21T08:50:08.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.032+0000 7f1e96754640 1 -- 172.21.15.94:0/1749276391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e9004fce0 con 0x7f1e901010f0 2024-04-21T08:50:08.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.034+0000 7f1e8d7fa640 1 -- 172.21.15.94:0/1749276391 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1e80004f80 con 0x7f1e901010f0 2024-04-21T08:50:08.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.034+0000 7f1e8d7fa640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1e780a3dc0 0x7f1e780a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.034+0000 7f1e8f7fe640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1e780a3dc0 0x7f1e780a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.034+0000 7f1e8d7fa640 1 -- 172.21.15.94:0/1749276391 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9687+0+0 (secure 0 0 0) 0x7f1e800d13f0 con 0x7f1e901010f0 2024-04-21T08:50:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.035+0000 7f1e8f7fe640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1e780a3dc0 0x7f1e780a6280 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009830 tx=0x7f1e840058d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.038+0000 7f1e8d7fa640 1 -- 172.21.15.94:0/1749276391 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e8009d600 con 0x7f1e901010f0 2024-04-21T08:50:08.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.244+0000 7f1e96754640 1 -- 172.21.15.94:0/1749276391 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f1e90109410 con 0x7f1e780a3dc0 2024-04-21T08:50:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.252+0000 7f1e8d7fa640 1 -- 172.21.15.94:0/1749276391 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f1e840346f0 con 0x7f1e780a3dc0 2024-04-21T08:50:08.252 INFO:teuthology.orchestra.run.smithi094.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T08:40:06", "first": "2024-04-21T08:41:00", "last": "2024-04-21T08:50:00", "last_pruned": "2024-04-21T08:50:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-04-21T08:50:08.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 -- 172.21.15.94:0/1749276391 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1e780a3dc0 msgr2=0x7f1e780a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1e780a3dc0 0x7f1e780a6280 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009830 tx=0x7f1e840058d0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 -- 172.21.15.94:0/1749276391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e901010f0 msgr2=0x7f1e90199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e901010f0 0x7f1e90199500 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000e9c0 tx=0x7f1e8000ee80 comp rx=0 tx=0).stop 2024-04-21T08:50:08.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 -- 172.21.15.94:0/1749276391 shutdown_connections 2024-04-21T08:50:08.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1e780a3dc0 0x7f1e780a6280 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e90106170 0x7f1e90199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e90199fd0 0x7f1e901a4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 --2- 172.21.15.94:0/1749276391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e901010f0 0x7f1e90199500 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 -- 172.21.15.94:0/1749276391 >> 172.21.15.94:0/1749276391 conn(0x7f1e90077fd0 msgr2=0x7f1e900ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:08.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 -- 172.21.15.94:0/1749276391 shutdown_connections 2024-04-21T08:50:08.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.254+0000 7f1e96754640 1 -- 172.21.15.94:0/1749276391 wait complete. 2024-04-21T08:50:08.265 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T08:50:08.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.366+0000 7f99d8b66640 1 -- 172.21.15.94:0/2845652064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d4101110 msgr2=0x7f99d41014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.366+0000 7f99d8b66640 1 --2- 172.21.15.94:0/2845652064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d4101110 0x7f99d41014f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009900 tx=0x7f99c402bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.366+0000 7f99d8b66640 1 -- 172.21.15.94:0/2845652064 shutdown_connections 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.366+0000 7f99d8b66640 1 --2- 172.21.15.94:0/2845652064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4101ac0 0x7f99d4105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.366+0000 7f99d8b66640 1 --2- 172.21.15.94:0/2845652064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d4106190 0x7f99d410dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.366+0000 7f99d8b66640 1 --2- 172.21.15.94:0/2845652064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d4101110 0x7f99d41014f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.366+0000 7f99d8b66640 1 -- 172.21.15.94:0/2845652064 >> 172.21.15.94:0/2845652064 conn(0x7f99d4077fd0 msgr2=0x7f99d40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.366+0000 7f99d8b66640 1 -- 172.21.15.94:0/2845652064 shutdown_connections 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.367+0000 7f99d8b66640 1 -- 172.21.15.94:0/2845652064 wait complete. 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.367+0000 7f99d8b66640 1 Processor -- start 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.367+0000 7f99d8b66640 1 -- start start 2024-04-21T08:50:08.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.367+0000 7f99d8b66640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4101110 0x7f99d41974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.367+0000 7f99d8b66640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d4106190 0x7f99d4197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.367+0000 7f99d8b66640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d419bd90 0x7f99d419c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.367+0000 7f99d8b66640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f99d41110d0 con 0x7f99d419bd90 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.367+0000 7f99d8b66640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f99d4110f50 con 0x7f99d4106190 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.367+0000 7f99d8b66640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f99d4111250 con 0x7f99d4101110 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.367+0000 7f99d2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d419bd90 0x7f99d419c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d419bd90 0x7f99d419c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34080/0 (socket says 172.21.15.94:34080) 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d2d9d640 1 -- 172.21.15.94:0/3021220333 learned_addr learned my addr 172.21.15.94:0/3021220333 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d259c640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4101110 0x7f99d41974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d2d9d640 1 -- 172.21.15.94:0/3021220333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4101110 msgr2=0x7f99d41974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d2d9d640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4101110 0x7f99d41974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d2d9d640 1 -- 172.21.15.94:0/3021220333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d4106190 msgr2=0x7f99d4197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d1d9b640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d4106190 0x7f99d4197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d2d9d640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d4106190 0x7f99d4197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d2d9d640 1 -- 172.21.15.94:0/3021220333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d419c880 con 0x7f99d419bd90 2024-04-21T08:50:08.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d259c640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4101110 0x7f99d41974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:08.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d2d9d640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d419bd90 0x7f99d419c240 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00eaa0 tx=0x7f99cc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:08.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99c37fe640 1 -- 172.21.15.94:0/3021220333 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99cc00cdb0 con 0x7f99d419bd90 2024-04-21T08:50:08.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99c37fe640 1 -- 172.21.15.94:0/3021220333 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99cc005020 con 0x7f99d419bd90 2024-04-21T08:50:08.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d8b66640 1 -- 172.21.15.94:0/3021220333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d419cb70 con 0x7f99d419bd90 2024-04-21T08:50:08.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99c37fe640 1 -- 172.21.15.94:0/3021220333 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99cc0116d0 con 0x7f99d419bd90 2024-04-21T08:50:08.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.368+0000 7f99d8b66640 1 -- 172.21.15.94:0/3021220333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d41a4310 con 0x7f99d419bd90 2024-04-21T08:50:08.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.369+0000 7f99d8b66640 1 -- 172.21.15.94:0/3021220333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d404fce0 con 0x7f99d419bd90 2024-04-21T08:50:08.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.370+0000 7f99c37fe640 1 -- 172.21.15.94:0/3021220333 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f99cc0051d0 con 0x7f99d419bd90 2024-04-21T08:50:08.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.371+0000 7f99c37fe640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f99bc0a3dc0 0x7f99bc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.371+0000 7f99d259c640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f99bc0a3dc0 0x7f99bc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.371+0000 7f99c37fe640 1 -- 172.21.15.94:0/3021220333 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9687+0+0 (secure 0 0 0) 0x7f99cc0d0ee0 con 0x7f99d419bd90 2024-04-21T08:50:08.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.371+0000 7f99d259c640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f99bc0a3dc0 0x7f99bc0a6280 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f99c4005ab0 tx=0x7f99c4005a20 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:08.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.375+0000 7f99c37fe640 1 -- 172.21.15.94:0/3021220333 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99cc016030 con 0x7f99d419bd90 2024-04-21T08:50:08.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:08 smithi094 ceph-mon[27988]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T08:50:08.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:08 smithi173 ceph-mon[31428]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T08:50:08.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.574+0000 7f99d8b66640 1 -- 172.21.15.94:0/3021220333 --> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f99d4101c00 con 0x7f99bc0a3dc0 2024-04-21T08:50:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.581+0000 7f99c37fe640 1 -- 172.21.15.94:0/3021220333 <== mgr.15453 v2:172.21.15.94:6800/4147145528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f99d4101c00 con 0x7f99bc0a3dc0 2024-04-21T08:50:08.581 INFO:teuthology.orchestra.run.smithi094.stdout:/ 1m 6m3h 2024-04-21T08:50:08.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 -- 172.21.15.94:0/3021220333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f99bc0a3dc0 msgr2=0x7f99bc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f99bc0a3dc0 0x7f99bc0a6280 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f99c4005ab0 tx=0x7f99c4005a20 comp rx=0 tx=0).stop 2024-04-21T08:50:08.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 -- 172.21.15.94:0/3021220333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d419bd90 msgr2=0x7f99d419c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d419bd90 0x7f99d419c240 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00eaa0 tx=0x7f99cc00ef60 comp rx=0 tx=0).stop 2024-04-21T08:50:08.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 -- 172.21.15.94:0/3021220333 shutdown_connections 2024-04-21T08:50:08.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f99bc0a3dc0 0x7f99bc0a6280 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d419bd90 0x7f99d419c240 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d4106190 0x7f99d4197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 --2- 172.21.15.94:0/3021220333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4101110 0x7f99d41974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 -- 172.21.15.94:0/3021220333 >> 172.21.15.94:0/3021220333 conn(0x7f99d4077fd0 msgr2=0x7f99d40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:08.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 -- 172.21.15.94:0/3021220333 shutdown_connections 2024-04-21T08:50:08.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:08.583+0000 7f99d8b66640 1 -- 172.21.15.94:0/3021220333 wait complete. 2024-04-21T08:50:08.594 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-21T08:50:08.606 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-21T08:50:08.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:08 smithi111 ceph-mon[31493]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T08:50:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:09 smithi094 ceph-mon[27988]: pgmap v315: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:50:09.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:09 smithi094 ceph-mon[27988]: from='client.18147 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:09.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:09 smithi094 ceph-mon[27988]: from='client.18153 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:09.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:09 smithi173 ceph-mon[31428]: pgmap v315: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:50:09.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:09 smithi173 ceph-mon[31428]: from='client.18147 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:09.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:09 smithi173 ceph-mon[31428]: from='client.18153 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:09.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:09 smithi111 ceph-mon[31493]: pgmap v315: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T08:50:09.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:09 smithi111 ceph-mon[31493]: from='client.18147 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:09.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:09 smithi111 ceph-mon[31493]: from='client.18153 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:11.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:11 smithi111 ceph-mon[31493]: pgmap v316: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 29 op/s 2024-04-21T08:50:11.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:11 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:50:11.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:11 smithi094 ceph-mon[27988]: pgmap v316: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 29 op/s 2024-04-21T08:50:11.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:11 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:50:11.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:11 smithi173 ceph-mon[31428]: pgmap v316: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 29 op/s 2024-04-21T08:50:11.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:11 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:50:13.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T08:50:13.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:13.189 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:12 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:13.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:13.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:12 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:50:13.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:13.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:13.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:12 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' 2024-04-21T08:50:14.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:13 smithi111 ceph-mon[31493]: pgmap v318: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 KiB/s wr, 29 op/s 2024-04-21T08:50:14.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:13 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T08:50:14.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:13 smithi111 ceph-mon[31493]: Unable to set osd_memory_target on smithi111 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-21T08:50:14.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:13 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi094 to 176.2M 2024-04-21T08:50:14.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:13 smithi111 ceph-mon[31493]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-21T08:50:14.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:13 smithi111 ceph-mon[31493]: Unable to set osd_memory_target on smithi094 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T08:50:14.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:13 smithi111 ceph-mon[31493]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-04-21T08:50:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:13 smithi094 ceph-mon[27988]: pgmap v318: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 KiB/s wr, 29 op/s 2024-04-21T08:50:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:13 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T08:50:14.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:13 smithi094 ceph-mon[27988]: Unable to set osd_memory_target on smithi111 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-21T08:50:14.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:13 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi094 to 176.2M 2024-04-21T08:50:14.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:13 smithi094 ceph-mon[27988]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-21T08:50:14.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:13 smithi094 ceph-mon[27988]: Unable to set osd_memory_target on smithi094 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T08:50:14.261 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:13 smithi094 ceph-mon[27988]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-04-21T08:50:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:13 smithi173 ceph-mon[31428]: pgmap v318: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 KiB/s wr, 29 op/s 2024-04-21T08:50:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:13 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T08:50:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:13 smithi173 ceph-mon[31428]: Unable to set osd_memory_target on smithi111 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-21T08:50:14.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:13 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi094 to 176.2M 2024-04-21T08:50:14.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:13 smithi173 ceph-mon[31428]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-21T08:50:14.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:13 smithi173 ceph-mon[31428]: Unable to set osd_memory_target on smithi094 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T08:50:14.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:13 smithi173 ceph-mon[31428]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-04-21T08:50:16.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:15 smithi111 ceph-mon[31493]: pgmap v319: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-21T08:50:16.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:15 smithi094 ceph-mon[27988]: pgmap v319: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-21T08:50:16.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:15 smithi173 ceph-mon[31428]: pgmap v319: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-21T08:50:17.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:50:17.426 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:50:17.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.542+0000 7f65a7d4a640 1 -- 172.21.15.94:0/2642168707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0101a50 msgr2=0x7f65a0105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.542+0000 7f65a7d4a640 1 --2- 172.21.15.94:0/2642168707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0101a50 0x7f65a0105a90 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f659c009900 tx=0x7f659c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:17.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.542+0000 7f65a7d4a640 1 -- 172.21.15.94:0/2642168707 shutdown_connections 2024-04-21T08:50:17.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.542+0000 7f65a7d4a640 1 --2- 172.21.15.94:0/2642168707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0106170 0x7f65a010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.542+0000 7f65a7d4a640 1 --2- 172.21.15.94:0/2642168707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a0101a50 0x7f65a0105a90 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.542+0000 7f65a7d4a640 1 --2- 172.21.15.94:0/2642168707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a01010a0 0x7f65a0101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.542+0000 7f65a7d4a640 1 -- 172.21.15.94:0/2642168707 >> 172.21.15.94:0/2642168707 conn(0x7f65a0077f70 msgr2=0x7f65a0078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.542+0000 7f65a7d4a640 1 -- 172.21.15.94:0/2642168707 shutdown_connections 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.542+0000 7f65a7d4a640 1 -- 172.21.15.94:0/2642168707 wait complete. 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a7d4a640 1 Processor -- start 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a7d4a640 1 -- start start 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a7d4a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a01010a0 0x7f65a0102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a7d4a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0106170 0x7f65a0103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a7d4a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a01035b0 0x7f65a01a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a7d4a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f65a0111090 con 0x7f65a01035b0 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a7d4a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f65a0110f10 con 0x7f65a0106170 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a7d4a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65a0111210 con 0x7f65a01010a0 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a62e7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a01035b0 0x7f65a01a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a62e7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a01035b0 0x7f65a01a6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57732/0 (socket says 172.21.15.94:57732) 2024-04-21T08:50:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a62e7640 1 -- 172.21.15.94:0/3511476571 learned_addr learned my addr 172.21.15.94:0/3511476571 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a52e5640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0106170 0x7f65a0103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a01010a0 0x7f65a0102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a62e7640 1 -- 172.21.15.94:0/3511476571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a01010a0 msgr2=0x7f65a0102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a62e7640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a01010a0 0x7f65a0102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a62e7640 1 -- 172.21.15.94:0/3511476571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0106170 msgr2=0x7f65a0103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a62e7640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0106170 0x7f65a0103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.543+0000 7f65a62e7640 1 -- 172.21.15.94:0/3511476571 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a01a6f70 con 0x7f65a01035b0 2024-04-21T08:50:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.544+0000 7f65a62e7640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a01035b0 0x7f65a01a6870 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f659800cad0 tx=0x7f659800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.544+0000 7f6596ffd640 1 -- 172.21.15.94:0/3511476571 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659800ae20 con 0x7f65a01035b0 2024-04-21T08:50:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.544+0000 7f6596ffd640 1 -- 172.21.15.94:0/3511476571 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6598005020 con 0x7f65a01035b0 2024-04-21T08:50:17.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.544+0000 7f65a7d4a640 1 -- 172.21.15.94:0/3511476571 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a01a7260 con 0x7f65a01035b0 2024-04-21T08:50:17.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.544+0000 7f6596ffd640 1 -- 172.21.15.94:0/3511476571 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659800f6b0 con 0x7f65a01035b0 2024-04-21T08:50:17.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.544+0000 7f65a7d4a640 1 -- 172.21.15.94:0/3511476571 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a01a7730 con 0x7f65a01035b0 2024-04-21T08:50:17.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.547+0000 7f6596ffd640 1 -- 172.21.15.94:0/3511476571 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f65980051d0 con 0x7f65a01035b0 2024-04-21T08:50:17.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.547+0000 7f65aaa10740 1 Processor -- start 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.547+0000 7f65aaa10740 1 -- start start 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.547+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 0x565382430960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382439540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382439a80 0x56538243be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65aaa10740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5653823951e0 con 0x565382430ea0 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65aaa10740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x565382332390 con 0x565382439a80 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65aaa10740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5653823911a0 con 0x565382430580 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a5ae6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382439540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f6596ffd640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f658c0a3d50 0x7f658c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a5ae6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382439540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57742/0 (socket says 172.21.15.94:57742) 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a5ae6640 1 -- 172.21.15.94:0/166236461 learned_addr learned my addr 172.21.15.94:0/166236461 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:17.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a52e5640 1 --2- 172.21.15.94:0/166236461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 0x565382430960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a62e7640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f658c0a3d50 0x7f658c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a52e5640 1 --2- 172.21.15.94:0/166236461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382439a80 0x56538243be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a5ae6640 1 -- 172.21.15.94:0/166236461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 msgr2=0x565382430960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/166236461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 0x565382430960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a5ae6640 1 -- 172.21.15.94:0/166236461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382439a80 msgr2=0x56538243be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f6596ffd640 1 -- 172.21.15.94:0/3511476571 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f65980d0c50 con 0x7f65a01035b0 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/166236461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382439a80 0x56538243be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a5ae6640 1 -- 172.21.15.94:0/166236461 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653821717d0 con 0x565382430ea0 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a52e5640 1 --2- 172.21.15.94:0/166236461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382439a80 0x56538243be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/166236461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382439540 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f659000cc60 tx=0x7f659000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d263f959fc573d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f65a62e7640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f658c0a3d50 0x7f658c0a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f65980ce020 tx=0x7f6598087040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.548+0000 7f658bfff640 1 -- 172.21.15.94:0/166236461 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590014070 con 0x565382430ea0 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f658bfff640 1 -- 172.21.15.94:0/166236461 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65900047a0 con 0x565382430ea0 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f658bfff640 1 -- 172.21.15.94:0/166236461 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590010690 con 0x565382430ea0 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f65aaa10740 1 -- 172.21.15.94:0/166236461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 msgr2=0x565382439540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f65aaa10740 1 --2- 172.21.15.94:0/166236461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382439540 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f659000cc60 tx=0x7f659000e720 comp rx=0 tx=0).stop 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f65aaa10740 1 -- 172.21.15.94:0/166236461 shutdown_connections 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f65aaa10740 1 --2- 172.21.15.94:0/166236461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382439a80 0x56538243be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f65aaa10740 1 --2- 172.21.15.94:0/166236461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382439540 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f65aaa10740 1 --2- 172.21.15.94:0/166236461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 0x565382430960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f65aaa10740 1 -- 172.21.15.94:0/166236461 >> 172.21.15.94:0/166236461 conn(0x56538242acb0 msgr2=0x56538242b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f65aaa10740 1 -- 172.21.15.94:0/166236461 shutdown_connections 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.549+0000 7f65aaa10740 1 -- 172.21.15.94:0/166236461 wait complete. 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65aaa10740 1 -- start start 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65aaa10740 1 Processor -- start 2024-04-21T08:50:17.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382430580 0x565382436af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382437030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382439a80 0x565382437570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65aaa10740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56538243c870 con 0x565382430ea0 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65aaa10740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56538234e2f0 con 0x565382430580 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65aaa10740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56538243c9f0 con 0x565382439a80 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65aaa10740 10 client.0 _ll_register_callbacks cb 0x56538235b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65a5ae6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382437030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65a5ae6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382437030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57756/0 (socket says 172.21.15.94:57756) 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.550+0000 7f65a5ae6640 1 -- 172.21.15.94:0/4120724247 learned_addr learned my addr 172.21.15.94:0/4120724247 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65a52e5640 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382439a80 0x565382437570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65a5ae6640 1 -- 172.21.15.94:0/4120724247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382439a80 msgr2=0x565382437570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382439a80 0x565382437570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65a52e5640 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382430580 0x565382436af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65a5ae6640 1 -- 172.21.15.94:0/4120724247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382430580 msgr2=0x565382436af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382430580 0x565382436af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65a5ae6640 1 -- 172.21.15.94:0/4120724247 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653820a73f0 con 0x565382430ea0 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382437030 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f6590010bb0 tx=0x7f6590010be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569c966b2a3ddd04 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65897fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65897fa640 1 -- 172.21.15.94:0/4120724247 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590014070 con 0x565382430ea0 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65897fa640 1 -- 172.21.15.94:0/4120724247 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f6590004030 con 0x565382430ea0 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65aaa10740 1 -- 172.21.15.94:0/4120724247 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65900047a0 con 0x565382430ea0 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65897fa640 1 -- 172.21.15.94:0/4120724247 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f659002d050 con 0x565382430ea0 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65aaa10740 10 client.18165 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65aaa10740 1 -- 172.21.15.94:0/4120724247 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5653820670f0 con 0x565382430ea0 2024-04-21T08:50:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.551+0000 7f65897fa640 1 -- 172.21.15.94:0/4120724247 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65900047a0 con 0x565382430ea0 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.552+0000 7f65897fa640 1 -- 172.21.15.94:0/4120724247 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6590003c60 con 0x565382430ea0 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.552+0000 7f65aaa10740 10 client.18165 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.552+0000 7f65aaa10740 10 client.18165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.552+0000 7f65aaa10740 20 client.18165 populate_metadata read hostname 'smithi094' 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.552+0000 7f65aaa10740 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565382449a60 0x56538244be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.552+0000 7f65aaa10740 4 client.18165 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.552+0000 7f65aaa10740 1 -- 172.21.15.94:0/4120724247 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653823a2c00 con 0x565382449a60 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.552+0000 7f65a52e5640 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565382449a60 0x56538244be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.552+0000 7f65897fa640 10 client.18165 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.552+0000 7f65a52e5640 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565382449a60 0x56538244be50 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=138804b5bfbad44c server_cookie=b37d1a316693ac1c in_seq=0 out_seq=0 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65897fa640 1 -- 172.21.15.94:0/4120724247 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653823a2c00 con 0x565382449a60 2024-04-21T08:50:17.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65897fa640 10 client.18165 handle_command_reply: tid=0 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65aaa10740 1 client.18165 shutdown 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65aaa10740 1 -- 172.21.15.94:0/4120724247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 msgr2=0x565382437030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65aaa10740 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382437030 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f6590010bb0 tx=0x7f6590010be0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65aaa10740 1 -- 172.21.15.94:0/4120724247 shutdown_connections 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65aaa10740 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x565382449a60 0x56538244be50 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65aaa10740 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382439a80 0x565382437570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65897fa640 0 client.18165 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65aaa10740 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382437030 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65aaa10740 1 --2- 172.21.15.94:0/4120724247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382430580 0x565382436af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.553+0000 7f65aaa10740 1 -- 172.21.15.94:0/4120724247 >> 172.21.15.94:0/4120724247 conn(0x56538242acb0 msgr2=0x56538242ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.554+0000 7f65aaa10740 1 -- 172.21.15.94:0/4120724247 shutdown_connections 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.554+0000 7f65aaa10740 1 -- 172.21.15.94:0/4120724247 wait complete. 2024-04-21T08:50:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.554+0000 7f65aaa10740 20 client.18165 trim_cache size 0 max 16384 2024-04-21T08:50:17.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.571+0000 7f65aaa10740 1 Processor -- start 2024-04-21T08:50:17.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.571+0000 7f65aaa10740 1 -- start start 2024-04-21T08:50:17.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.571+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 0x565382439220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.571+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382435a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.571+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382439a80 0x565382445730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.571+0000 7f65aaa10740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01b3fd0 con 0x565382430ea0 2024-04-21T08:50:17.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.571+0000 7f65aaa10740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01b18a0 con 0x565382439a80 2024-04-21T08:50:17.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.571+0000 7f65aaa10740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01af710 con 0x565382430580 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a5ae6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382435a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a5ae6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382435a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57770/0 (socket says 172.21.15.94:57770) 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a5ae6640 1 -- 172.21.15.94:0/753963540 learned_addr learned my addr 172.21.15.94:0/753963540 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a52e5640 1 --2- 172.21.15.94:0/753963540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 0x565382439220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a5ae6640 1 -- 172.21.15.94:0/753963540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 msgr2=0x565382439220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/753963540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 0x565382439220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a5ae6640 1 -- 172.21.15.94:0/753963540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382439a80 msgr2=0x565382445730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/753963540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382439a80 0x565382445730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a5ae6640 1 -- 172.21.15.94:0/753963540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565382435040 con 0x565382430ea0 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a52e5640 1 --2- 172.21.15.94:0/753963540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 0x565382439220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/753963540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382435a30 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f6590006010 tx=0x7f6590005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa466a6960d291da server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f658b7fe640 1 -- 172.21.15.94:0/753963540 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659001c020 con 0x565382430ea0 2024-04-21T08:50:17.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f658b7fe640 1 -- 172.21.15.94:0/753963540 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f659000c770 con 0x565382430ea0 2024-04-21T08:50:17.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.572+0000 7f658b7fe640 1 -- 172.21.15.94:0/753963540 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590014070 con 0x565382430ea0 2024-04-21T08:50:17.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 -- 172.21.15.94:0/753963540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 msgr2=0x565382435a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 --2- 172.21.15.94:0/753963540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382435a30 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f6590006010 tx=0x7f6590005fa0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 -- 172.21.15.94:0/753963540 shutdown_connections 2024-04-21T08:50:17.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 --2- 172.21.15.94:0/753963540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382439a80 0x565382445730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 --2- 172.21.15.94:0/753963540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382430ea0 0x565382435a30 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 --2- 172.21.15.94:0/753963540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430580 0x565382439220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 -- 172.21.15.94:0/753963540 >> 172.21.15.94:0/753963540 conn(0x56538242acb0 msgr2=0x565382429d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 -- 172.21.15.94:0/753963540 shutdown_connections 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 -- 172.21.15.94:0/753963540 wait complete. 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 -- start start 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.573+0000 7f65aaa10740 1 Processor -- start 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382430580 0x56538243d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430ea0 0x56538243dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65aaa10740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382439a80 0x56538243e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65aaa10740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01afd60 con 0x565382439a80 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65aaa10740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01b11d0 con 0x565382430580 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65aaa10740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01b0430 con 0x565382430ea0 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65aaa10740 10 client.0 _ll_register_callbacks cb 0x7f659c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a52e5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382439a80 0x56538243e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a52e5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382439a80 0x56538243e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57778/0 (socket says 172.21.15.94:57778) 2024-04-21T08:50:17.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a52e5640 1 -- 172.21.15.94:0/1641879946 learned_addr learned my addr 172.21.15.94:0/1641879946 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:17.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a52e5640 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382430580 0x56538243d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430ea0 0x56538243dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a52e5640 1 -- 172.21.15.94:0/1641879946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430ea0 msgr2=0x56538243dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a52e5640 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430ea0 0x56538243dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a52e5640 1 -- 172.21.15.94:0/1641879946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382430580 msgr2=0x56538243d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a52e5640 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382430580 0x56538243d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a52e5640 1 -- 172.21.15.94:0/1641879946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653824af040 con 0x565382439a80 2024-04-21T08:50:17.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65a52e5640 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382439a80 0x56538243e460 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f659c02c680 tx=0x7f659c02c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93baccc14b028c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65897fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65897fa640 1 -- 172.21.15.94:0/1641879946 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c005250 con 0x565382439a80 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65897fa640 1 -- 172.21.15.94:0/1641879946 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f659c0053f0 con 0x565382439a80 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.574+0000 7f65aaa10740 1 -- 172.21.15.94:0/1641879946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f659000c770 con 0x565382439a80 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65897fa640 1 -- 172.21.15.94:0/1641879946 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f659c0055f0 con 0x565382439a80 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65aaa10740 10 client.18171 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65aaa10740 1 -- 172.21.15.94:0/1641879946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5653824b02c0 con 0x565382439a80 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65897fa640 1 -- 172.21.15.94:0/1641879946 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f659000c770 con 0x565382439a80 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65897fa640 1 -- 172.21.15.94:0/1641879946 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f659c004ad0 con 0x565382439a80 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65aaa10740 10 client.18171 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65aaa10740 10 client.18171 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65aaa10740 20 client.18171 populate_metadata read hostname 'smithi094' 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65aaa10740 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5653824b09c0 0x5653824b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65aaa10740 4 client.18171 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65aaa10740 1 -- 172.21.15.94:0/1641879946 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56538239ca70 con 0x5653824b09c0 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.575+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5653824b09c0 0x5653824b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65897fa640 10 client.18171 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65a5ae6640 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5653824b09c0 0x5653824b2de0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=570c7b120c972917 server_cookie=c6850c37f7b98c4f in_seq=0 out_seq=0 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65897fa640 1 -- 172.21.15.94:0/1641879946 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56538239ca70 con 0x5653824b09c0 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65897fa640 10 client.18171 handle_command_reply: tid=0 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 client.18171 shutdown 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 -- 172.21.15.94:0/1641879946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382439a80 msgr2=0x56538243e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382439a80 0x56538243e460 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f659c02c680 tx=0x7f659c02c610 comp rx=0 tx=0).stop 2024-04-21T08:50:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 -- 172.21.15.94:0/1641879946 shutdown_connections 2024-04-21T08:50:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5653824b09c0 0x5653824b2de0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x565382430ea0 0x56538243dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x565382439a80 0x56538243e460 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 --2- 172.21.15.94:0/1641879946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565382430580 0x56538243d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65897fa640 0 client.18171 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:50:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 -- 172.21.15.94:0/1641879946 >> 172.21.15.94:0/1641879946 conn(0x56538242acb0 msgr2=0x5653824368e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 -- 172.21.15.94:0/1641879946 shutdown_connections 2024-04-21T08:50:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 1 -- 172.21.15.94:0/1641879946 wait complete. 2024-04-21T08:50:17.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.576+0000 7f65aaa10740 20 client.18171 trim_cache size 0 max 16384 2024-04-21T08:50:17.587 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:50:17.587 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "no active scrubs running", 2024-04-21T08:50:17.587 INFO:teuthology.orchestra.run.smithi094.stdout: "scrubs": {} 2024-04-21T08:50:17.587 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:50:17.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 -- 172.21.15.94:0/3511476571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f658c0a3d50 msgr2=0x7f658c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f658c0a3d50 0x7f658c0a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f65980ce020 tx=0x7f6598087040 comp rx=0 tx=0).stop 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 -- 172.21.15.94:0/3511476571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a01035b0 msgr2=0x7f65a01a6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a01035b0 0x7f65a01a6870 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f659800cad0 tx=0x7f659800cf90 comp rx=0 tx=0).stop 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 -- 172.21.15.94:0/3511476571 shutdown_connections 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f658c0a3d50 0x7f658c0a6210 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65a01035b0 0x7f65a01a6870 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a0106170 0x7f65a0103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 --2- 172.21.15.94:0/3511476571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65a01010a0 0x7f65a0102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 -- 172.21.15.94:0/3511476571 >> 172.21.15.94:0/3511476571 conn(0x7f65a0077f70 msgr2=0x7f65a006a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 -- 172.21.15.94:0/3511476571 shutdown_connections 2024-04-21T08:50:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.577+0000 7f65a7d4a640 1 -- 172.21.15.94:0/3511476571 wait complete. 2024-04-21T08:50:17.590 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:50:17.590 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:50:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.688+0000 7f58d0150640 1 -- 172.21.15.94:0/1897926345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58c8106f50 msgr2=0x7f58c8107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.688+0000 7f58d0150640 1 --2- 172.21.15.94:0/1897926345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58c8106f50 0x7f58c8107330 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009930 tx=0x7f58c402bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.688+0000 7f58d0150640 1 -- 172.21.15.94:0/1897926345 shutdown_connections 2024-04-21T08:50:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.688+0000 7f58d0150640 1 --2- 172.21.15.94:0/1897926345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58c8100e80 0x7f58c810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.688+0000 7f58d0150640 1 --2- 172.21.15.94:0/1897926345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58c81004c0 0x7f58c8100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.688+0000 7f58d0150640 1 --2- 172.21.15.94:0/1897926345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58c8106f50 0x7f58c8107330 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.688+0000 7f58d0150640 1 -- 172.21.15.94:0/1897926345 >> 172.21.15.94:0/1897926345 conn(0x7f58c8077450 msgr2=0x7f58c80fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.688+0000 7f58d0150640 1 -- 172.21.15.94:0/1897926345 shutdown_connections 2024-04-21T08:50:17.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.688+0000 7f58d0150640 1 -- 172.21.15.94:0/1897926345 wait complete. 2024-04-21T08:50:17.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58d0150640 1 Processor -- start 2024-04-21T08:50:17.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58d0150640 1 -- start start 2024-04-21T08:50:17.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58d0150640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58c81004c0 0x7f58c819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58d0150640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58c8100e80 0x7f58c819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58d0150640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58c8106f50 0x7f58c8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58d0150640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f58c8112280 con 0x7f58c81004c0 2024-04-21T08:50:17.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58d0150640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f58c8112100 con 0x7f58c8100e80 2024-04-21T08:50:17.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58d0150640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f58c8112400 con 0x7f58c8106f50 2024-04-21T08:50:17.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58cdeec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58c81004c0 0x7f58c819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58cdeec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58c81004c0 0x7f58c819e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57792/0 (socket says 172.21.15.94:57792) 2024-04-21T08:50:17.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58cdeec640 1 -- 172.21.15.94:0/2371913411 learned_addr learned my addr 172.21.15.94:0/2371913411 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:17.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58cd6eb640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58c8100e80 0x7f58c819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58ce6ed640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58c8106f50 0x7f58c8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58cdeec640 1 -- 172.21.15.94:0/2371913411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58c8106f50 msgr2=0x7f58c8198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58cdeec640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58c8106f50 0x7f58c8198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58cdeec640 1 -- 172.21.15.94:0/2371913411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58c8100e80 msgr2=0x7f58c819e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58cdeec640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58c8100e80 0x7f58c819e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.689+0000 7f58cdeec640 1 -- 172.21.15.94:0/2371913411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c8198da0 con 0x7f58c81004c0 2024-04-21T08:50:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.690+0000 7f58cdeec640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58c81004c0 0x7f58c819e420 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f58c4005940 tx=0x7f58c40040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.690+0000 7f58beffd640 1 -- 172.21.15.94:0/2371913411 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4043020 con 0x7f58c81004c0 2024-04-21T08:50:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.690+0000 7f58beffd640 1 -- 172.21.15.94:0/2371913411 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58c40044c0 con 0x7f58c81004c0 2024-04-21T08:50:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.690+0000 7f58d0150640 1 -- 172.21.15.94:0/2371913411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c8199030 con 0x7f58c81004c0 2024-04-21T08:50:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.690+0000 7f58beffd640 1 -- 172.21.15.94:0/2371913411 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4003bf0 con 0x7f58c81004c0 2024-04-21T08:50:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.690+0000 7f58d0150640 1 -- 172.21.15.94:0/2371913411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c81a52e0 con 0x7f58c81004c0 2024-04-21T08:50:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.691+0000 7f58d0150640 1 -- 172.21.15.94:0/2371913411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c8069230 con 0x7f58c81004c0 2024-04-21T08:50:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.692+0000 7f58beffd640 1 -- 172.21.15.94:0/2371913411 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f58c4046020 con 0x7f58c81004c0 2024-04-21T08:50:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.692+0000 7f58beffd640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58b40a3d20 0x7f58b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.692+0000 7f58cd6eb640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58b40a3d20 0x7f58b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.692+0000 7f58beffd640 1 -- 172.21.15.94:0/2371913411 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f58c40f27a0 con 0x7f58c81004c0 2024-04-21T08:50:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.693+0000 7f58cd6eb640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58b40a3d20 0x7f58b40a61e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f58b8005fd0 tx=0x7f58b8005850 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.697+0000 7f58beffd640 1 -- 172.21.15.94:0/2371913411 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58c404d070 con 0x7f58c81004c0 2024-04-21T08:50:17.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.915+0000 7f58d0150640 1 -- 172.21.15.94:0/2371913411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58c810a4a0 con 0x7f58c81004c0 2024-04-21T08:50:17.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.916+0000 7f58beffd640 1 -- 172.21.15.94:0/2371913411 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f58c403d3b0 con 0x7f58c81004c0 2024-04-21T08:50:17.923 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:50:17.923 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:50:17.924 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:50:17.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 -- 172.21.15.94:0/2371913411 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58b40a3d20 msgr2=0x7f58b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58b40a3d20 0x7f58b40a61e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f58b8005fd0 tx=0x7f58b8005850 comp rx=0 tx=0).stop 2024-04-21T08:50:17.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 -- 172.21.15.94:0/2371913411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58c81004c0 msgr2=0x7f58c819e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58c81004c0 0x7f58c819e420 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f58c4005940 tx=0x7f58c40040c0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 -- 172.21.15.94:0/2371913411 shutdown_connections 2024-04-21T08:50:17.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58b40a3d20 0x7f58b40a61e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58c8106f50 0x7f58c8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58c8100e80 0x7f58c819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 --2- 172.21.15.94:0/2371913411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58c81004c0 0x7f58c819e420 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 -- 172.21.15.94:0/2371913411 >> 172.21.15.94:0/2371913411 conn(0x7f58c8077450 msgr2=0x7f58c81050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 -- 172.21.15.94:0/2371913411 shutdown_connections 2024-04-21T08:50:17.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:17.918+0000 7f58d0150640 1 -- 172.21.15.94:0/2371913411 wait complete. 2024-04-21T08:50:17.930 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:50:18.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.047+0000 7f587369a640 1 -- 172.21.15.94:0/711960587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f586c107260 msgr2=0x7f586c10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.047+0000 7f587369a640 1 --2- 172.21.15.94:0/711960587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f586c107260 0x7f586c10ee30 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f5864009ca0 tx=0x7f586402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.047+0000 7f587369a640 1 -- 172.21.15.94:0/711960587 shutdown_connections 2024-04-21T08:50:18.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.047+0000 7f587369a640 1 --2- 172.21.15.94:0/711960587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586c102b40 0x7f586c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.047+0000 7f587369a640 1 --2- 172.21.15.94:0/711960587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f586c107260 0x7f586c10ee30 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.047+0000 7f587369a640 1 --2- 172.21.15.94:0/711960587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c102190 0x7f586c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.047+0000 7f587369a640 1 -- 172.21.15.94:0/711960587 >> 172.21.15.94:0/711960587 conn(0x7f586c0793a0 msgr2=0x7f586c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.047+0000 7f587369a640 1 -- 172.21.15.94:0/711960587 shutdown_connections 2024-04-21T08:50:18.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.047+0000 7f587369a640 1 -- 172.21.15.94:0/711960587 wait complete. 2024-04-21T08:50:18.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f587369a640 1 Processor -- start 2024-04-21T08:50:18.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f587369a640 1 -- start start 2024-04-21T08:50:18.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f587369a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586c102190 0x7f586c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f587369a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f586c107260 0x7f586c198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f587369a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c19cff0 0x7f586c19d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f587369a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f586c1121d0 con 0x7f586c107260 2024-04-21T08:50:18.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f587369a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f586c112050 con 0x7f586c19cff0 2024-04-21T08:50:18.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f587369a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f586c112350 con 0x7f586c102190 2024-04-21T08:50:18.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f5870c35640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f586c107260 0x7f586c198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f5870c35640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f586c107260 0x7f586c198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57812/0 (socket says 172.21.15.94:57812) 2024-04-21T08:50:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.048+0000 7f5870c35640 1 -- 172.21.15.94:0/470490789 learned_addr learned my addr 172.21.15.94:0/470490789 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f5871436640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586c102190 0x7f586c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f5871c37640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c19cff0 0x7f586c19d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f5870c35640 1 -- 172.21.15.94:0/470490789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586c102190 msgr2=0x7f586c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f5870c35640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586c102190 0x7f586c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f5870c35640 1 -- 172.21.15.94:0/470490789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c19cff0 msgr2=0x7f586c19d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f5870c35640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c19cff0 0x7f586c19d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f5870c35640 1 -- 172.21.15.94:0/470490789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586c19dae0 con 0x7f586c107260 2024-04-21T08:50:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f5871436640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586c102190 0x7f586c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:18.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f5870c35640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f586c107260 0x7f586c198c70 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f585c00d740 tx=0x7f585c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f58627fc640 1 -- 172.21.15.94:0/470490789 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c017070 con 0x7f586c107260 2024-04-21T08:50:18.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f587369a640 1 -- 172.21.15.94:0/470490789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586c19ddd0 con 0x7f586c107260 2024-04-21T08:50:18.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f58627fc640 1 -- 172.21.15.94:0/470490789 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f585c00de50 con 0x7f586c107260 2024-04-21T08:50:18.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f58627fc640 1 -- 172.21.15.94:0/470490789 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c004ce0 con 0x7f586c107260 2024-04-21T08:50:18.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.049+0000 7f587369a640 1 -- 172.21.15.94:0/470490789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f586c1a55c0 con 0x7f586c107260 2024-04-21T08:50:18.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.051+0000 7f58627fc640 1 -- 172.21.15.94:0/470490789 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f585c004e80 con 0x7f586c107260 2024-04-21T08:50:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f587636c740 1 Processor -- start 2024-04-21T08:50:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f587636c740 1 -- start start 2024-04-21T08:50:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f58627fc640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58580a3d20 0x7f58580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f587636c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 0x55c5e1fc8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f587636c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fd5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871c37640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58580a3d20 0x7f58580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f587636c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f587636c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c5e1f2d1f0 con 0x55c5e1fd5e10 2024-04-21T08:50:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f587636c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c5e1eca4b0 con 0x55c5e1fc8c00 2024-04-21T08:50:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f587636c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c5e1f291b0 con 0x55c5e1fc8250 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f58627fc640 1 -- 172.21.15.94:0/470490789 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f585c0d0610 con 0x7f586c107260 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871c37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871c37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57820/0 (socket says 172.21.15.94:57820) 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871c37640 1 -- 172.21.15.94:0/4182576554 learned_addr learned my addr 172.21.15.94:0/4182576554 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5870c35640 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fd5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871436640 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 0x55c5e1fc8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871c37640 1 -- 172.21.15.94:0/4182576554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 msgr2=0x55c5e1fc8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871c37640 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 0x55c5e1fc8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871c37640 1 -- 172.21.15.94:0/4182576554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8c00 msgr2=0x55c5e1fd5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871c37640 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fd5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871c37640 1 -- 172.21.15.94:0/4182576554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5e1ebc1b0 con 0x55c5e1fd5e10 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5871436640 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 0x55c5e1fc8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.052+0000 7f5870c35640 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fd5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f5871c37640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58580a3d20 0x7f58580a61e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5864032e30 tx=0x7f5864009880 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f5871c37640 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd8220 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f5864032e60 tx=0x7f5864045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bffcaa6fca467a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f58577fe640 1 -- 172.21.15.94:0/4182576554 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864052070 con 0x55c5e1fd5e10 2024-04-21T08:50:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f58577fe640 1 -- 172.21.15.94:0/4182576554 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f586403a040 con 0x55c5e1fd5e10 2024-04-21T08:50:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f58577fe640 1 -- 172.21.15.94:0/4182576554 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864039050 con 0x55c5e1fd5e10 2024-04-21T08:50:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f587636c740 1 -- 172.21.15.94:0/4182576554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 msgr2=0x55c5e1fd8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f587636c740 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd8220 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f5864032e60 tx=0x7f5864045000 comp rx=0 tx=0).stop 2024-04-21T08:50:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f587636c740 1 -- 172.21.15.94:0/4182576554 shutdown_connections 2024-04-21T08:50:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f587636c740 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd8220 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f587636c740 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fd5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f587636c740 1 --2- 172.21.15.94:0/4182576554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 0x55c5e1fc8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f587636c740 1 -- 172.21.15.94:0/4182576554 >> 172.21.15.94:0/4182576554 conn(0x55c5e1fc2ba0 msgr2=0x55c5e1fc2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.053+0000 7f587636c740 1 -- 172.21.15.94:0/4182576554 shutdown_connections 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f587636c740 1 -- 172.21.15.94:0/4182576554 wait complete. 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f587636c740 1 -- start start 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f587636c740 1 Processor -- start 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f587636c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8250 0x55c5e1fd5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f587636c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fd2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f587636c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f587636c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c5e1fd8ce0 con 0x55c5e1fd5e10 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f587636c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c5e1ee6500 con 0x55c5e1fc8250 2024-04-21T08:50:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f587636c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c5e1fd8e60 con 0x55c5e1fc8c00 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f587636c740 10 client.0 _ll_register_callbacks cb 0x55c5e1eb4b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f5871c37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f5871c37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57836/0 (socket says 172.21.15.94:57836) 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f5871c37640 1 -- 172.21.15.94:0/451229167 learned_addr learned my addr 172.21.15.94:0/451229167 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.054+0000 7f5871436640 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8250 0x55c5e1fd5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5870c35640 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fd2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5871c37640 1 -- 172.21.15.94:0/451229167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 msgr2=0x55c5e1fd2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5871c37640 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fd2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5871c37640 1 -- 172.21.15.94:0/451229167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8250 msgr2=0x55c5e1fd5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5871c37640 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8250 0x55c5e1fd5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5871c37640 1 -- 172.21.15.94:0/451229167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5e1c3f3f0 con 0x55c5e1fd5e10 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5871436640 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8250 0x55c5e1fd5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5870c35640 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fd2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5871c37640 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd2f70 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f5864005de0 tx=0x7f58640095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c26ba4e74b0244 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5854ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5854ff9640 1 -- 172.21.15.94:0/451229167 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864052040 con 0x55c5e1fd5e10 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5854ff9640 1 -- 172.21.15.94:0/451229167 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f5864004240 con 0x55c5e1fd5e10 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f587636c740 1 -- 172.21.15.94:0/451229167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f586403a040 con 0x55c5e1fd5e10 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5854ff9640 1 -- 172.21.15.94:0/451229167 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5864064050 con 0x55c5e1fd5e10 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f587636c740 10 client.18189 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f587636c740 1 -- 172.21.15.94:0/451229167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c5e1d097d0 con 0x55c5e1fd5e10 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5854ff9640 1 -- 172.21.15.94:0/451229167 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f586403a040 con 0x55c5e1fd5e10 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f5854ff9640 1 -- 172.21.15.94:0/451229167 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5864034cc0 con 0x55c5e1fd5e10 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f587636c740 10 client.18189 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:18.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f587636c740 10 client.18189 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f587636c740 20 client.18189 populate_metadata read hostname 'smithi094' 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f587636c740 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c5e1fe5d30 0x55c5e1fe8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.055+0000 7f587636c740 4 client.18189 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.056+0000 7f587636c740 1 -- 172.21.15.94:0/451229167 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5e1ecc930 con 0x55c5e1fe5d30 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.056+0000 7f5871436640 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c5e1fe5d30 0x55c5e1fe8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.056+0000 7f5854ff9640 10 client.18189 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.056+0000 7f5871436640 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c5e1fe5d30 0x55c5e1fe8120 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ffc3ea88afa9bf17 server_cookie=f5dec897930f50a1 in_seq=0 out_seq=0 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f5854ff9640 1 -- 172.21.15.94:0/451229167 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c5e1ecc930 con 0x55c5e1fe5d30 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f5854ff9640 10 client.18189 handle_command_reply: tid=0 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f587636c740 1 client.18189 shutdown 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f587636c740 1 -- 172.21.15.94:0/451229167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 msgr2=0x55c5e1fd2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f587636c740 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd2f70 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f5864005de0 tx=0x7f58640095c0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f587636c740 1 -- 172.21.15.94:0/451229167 shutdown_connections 2024-04-21T08:50:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f587636c740 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c5e1fe5d30 0x55c5e1fe8120 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f587636c740 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fd2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f5854ff9640 0 client.18189 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:50:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f587636c740 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fd2f70 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f587636c740 1 --2- 172.21.15.94:0/451229167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fc8250 0x55c5e1fd5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.057+0000 7f587636c740 1 -- 172.21.15.94:0/451229167 >> 172.21.15.94:0/451229167 conn(0x55c5e1fc2ba0 msgr2=0x55c5e1fc48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.058+0000 7f587636c740 1 -- 172.21.15.94:0/451229167 shutdown_connections 2024-04-21T08:50:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.058+0000 7f587636c740 1 -- 172.21.15.94:0/451229167 wait complete. 2024-04-21T08:50:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.058+0000 7f587636c740 20 client.18189 trim_cache size 0 max 16384 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f587636c740 1 Processor -- start 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f587636c740 1 -- start start 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f587636c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8250 0x55c5e1fbdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f587636c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fe1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f587636c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fed750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f587636c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f586c1afc30 con 0x55c5e1fc8250 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f587636c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f586c1ad500 con 0x55c5e1fd5e10 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f587636c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f586c1ab370 con 0x55c5e1fc8c00 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871436640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8250 0x55c5e1fbdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871436640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8250 0x55c5e1fbdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57846/0 (socket says 172.21.15.94:57846) 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871436640 1 -- 172.21.15.94:0/2918915900 learned_addr learned my addr 172.21.15.94:0/2918915900 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5870c35640 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fe1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871c37640 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fed750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871436640 1 -- 172.21.15.94:0/2918915900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 msgr2=0x55c5e1fe1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871436640 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fe1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871436640 1 -- 172.21.15.94:0/2918915900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 msgr2=0x55c5e1fed750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871436640 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fed750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871436640 1 -- 172.21.15.94:0/2918915900 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5e1fbe5f0 con 0x55c5e1fc8250 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5870c35640 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fe1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871c37640 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fed750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5871436640 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8250 0x55c5e1fbdec0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f5868005ef0 tx=0x7f5868005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bb85ee9dc11dc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5856ffd640 1 -- 172.21.15.94:0/2918915900 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586801c030 con 0x55c5e1fc8250 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.071+0000 7f5856ffd640 1 -- 172.21.15.94:0/2918915900 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5868018c80 con 0x55c5e1fc8250 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f5856ffd640 1 -- 172.21.15.94:0/2918915900 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586800e670 con 0x55c5e1fc8250 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 -- 172.21.15.94:0/2918915900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8250 msgr2=0x55c5e1fbdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8250 0x55c5e1fbdec0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f5868005ef0 tx=0x7f5868005e80 comp rx=0 tx=0).stop 2024-04-21T08:50:18.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 -- 172.21.15.94:0/2918915900 shutdown_connections 2024-04-21T08:50:18.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8c00 0x55c5e1fe1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fed750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 --2- 172.21.15.94:0/2918915900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8250 0x55c5e1fbdec0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 -- 172.21.15.94:0/2918915900 >> 172.21.15.94:0/2918915900 conn(0x55c5e1fc2ba0 msgr2=0x55c5e1fc4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 -- 172.21.15.94:0/2918915900 shutdown_connections 2024-04-21T08:50:18.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 -- 172.21.15.94:0/2918915900 wait complete. 2024-04-21T08:50:18.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 -- start start 2024-04-21T08:50:18.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.072+0000 7f587636c740 1 Processor -- start 2024-04-21T08:50:18.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f587636c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 0x55c5e20443b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f587636c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8c00 0x55c5e20448f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f587636c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fdb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f587636c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f586c1ab9c0 con 0x55c5e1fc8c00 2024-04-21T08:50:18.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f587636c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f586c1ace30 con 0x55c5e1fd5e10 2024-04-21T08:50:18.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f587636c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f586c1ac090 con 0x55c5e1fc8250 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f587636c740 10 client.0 _ll_register_callbacks cb 0x7f5864004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5870c35640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8c00 0x55c5e20448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5870c35640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8c00 0x55c5e20448f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57850/0 (socket says 172.21.15.94:57850) 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5870c35640 1 -- 172.21.15.94:0/2222350964 learned_addr learned my addr 172.21.15.94:0/2222350964 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5871c37640 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fdb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5871436640 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 0x55c5e20443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5870c35640 1 -- 172.21.15.94:0/2222350964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 msgr2=0x55c5e20443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5870c35640 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 0x55c5e20443b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5870c35640 1 -- 172.21.15.94:0/2222350964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 msgr2=0x55c5e1fdb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5870c35640 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fdb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5870c35640 1 -- 172.21.15.94:0/2222350964 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5e1fd8200 con 0x55c5e1fc8c00 2024-04-21T08:50:18.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5871c37640 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fdb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:18.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5870c35640 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8c00 0x55c5e20448f0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f585c0cd9e0 tx=0x7f585c09c240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b5875fa8c1ba59 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5854ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:50:18.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.073+0000 7f5854ff9640 1 -- 172.21.15.94:0/2222350964 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c020040 con 0x55c5e1fc8c00 2024-04-21T08:50:18.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f587636c740 1 -- 172.21.15.94:0/2222350964 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5868018c80 con 0x55c5e1fc8c00 2024-04-21T08:50:18.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f5854ff9640 1 -- 172.21.15.94:0/2222350964 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f585c0987f0 con 0x55c5e1fc8c00 2024-04-21T08:50:18.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f5854ff9640 1 -- 172.21.15.94:0/2222350964 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f585c0bb120 con 0x55c5e1fc8c00 2024-04-21T08:50:18.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f587636c740 10 client.18195 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:18.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f587636c740 1 -- 172.21.15.94:0/2222350964 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c5e204b9d0 con 0x55c5e1fc8c00 2024-04-21T08:50:18.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f5854ff9640 1 -- 172.21.15.94:0/2222350964 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5868018c80 con 0x55c5e1fc8c00 2024-04-21T08:50:18.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f5854ff9640 1 -- 172.21.15.94:0/2222350964 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f585c09b760 con 0x55c5e1fc8c00 2024-04-21T08:50:18.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f587636c740 10 client.18195 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:18.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f587636c740 10 client.18195 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T08:50:18.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f587636c740 20 client.18195 populate_metadata read hostname 'smithi094' 2024-04-21T08:50:18.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f587636c740 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c5e1fe5d30 0x55c5e204e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f587636c740 4 client.18195 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f587636c740 1 -- 172.21.15.94:0/2222350964 --> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c5e1f3ac10 con 0x55c5e1fe5d30 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.074+0000 7f5871436640 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c5e1fe5d30 0x55c5e204e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f5871436640 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c5e1fe5d30 0x55c5e204e190 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c06d06f5ea16f955 server_cookie=5aa4701058e239d8 in_seq=0 out_seq=0 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f5854ff9640 10 client.18195 ms_handle_connect on v2:172.21.15.111:6836/550365815 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f5854ff9640 1 -- 172.21.15.94:0/2222350964 <== mds.2 v2:172.21.15.111:6836/550365815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c5e1f3ac10 con 0x55c5e1fe5d30 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f5854ff9640 10 client.18195 handle_command_reply: tid=0 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 client.18195 shutdown 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 -- 172.21.15.94:0/2222350964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8c00 msgr2=0x55c5e20448f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8c00 0x55c5e20448f0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f585c0cd9e0 tx=0x7f585c09c240 comp rx=0 tx=0).stop 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 -- 172.21.15.94:0/2222350964 shutdown_connections 2024-04-21T08:50:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.111:6836/550365815,v1:172.21.15.111:6837/550365815] conn(0x55c5e1fe5d30 0x55c5e204e190 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c5e1fd5e10 0x55c5e1fdb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f5854ff9640 0 client.18195 ms_handle_reset on v2:172.21.15.111:6836/550365815 2024-04-21T08:50:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c5e1fc8c00 0x55c5e20448f0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 --2- 172.21.15.94:0/2222350964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5e1fc8250 0x55c5e20443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 -- 172.21.15.94:0/2222350964 >> 172.21.15.94:0/2222350964 conn(0x55c5e1fc2ba0 msgr2=0x55c5e1fc3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 -- 172.21.15.94:0/2222350964 shutdown_connections 2024-04-21T08:50:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 1 -- 172.21.15.94:0/2222350964 wait complete. 2024-04-21T08:50:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.075+0000 7f587636c740 20 client.18195 trim_cache size 0 max 16384 2024-04-21T08:50:18.086 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:50:18.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 -- 172.21.15.94:0/470490789 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58580a3d20 msgr2=0x7f58580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58580a3d20 0x7f58580a61e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5864032e30 tx=0x7f5864009880 comp rx=0 tx=0).stop 2024-04-21T08:50:18.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 -- 172.21.15.94:0/470490789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f586c107260 msgr2=0x7f586c198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f586c107260 0x7f586c198c70 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f585c00d740 tx=0x7f585c00dc00 comp rx=0 tx=0).stop 2024-04-21T08:50:18.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 -- 172.21.15.94:0/470490789 shutdown_connections 2024-04-21T08:50:18.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58580a3d20 0x7f58580a61e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c19cff0 0x7f586c19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f586c107260 0x7f586c198c70 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 --2- 172.21.15.94:0/470490789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586c102190 0x7f586c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 -- 172.21.15.94:0/470490789 >> 172.21.15.94:0/470490789 conn(0x7f586c0793a0 msgr2=0x7f586c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 -- 172.21.15.94:0/470490789 shutdown_connections 2024-04-21T08:50:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.076+0000 7f587369a640 1 -- 172.21.15.94:0/470490789 wait complete. 2024-04-21T08:50:18.089 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:50:18.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:17 smithi094 ceph-mon[27988]: pgmap v320: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 10 op/s 2024-04-21T08:50:18.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:17 smithi111 ceph-mon[31493]: pgmap v320: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 10 op/s 2024-04-21T08:50:18.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/1930086349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb98101a40 msgr2=0x7ffb98105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 --2- 172.21.15.94:0/1930086349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb98101a40 0x7ffb98105a80 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7ffb80009900 tx=0x7ffb8002bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:18.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/1930086349 shutdown_connections 2024-04-21T08:50:18.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 --2- 172.21.15.94:0/1930086349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb98106160 0x7ffb9810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 --2- 172.21.15.94:0/1930086349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb98101a40 0x7ffb98105a80 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 --2- 172.21.15.94:0/1930086349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98101090 0x7ffb98101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/1930086349 >> 172.21.15.94:0/1930086349 conn(0x7ffb98077f50 msgr2=0x7ffb98078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/1930086349 shutdown_connections 2024-04-21T08:50:18.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/1930086349 wait complete. 2024-04-21T08:50:18.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 Processor -- start 2024-04-21T08:50:18.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.190+0000 7ffb9ccc1640 1 -- start start 2024-04-21T08:50:18.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb9ccc1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98101090 0x7ffb98197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb9ccc1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb98106160 0x7ffb98197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb9ccc1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9819bf20 0x7ffb9819c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb9ccc1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffb981110a0 con 0x7ffb9819bf20 2024-04-21T08:50:18.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb9ccc1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffb98110f20 con 0x7ffb98101090 2024-04-21T08:50:18.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb9ccc1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffb98111220 con 0x7ffb98106160 2024-04-21T08:50:18.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb96d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9819bf20 0x7ffb9819c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb96d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9819bf20 0x7ffb9819c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57868/0 (socket says 172.21.15.94:57868) 2024-04-21T08:50:18.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb96d9d640 1 -- 172.21.15.94:0/2719359654 learned_addr learned my addr 172.21.15.94:0/2719359654 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb8ffff640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb98106160 0x7ffb98197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb9659c640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98101090 0x7ffb98197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb96d9d640 1 -- 172.21.15.94:0/2719359654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb98106160 msgr2=0x7ffb98197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb96d9d640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb98106160 0x7ffb98197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb96d9d640 1 -- 172.21.15.94:0/2719359654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98101090 msgr2=0x7ffb98197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb96d9d640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98101090 0x7ffb98197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb96d9d640 1 -- 172.21.15.94:0/2719359654 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9819ca10 con 0x7ffb9819bf20 2024-04-21T08:50:18.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb9659c640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98101090 0x7ffb98197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:18.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb96d9d640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9819bf20 0x7ffb9819c3d0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000eaa0 tx=0x7ffb9000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb8f7fe640 1 -- 172.21.15.94:0/2719359654 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9000cdb0 con 0x7ffb9819bf20 2024-04-21T08:50:18.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/2719359654 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb9819cd00 con 0x7ffb9819bf20 2024-04-21T08:50:18.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb8f7fe640 1 -- 172.21.15.94:0/2719359654 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb90005020 con 0x7ffb9819bf20 2024-04-21T08:50:18.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb8f7fe640 1 -- 172.21.15.94:0/2719359654 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb900116d0 con 0x7ffb9819bf20 2024-04-21T08:50:18.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.191+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/2719359654 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb981a44c0 con 0x7ffb9819bf20 2024-04-21T08:50:18.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.193+0000 7ffb8f7fe640 1 -- 172.21.15.94:0/2719359654 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7ffb900051d0 con 0x7ffb9819bf20 2024-04-21T08:50:18.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb9f993740 1 Processor -- start 2024-04-21T08:50:18.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb9f993740 1 -- start start 2024-04-21T08:50:18.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb8f7fe640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffb840a3d20 0x7ffb840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa4884d0 0x55c4fa4888b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa488e80 0x55c4fa491430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa491a00 0x55c4fa493e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb9f993740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c4fa3ed1f0 con 0x55c4fa488e80 2024-04-21T08:50:18.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb9f993740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c4fa38a4b0 con 0x55c4fa4884d0 2024-04-21T08:50:18.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb9f993740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c4fa3e91b0 con 0x55c4fa491a00 2024-04-21T08:50:18.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb8ffff640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffb840a3d20 0x7ffb840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.194+0000 7ffb8f7fe640 1 -- 172.21.15.94:0/2719359654 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7ffb900d0d30 con 0x7ffb9819bf20 2024-04-21T08:50:18.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa488e80 0x55c4fa491430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa488e80 0x55c4fa491430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57872/0 (socket says 172.21.15.94:57872) 2024-04-21T08:50:18.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8ffff640 1 -- 172.21.15.94:0/1023210847 learned_addr learned my addr 172.21.15.94:0/1023210847 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb9659c640 1 --2- 172.21.15.94:0/1023210847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa4884d0 0x55c4fa4888b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb96d9d640 1 --2- 172.21.15.94:0/1023210847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa491a00 0x55c4fa493e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8ffff640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffb840a3d20 0x7ffb840a61e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffb80005ec0 tx=0x7ffb80004310 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8ffff640 1 -- 172.21.15.94:0/1023210847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa491a00 msgr2=0x55c4fa493e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8ffff640 1 --2- 172.21.15.94:0/1023210847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa491a00 0x55c4fa493e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8ffff640 1 -- 172.21.15.94:0/1023210847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa4884d0 msgr2=0x55c4fa4888b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8ffff640 1 --2- 172.21.15.94:0/1023210847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa4884d0 0x55c4fa4888b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8ffff640 1 -- 172.21.15.94:0/1023210847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4fa37c1b0 con 0x55c4fa488e80 2024-04-21T08:50:18.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb9659c640 1 --2- 172.21.15.94:0/1023210847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa4884d0 0x55c4fa4888b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:18.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8ffff640 1 --2- 172.21.15.94:0/1023210847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa488e80 0x55c4fa491430 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7ffb80009900 tx=0x7ffb8004a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6c489ffe7a53ea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8cff9640 1 -- 172.21.15.94:0/1023210847 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb80049070 con 0x55c4fa488e80 2024-04-21T08:50:18.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8cff9640 1 -- 172.21.15.94:0/1023210847 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb80052070 con 0x55c4fa488e80 2024-04-21T08:50:18.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.195+0000 7ffb8cff9640 1 -- 172.21.15.94:0/1023210847 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb80004e40 con 0x55c4fa488e80 2024-04-21T08:50:18.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 -- 172.21.15.94:0/1023210847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa488e80 msgr2=0x55c4fa491430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 --2- 172.21.15.94:0/1023210847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa488e80 0x55c4fa491430 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7ffb80009900 tx=0x7ffb8004a000 comp rx=0 tx=0).stop 2024-04-21T08:50:18.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 -- 172.21.15.94:0/1023210847 shutdown_connections 2024-04-21T08:50:18.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 --2- 172.21.15.94:0/1023210847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa491a00 0x55c4fa493e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 --2- 172.21.15.94:0/1023210847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa488e80 0x55c4fa491430 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 --2- 172.21.15.94:0/1023210847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa4884d0 0x55c4fa4888b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 -- 172.21.15.94:0/1023210847 >> 172.21.15.94:0/1023210847 conn(0x55c4fa482ba0 msgr2=0x55c4fa482f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 -- 172.21.15.94:0/1023210847 shutdown_connections 2024-04-21T08:50:18.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 -- 172.21.15.94:0/1023210847 wait complete. 2024-04-21T08:50:18.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 -- start start 2024-04-21T08:50:18.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.196+0000 7ffb9f993740 1 Processor -- start 2024-04-21T08:50:18.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa4884d0 0x55c4fa48cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa488e80 0x55c4fa48d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa48d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9f993740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55c4fa4947d0 con 0x55c4fa491a00 2024-04-21T08:50:18.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9f993740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c4fa3a6500 con 0x55c4fa488e80 2024-04-21T08:50:18.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9f993740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c4fa494950 con 0x55c4fa4884d0 2024-04-21T08:50:18.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9f993740 10 client.0 _ll_register_callbacks cb 0x55c4fa374b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:18.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb96d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa48d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb96d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa48d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57886/0 (socket says 172.21.15.94:57886) 2024-04-21T08:50:18.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb96d9d640 1 -- 172.21.15.94:0/1928480391 learned_addr learned my addr 172.21.15.94:0/1928480391 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9659c640 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa4884d0 0x55c4fa48cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb96d9d640 1 -- 172.21.15.94:0/1928480391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa4884d0 msgr2=0x55c4fa48cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb96d9d640 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa4884d0 0x55c4fa48cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb96d9d640 1 -- 172.21.15.94:0/1928480391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa488e80 msgr2=0x55c4fa48d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb96d9d640 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa488e80 0x55c4fa48d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb96d9d640 1 -- 172.21.15.94:0/1928480391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4fa0ff3f0 con 0x55c4fa491a00 2024-04-21T08:50:18.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb96d9d640 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa48d5a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7ffb900ce100 tx=0x7ffb9009c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e13ff6f6d930ad server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb7dffb640 1 -- 172.21.15.94:0/1928480391 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9001f040 con 0x55c4fa491a00 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb7dffb640 1 -- 172.21.15.94:0/1928480391 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7ffb900870b0 con 0x55c4fa491a00 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9f993740 1 -- 172.21.15.94:0/1928480391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb80052070 con 0x55c4fa491a00 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb7dffb640 1 -- 172.21.15.94:0/1928480391 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffb9009dbd0 con 0x55c4fa491a00 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9f993740 10 client.18207 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb9f993740 1 -- 172.21.15.94:0/1928480391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4fa1c97d0 con 0x55c4fa491a00 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.197+0000 7ffb7dffb640 1 -- 172.21.15.94:0/1928480391 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb80052070 con 0x55c4fa491a00 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.198+0000 7ffb7dffb640 1 -- 172.21.15.94:0/1928480391 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ffb9000f040 con 0x55c4fa491a00 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.198+0000 7ffb9f993740 10 client.18207 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.198+0000 7ffb9f993740 10 client.18207 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.198+0000 7ffb9f993740 20 client.18207 populate_metadata read hostname 'smithi094' 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.198+0000 7ffb9f993740 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c4fa4a1be0 0x55c4fa4a4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.198+0000 7ffb9f993740 4 client.18207 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:18.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.198+0000 7ffb9f993740 1 -- 172.21.15.94:0/1928480391 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4fa38c930 con 0x55c4fa4a1be0 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.198+0000 7ffb9659c640 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c4fa4a1be0 0x55c4fa4a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9659c640 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c4fa4a1be0 0x55c4fa4a4080 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a1c3a43e5f9be70a server_cookie=2d9468d21e8a74c1 in_seq=0 out_seq=0 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb7dffb640 10 client.18207 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb7dffb640 1 -- 172.21.15.94:0/1928480391 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c4fa38c930 con 0x55c4fa4a1be0 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb7dffb640 10 client.18207 handle_command_reply: tid=0 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9f993740 1 client.18207 shutdown 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9f993740 1 -- 172.21.15.94:0/1928480391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 msgr2=0x55c4fa48d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9f993740 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa48d5a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7ffb900ce100 tx=0x7ffb9009c9e0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9f993740 1 -- 172.21.15.94:0/1928480391 shutdown_connections 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9f993740 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c4fa4a1be0 0x55c4fa4a4080 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9f993740 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa48d5a0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb7dffb640 0 client.18207 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9f993740 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa488e80 0x55c4fa48d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9f993740 1 --2- 172.21.15.94:0/1928480391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa4884d0 0x55c4fa48cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9f993740 1 -- 172.21.15.94:0/1928480391 >> 172.21.15.94:0/1928480391 conn(0x55c4fa482ba0 msgr2=0x55c4fa4848f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.199+0000 7ffb9f993740 1 -- 172.21.15.94:0/1928480391 shutdown_connections 2024-04-21T08:50:18.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.200+0000 7ffb9f993740 1 -- 172.21.15.94:0/1928480391 wait complete. 2024-04-21T08:50:18.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.200+0000 7ffb9f993740 20 client.18207 trim_cache size 0 max 16384 2024-04-21T08:50:18.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb9f993740 1 Processor -- start 2024-04-21T08:50:18.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb9f993740 1 -- start start 2024-04-21T08:50:18.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa4884d0 0x55c4fa491260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa488e80 0x55c4fa495b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa484700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb9f993740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffb981afcb0 con 0x55c4fa491a00 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb9f993740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffb981ad580 con 0x55c4fa4884d0 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb9f993740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffb981ab3f0 con 0x55c4fa488e80 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb96d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa484700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb96d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa484700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57898/0 (socket says 172.21.15.94:57898) 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb96d9d640 1 -- 172.21.15.94:0/3553498584 learned_addr learned my addr 172.21.15.94:0/3553498584 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb96d9d640 1 -- 172.21.15.94:0/3553498584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa488e80 msgr2=0x55c4fa495b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb96d9d640 1 --2- 172.21.15.94:0/3553498584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa488e80 0x55c4fa495b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb96d9d640 1 -- 172.21.15.94:0/3553498584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa4884d0 msgr2=0x55c4fa491260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb96d9d640 1 --2- 172.21.15.94:0/3553498584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa4884d0 0x55c4fa491260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb96d9d640 1 -- 172.21.15.94:0/3553498584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4fa48b070 con 0x55c4fa491a00 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.213+0000 7ffb96d9d640 1 --2- 172.21.15.94:0/3553498584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa484700 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7ffb90098d00 tx=0x7ffb90098a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33970884b3053daa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb7ffff640 1 -- 172.21.15.94:0/3553498584 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb900870b0 con 0x55c4fa491a00 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb7ffff640 1 -- 172.21.15.94:0/3553498584 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb90087250 con 0x55c4fa491a00 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb7ffff640 1 -- 172.21.15.94:0/3553498584 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90098040 con 0x55c4fa491a00 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb9f993740 1 -- 172.21.15.94:0/3553498584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 msgr2=0x55c4fa484700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb9f993740 1 --2- 172.21.15.94:0/3553498584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa484700 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7ffb90098d00 tx=0x7ffb90098a80 comp rx=0 tx=0).stop 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb9f993740 1 -- 172.21.15.94:0/3553498584 shutdown_connections 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb9f993740 1 --2- 172.21.15.94:0/3553498584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa488e80 0x55c4fa495b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb9f993740 1 --2- 172.21.15.94:0/3553498584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa491a00 0x55c4fa484700 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb9f993740 1 --2- 172.21.15.94:0/3553498584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa4884d0 0x55c4fa491260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb9f993740 1 -- 172.21.15.94:0/3553498584 >> 172.21.15.94:0/3553498584 conn(0x55c4fa482ba0 msgr2=0x55c4fa483ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb9f993740 1 -- 172.21.15.94:0/3553498584 shutdown_connections 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb9f993740 1 -- 172.21.15.94:0/3553498584 wait complete. 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.214+0000 7ffb9f993740 1 -- start start 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9f993740 1 Processor -- start 2024-04-21T08:50:18.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa4884d0 0x55c4fa489520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa488e80 0x55c4fa489a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9f993740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa491a00 0x55c4fa49c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9f993740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffb981aba40 con 0x55c4fa4884d0 2024-04-21T08:50:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9f993740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffb981aceb0 con 0x55c4fa491a00 2024-04-21T08:50:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9f993740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffb981ac110 con 0x55c4fa488e80 2024-04-21T08:50:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9f993740 10 client.0 _ll_register_callbacks cb 0x7ffb9009d010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa4884d0 0x55c4fa489520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa4884d0 0x55c4fa489520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57902/0 (socket says 172.21.15.94:57902) 2024-04-21T08:50:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9659c640 1 -- 172.21.15.94:0/2038568469 learned_addr learned my addr 172.21.15.94:0/2038568469 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9659c640 1 -- 172.21.15.94:0/2038568469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa488e80 msgr2=0x55c4fa489a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9659c640 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa488e80 0x55c4fa489a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9659c640 1 -- 172.21.15.94:0/2038568469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa491a00 msgr2=0x55c4fa49c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9659c640 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa491a00 0x55c4fa49c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9659c640 1 -- 172.21.15.94:0/2038568469 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4fa49cef0 con 0x55c4fa4884d0 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9659c640 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa4884d0 0x55c4fa489520 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x55c4fa4a4890 tx=0x7ffb88014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6faffd24d76ad681 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb7dffb640 1 -- 172.21.15.94:0/2038568469 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb88004230 con 0x55c4fa4884d0 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb7dffb640 1 -- 172.21.15.94:0/2038568469 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7ffb880043d0 con 0x55c4fa4884d0 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.215+0000 7ffb9f993740 1 -- 172.21.15.94:0/2038568469 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb90087250 con 0x55c4fa4884d0 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb7dffb640 1 -- 172.21.15.94:0/2038568469 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffb8802d050 con 0x55c4fa4884d0 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb9f993740 10 client.18213 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb9f993740 1 -- 172.21.15.94:0/2038568469 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4fa508af0 con 0x55c4fa4884d0 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb7dffb640 1 -- 172.21.15.94:0/2038568469 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb90087250 con 0x55c4fa4884d0 2024-04-21T08:50:18.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb7dffb640 1 -- 172.21.15.94:0/2038568469 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ffb8800f040 con 0x55c4fa4884d0 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb9f993740 10 client.18213 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb9f993740 10 client.18213 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb9f993740 20 client.18213 populate_metadata read hostname 'smithi094' 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb9f993740 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c4fa5092c0 0x55c4fa50b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb9f993740 4 client.18213 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb9f993740 1 -- 172.21.15.94:0/2038568469 --> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4fa3fac10 con 0x55c4fa5092c0 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.216+0000 7ffb8ffff640 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c4fa5092c0 0x55c4fa50b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb7dffb640 10 client.18213 ms_handle_connect on v2:172.21.15.173:6838/2987518473 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb8ffff640 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c4fa5092c0 0x55c4fa50b720 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1a5d40dd57267539 server_cookie=8a80552fb4b247d7 in_seq=0 out_seq=0 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb7dffb640 1 -- 172.21.15.94:0/2038568469 <== mds.1 v2:172.21.15.173:6838/2987518473 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4fa3fac10 con 0x55c4fa5092c0 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb7dffb640 10 client.18213 handle_command_reply: tid=0 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 client.18213 shutdown 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 -- 172.21.15.94:0/2038568469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa4884d0 msgr2=0x55c4fa489520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa4884d0 0x55c4fa489520 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x55c4fa4a4890 tx=0x7ffb88014840 comp rx=0 tx=0).stop 2024-04-21T08:50:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 -- 172.21.15.94:0/2038568469 shutdown_connections 2024-04-21T08:50:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c4fa488e80 0x55c4fa489a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.173:6838/2987518473,v1:172.21.15.173:6839/2987518473] conn(0x55c4fa5092c0 0x55c4fa50b720 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fa491a00 0x55c4fa49c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb7dffb640 0 client.18213 ms_handle_reset on v2:172.21.15.173:6838/2987518473 2024-04-21T08:50:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 --2- 172.21.15.94:0/2038568469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4fa4884d0 0x55c4fa489520 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 -- 172.21.15.94:0/2038568469 >> 172.21.15.94:0/2038568469 conn(0x55c4fa482ba0 msgr2=0x55c4fa481c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 -- 172.21.15.94:0/2038568469 shutdown_connections 2024-04-21T08:50:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 1 -- 172.21.15.94:0/2038568469 wait complete. 2024-04-21T08:50:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9f993740 20 client.18213 trim_cache size 0 max 16384 2024-04-21T08:50:18.230 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:50:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/2719359654 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffb840a3d20 msgr2=0x7ffb840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffb840a3d20 0x7ffb840a61e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffb80005ec0 tx=0x7ffb80004310 comp rx=0 tx=0).stop 2024-04-21T08:50:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/2719359654 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9819bf20 msgr2=0x7ffb9819c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9819bf20 0x7ffb9819c3d0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000eaa0 tx=0x7ffb9000ef60 comp rx=0 tx=0).stop 2024-04-21T08:50:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/2719359654 shutdown_connections 2024-04-21T08:50:18.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7ffb840a3d20 0x7ffb840a61e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb98106160 0x7ffb98197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9819bf20 0x7ffb9819c3d0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 --2- 172.21.15.94:0/2719359654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98101090 0x7ffb98197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/2719359654 >> 172.21.15.94:0/2719359654 conn(0x7ffb98077f50 msgr2=0x7ffb980ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.217+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/2719359654 shutdown_connections 2024-04-21T08:50:18.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.218+0000 7ffb9ccc1640 1 -- 172.21.15.94:0/2719359654 wait complete. 2024-04-21T08:50:18.232 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:50:18.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:17 smithi173 ceph-mon[31428]: pgmap v320: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 10 op/s 2024-04-21T08:50:18.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 -- 172.21.15.94:0/1486755723 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63ac102190 msgr2=0x7f63ac102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 --2- 172.21.15.94:0/1486755723 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63ac102190 0x7f63ac102570 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009900 tx=0x7f63a802bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:18.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 -- 172.21.15.94:0/1486755723 shutdown_connections 2024-04-21T08:50:18.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 --2- 172.21.15.94:0/1486755723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63ac102b40 0x7f63ac106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 --2- 172.21.15.94:0/1486755723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac107260 0x7f63ac10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 --2- 172.21.15.94:0/1486755723 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63ac102190 0x7f63ac102570 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 -- 172.21.15.94:0/1486755723 >> 172.21.15.94:0/1486755723 conn(0x7f63ac0fd990 msgr2=0x7f63ac0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 -- 172.21.15.94:0/1486755723 shutdown_connections 2024-04-21T08:50:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 -- 172.21.15.94:0/1486755723 wait complete. 2024-04-21T08:50:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 Processor -- start 2024-04-21T08:50:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 -- start start 2024-04-21T08:50:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63ac102190 0x7f63ac198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63ac107260 0x7f63ac198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac19cfe0 0x7f63ac19d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1121c0 con 0x7f63ac102190 2024-04-21T08:50:18.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac112040 con 0x7f63ac19cfe0 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b40f2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac112340 con 0x7f63ac107260 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.373+0000 7f63b1e8e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63ac102190 0x7f63ac198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b1e8e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63ac102190 0x7f63ac198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57908/0 (socket says 172.21.15.94:57908) 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b1e8e640 1 -- 172.21.15.94:0/1795100063 learned_addr learned my addr 172.21.15.94:0/1795100063 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b168d640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63ac107260 0x7f63ac198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b268f640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac19cfe0 0x7f63ac19d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b1e8e640 1 -- 172.21.15.94:0/1795100063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63ac107260 msgr2=0x7f63ac198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63ac107260 0x7f63ac198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b1e8e640 1 -- 172.21.15.94:0/1795100063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac19cfe0 msgr2=0x7f63ac19d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac19cfe0 0x7f63ac19d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b1e8e640 1 -- 172.21.15.94:0/1795100063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ac19dad0 con 0x7f63ac102190 2024-04-21T08:50:18.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b168d640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63ac107260 0x7f63ac198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:18.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63ac102190 0x7f63ac198720 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f63a8005940 tx=0x7f63a802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63a2ffd640 1 -- 172.21.15.94:0/1795100063 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a8043020 con 0x7f63ac102190 2024-04-21T08:50:18.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b40f2640 1 -- 172.21.15.94:0/1795100063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ac19dd60 con 0x7f63ac102190 2024-04-21T08:50:18.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63a2ffd640 1 -- 172.21.15.94:0/1795100063 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a802cbe0 con 0x7f63ac102190 2024-04-21T08:50:18.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63a2ffd640 1 -- 172.21.15.94:0/1795100063 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a8003bf0 con 0x7f63ac102190 2024-04-21T08:50:18.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.374+0000 7f63b40f2640 1 -- 172.21.15.94:0/1795100063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ac1a5580 con 0x7f63ac102190 2024-04-21T08:50:18.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.377+0000 7f63b6dc1740 1 Processor -- start 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.377+0000 7f63a2ffd640 1 -- 172.21.15.94:0/1795100063 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f63a8046020 con 0x7f63ac102190 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.377+0000 7f63b6dc1740 1 -- start start 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.377+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 0x5626663db8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.377+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663e4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.377+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663e4a00 0x5626663e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.377+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5626663401f0 con 0x5626663dbe80 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.377+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5626662dd4b0 con 0x5626663db4d0 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.377+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56266633c1b0 con 0x5626663e4a00 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b1e8e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663e4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b1e8e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663e4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57924/0 (socket says 172.21.15.94:57924) 2024-04-21T08:50:18.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b1e8e640 1 -- 172.21.15.94:0/1734294988 learned_addr learned my addr 172.21.15.94:0/1734294988 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b268f640 1 --2- 172.21.15.94:0/1734294988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 0x5626663db8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b168d640 1 --2- 172.21.15.94:0/1734294988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663e4a00 0x5626663e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b1e8e640 1 -- 172.21.15.94:0/1734294988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663e4a00 msgr2=0x5626663e6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/1734294988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663e4a00 0x5626663e6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b1e8e640 1 -- 172.21.15.94:0/1734294988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 msgr2=0x5626663db8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/1734294988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 0x5626663db8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b1e8e640 1 -- 172.21.15.94:0/1734294988 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626662cf1b0 con 0x5626663dbe80 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b268f640 1 --2- 172.21.15.94:0/1734294988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 0x5626663db8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63a2ffd640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63980a3d20 0x7f63980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/1734294988 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663e4430 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f63a80f01b0 tx=0x7f63a80f6ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7914c4ac20a6e3cf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63b268f640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63980a3d20 0x7f63980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f63a2ffd640 1 -- 172.21.15.94:0/1795100063 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f63a80f2e10 con 0x7f63ac102190 2024-04-21T08:50:18.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f6397fff640 1 -- 172.21.15.94:0/1734294988 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a80bad30 con 0x5626663dbe80 2024-04-21T08:50:18.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f6397fff640 1 -- 172.21.15.94:0/1734294988 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a80bf190 con 0x5626663dbe80 2024-04-21T08:50:18.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.378+0000 7f6397fff640 1 -- 172.21.15.94:0/1734294988 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a80be3f0 con 0x5626663dbe80 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 -- 172.21.15.94:0/1734294988 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 msgr2=0x5626663e4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/1734294988 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663e4430 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f63a80f01b0 tx=0x7f63a80f6ec0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b268f640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63980a3d20 0x7f63980a61e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f63a40073f0 tx=0x7f63a4007380 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 -- 172.21.15.94:0/1734294988 shutdown_connections 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/1734294988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663e4a00 0x5626663e6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/1734294988 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663e4430 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/1734294988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 0x5626663db8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 -- 172.21.15.94:0/1734294988 >> 172.21.15.94:0/1734294988 conn(0x5626663d5ba0 msgr2=0x5626663d5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 -- 172.21.15.94:0/1734294988 shutdown_connections 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 -- 172.21.15.94:0/1734294988 wait complete. 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 -- start start 2024-04-21T08:50:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.379+0000 7f63b6dc1740 1 Processor -- start 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663db4d0 0x5626663e4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663dbe80 0x5626663e16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663e4a00 0x5626663e1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x5626663e7730 con 0x5626663db4d0 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5626662f9500 con 0x5626663dbe80 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5626663e78b0 con 0x5626663e4a00 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b6dc1740 10 client.0 _ll_register_callbacks cb 0x5626662c7b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b1e8e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663db4d0 0x5626663e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b1e8e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663db4d0 0x5626663e4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57936/0 (socket says 172.21.15.94:57936) 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b1e8e640 1 -- 172.21.15.94:0/175586712 learned_addr learned my addr 172.21.15.94:0/175586712 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b168d640 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663dbe80 0x5626663e16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b268f640 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663e4a00 0x5626663e1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b1e8e640 1 -- 172.21.15.94:0/175586712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663e4a00 msgr2=0x5626663e1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663e4a00 0x5626663e1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b1e8e640 1 -- 172.21.15.94:0/175586712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663dbe80 msgr2=0x5626663e16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663dbe80 0x5626663e16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b1e8e640 1 -- 172.21.15.94:0/175586712 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626660523f0 con 0x5626663db4d0 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663db4d0 0x5626663e4320 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f63a8005ec0 tx=0x7f63a8009300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa42e95e74c58ca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63957fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.380+0000 7f63957fa640 1 -- 172.21.15.94:0/175586712 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a80be5d0 con 0x5626663db4d0 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63957fa640 1 -- 172.21.15.94:0/175586712 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f63a80bb100 con 0x5626663db4d0 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63b6dc1740 1 -- 172.21.15.94:0/175586712 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63a80bf190 con 0x5626663db4d0 2024-04-21T08:50:18.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63957fa640 1 -- 172.21.15.94:0/175586712 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f63a80be860 con 0x5626663db4d0 2024-04-21T08:50:18.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63b6dc1740 10 client.18225 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:18.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63b6dc1740 1 -- 172.21.15.94:0/175586712 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56266611c7d0 con 0x5626663db4d0 2024-04-21T08:50:18.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63957fa640 1 -- 172.21.15.94:0/175586712 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a80bf190 con 0x5626663db4d0 2024-04-21T08:50:18.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63957fa640 1 -- 172.21.15.94:0/175586712 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f63a80becf0 con 0x5626663db4d0 2024-04-21T08:50:18.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63b6dc1740 10 client.18225 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:18.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63b6dc1740 10 client.18225 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:18.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63b6dc1740 20 client.18225 populate_metadata read hostname 'smithi094' 2024-04-21T08:50:18.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626663f47d0 0x5626663f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63b6dc1740 4 client.18225 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63b6dc1740 1 -- 172.21.15.94:0/175586712 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626662df930 con 0x5626663f47d0 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.381+0000 7f63b168d640 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626663f47d0 0x5626663f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.382+0000 7f63b168d640 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626663f47d0 0x5626663f6bc0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df1cfaff52d4c1b4 server_cookie=e09c0327c2b9d872 in_seq=0 out_seq=0 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.382+0000 7f63957fa640 10 client.18225 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.382+0000 7f63957fa640 1 -- 172.21.15.94:0/175586712 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5626662df930 con 0x5626663f47d0 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.382+0000 7f63957fa640 10 client.18225 handle_command_reply: tid=0 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.382+0000 7f63b6dc1740 1 client.18225 shutdown 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 1 -- 172.21.15.94:0/175586712 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663db4d0 msgr2=0x5626663e4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663db4d0 0x5626663e4320 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f63a8005ec0 tx=0x7f63a8009300 comp rx=0 tx=0).stop 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 1 -- 172.21.15.94:0/175586712 shutdown_connections 2024-04-21T08:50:18.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626663f47d0 0x5626663f6bc0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663e4a00 0x5626663e1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663dbe80 0x5626663e16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/175586712 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663db4d0 0x5626663e4320 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63957fa640 0 client.18225 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:50:18.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 1 -- 172.21.15.94:0/175586712 >> 172.21.15.94:0/175586712 conn(0x5626663d5ba0 msgr2=0x5626663d78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 1 -- 172.21.15.94:0/175586712 shutdown_connections 2024-04-21T08:50:18.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 1 -- 172.21.15.94:0/175586712 wait complete. 2024-04-21T08:50:18.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.383+0000 7f63b6dc1740 20 client.18225 trim_cache size 0 max 16384 2024-04-21T08:50:18.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b6dc1740 1 Processor -- start 2024-04-21T08:50:18.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b6dc1740 1 -- start start 2024-04-21T08:50:18.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663d5ba0 0x5626663f0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663db4d0 0x5626663d7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663dbe80 0x5626663e3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1b1d90 con 0x5626663d5ba0 2024-04-21T08:50:18.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1af660 con 0x5626663dbe80 2024-04-21T08:50:18.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1ad4d0 con 0x5626663db4d0 2024-04-21T08:50:18.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b1e8e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663d5ba0 0x5626663f0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b1e8e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663d5ba0 0x5626663f0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57950/0 (socket says 172.21.15.94:57950) 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b1e8e640 1 -- 172.21.15.94:0/678389870 learned_addr learned my addr 172.21.15.94:0/678389870 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b268f640 1 --2- 172.21.15.94:0/678389870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663dbe80 0x5626663e3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b1e8e640 1 -- 172.21.15.94:0/678389870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663db4d0 msgr2=0x5626663d7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/678389870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663db4d0 0x5626663d7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b1e8e640 1 -- 172.21.15.94:0/678389870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663dbe80 msgr2=0x5626663e3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/678389870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663dbe80 0x5626663e3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b1e8e640 1 -- 172.21.15.94:0/678389870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626663e1fc0 con 0x5626663d5ba0 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.396+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/678389870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663d5ba0 0x5626663f0560 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f63a80be790 tx=0x7f63a802cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84a240375dfa1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63977fe640 1 -- 172.21.15.94:0/678389870 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a80be5d0 con 0x5626663d5ba0 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63977fe640 1 -- 172.21.15.94:0/678389870 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a80f6470 con 0x5626663d5ba0 2024-04-21T08:50:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63977fe640 1 -- 172.21.15.94:0/678389870 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a80c7700 con 0x5626663d5ba0 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63b6dc1740 1 -- 172.21.15.94:0/678389870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663d5ba0 msgr2=0x5626663f0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/678389870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663d5ba0 0x5626663f0560 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f63a80be790 tx=0x7f63a802cd90 comp rx=0 tx=0).stop 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63b6dc1740 1 -- 172.21.15.94:0/678389870 shutdown_connections 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/678389870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663db4d0 0x5626663d7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/678389870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663dbe80 0x5626663e3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/678389870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663d5ba0 0x5626663f0560 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63b6dc1740 1 -- 172.21.15.94:0/678389870 >> 172.21.15.94:0/678389870 conn(0x5626663e1850 msgr2=0x5626663e06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63b6dc1740 1 -- 172.21.15.94:0/678389870 shutdown_connections 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63b6dc1740 1 -- 172.21.15.94:0/678389870 wait complete. 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.397+0000 7f63b6dc1740 1 -- start start 2024-04-21T08:50:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b6dc1740 1 Processor -- start 2024-04-21T08:50:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663d5ba0 0x5626663de3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 0x5626663de8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b6dc1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663dee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1adb20 con 0x5626663dbe80 2024-04-21T08:50:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1aef90 con 0x5626663db4d0 2024-04-21T08:50:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b6dc1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1ae1f0 con 0x5626663d5ba0 2024-04-21T08:50:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b6dc1740 10 client.0 _ll_register_callbacks cb 0x7f63a80f3110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b268f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663dee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b268f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663dee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57962/0 (socket says 172.21.15.94:57962) 2024-04-21T08:50:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b268f640 1 -- 172.21.15.94:0/3209948631 learned_addr learned my addr 172.21.15.94:0/3209948631 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663d5ba0 0x5626663de3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b168d640 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 0x5626663de8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b268f640 1 -- 172.21.15.94:0/3209948631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663d5ba0 msgr2=0x5626663de3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b268f640 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663d5ba0 0x5626663de3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b268f640 1 -- 172.21.15.94:0/3209948631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 msgr2=0x5626663de8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b268f640 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 0x5626663de8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.398+0000 7f63b268f640 1 -- 172.21.15.94:0/3209948631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562666459bc0 con 0x5626663dbe80 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b268f640 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663dee20 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f63a4020e10 tx=0x7f63a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbda6be1926c550 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63957fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63957fa640 1 -- 172.21.15.94:0/3209948631 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4003b70 con 0x5626663dbe80 2024-04-21T08:50:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63957fa640 1 -- 172.21.15.94:0/3209948631 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f63a4003d10 con 0x5626663dbe80 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b6dc1740 1 -- 172.21.15.94:0/3209948631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63a80f6470 con 0x5626663dbe80 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63957fa640 1 -- 172.21.15.94:0/3209948631 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f63a4004020 con 0x5626663dbe80 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b6dc1740 10 client.18231 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b6dc1740 1 -- 172.21.15.94:0/3209948631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56266645ae80 con 0x5626663dbe80 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63957fa640 1 -- 172.21.15.94:0/3209948631 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a80f6470 con 0x5626663dbe80 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63957fa640 1 -- 172.21.15.94:0/3209948631 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f63a4027070 con 0x5626663dbe80 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b6dc1740 10 client.18231 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b6dc1740 10 client.18231 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b6dc1740 20 client.18231 populate_metadata read hostname 'smithi094' 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626663e4a00 0x56266645d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b6dc1740 4 client.18231 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:50:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b6dc1740 1 -- 172.21.15.94:0/3209948631 --> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56266634dc10 con 0x5626663e4a00 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.399+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626663e4a00 0x56266645d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63957fa640 10 client.18231 ms_handle_connect on v2:172.21.15.111:6838/3878872423 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b1e8e640 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626663e4a00 0x56266645d690 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d22a30a9571c3590 server_cookie=85433dde6e1b120e in_seq=0 out_seq=0 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63957fa640 1 -- 172.21.15.94:0/3209948631 <== mds.0 v2:172.21.15.111:6838/3878872423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56266634dc10 con 0x5626663e4a00 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63957fa640 10 client.18231 handle_command_reply: tid=0 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 client.18231 shutdown 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 -- 172.21.15.94:0/3209948631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 msgr2=0x5626663dee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663dee20 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f63a4020e10 tx=0x7f63a4004a40 comp rx=0 tx=0).stop 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 -- 172.21.15.94:0/3209948631 shutdown_connections 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.111:6838/3878872423,v1:172.21.15.111:6839/3878872423] conn(0x5626663e4a00 0x56266645d690 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626663dbe80 0x5626663dee20 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626663db4d0 0x5626663de8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63957fa640 0 client.18231 ms_handle_reset on v2:172.21.15.111:6838/3878872423 2024-04-21T08:50:18.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 --2- 172.21.15.94:0/3209948631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626663d5ba0 0x5626663de3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 -- 172.21.15.94:0/3209948631 >> 172.21.15.94:0/3209948631 conn(0x5626663e1850 msgr2=0x5626663d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 -- 172.21.15.94:0/3209948631 shutdown_connections 2024-04-21T08:50:18.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 1 -- 172.21.15.94:0/3209948631 wait complete. 2024-04-21T08:50:18.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.400+0000 7f63b6dc1740 20 client.18231 trim_cache size 0 max 16384 2024-04-21T08:50:18.412 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2024-04-21T08:50:18.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 -- 172.21.15.94:0/1795100063 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63980a3d20 msgr2=0x7f63980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63980a3d20 0x7f63980a61e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f63a40073f0 tx=0x7f63a4007380 comp rx=0 tx=0).stop 2024-04-21T08:50:18.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 -- 172.21.15.94:0/1795100063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63ac102190 msgr2=0x7f63ac198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63ac102190 0x7f63ac198720 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f63a8005940 tx=0x7f63a802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 -- 172.21.15.94:0/1795100063 shutdown_connections 2024-04-21T08:50:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f63980a3d20 0x7f63980a61e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63ac107260 0x7f63ac198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac19cfe0 0x7f63ac19d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 --2- 172.21.15.94:0/1795100063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63ac102190 0x7f63ac198720 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 -- 172.21.15.94:0/1795100063 >> 172.21.15.94:0/1795100063 conn(0x7f63ac0fd990 msgr2=0x7f63ac0fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 -- 172.21.15.94:0/1795100063 shutdown_connections 2024-04-21T08:50:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.401+0000 7f63b40f2640 1 -- 172.21.15.94:0/1795100063 wait complete. 2024-04-21T08:50:18.415 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/thrasher.py", line 43, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-04-21T08:50:18.416 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 155, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 90, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: 2024-04-21T08:50:18.416 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-21T08:50:18.467 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:50:18.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 -- 172.21.15.94:0/3743850935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658406b040 msgr2=0x7f658410edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 --2- 172.21.15.94:0/3743850935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658406b040 0x7f658410edb0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f656c009900 tx=0x7f656c02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 -- 172.21.15.94:0/3743850935 shutdown_connections 2024-04-21T08:50:18.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 --2- 172.21.15.94:0/3743850935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f658406a680 0x7f658406ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 --2- 172.21.15.94:0/3743850935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658406b040 0x7f658410edb0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 --2- 172.21.15.94:0/3743850935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6584108030 0x7f6584108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 -- 172.21.15.94:0/3743850935 >> 172.21.15.94:0/3743850935 conn(0x7f6584077e30 msgr2=0x7f6584078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 -- 172.21.15.94:0/3743850935 shutdown_connections 2024-04-21T08:50:18.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 -- 172.21.15.94:0/3743850935 wait complete. 2024-04-21T08:50:18.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 Processor -- start 2024-04-21T08:50:18.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.565+0000 7f658b175640 1 -- start start 2024-04-21T08:50:18.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f658b175640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658406a680 0x7f65841986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f658b175640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f658406b040 0x7f6584198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f658b175640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6584108030 0x7f658419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f658b175640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f65841121d0 con 0x7f658406a680 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f658b175640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6584112050 con 0x7f658406b040 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f658b175640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6584112350 con 0x7f6584108030 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6588f11640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658406a680 0x7f65841986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6588f11640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658406a680 0x7f65841986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57978/0 (socket says 172.21.15.94:57978) 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6588f11640 1 -- 172.21.15.94:0/981871338 learned_addr learned my addr 172.21.15.94:0/981871338 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6588f11640 1 -- 172.21.15.94:0/981871338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6584108030 msgr2=0x7f658419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6588f11640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6584108030 0x7f658419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6588f11640 1 -- 172.21.15.94:0/981871338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f658406b040 msgr2=0x7f6584198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6588f11640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f658406b040 0x7f6584198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6588f11640 1 -- 172.21.15.94:0/981871338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f658419d690 con 0x7f658406a680 2024-04-21T08:50:18.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6588f11640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658406a680 0x7f65841986f0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f657c00ba50 tx=0x7f657c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6581ffb640 1 -- 172.21.15.94:0/981871338 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657c019070 con 0x7f658406a680 2024-04-21T08:50:18.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f658b175640 1 -- 172.21.15.94:0/981871338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f658419d980 con 0x7f658406a680 2024-04-21T08:50:18.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6581ffb640 1 -- 172.21.15.94:0/981871338 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f657c00cbe0 con 0x7f658406a680 2024-04-21T08:50:18.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.566+0000 7f6581ffb640 1 -- 172.21.15.94:0/981871338 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657c0075d0 con 0x7f658406a680 2024-04-21T08:50:18.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.567+0000 7f658b175640 1 -- 172.21.15.94:0/981871338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65841a5290 con 0x7f658406a680 2024-04-21T08:50:18.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.567+0000 7f658b175640 1 -- 172.21.15.94:0/981871338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6584069230 con 0x7f658406a680 2024-04-21T08:50:18.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.569+0000 7f6581ffb640 1 -- 172.21.15.94:0/981871338 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f657c01c070 con 0x7f658406a680 2024-04-21T08:50:18.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.570+0000 7f6581ffb640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d20 0x7f65700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.570+0000 7f6583fff640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d20 0x7f65700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.570+0000 7f6581ffb640 1 -- 172.21.15.94:0/981871338 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f657c0d0650 con 0x7f658406a680 2024-04-21T08:50:18.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.570+0000 7f6583fff640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d20 0x7f65700a61e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6574005fd0 tx=0x7f6574005850 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.573+0000 7f6581ffb640 1 -- 172.21.15.94:0/981871338 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f657c09c9d0 con 0x7f658406a680 2024-04-21T08:50:18.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.753+0000 7f658b175640 1 -- 172.21.15.94:0/981871338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6584109ee0 con 0x7f658406a680 2024-04-21T08:50:18.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.754+0000 7f6581ffb640 1 -- 172.21.15.94:0/981871338 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+25452 (secure 0 0 0) 0x7f657c09c2d0 con 0x7f658406a680 2024-04-21T08:50:18.755 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:50:18.755 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":102,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:50:11.852327+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:50:18.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 -- 172.21.15.94:0/981871338 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d20 msgr2=0x7f65700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d20 0x7f65700a61e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6574005fd0 tx=0x7f6574005850 comp rx=0 tx=0).stop 2024-04-21T08:50:18.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 -- 172.21.15.94:0/981871338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658406a680 msgr2=0x7f65841986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658406a680 0x7f65841986f0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f657c00ba50 tx=0x7f657c00bf10 comp rx=0 tx=0).stop 2024-04-21T08:50:18.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 -- 172.21.15.94:0/981871338 shutdown_connections 2024-04-21T08:50:18.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f65700a3d20 0x7f65700a61e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6584108030 0x7f658419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f658406b040 0x7f6584198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 --2- 172.21.15.94:0/981871338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658406a680 0x7f65841986f0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 -- 172.21.15.94:0/981871338 >> 172.21.15.94:0/981871338 conn(0x7f6584077e30 msgr2=0x7f65840ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 -- 172.21.15.94:0/981871338 shutdown_connections 2024-04-21T08:50:18.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.757+0000 7f658b175640 1 -- 172.21.15.94:0/981871338 wait complete. 2024-04-21T08:50:18.769 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:50:18.770 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:50:18.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.887+0000 7f263cdce640 1 -- 172.21.15.94:0/4067205768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26381010a0 msgr2=0x7f2638101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.887+0000 7f263cdce640 1 --2- 172.21.15.94:0/4067205768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26381010a0 0x7f2638101480 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2628009900 tx=0x7f262802bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:18.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.887+0000 7f263cdce640 1 -- 172.21.15.94:0/4067205768 shutdown_connections 2024-04-21T08:50:18.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.887+0000 7f263cdce640 1 --2- 172.21.15.94:0/4067205768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2638101a50 0x7f2638105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.887+0000 7f263cdce640 1 --2- 172.21.15.94:0/4067205768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2638106170 0x7f263810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.887+0000 7f263cdce640 1 --2- 172.21.15.94:0/4067205768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26381010a0 0x7f2638101480 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.887+0000 7f263cdce640 1 -- 172.21.15.94:0/4067205768 >> 172.21.15.94:0/4067205768 conn(0x7f2638077eb0 msgr2=0x7f26380782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.887+0000 7f263cdce640 1 -- 172.21.15.94:0/4067205768 shutdown_connections 2024-04-21T08:50:18.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.887+0000 7f263cdce640 1 -- 172.21.15.94:0/4067205768 wait complete. 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f263cdce640 1 Processor -- start 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f263cdce640 1 -- start start 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f263cdce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381010a0 0x7f26381975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f263cdce640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2638106170 0x7f2638197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f263cdce640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263819be90 0x7f263819c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f263cdce640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f26381110b0 con 0x7f2638106170 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f263cdce640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2638110f30 con 0x7f263819be90 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f263cdce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2638111230 con 0x7f26381010a0 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f262ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2638106170 0x7f2638197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f262ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2638106170 0x7f2638197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58002/0 (socket says 172.21.15.94:58002) 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f262ffff640 1 -- 172.21.15.94:0/1494377397 learned_addr learned my addr 172.21.15.94:0/1494377397 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f263659c640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381010a0 0x7f26381975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.888+0000 7f2636d9d640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263819be90 0x7f263819c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f262ffff640 1 -- 172.21.15.94:0/1494377397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381010a0 msgr2=0x7f26381975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f262ffff640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381010a0 0x7f26381975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f262ffff640 1 -- 172.21.15.94:0/1494377397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263819be90 msgr2=0x7f263819c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f262ffff640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263819be90 0x7f263819c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f262ffff640 1 -- 172.21.15.94:0/1494377397 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263819c980 con 0x7f2638106170 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f262ffff640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2638106170 0x7f2638197b10 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f262000c970 tx=0x7f262000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f262f7fe640 1 -- 172.21.15.94:0/1494377397 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2620007bb0 con 0x7f2638106170 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f263cdce640 1 -- 172.21.15.94:0/1494377397 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263819cc70 con 0x7f2638106170 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f262f7fe640 1 -- 172.21.15.94:0/1494377397 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2620007d50 con 0x7f2638106170 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f262f7fe640 1 -- 172.21.15.94:0/1494377397 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262000f450 con 0x7f2638106170 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.889+0000 7f263cdce640 1 -- 172.21.15.94:0/1494377397 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26381a44b0 con 0x7f2638106170 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.890+0000 7f263cdce640 1 -- 172.21.15.94:0/1494377397 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26380691c0 con 0x7f2638106170 2024-04-21T08:50:18.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.891+0000 7f262f7fe640 1 -- 172.21.15.94:0/1494377397 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f2620004ec0 con 0x7f2638106170 2024-04-21T08:50:18.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.891+0000 7f262f7fe640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f26240a3d20 0x7f26240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.892+0000 7f263659c640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f26240a3d20 0x7f26240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.892+0000 7f262f7fe640 1 -- 172.21.15.94:0/1494377397 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f26200d08a0 con 0x7f2638106170 2024-04-21T08:50:18.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.892+0000 7f263659c640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f26240a3d20 0x7f26240a61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2628009230 tx=0x7f2628005fd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:18.896+0000 7f262f7fe640 1 -- 172.21.15.94:0/1494377397 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2620014030 con 0x7f2638106170 2024-04-21T08:50:19.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.073+0000 7f263cdce640 1 -- 172.21.15.94:0/1494377397 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2638109400 con 0x7f2638106170 2024-04-21T08:50:19.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.074+0000 7f262f7fe640 1 -- 172.21.15.94:0/1494377397 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7f262009c4c0 con 0x7f2638106170 2024-04-21T08:50:19.075 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:50:19.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 -- 172.21.15.94:0/1494377397 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f26240a3d20 msgr2=0x7f26240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f26240a3d20 0x7f26240a61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2628009230 tx=0x7f2628005fd0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 -- 172.21.15.94:0/1494377397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2638106170 msgr2=0x7f2638197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2638106170 0x7f2638197b10 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f262000c970 tx=0x7f262000ce30 comp rx=0 tx=0).stop 2024-04-21T08:50:19.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 -- 172.21.15.94:0/1494377397 shutdown_connections 2024-04-21T08:50:19.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f26240a3d20 0x7f26240a61e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263819be90 0x7f263819c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2638106170 0x7f2638197b10 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 --2- 172.21.15.94:0/1494377397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381010a0 0x7f26381975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 -- 172.21.15.94:0/1494377397 >> 172.21.15.94:0/1494377397 conn(0x7f2638077eb0 msgr2=0x7f26380ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 -- 172.21.15.94:0/1494377397 shutdown_connections 2024-04-21T08:50:19.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.076+0000 7f263cdce640 1 -- 172.21.15.94:0/1494377397 wait complete. 2024-04-21T08:50:19.086 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:18 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2371913411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:19.086 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:18 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/981871338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:19.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:50:19.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:18 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2371913411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:19.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:18 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/981871338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:19.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.188+0000 7f3182ef1640 1 -- 172.21.15.94:0/2106110292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f317c1010a0 msgr2=0x7f317c101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.188+0000 7f3182ef1640 1 --2- 172.21.15.94:0/2106110292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f317c1010a0 0x7f317c101480 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f3170009900 tx=0x7f317002bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:19.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.188+0000 7f3182ef1640 1 -- 172.21.15.94:0/2106110292 shutdown_connections 2024-04-21T08:50:19.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.188+0000 7f3182ef1640 1 --2- 172.21.15.94:0/2106110292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f317c106170 0x7f317c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.188+0000 7f3182ef1640 1 --2- 172.21.15.94:0/2106110292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f317c101a50 0x7f317c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.188+0000 7f3182ef1640 1 --2- 172.21.15.94:0/2106110292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f317c1010a0 0x7f317c101480 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.188+0000 7f3182ef1640 1 -- 172.21.15.94:0/2106110292 >> 172.21.15.94:0/2106110292 conn(0x7f317c077eb0 msgr2=0x7f317c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.188+0000 7f3182ef1640 1 -- 172.21.15.94:0/2106110292 shutdown_connections 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.188+0000 7f3182ef1640 1 -- 172.21.15.94:0/2106110292 wait complete. 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3182ef1640 1 Processor -- start 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3182ef1640 1 -- start start 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3182ef1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f317c1010a0 0x7f317c1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3182ef1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f317c106170 0x7f317c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3182ef1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f317c19be80 0x7f317c19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3182ef1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f317c1110e0 con 0x7f317c106170 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3182ef1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f317c110f60 con 0x7f317c1010a0 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3182ef1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f317c111260 con 0x7f317c19be80 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f317bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f317c106170 0x7f317c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f317bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f317c106170 0x7f317c197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58018/0 (socket says 172.21.15.94:58018) 2024-04-21T08:50:19.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f317bfff640 1 -- 172.21.15.94:0/544870003 learned_addr learned my addr 172.21.15.94:0/544870003 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f317bfff640 1 -- 172.21.15.94:0/544870003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f317c19be80 msgr2=0x7f317c19c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f317bfff640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f317c19be80 0x7f317c19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f317bfff640 1 -- 172.21.15.94:0/544870003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f317c1010a0 msgr2=0x7f317c1975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f317bfff640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f317c1010a0 0x7f317c1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f317bfff640 1 -- 172.21.15.94:0/544870003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317c19c970 con 0x7f317c106170 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f317bfff640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f317c106170 0x7f317c197b00 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f316800c970 tx=0x7f316800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3179ffb640 1 -- 172.21.15.94:0/544870003 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168014070 con 0x7f317c106170 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3182ef1640 1 -- 172.21.15.94:0/544870003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f317c19cc60 con 0x7f317c106170 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3179ffb640 1 -- 172.21.15.94:0/544870003 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3168004440 con 0x7f317c106170 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.189+0000 7f3179ffb640 1 -- 172.21.15.94:0/544870003 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168003e50 con 0x7f317c106170 2024-04-21T08:50:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.190+0000 7f3182ef1640 1 -- 172.21.15.94:0/544870003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f317c1a4450 con 0x7f317c106170 2024-04-21T08:50:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.191+0000 7f3182ef1640 1 -- 172.21.15.94:0/544870003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f317c0691c0 con 0x7f317c106170 2024-04-21T08:50:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.192+0000 7f3179ffb640 1 -- 172.21.15.94:0/544870003 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f3168017070 con 0x7f317c106170 2024-04-21T08:50:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.192+0000 7f3179ffb640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f316c0a3ec0 0x7f316c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.192+0000 7f3180c8d640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f316c0a3ec0 0x7f316c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.193+0000 7f3179ffb640 1 -- 172.21.15.94:0/544870003 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f31680d0b30 con 0x7f317c106170 2024-04-21T08:50:19.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.193+0000 7f3180c8d640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f316c0a3ec0 0x7f316c0a6380 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3170005770 tx=0x7f3170000bc0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.197+0000 7f3179ffb640 1 -- 172.21.15.94:0/544870003 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f316801e020 con 0x7f317c106170 2024-04-21T08:50:19.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:18 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2371913411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:19.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:18 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/981871338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:19.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.375+0000 7f3182ef1640 1 -- 172.21.15.94:0/544870003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f317c101a50 con 0x7f317c106170 2024-04-21T08:50:19.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.375+0000 7f3179ffb640 1 -- 172.21.15.94:0/544870003 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7f316809cda0 con 0x7f317c106170 2024-04-21T08:50:19.376 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 -- 172.21.15.94:0/544870003 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f316c0a3ec0 msgr2=0x7f316c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f316c0a3ec0 0x7f316c0a6380 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3170005770 tx=0x7f3170000bc0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 -- 172.21.15.94:0/544870003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f317c106170 msgr2=0x7f317c197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f317c106170 0x7f317c197b00 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f316800c970 tx=0x7f316800ce30 comp rx=0 tx=0).stop 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 -- 172.21.15.94:0/544870003 shutdown_connections 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f316c0a3ec0 0x7f316c0a6380 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f317c19be80 0x7f317c19c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f317c106170 0x7f317c197b00 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 --2- 172.21.15.94:0/544870003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f317c1010a0 0x7f317c1975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 -- 172.21.15.94:0/544870003 >> 172.21.15.94:0/544870003 conn(0x7f317c077eb0 msgr2=0x7f317c0fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 -- 172.21.15.94:0/544870003 shutdown_connections 2024-04-21T08:50:19.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.378+0000 7f3182ef1640 1 -- 172.21.15.94:0/544870003 wait complete. 2024-04-21T08:50:19.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:50:19.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.491+0000 7f3852532640 1 -- 172.21.15.94:0/3813285102 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384c103af0 msgr2=0x7f384c10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.492+0000 7f3852532640 1 --2- 172.21.15.94:0/3813285102 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384c103af0 0x7f384c10bfd0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f383c009900 tx=0x7f383c02bd60 comp rx=0 tx=0).stop 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.492+0000 7f3852532640 1 -- 172.21.15.94:0/3813285102 shutdown_connections 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.492+0000 7f3852532640 1 --2- 172.21.15.94:0/3813285102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f384c10c510 0x7f384c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.492+0000 7f3852532640 1 --2- 172.21.15.94:0/3813285102 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384c103af0 0x7f384c10bfd0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.492+0000 7f3852532640 1 --2- 172.21.15.94:0/3813285102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f384c1031d0 0x7f384c1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.492+0000 7f3852532640 1 -- 172.21.15.94:0/3813285102 >> 172.21.15.94:0/3813285102 conn(0x7f384c077f50 msgr2=0x7f384c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.492+0000 7f3852532640 1 -- 172.21.15.94:0/3813285102 shutdown_connections 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.492+0000 7f3852532640 1 -- 172.21.15.94:0/3813285102 wait complete. 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.492+0000 7f3852532640 1 Processor -- start 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.492+0000 7f3852532640 1 -- start start 2024-04-21T08:50:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f3852532640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f384c1031d0 0x7f384c1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f3852532640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384c103af0 0x7f384c1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f3852532640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f384c10c510 0x7f384c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f3852532640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f384c1121a0 con 0x7f384c103af0 2024-04-21T08:50:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f3852532640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f384c112020 con 0x7f384c10c510 2024-04-21T08:50:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f3852532640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f384c112320 con 0x7f384c1031d0 2024-04-21T08:50:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f384b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384c103af0 0x7f384c1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f384b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384c103af0 0x7f384c1a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58040/0 (socket says 172.21.15.94:58040) 2024-04-21T08:50:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f384b7fe640 1 -- 172.21.15.94:0/3927732333 learned_addr learned my addr 172.21.15.94:0/3927732333 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f384bfff640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f384c1031d0 0x7f384c1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f384b7fe640 1 -- 172.21.15.94:0/3927732333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f384c1031d0 msgr2=0x7f384c1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f384b7fe640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f384c1031d0 0x7f384c1a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f384b7fe640 1 -- 172.21.15.94:0/3927732333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f384c10c510 msgr2=0x7f384c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f3850acf640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f384c10c510 0x7f384c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f384b7fe640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f384c10c510 0x7f384c19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f384b7fe640 1 -- 172.21.15.94:0/3927732333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f384c19af50 con 0x7f384c103af0 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f384b7fe640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384c103af0 0x7f384c1a0b50 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f383c005940 tx=0x7f383c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f38497fa640 1 -- 172.21.15.94:0/3927732333 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c043020 con 0x7f384c103af0 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f3852532640 1 -- 172.21.15.94:0/3927732333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f384c19b1e0 con 0x7f384c103af0 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f38497fa640 1 -- 172.21.15.94:0/3927732333 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f383c02dc30 con 0x7f384c103af0 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f38497fa640 1 -- 172.21.15.94:0/3927732333 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c003a10 con 0x7f384c103af0 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.493+0000 7f3852532640 1 -- 172.21.15.94:0/3927732333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f384c1a7460 con 0x7f384c103af0 2024-04-21T08:50:19.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.494+0000 7f3852532640 1 -- 172.21.15.94:0/3927732333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f384c0698e0 con 0x7f384c103af0 2024-04-21T08:50:19.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.495+0000 7f38497fa640 1 -- 172.21.15.94:0/3927732333 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f383c046020 con 0x7f384c103af0 2024-04-21T08:50:19.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.496+0000 7f38497fa640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f38380a3d20 0x7f38380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.496+0000 7f384bfff640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f38380a3d20 0x7f38380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.496+0000 7f38497fa640 1 -- 172.21.15.94:0/3927732333 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f383c0f2c70 con 0x7f384c103af0 2024-04-21T08:50:19.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.497+0000 7f384bfff640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f38380a3d20 0x7f38380a61e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f383400bbe0 tx=0x7f383400b390 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.501+0000 7f38497fa640 1 -- 172.21.15.94:0/3927732333 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f383c04d020 con 0x7f384c103af0 2024-04-21T08:50:19.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.679+0000 7f3852532640 1 -- 172.21.15.94:0/3927732333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f384c1a7630 con 0x7f384c103af0 2024-04-21T08:50:19.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.679+0000 7f38497fa640 1 -- 172.21.15.94:0/3927732333 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7f383c0beff0 con 0x7f384c103af0 2024-04-21T08:50:19.679 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:50:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 -- 172.21.15.94:0/3927732333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f38380a3d20 msgr2=0x7f38380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f38380a3d20 0x7f38380a61e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f383400bbe0 tx=0x7f383400b390 comp rx=0 tx=0).stop 2024-04-21T08:50:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 -- 172.21.15.94:0/3927732333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384c103af0 msgr2=0x7f384c1a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384c103af0 0x7f384c1a0b50 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f383c005940 tx=0x7f383c0045e0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 -- 172.21.15.94:0/3927732333 shutdown_connections 2024-04-21T08:50:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f38380a3d20 0x7f38380a61e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f384c10c510 0x7f384c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384c103af0 0x7f384c1a0b50 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 --2- 172.21.15.94:0/3927732333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f384c1031d0 0x7f384c1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 -- 172.21.15.94:0/3927732333 >> 172.21.15.94:0/3927732333 conn(0x7f384c077f50 msgr2=0x7f384c108480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 -- 172.21.15.94:0/3927732333 shutdown_connections 2024-04-21T08:50:19.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.681+0000 7f3852532640 1 -- 172.21.15.94:0/3927732333 wait complete. 2024-04-21T08:50:19.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:50:19.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.793+0000 7f915a559640 1 -- 172.21.15.94:0/2271869921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9154102b40 msgr2=0x7f9154106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.793+0000 7f915a559640 1 --2- 172.21.15.94:0/2271869921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9154102b40 0x7f9154106b80 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f9148009900 tx=0x7f914802bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:19.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.793+0000 7f915a559640 1 -- 172.21.15.94:0/2271869921 shutdown_connections 2024-04-21T08:50:19.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.793+0000 7f915a559640 1 --2- 172.21.15.94:0/2271869921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9154107260 0x7f915410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.793+0000 7f915a559640 1 --2- 172.21.15.94:0/2271869921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9154102b40 0x7f9154106b80 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.793+0000 7f915a559640 1 --2- 172.21.15.94:0/2271869921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9154102190 0x7f9154102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.793+0000 7f915a559640 1 -- 172.21.15.94:0/2271869921 >> 172.21.15.94:0/2271869921 conn(0x7f91540793a0 msgr2=0x7f91540797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.793+0000 7f915a559640 1 -- 172.21.15.94:0/2271869921 shutdown_connections 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.793+0000 7f915a559640 1 -- 172.21.15.94:0/2271869921 wait complete. 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f915a559640 1 Processor -- start 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f915a559640 1 -- start start 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f915a559640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9154102190 0x7f9154198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f915a559640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9154107260 0x7f9154198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f915a559640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f915419cfc0 0x7f915419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f915a559640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f91541121a0 con 0x7f9154107260 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f915a559640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9154112020 con 0x7f915419cfc0 2024-04-21T08:50:19.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f915a559640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9154112320 con 0x7f9154102190 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f91537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9154107260 0x7f9154198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f91537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9154107260 0x7f9154198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58070/0 (socket says 172.21.15.94:58070) 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f91537fe640 1 -- 172.21.15.94:0/522484407 learned_addr learned my addr 172.21.15.94:0/522484407 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f9158af6640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f915419cfc0 0x7f915419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f9153fff640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9154102190 0x7f9154198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f91537fe640 1 -- 172.21.15.94:0/522484407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9154102190 msgr2=0x7f9154198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f91537fe640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9154102190 0x7f9154198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f91537fe640 1 -- 172.21.15.94:0/522484407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f915419cfc0 msgr2=0x7f915419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.794+0000 7f91537fe640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f915419cfc0 0x7f915419d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.795+0000 7f91537fe640 1 -- 172.21.15.94:0/522484407 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915419dab0 con 0x7f9154107260 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.795+0000 7f91537fe640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9154107260 0x7f9154198c40 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f9148036040 tx=0x7f914802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.795+0000 7f91517fa640 1 -- 172.21.15.94:0/522484407 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148043020 con 0x7f9154107260 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.795+0000 7f915a559640 1 -- 172.21.15.94:0/522484407 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915419dd40 con 0x7f9154107260 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.795+0000 7f91517fa640 1 -- 172.21.15.94:0/522484407 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f914802cbe0 con 0x7f9154107260 2024-04-21T08:50:19.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.795+0000 7f91517fa640 1 -- 172.21.15.94:0/522484407 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148003b80 con 0x7f9154107260 2024-04-21T08:50:19.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.795+0000 7f915a559640 1 -- 172.21.15.94:0/522484407 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91541a5590 con 0x7f9154107260 2024-04-21T08:50:19.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.796+0000 7f915a559640 1 -- 172.21.15.94:0/522484407 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91540691c0 con 0x7f9154107260 2024-04-21T08:50:19.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.797+0000 7f91517fa640 1 -- 172.21.15.94:0/522484407 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9148046020 con 0x7f9154107260 2024-04-21T08:50:19.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.798+0000 7f91517fa640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f913c0a3d20 0x7f913c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.798+0000 7f9153fff640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f913c0a3d20 0x7f913c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.798+0000 7f91517fa640 1 -- 172.21.15.94:0/522484407 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f91480f2d30 con 0x7f9154107260 2024-04-21T08:50:19.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.798+0000 7f9153fff640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f913c0a3d20 0x7f913c0a61e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f914400abe0 tx=0x7f914400a2d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.802+0000 7f91517fa640 1 -- 172.21.15.94:0/522484407 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f914804d020 con 0x7f9154107260 2024-04-21T08:50:19.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.979+0000 7f915a559640 1 -- 172.21.15.94:0/522484407 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f915410a4c0 con 0x7f9154107260 2024-04-21T08:50:19.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.979+0000 7f91517fa640 1 -- 172.21.15.94:0/522484407 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) ==== 103+0+11 (secure 0 0 0) 0x7f91480bf0b0 con 0x7f9154107260 2024-04-21T08:50:19.979 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:50:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 -- 172.21.15.94:0/522484407 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f913c0a3d20 msgr2=0x7f913c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f913c0a3d20 0x7f913c0a61e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f914400abe0 tx=0x7f914400a2d0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 -- 172.21.15.94:0/522484407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9154107260 msgr2=0x7f9154198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9154107260 0x7f9154198c40 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f9148036040 tx=0x7f914802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 -- 172.21.15.94:0/522484407 shutdown_connections 2024-04-21T08:50:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f913c0a3d20 0x7f913c0a61e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f915419cfc0 0x7f915419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9154107260 0x7f9154198c40 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 --2- 172.21.15.94:0/522484407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9154102190 0x7f9154198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 -- 172.21.15.94:0/522484407 >> 172.21.15.94:0/522484407 conn(0x7f91540793a0 msgr2=0x7f91540ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 -- 172.21.15.94:0/522484407 shutdown_connections 2024-04-21T08:50:19.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:19.981+0000 7f915a559640 1 -- 172.21.15.94:0/522484407 wait complete. 2024-04-21T08:50:19.991 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:19 smithi094 ceph-mon[27988]: pgmap v321: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 9 op/s 2024-04-21T08:50:19.991 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:19 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1494377397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:19.991 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:19 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/544870003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:19.991 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:19 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3927732333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:19.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-21T08:50:20.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/763325851 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d18106160 msgr2=0x7f2d1810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:20.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 --2- 172.21.15.94:0/763325851 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d18106160 0x7f2d1810dd30 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f2d08009900 tx=0x7f2d0802bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:20.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/763325851 shutdown_connections 2024-04-21T08:50:20.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 --2- 172.21.15.94:0/763325851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d18101a40 0x7f2d18105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:20.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 --2- 172.21.15.94:0/763325851 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d18106160 0x7f2d1810dd30 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:20.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 --2- 172.21.15.94:0/763325851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d18101090 0x7f2d18101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:20.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/763325851 >> 172.21.15.94:0/763325851 conn(0x7f2d180771d0 msgr2=0x7f2d180775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:20.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/763325851 shutdown_connections 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/763325851 wait complete. 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 Processor -- start 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 -- start start 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d18101090 0x7f2d181975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d18106160 0x7f2d18197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d1819be90 0x7f2d1819c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2d181110b0 con 0x7f2d18101090 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.092+0000 7f2d1ee3a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2d18110f30 con 0x7f2d1819be90 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1ee3a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2d18111230 con 0x7f2d18106160 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1cbd6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d18101090 0x7f2d181975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1cbd6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d18101090 0x7f2d181975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58100/0 (socket says 172.21.15.94:58100) 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1cbd6640 1 -- 172.21.15.94:0/3927849834 learned_addr learned my addr 172.21.15.94:0/3927849834 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1cbd6640 1 -- 172.21.15.94:0/3927849834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d18106160 msgr2=0x7f2d18197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:20.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1cbd6640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d18106160 0x7f2d18197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:20.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1cbd6640 1 -- 172.21.15.94:0/3927849834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d1819be90 msgr2=0x7f2d1819c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:20.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1d3d7640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d1819be90 0x7f2d1819c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:20.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1cbd6640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d1819be90 0x7f2d1819c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:20.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1cbd6640 1 -- 172.21.15.94:0/3927849834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1819c980 con 0x7f2d18101090 2024-04-21T08:50:20.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1cbd6640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d18101090 0x7f2d181975d0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000eab0 tx=0x7f2d1000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:20.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d15ffb640 1 -- 172.21.15.94:0/3927849834 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1000ce40 con 0x7f2d18101090 2024-04-21T08:50:20.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/3927849834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1819cc70 con 0x7f2d18101090 2024-04-21T08:50:20.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d15ffb640 1 -- 172.21.15.94:0/3927849834 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d10011920 con 0x7f2d18101090 2024-04-21T08:50:20.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d15ffb640 1 -- 172.21.15.94:0/3927849834 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d10010a30 con 0x7f2d18101090 2024-04-21T08:50:20.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.093+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/3927849834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1806a960 con 0x7f2d18101090 2024-04-21T08:50:20.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.095+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/3927849834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d180691c0 con 0x7f2d18101090 2024-04-21T08:50:20.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.096+0000 7f2d15ffb640 1 -- 172.21.15.94:0/3927849834 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f2d10010bd0 con 0x7f2d18101090 2024-04-21T08:50:20.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.096+0000 7f2d15ffb640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2d040a3d20 0x7f2d040a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:20.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.096+0000 7f2d17fff640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2d040a3d20 0x7f2d040a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:20.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.096+0000 7f2d15ffb640 1 -- 172.21.15.94:0/3927849834 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f2d100d12f0 con 0x7f2d18101090 2024-04-21T08:50:20.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.097+0000 7f2d17fff640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2d040a3d20 0x7f2d040a61e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2d181006c0 tx=0x7f2d00009590 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:20.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.101+0000 7f2d15ffb640 1 -- 172.21.15.94:0/3927849834 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d10016030 con 0x7f2d18101090 2024-04-21T08:50:20.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:19 smithi111 ceph-mon[31493]: pgmap v321: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 9 op/s 2024-04-21T08:50:20.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:19 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1494377397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:20.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:19 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/544870003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:20.188 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:19 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3927732333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:20.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:19 smithi173 ceph-mon[31428]: pgmap v321: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 9 op/s 2024-04-21T08:50:20.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:19 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1494377397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:20.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:19 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/544870003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:20.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:19 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3927732333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:20.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.326+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/3927849834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f2d18198920 con 0x7f2d18101090 2024-04-21T08:50:20.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.334+0000 7f2d15ffb640 1 -- 172.21.15.94:0/3927849834 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+746060 (secure 0 0 0) 0x7f2d1009d560 con 0x7f2d18101090 2024-04-21T08:50:20.348 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:50:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 25, 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: "active_gid": 15453, 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: "active_name": "x", 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: "active_addrs": { 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v2", 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6800", 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 4147145528 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:6801", 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 4147145528 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: "active_addr": "172.21.15.94:6801/4147145528", 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: "active_change": "2024-04-21T08:39:57.898973+0000", 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: "active_mgr_features": 4540701547738038271, 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: "available": true, 2024-04-21T08:50:20.350 INFO:teuthology.orchestra.run.smithi094.stdout: "standbys": [ 2024-04-21T08:50:20.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.351 INFO:teuthology.orchestra.run.smithi094.stdout: "gid": 24923, 2024-04-21T08:50:20.351 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "y", 2024-04-21T08:50:20.351 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr_features": 4540701547738038271, 2024-04-21T08:50:20.351 INFO:teuthology.orchestra.run.smithi094.stdout: "available_modules": [ 2024-04-21T08:50:20.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.351 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "alerts", 2024-04-21T08:50:20.351 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.352 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.352 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.352 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:20.352 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:20.352 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.354 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.355 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.356 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.358 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_destination": { 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_destination", 2024-04-21T08:50:20.359 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Email address to send alerts to", 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.360 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_from_name": { 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_from_name", 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "Ceph", 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Email From: name", 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.361 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_host": { 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_host", 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SMTP server", 2024-04-21T08:50:20.362 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_password": { 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_password", 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.363 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Password to authenticate with", 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_port": { 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_port", 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "465", 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.364 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SMTP port", 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_sender": { 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_sender", 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.365 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.366 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.366 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.366 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.366 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SMTP envelope sender", 2024-04-21T08:50:20.366 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.366 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.366 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.366 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_ssl": { 2024-04-21T08:50:20.366 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_ssl", 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_user": { 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_user", 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "User to authenticate as", 2024-04-21T08:50:20.368 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "balancer", 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: "active": { 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "active", 2024-04-21T08:50:20.369 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.370 INFO:teuthology.orchestra.run.smithi094.stdout: "begin_time": { 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "begin_time", 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0000", 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "begin_weekday": { 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "begin_weekday", 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "0", 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "6", 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T08:50:20.372 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_compat_max_iterations": { 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crush_compat_max_iterations", 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "25", 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "250", 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.373 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_compat_metrics": { 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crush_compat_metrics", 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T08:50:20.374 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.375 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.375 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.375 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T08:50:20.375 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T08:50:20.375 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.375 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.375 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_compat_step": { 2024-04-21T08:50:20.375 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crush_compat_step", 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0.5", 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "0.001", 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "0.999", 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "aggressiveness of optimization", 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.377 INFO:teuthology.orchestra.run.smithi094.stdout: "end_time": { 2024-04-21T08:50:20.377 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "end_time", 2024-04-21T08:50:20.377 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.377 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.377 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.377 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "2359", 2024-04-21T08:50:20.377 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.377 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.377 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.378 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T08:50:20.378 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:50:20.378 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.378 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.378 INFO:teuthology.orchestra.run.smithi094.stdout: "end_weekday": { 2024-04-21T08:50:20.378 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "end_weekday", 2024-04-21T08:50:20.378 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:20.378 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.378 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.379 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:20.379 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "0", 2024-04-21T08:50:20.379 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "6", 2024-04-21T08:50:20.379 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.379 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T08:50:20.379 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:50:20.379 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.379 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.380 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.381 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.381 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.381 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.381 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.381 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.381 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.381 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.381 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.381 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.382 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.382 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.382 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.382 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.382 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.382 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.382 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.382 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.382 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.383 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.383 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.383 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.383 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.383 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.383 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.383 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.383 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.384 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.384 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.384 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.384 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.384 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.384 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.384 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.384 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.384 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.385 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.385 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.385 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.385 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.385 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.385 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.385 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.385 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.386 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.386 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.386 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.386 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.386 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.386 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.386 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.386 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.387 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.387 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.387 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.387 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.387 INFO:teuthology.orchestra.run.smithi094.stdout: "min_score": { 2024-04-21T08:50:20.387 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "min_score", 2024-04-21T08:50:20.387 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.387 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": { 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mode", 2024-04-21T08:50:20.388 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "upmap", 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "crush-compat", 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "none", 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "read", 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap", 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap-read" 2024-04-21T08:50:20.389 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Balancer mode", 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: "pool_ids": { 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pool_ids", 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.390 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.391 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap_max_deviation": { 2024-04-21T08:50:20.392 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "upmap_max_deviation", 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap_max_optimizations": { 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "upmap_max_optimizations", 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.394 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cephadm", 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: "agent_down_multiplier": { 2024-04-21T08:50:20.395 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "agent_down_multiplier", 2024-04-21T08:50:20.396 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.396 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.396 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.396 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "3.0", 2024-04-21T08:50:20.396 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.396 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.396 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.396 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T08:50:20.396 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.396 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "agent_refresh_rate": { 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "agent_refresh_rate", 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "20", 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.397 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: "agent_starting_port": { 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "agent_starting_port", 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "4721", 2024-04-21T08:50:20.398 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.399 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.399 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.399 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T08:50:20.399 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.399 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.399 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.399 INFO:teuthology.orchestra.run.smithi094.stdout: "allow_ptrace": { 2024-04-21T08:50:20.399 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "allow_ptrace", 2024-04-21T08:50:20.399 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "autotune_interval": { 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "autotune_interval", 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.401 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: "autotune_memory_target_ratio": { 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0.7", 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.402 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T08:50:20.403 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.403 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.403 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.403 INFO:teuthology.orchestra.run.smithi094.stdout: "cephadm_log_destination": { 2024-04-21T08:50:20.403 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cephadm_log_destination", 2024-04-21T08:50:20.403 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.403 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.403 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.403 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: "file", 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: "file,syslog", 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: "syslog" 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.404 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: "cgroups_split": { 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cgroups_split", 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.405 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: "config_checks_enabled": { 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "config_checks_enabled", 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.406 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: "config_dashboard": { 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "config_dashboard", 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.407 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.408 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.408 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.408 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.408 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.408 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.408 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T08:50:20.408 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.408 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.408 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_alertmanager": { 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_alertmanager", 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.409 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:20.410 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.410 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.410 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.410 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_base": { 2024-04-21T08:50:20.410 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_base", 2024-04-21T08:50:20.410 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.410 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Container image name, without the tag", 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.411 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_elasticsearch": { 2024-04-21T08:50:20.412 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_elasticsearch", 2024-04-21T08:50:20.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.412 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.412 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.412 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T08:50:20.412 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.412 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.412 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "elasticsearch container image", 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_grafana": { 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_grafana", 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.413 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_haproxy": { 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_haproxy", 2024-04-21T08:50:20.414 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.415 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.415 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.415 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T08:50:20.415 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.415 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.415 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.415 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "HAproxy container image", 2024-04-21T08:50:20.415 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.415 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.415 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_jaeger_agent": { 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_jaeger_agent", 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.416 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Jaeger agent container image", 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_jaeger_collector": { 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_jaeger_collector", 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T08:50:20.417 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Jaeger collector container image", 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_jaeger_query": { 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_jaeger_query", 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.418 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Jaeger query container image", 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.419 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_keepalived": { 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_keepalived", 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Keepalived container image", 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.420 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_loki": { 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_loki", 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.421 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Loki container image", 2024-04-21T08:50:20.422 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.422 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.422 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.422 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_node_exporter": { 2024-04-21T08:50:20.422 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_node_exporter", 2024-04-21T08:50:20.422 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.422 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.422 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.422 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_nvmeof": { 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_nvmeof", 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Nvme-of container image", 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.424 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_prometheus": { 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_prometheus", 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.425 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_promtail": { 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_promtail", 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.426 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.427 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Promtail container image", 2024-04-21T08:50:20.427 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.427 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.427 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.427 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_samba": { 2024-04-21T08:50:20.427 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_samba", 2024-04-21T08:50:20.427 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.427 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.427 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Samba/SMB container image", 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_snmp_gateway": { 2024-04-21T08:50:20.428 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_snmp_gateway", 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SNMP Gateway container image", 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.430 INFO:teuthology.orchestra.run.smithi094.stdout: "container_init": { 2024-04-21T08:50:20.430 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_init", 2024-04-21T08:50:20.430 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.430 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.430 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.430 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.430 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.430 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.430 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.431 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T08:50:20.431 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.431 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.431 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.431 INFO:teuthology.orchestra.run.smithi094.stdout: "daemon_cache_timeout": { 2024-04-21T08:50:20.431 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "daemon_cache_timeout", 2024-04-21T08:50:20.431 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.431 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.431 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: "default_cephadm_command_timeout": { 2024-04-21T08:50:20.432 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T08:50:20.433 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.433 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.433 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.433 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "900", 2024-04-21T08:50:20.433 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.433 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.433 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.433 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T08:50:20.433 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.433 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "default_registry": { 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default_registry", 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io", 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.434 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "device_cache_timeout": { 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "device_cache_timeout", 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "1800", 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.435 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "seconds to cache device inventory", 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "device_enhanced_scan": { 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "device_enhanced_scan", 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.436 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "facts_cache_timeout": { 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "facts_cache_timeout", 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.437 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "seconds to cache host facts data", 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "grafana_dashboards_path": { 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "grafana_dashboards_path", 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.438 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "host_check_interval": { 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "host_check_interval", 2024-04-21T08:50:20.439 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to perform a host check", 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.440 INFO:teuthology.orchestra.run.smithi094.stdout: "hw_monitoring": { 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "hw_monitoring", 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "inventory_list_all": { 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "inventory_list_all", 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.442 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.443 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: "log_refresh_metadata": { 2024-04-21T08:50:20.444 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_refresh_metadata", 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.446 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.447 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.448 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.449 INFO:teuthology.orchestra.run.smithi094.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "*", 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.451 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "max_count_per_host": { 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_count_per_host", 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "max number of daemons per service per host", 2024-04-21T08:50:20.452 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "max_osd_draining_count": { 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_osd_draining_count", 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.453 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "migration_current": { 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "migration_current", 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.454 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "internal - do not modify", 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": { 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mode", 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "root", 2024-04-21T08:50:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "cephadm-package", 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "root" 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "oob_default_addr": { 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "oob_default_addr", 2024-04-21T08:50:20.456 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "169.254.1.1", 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "prometheus_alerts_path": { 2024-04-21T08:50:20.457 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "prometheus_alerts_path", 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_insecure": { 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_insecure", 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.459 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_password": { 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_password", 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T08:50:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_url": { 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_url", 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.461 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_username": { 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_username", 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.462 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "secure_monitoring_stack": { 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "secure_monitoring_stack", 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.463 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "service_discovery_port": { 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "service_discovery_port", 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.464 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8765", 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "cephadm service discovery port", 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "ssh_config_file": { 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssh_config_file", 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.465 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "use_agent": { 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "use_agent", 2024-04-21T08:50:20.466 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "use_repo_digest": { 2024-04-21T08:50:20.467 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "use_repo_digest", 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.468 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_on_failed_host_check": { 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_on_failed_host_check", 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.469 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_on_stray_daemons": { 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_on_stray_daemons", 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T08:50:20.470 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_on_stray_hosts": { 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_on_stray_hosts", 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.471 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crash", 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.472 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.473 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.474 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.475 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.477 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.478 INFO:teuthology.orchestra.run.smithi094.stdout: "retain_interval": { 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "retain_interval", 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "31536000", 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_recent_interval": { 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_recent_interval", 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "1209600", 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.480 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "dashboard", 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T08:50:20.481 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.482 INFO:teuthology.orchestra.run.smithi094.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.483 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.484 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "AUDIT_API_ENABLED": { 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.485 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T08:50:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T08:50:20.486 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.487 INFO:teuthology.orchestra.run.smithi094.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T08:50:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T08:50:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.488 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.489 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.490 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T08:50:20.491 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.492 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.493 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T08:50:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T08:50:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.494 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T08:50:20.495 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.497 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T08:50:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T08:50:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.498 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.499 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "admin", 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T08:50:20.500 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_URL": { 2024-04-21T08:50:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_URL", 2024-04-21T08:50:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.502 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "admin", 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.503 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.504 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T08:50:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T08:50:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T08:50:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T08:50:20.509 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "[]", 2024-04-21T08:50:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.510 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "{}", 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.511 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.512 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.513 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T08:50:20.514 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.515 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.516 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.517 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.518 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.519 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T08:50:20.520 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.522 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.523 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T08:50:20.524 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.525 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T08:50:20.526 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8", 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T08:50:20.527 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "45", 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.529 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "admin", 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.530 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.531 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.532 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.533 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.534 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T08:50:20.535 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.536 INFO:teuthology.orchestra.run.smithi094.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.537 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "cross_origin_url": { 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cross_origin_url", 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.538 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "crt_file": { 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crt_file", 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.539 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "debug": { 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "debug", 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.540 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Enable/disable debug options", 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "jwt_token_ttl": { 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "jwt_token_ttl", 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.541 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "28800", 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "key_file": { 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "key_file", 2024-04-21T08:50:20.542 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.543 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.544 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.545 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.546 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.547 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.548 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "motd": { 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "motd", 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.549 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "The message of the day", 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_resolve_ip_addr": { 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.550 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "server_addr": { 2024-04-21T08:50:20.551 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_addr", 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "::", 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.552 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "server_port": { 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_port", 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8080", 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.553 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: "ssl": { 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssl", 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.554 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: "ssl_server_port": { 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssl_server_port", 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.555 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.556 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8443", 2024-04-21T08:50:20.556 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.556 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.556 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.556 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.556 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.556 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.556 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.556 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_behaviour": { 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_behaviour", 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "redirect", 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect" 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.558 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.558 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.558 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.558 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_error_status_code": { 2024-04-21T08:50:20.558 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_error_status_code", 2024-04-21T08:50:20.558 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.558 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.558 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.558 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "500", 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "400", 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "599", 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "url_prefix": { 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "url_prefix", 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.559 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.560 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "devicehealth", 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: "enable_monitoring": { 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enable_monitoring", 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.561 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.562 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.562 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.562 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.562 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.562 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "monitor device health metrics", 2024-04-21T08:50:20.562 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.562 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.562 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.562 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.563 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.564 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.566 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.566 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.566 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.566 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.566 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.566 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.566 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.566 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.566 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.566 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.567 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.568 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: "mark_out_threshold": { 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mark_out_threshold", 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "2419200", 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.569 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.570 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T08:50:20.570 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.570 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.570 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.570 INFO:teuthology.orchestra.run.smithi094.stdout: "pool_name": { 2024-04-21T08:50:20.570 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pool_name", 2024-04-21T08:50:20.570 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.570 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.570 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "device_health_metrics", 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: "retention_period": { 2024-04-21T08:50:20.571 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "retention_period", 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "15552000", 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how long to retain device health metrics", 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.572 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: "scrape_frequency": { 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "scrape_frequency", 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "86400", 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.573 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "self_heal": { 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "self_heal", 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.574 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.575 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_threshold": { 2024-04-21T08:50:20.576 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_threshold", 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "7257600", 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.577 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "diskprediction_local", 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.578 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.579 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.580 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.582 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.583 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "predict_interval": { 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "predict_interval", 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "86400", 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.584 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: "predictor_model": { 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "predictor_model", 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.585 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "prophetstor", 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.586 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "influx", 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": false, 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "influxdb python module not found", 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: "batch_size": { 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "batch_size", 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.588 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5000", 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: "database": { 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "database", 2024-04-21T08:50:20.589 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "ceph", 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.590 INFO:teuthology.orchestra.run.smithi094.stdout: "hostname": { 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "hostname", 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "InfluxDB server hostname", 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.591 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "30", 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "5", 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T08:50:20.592 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.593 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.594 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.595 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.596 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.597 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "password": { 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "password", 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "password of InfluxDB server user", 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "port": { 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "port", 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8086", 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "InfluxDB server port", 2024-04-21T08:50:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: "ssl": { 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssl", 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "false", 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.601 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: "threads": { 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "threads", 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.602 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "32", 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "username": { 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "username", 2024-04-21T08:50:20.603 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "username of InfluxDB server user", 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.604 INFO:teuthology.orchestra.run.smithi094.stdout: "verify_ssl": { 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "verify_ssl", 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "true", 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.605 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "insights", 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.606 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.607 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.608 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.609 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.610 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "iostat", 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.612 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.613 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.614 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.615 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.616 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "k8sevents", 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.618 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_event_retention_days": { 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ceph_event_retention_days", 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "7", 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T08:50:20.619 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "config_check_secs": { 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "config_check_secs", 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "10", 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.620 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.621 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.623 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.624 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.625 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "localpool", 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "failure_domain": { 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "failure_domain", 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "host", 2024-04-21T08:50:20.627 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "failure domain for any created local pool", 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.628 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.629 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.630 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.631 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.633 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "min_size": { 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "min_size", 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "default min_size for any created local pool", 2024-04-21T08:50:20.634 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "num_rep": { 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "num_rep", 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "3", 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.635 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "default replica count for any created local pool", 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "pg_num": { 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pg_num", 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "128", 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.636 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "prefix": { 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "prefix", 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.637 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "name prefix for any created local pool", 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "subtree": { 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "subtree", 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.638 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "rack", 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mds_autoscaler", 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.640 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.641 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.643 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.644 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.645 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mirroring", 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.646 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.647 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.648 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.649 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.650 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.650 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.650 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.650 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.650 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.650 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.650 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.650 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.650 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.651 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.651 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.651 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.651 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.651 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.651 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.651 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.651 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.651 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.651 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.652 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "nfs", 2024-04-21T08:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.654 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.655 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.656 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.657 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.658 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.659 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "orchestrator", 2024-04-21T08:50:20.660 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "fail_fs": { 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "fail_fs", 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.661 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.662 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.663 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.664 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.665 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.666 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.667 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "orchestrator": { 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "orchestrator", 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "cephadm", 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "rook", 2024-04-21T08:50:20.669 INFO:teuthology.orchestra.run.smithi094.stdout: "test_orchestrator" 2024-04-21T08:50:20.670 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.670 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Orchestrator backend", 2024-04-21T08:50:20.670 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.670 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.670 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.670 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "osd_perf_query", 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.671 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.672 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.673 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.674 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.675 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.676 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "osd_support", 2024-04-21T08:50:20.677 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.678 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.679 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.682 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.683 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.685 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pg_autoscaler", 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.687 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.688 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.689 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.690 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.691 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.692 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.693 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.694 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "threshold": { 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "threshold", 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "3.0", 2024-04-21T08:50:20.695 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1.0", 2024-04-21T08:50:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "scaling threshold", 2024-04-21T08:50:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T08:50:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "progress", 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "allow_pg_recovery_event": { 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "allow_pg_recovery_event", 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.697 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": { 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enabled", 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.698 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.699 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.700 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.701 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.703 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.704 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.705 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "max_completed_events": { 2024-04-21T08:50:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_completed_events", 2024-04-21T08:50:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "50", 2024-04-21T08:50:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.706 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "number of past completed events to remember", 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.707 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how long the module is going to sleep", 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "prometheus", 2024-04-21T08:50:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "cache": { 2024-04-21T08:50:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cache", 2024-04-21T08:50:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.709 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "exclude_perf_counters": { 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "exclude_perf_counters", 2024-04-21T08:50:20.710 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.711 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.712 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.713 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.714 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.715 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.716 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_stats_pools": { 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rbd_stats_pools", 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.718 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "300", 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.719 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "scrape_interval": { 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "scrape_interval", 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "15.0", 2024-04-21T08:50:20.720 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "server_addr": { 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_addr", 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.721 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "::", 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.722 INFO:teuthology.orchestra.run.smithi094.stdout: "server_port": { 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_port", 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "9283", 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "stale_cache_strategy": { 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "stale_cache_strategy", 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "log", 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.724 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_behaviour": { 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_behaviour", 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "default", 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "default", 2024-04-21T08:50:20.725 INFO:teuthology.orchestra.run.smithi094.stdout: "error" 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_error_status_code": { 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_error_status_code", 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.726 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "500", 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "400", 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "599", 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rbd_support", 2024-04-21T08:50:20.727 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.729 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.730 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.731 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.732 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "max_concurrent_snap_create": { 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_concurrent_snap_create", 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.733 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "mirror_snapshot_schedule": { 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T08:50:20.734 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.735 INFO:teuthology.orchestra.run.smithi094.stdout: "trash_purge_schedule": { 2024-04-21T08:50:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "trash_purge_schedule", 2024-04-21T08:50:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.736 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "restful", 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "enable_auth": { 2024-04-21T08:50:20.737 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enable_auth", 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.738 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "key_file": { 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "key_file", 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.739 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.740 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.741 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.742 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.743 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.744 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "server_addr": { 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_addr", 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.745 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "server_port": { 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_port", 2024-04-21T08:50:20.746 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.747 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rgw", 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.748 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.749 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.750 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.751 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.753 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rook", 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.754 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.755 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.756 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.757 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.758 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.759 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_class": { 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "storage_class", 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "local", 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "selftest", 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.761 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.762 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.763 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.764 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.765 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.766 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "roption1": { 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "roption1", 2024-04-21T08:50:20.767 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.768 INFO:teuthology.orchestra.run.smithi094.stdout: "roption2": { 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "roption2", 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "xyz", 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption1": { 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption1", 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.770 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption2": { 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption2", 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.771 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption3": { 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption3", 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption4": { 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption4", 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.773 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption5": { 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption5", 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.775 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption6": { 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption6", 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.776 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.777 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.777 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.777 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.777 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.777 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption7": { 2024-04-21T08:50:20.777 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption7", 2024-04-21T08:50:20.777 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.777 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.777 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "42", 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: "testkey": { 2024-04-21T08:50:20.778 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "testkey", 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.779 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "testlkey": { 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "testlkey", 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.780 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.781 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.781 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.781 INFO:teuthology.orchestra.run.smithi094.stdout: "testnewline": { 2024-04-21T08:50:20.781 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "testnewline", 2024-04-21T08:50:20.781 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.781 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.781 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.781 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.781 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.782 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.782 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.782 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.782 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.782 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.782 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.782 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.782 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snap_schedule", 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "allow_m_granularity": { 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "allow_m_granularity", 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.783 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: "dump_on_update": { 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "dump_on_update", 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "dump database to debug log on update", 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.785 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.786 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.788 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.789 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.790 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "stats", 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.792 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.794 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.795 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.796 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.797 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "status", 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.798 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.799 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.800 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.801 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.803 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "telegraf", 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: "address": { 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "address", 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.804 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:20.805 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "15", 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.806 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.807 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.809 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.811 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.812 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "telemetry", 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_basic": { 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_basic", 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.813 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.814 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_crash": { 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_crash", 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_device": { 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_device", 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T08:50:20.816 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.817 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.817 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.817 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_ident": { 2024-04-21T08:50:20.817 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_ident", 2024-04-21T08:50:20.817 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.817 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.817 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.817 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.817 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_perf": { 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_perf", 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.818 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "contact": { 2024-04-21T08:50:20.819 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "contact", 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.820 INFO:teuthology.orchestra.run.smithi094.stdout: "description": { 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "description", 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.821 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "device_url": { 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "device_url", 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": { 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enabled", 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.823 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.824 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "24", 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "8", 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "last_opt_revision": { 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "last_opt_revision", 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.825 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "1", 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "leaderboard": { 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "leaderboard", 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.826 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "leaderboard_description": { 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "leaderboard_description", 2024-04-21T08:50:20.827 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.828 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.829 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.830 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.831 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.832 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.833 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "organization": { 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "organization", 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.834 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "proxy": { 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "proxy", 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.835 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "url": { 2024-04-21T08:50:20.836 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "url", 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test_orchestrator", 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.838 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.839 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.840 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.841 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.842 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.843 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "volumes", 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.844 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.845 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.846 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.846 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.846 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.846 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.846 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.846 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.846 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.846 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.846 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.847 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.847 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.847 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.847 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.847 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.847 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.847 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.847 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.847 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.847 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.848 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.849 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.849 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.849 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.849 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.849 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.849 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.849 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.849 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.849 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.849 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.850 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.850 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.850 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.850 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.850 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.850 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.851 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.851 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.851 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.851 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.851 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.851 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.851 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.851 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.851 INFO:teuthology.orchestra.run.smithi094.stdout: "max_concurrent_clones": { 2024-04-21T08:50:20.852 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_concurrent_clones", 2024-04-21T08:50:20.852 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.852 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.852 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.852 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "4", 2024-04-21T08:50:20.852 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.852 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.852 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.852 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T08:50:20.852 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: "periodic_async_work": { 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "periodic_async_work", 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.853 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.854 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.854 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Periodically check for async work", 2024-04-21T08:50:20.854 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.854 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.854 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.854 INFO:teuthology.orchestra.run.smithi094.stdout: "snapshot_clone_delay": { 2024-04-21T08:50:20.854 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snapshot_clone_delay", 2024-04-21T08:50:20.854 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.855 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.855 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.855 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:20.855 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.855 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.855 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.855 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T08:50:20.855 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.855 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: "snapshot_clone_no_wait": { 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.856 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix", 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.857 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.858 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.858 INFO:teuthology.orchestra.run.smithi094.stdout: "discovery_interval": { 2024-04-21T08:50:20.858 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "discovery_interval", 2024-04-21T08:50:20.858 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:20.858 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.858 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.858 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "100", 2024-04-21T08:50:20.858 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.858 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.858 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: "identifier": { 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "identifier", 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.859 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:20.860 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.862 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.862 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.862 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.862 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.862 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.862 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.862 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.862 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.862 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.862 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.863 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.863 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.863 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.863 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.863 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.863 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.863 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.863 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.863 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.863 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.864 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.865 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.866 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.866 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.866 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.866 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.866 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.866 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.866 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.866 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.866 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.866 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.867 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: "zabbix_host": { 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix_host", 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.868 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.869 INFO:teuthology.orchestra.run.smithi094.stdout: "zabbix_port": { 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix_port", 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10051", 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.870 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.871 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.871 INFO:teuthology.orchestra.run.smithi094.stdout: "zabbix_sender": { 2024-04-21T08:50:20.871 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix_sender", 2024-04-21T08:50:20.871 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.871 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.871 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.871 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T08:50:20.871 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.871 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.872 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.872 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.872 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.872 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.872 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.872 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: "gid": 25042, 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "z", 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr_features": 4540701547738038271, 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: "available_modules": [ 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "alerts", 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.873 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.874 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.875 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.876 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.877 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.878 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_destination": { 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_destination", 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Email address to send alerts to", 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.881 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_from_name": { 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_from_name", 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "Ceph", 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Email From: name", 2024-04-21T08:50:20.882 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_host": { 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_host", 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.883 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SMTP server", 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_password": { 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_password", 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.884 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Password to authenticate with", 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_port": { 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_port", 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.885 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "465", 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SMTP port", 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_sender": { 2024-04-21T08:50:20.886 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_sender", 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SMTP envelope sender", 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_ssl": { 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_ssl", 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.888 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_user": { 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_user", 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "User to authenticate as", 2024-04-21T08:50:20.889 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "balancer", 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: "active": { 2024-04-21T08:50:20.890 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "active", 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "begin_time": { 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "begin_time", 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0000", 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.892 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "begin_weekday": { 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "begin_weekday", 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "0", 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "6", 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T08:50:20.893 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_compat_max_iterations": { 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crush_compat_max_iterations", 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "25", 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "250", 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_compat_metrics": { 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crush_compat_metrics", 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.895 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_compat_step": { 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crush_compat_step", 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.896 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0.5", 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "0.001", 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "0.999", 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "aggressiveness of optimization", 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "end_time": { 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "end_time", 2024-04-21T08:50:20.897 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "2359", 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.898 INFO:teuthology.orchestra.run.smithi094.stdout: "end_weekday": { 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "end_weekday", 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "0", 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "6", 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.899 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.900 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.901 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.902 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.903 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.904 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "min_score": { 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "min_score", 2024-04-21T08:50:20.905 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.906 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": { 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mode", 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "upmap", 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "crush-compat", 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "none", 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "read", 2024-04-21T08:50:20.907 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap", 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap-read" 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Balancer mode", 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "pool_ids": { 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pool_ids", 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.908 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap_max_deviation": { 2024-04-21T08:50:20.910 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "upmap_max_deviation", 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.911 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap_max_optimizations": { 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "upmap_max_optimizations", 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cephadm", 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: "agent_down_multiplier": { 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "agent_down_multiplier", 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.913 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "3.0", 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "agent_refresh_rate": { 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "agent_refresh_rate", 2024-04-21T08:50:20.914 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "20", 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.915 INFO:teuthology.orchestra.run.smithi094.stdout: "agent_starting_port": { 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "agent_starting_port", 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "4721", 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.916 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "allow_ptrace": { 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "allow_ptrace", 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T08:50:20.917 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "autotune_interval": { 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "autotune_interval", 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: "autotune_memory_target_ratio": { 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0.7", 2024-04-21T08:50:20.919 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "cephadm_log_destination": { 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cephadm_log_destination", 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: "file", 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: "file,syslog", 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: "syslog" 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "cgroups_split": { 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cgroups_split", 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T08:50:20.922 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "config_checks_enabled": { 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "config_checks_enabled", 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.923 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.924 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T08:50:20.924 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.924 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.924 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.924 INFO:teuthology.orchestra.run.smithi094.stdout: "config_dashboard": { 2024-04-21T08:50:20.924 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "config_dashboard", 2024-04-21T08:50:20.924 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.925 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_alertmanager": { 2024-04-21T08:50:20.926 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_alertmanager", 2024-04-21T08:50:20.926 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.926 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.926 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.926 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T08:50:20.926 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.926 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.926 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.926 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_base": { 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_base", 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.927 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.928 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.928 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Container image name, without the tag", 2024-04-21T08:50:20.928 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.928 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.928 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.928 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_elasticsearch": { 2024-04-21T08:50:20.928 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_elasticsearch", 2024-04-21T08:50:20.928 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "elasticsearch container image", 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.930 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_grafana": { 2024-04-21T08:50:20.930 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_grafana", 2024-04-21T08:50:20.930 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.930 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.930 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.930 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T08:50:20.930 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.930 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.930 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.931 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:20.931 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.931 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.931 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.931 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_haproxy": { 2024-04-21T08:50:20.931 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_haproxy", 2024-04-21T08:50:20.931 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.931 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.931 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.932 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T08:50:20.932 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.932 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.932 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.932 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "HAproxy container image", 2024-04-21T08:50:20.932 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.932 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.932 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.933 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_jaeger_agent": { 2024-04-21T08:50:20.933 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_jaeger_agent", 2024-04-21T08:50:20.933 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.933 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.933 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.933 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T08:50:20.933 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.933 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.933 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Jaeger agent container image", 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_jaeger_collector": { 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_jaeger_collector", 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.934 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Jaeger collector container image", 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_jaeger_query": { 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_jaeger_query", 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.935 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Jaeger query container image", 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_keepalived": { 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_keepalived", 2024-04-21T08:50:20.936 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Keepalived container image", 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_loki": { 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_loki", 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Loki container image", 2024-04-21T08:50:20.938 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_node_exporter": { 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_node_exporter", 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.939 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_nvmeof": { 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_nvmeof", 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.940 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.941 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T08:50:20.941 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.941 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.941 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.941 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Nvme-of container image", 2024-04-21T08:50:20.941 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.941 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.941 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_prometheus": { 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_prometheus", 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:20.942 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_promtail": { 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_promtail", 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Promtail container image", 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_samba": { 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_samba", 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.944 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Samba/SMB container image", 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_snmp_gateway": { 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_snmp_gateway", 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.945 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SNMP Gateway container image", 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "container_init": { 2024-04-21T08:50:20.946 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_init", 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run.smithi094.stdout: "daemon_cache_timeout": { 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "daemon_cache_timeout", 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.948 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "default_cephadm_command_timeout": { 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "900", 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T08:50:20.949 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: "default_registry": { 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default_registry", 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io", 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "device_cache_timeout": { 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "device_cache_timeout", 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.951 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "1800", 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "seconds to cache device inventory", 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "device_enhanced_scan": { 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "device_enhanced_scan", 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.952 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "facts_cache_timeout": { 2024-04-21T08:50:20.953 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "facts_cache_timeout", 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "seconds to cache host facts data", 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.954 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "grafana_dashboards_path": { 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "grafana_dashboards_path", 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T08:50:20.955 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "host_check_interval": { 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "host_check_interval", 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.956 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to perform a host check", 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "hw_monitoring": { 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "hw_monitoring", 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.958 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T08:50:20.958 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.958 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.958 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.958 INFO:teuthology.orchestra.run.smithi094.stdout: "inventory_list_all": { 2024-04-21T08:50:20.958 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "inventory_list_all", 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.959 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.960 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.961 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.961 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.961 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.961 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.961 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.961 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.961 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.961 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.961 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.961 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: "log_refresh_metadata": { 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_refresh_metadata", 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.962 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T08:50:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.964 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.964 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.964 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.964 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T08:50:20.964 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.964 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.964 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.964 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.964 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.965 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.965 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.965 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.965 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.965 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.965 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.965 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.965 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.965 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.965 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.966 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.966 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.966 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.966 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.966 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.966 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.966 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.966 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.966 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.967 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.967 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.967 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.967 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.967 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.967 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.967 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.967 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.967 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.968 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.968 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.968 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.968 INFO:teuthology.orchestra.run.smithi094.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T08:50:20.968 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T08:50:20.968 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.968 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.968 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T08:50:20.969 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "*", 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "max_count_per_host": { 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_count_per_host", 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "max number of daemons per service per host", 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.971 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "max_osd_draining_count": { 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_osd_draining_count", 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T08:50:20.972 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "migration_current": { 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "migration_current", 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.973 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "internal - do not modify", 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": { 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mode", 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "root", 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: "cephadm-package", 2024-04-21T08:50:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: "root" 2024-04-21T08:50:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T08:50:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: "oob_default_addr": { 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "oob_default_addr", 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "169.254.1.1", 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.976 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "prometheus_alerts_path": { 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "prometheus_alerts_path", 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T08:50:20.977 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_insecure": { 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_insecure", 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.978 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_password": { 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_password", 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.979 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_url": { 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_url", 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.980 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_username": { 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_username", 2024-04-21T08:50:20.981 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.982 INFO:teuthology.orchestra.run.smithi094.stdout: "secure_monitoring_stack": { 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "secure_monitoring_stack", 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "service_discovery_port": { 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "service_discovery_port", 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8765", 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "cephadm service discovery port", 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.984 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "ssh_config_file": { 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssh_config_file", 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T08:50:20.985 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "use_agent": { 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "use_agent", 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.986 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T08:50:20.987 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.987 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.987 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.987 INFO:teuthology.orchestra.run.smithi094.stdout: "use_repo_digest": { 2024-04-21T08:50:20.987 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "use_repo_digest", 2024-04-21T08:50:20.987 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.988 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.988 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.988 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.988 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.988 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.988 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.988 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T08:50:20.988 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_on_failed_host_check": { 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_on_failed_host_check", 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.989 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_on_stray_daemons": { 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_on_stray_daemons", 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_on_stray_hosts": { 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_on_stray_hosts", 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.991 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crash", 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.993 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.995 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:20.996 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:20.997 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:20.998 INFO:teuthology.orchestra.run.smithi094.stdout: "retain_interval": { 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "retain_interval", 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "31536000", 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:20.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_recent_interval": { 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_recent_interval", 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "1209600", 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.000 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "dashboard", 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.001 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.002 INFO:teuthology.orchestra.run.smithi094.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T08:50:21.003 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T08:50:21.003 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.003 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.003 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.003 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.003 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.003 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.003 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.003 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.004 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.004 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.004 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.004 INFO:teuthology.orchestra.run.smithi094.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T08:50:21.004 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T08:50:21.004 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.005 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.005 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.005 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.005 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.005 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.005 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.006 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.006 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.006 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.006 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.006 INFO:teuthology.orchestra.run.smithi094.stdout: "AUDIT_API_ENABLED": { 2024-04-21T08:50:21.007 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T08:50:21.007 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.007 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.007 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.007 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.007 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.008 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.008 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.008 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.008 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.008 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.008 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.009 INFO:teuthology.orchestra.run.smithi094.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T08:50:21.009 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T08:50:21.009 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.009 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.009 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.009 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.009 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.009 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.009 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.009 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.010 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.010 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.010 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.010 INFO:teuthology.orchestra.run.smithi094.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T08:50:21.010 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T08:50:21.010 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.010 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.010 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T08:50:21.011 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.012 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T08:50:21.013 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T08:50:21.013 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.013 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.013 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.013 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.013 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.013 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.013 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.013 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.013 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.014 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.015 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.016 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.017 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T08:50:21.018 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T08:50:21.019 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "admin", 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.021 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.022 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_URL": { 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_URL", 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.023 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "admin", 2024-04-21T08:50:21.024 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.025 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.026 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T08:50:21.027 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "[]", 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.030 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "{}", 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.031 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.032 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.033 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.034 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T08:50:21.035 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.036 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.037 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.038 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.039 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.040 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.041 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.042 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T08:50:21.043 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T08:50:21.044 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.045 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8", 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.046 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "45", 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.047 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.048 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "admin", 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.050 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.051 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.052 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.053 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.054 INFO:teuthology.orchestra.run.smithi094.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.055 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "cross_origin_url": { 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cross_origin_url", 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.057 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "crt_file": { 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crt_file", 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.058 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "debug": { 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "debug", 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.059 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Enable/disable debug options", 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "jwt_token_ttl": { 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "jwt_token_ttl", 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.060 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "28800", 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "key_file": { 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "key_file", 2024-04-21T08:50:21.061 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.062 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.063 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.064 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.065 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.066 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "motd": { 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "motd", 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "The message of the day", 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_resolve_ip_addr": { 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "server_addr": { 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_addr", 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "::", 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.071 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "server_port": { 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_port", 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8080", 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.072 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "ssl": { 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssl", 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: "ssl_server_port": { 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssl_server_port", 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8443", 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_behaviour": { 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_behaviour", 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "redirect", 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect" 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_error_status_code": { 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_error_status_code", 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "500", 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "400", 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "599", 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "url_prefix": { 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "url_prefix", 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "devicehealth", 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: "enable_monitoring": { 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enable_monitoring", 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "monitor device health metrics", 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.084 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.085 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: "mark_out_threshold": { 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mark_out_threshold", 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "2419200", 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "pool_name": { 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pool_name", 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.087 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "device_health_metrics", 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "retention_period": { 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "retention_period", 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.088 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "15552000", 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how long to retain device health metrics", 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "scrape_frequency": { 2024-04-21T08:50:21.089 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "scrape_frequency", 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "86400", 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "self_heal": { 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "self_heal", 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.091 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T08:50:21.092 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_threshold": { 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_threshold", 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "7257600", 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.093 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "diskprediction_local", 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.094 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.095 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.096 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.097 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.098 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.099 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "predict_interval": { 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "predict_interval", 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "86400", 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "predictor_model": { 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "predictor_model", 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "prophetstor", 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "influx", 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": false, 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "influxdb python module not found", 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: "batch_size": { 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "batch_size", 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5000", 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "database": { 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "database", 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "ceph", 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi094.stdout: "hostname": { 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "hostname", 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "InfluxDB server hostname", 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "30", 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "5", 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.109 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.110 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.111 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.112 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.113 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "password": { 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "password", 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.114 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "password of InfluxDB server user", 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "port": { 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "port", 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8086", 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "InfluxDB server port", 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "ssl": { 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssl", 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.116 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "false", 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "threads": { 2024-04-21T08:50:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "threads", 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "32", 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.118 INFO:teuthology.orchestra.run.smithi094.stdout: "username": { 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "username", 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "username of InfluxDB server user", 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.119 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "verify_ssl": { 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "verify_ssl", 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "true", 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.120 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "insights", 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.121 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.122 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.123 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.124 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.125 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.126 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "iostat", 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.127 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.128 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.129 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.130 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.131 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.132 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "k8sevents", 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_event_retention_days": { 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ceph_event_retention_days", 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "7", 2024-04-21T08:50:21.134 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "config_check_secs": { 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "config_check_secs", 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.135 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "10", 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.137 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.138 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.139 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.140 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "localpool", 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: "failure_domain": { 2024-04-21T08:50:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "failure_domain", 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "host", 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "failure domain for any created local pool", 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.143 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.144 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.145 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.145 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.145 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.145 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.145 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.145 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.145 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.145 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.146 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.146 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.146 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.146 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.146 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.146 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.146 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.146 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.146 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.146 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.147 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.148 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.149 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "min_size": { 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "min_size", 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "default min_size for any created local pool", 2024-04-21T08:50:21.150 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "num_rep": { 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "num_rep", 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "3", 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.151 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "default replica count for any created local pool", 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "pg_num": { 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pg_num", 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "128", 2024-04-21T08:50:21.152 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "prefix": { 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "prefix", 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.153 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "name prefix for any created local pool", 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "subtree": { 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "subtree", 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.154 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "rack", 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mds_autoscaler", 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.156 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.159 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.160 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mirroring", 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.162 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.163 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.164 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.165 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.166 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.167 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "nfs", 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.168 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.169 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.170 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.171 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.172 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.173 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "orchestrator", 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.174 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "fail_fs": { 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "fail_fs", 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.175 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.176 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.177 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.178 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.179 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.180 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "orchestrator": { 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "orchestrator", 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.181 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: "cephadm", 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: "rook", 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: "test_orchestrator" 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Orchestrator backend", 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.182 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "osd_perf_query", 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.183 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.184 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.185 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.186 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.187 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.188 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "osd_support", 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.190 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.191 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.192 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.193 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.194 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pg_autoscaler", 2024-04-21T08:50:21.195 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.196 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.197 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.198 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.199 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.200 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.201 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "threshold": { 2024-04-21T08:50:21.202 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "threshold", 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "3.0", 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1.0", 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "scaling threshold", 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "progress", 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: "allow_pg_recovery_event": { 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "allow_pg_recovery_event", 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.204 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": { 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enabled", 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.205 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.206 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.207 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.208 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.209 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.210 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: "max_completed_events": { 2024-04-21T08:50:21.212 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_completed_events", 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "50", 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "number of past completed events to remember", 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.213 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how long the module is going to sleep", 2024-04-21T08:50:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "prometheus", 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: "cache": { 2024-04-21T08:50:21.215 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cache", 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi094.stdout: "exclude_perf_counters": { 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "exclude_perf_counters", 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.221 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.222 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_stats_pools": { 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rbd_stats_pools", 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.224 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "300", 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.225 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: "scrape_interval": { 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "scrape_interval", 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "15.0", 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.226 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "server_addr": { 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_addr", 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "::", 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.227 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "server_port": { 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_port", 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.228 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "9283", 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "stale_cache_strategy": { 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "stale_cache_strategy", 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.229 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "log", 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_behaviour": { 2024-04-21T08:50:21.230 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_behaviour", 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "default", 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "default", 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "error" 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_error_status_code": { 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_error_status_code", 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "500", 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "400", 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "599", 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.232 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rbd_support", 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.233 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.234 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.235 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.236 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.237 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.238 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: "max_concurrent_snap_create": { 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_concurrent_snap_create", 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.239 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: "mirror_snapshot_schedule": { 2024-04-21T08:50:21.240 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.241 INFO:teuthology.orchestra.run.smithi094.stdout: "trash_purge_schedule": { 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "trash_purge_schedule", 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "restful", 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: "enable_auth": { 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enable_auth", 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "key_file": { 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "key_file", 2024-04-21T08:50:21.244 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.245 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.246 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.247 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.248 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.249 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.250 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "server_addr": { 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_addr", 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.251 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "server_port": { 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_port", 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.252 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rgw", 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.253 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.254 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.255 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.256 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.257 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.258 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rook", 2024-04-21T08:50:21.259 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.260 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.261 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.262 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.263 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.264 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_class": { 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "storage_class", 2024-04-21T08:50:21.265 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "local", 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "selftest", 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.267 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.270 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.271 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.272 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "roption1": { 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "roption1", 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.273 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "roption2": { 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "roption2", 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "xyz", 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.274 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption1": { 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption1", 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.275 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption2": { 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption2", 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.276 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption3": { 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption3", 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.277 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption4": { 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption4", 2024-04-21T08:50:21.278 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption5": { 2024-04-21T08:50:21.279 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption5", 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption6": { 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption6", 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.281 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption7": { 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption7", 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "42", 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.282 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "testkey": { 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "testkey", 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.283 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "testlkey": { 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "testlkey", 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.284 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "testnewline": { 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "testnewline", 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.285 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.286 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.286 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.286 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.286 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.286 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.286 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.286 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.286 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.286 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snap_schedule", 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "allow_m_granularity": { 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "allow_m_granularity", 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.287 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "dump_on_update": { 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "dump_on_update", 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.288 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "dump database to debug log on update", 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.289 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.290 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.291 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.292 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.293 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.294 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "stats", 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.295 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.296 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.297 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.298 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.299 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.300 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.301 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "status", 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.302 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.304 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.305 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.306 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "telegraf", 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: "address": { 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "address", 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.309 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "15", 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.310 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.311 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.312 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.313 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.314 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.315 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "telemetry", 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.316 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_basic": { 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_basic", 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_crash": { 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_crash", 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T08:50:21.318 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_device": { 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_device", 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.319 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_ident": { 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_ident", 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.320 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_perf": { 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_perf", 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.321 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "contact": { 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "contact", 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.322 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "description": { 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "description", 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.323 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: "device_url": { 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "device_url", 2024-04-21T08:50:21.324 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": { 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enabled", 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.326 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "24", 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "8", 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.327 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "last_opt_revision": { 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "last_opt_revision", 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "1", 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.328 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "leaderboard": { 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "leaderboard", 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.329 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "leaderboard_description": { 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "leaderboard_description", 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.330 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.331 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.332 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.333 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.334 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.335 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.336 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "organization": { 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "organization", 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.337 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "proxy": { 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "proxy", 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.338 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "url": { 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "url", 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.339 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test_orchestrator", 2024-04-21T08:50:21.340 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.341 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.342 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.343 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.344 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.345 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.346 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "volumes", 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.347 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.348 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.350 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.351 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "max_concurrent_clones": { 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_concurrent_clones", 2024-04-21T08:50:21.352 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "4", 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "periodic_async_work": { 2024-04-21T08:50:21.353 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "periodic_async_work", 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Periodically check for async work", 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "snapshot_clone_delay": { 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snapshot_clone_delay", 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.355 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: "snapshot_clone_no_wait": { 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.356 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix", 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: "discovery_interval": { 2024-04-21T08:50:21.357 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "discovery_interval", 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "100", 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "identifier": { 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "identifier", 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.359 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.361 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.362 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.363 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.364 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.365 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "zabbix_host": { 2024-04-21T08:50:21.366 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix_host", 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "zabbix_port": { 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix_port", 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10051", 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "zabbix_sender": { 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix_sender", 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: "modules": [ 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: "cephadm", 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: "dashboard", 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: "iostat", 2024-04-21T08:50:21.370 INFO:teuthology.orchestra.run.smithi094.stdout: "nfs", 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "restful", 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "snap_schedule" 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "available_modules": [ 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "alerts", 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.371 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.372 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.374 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.375 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.376 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.377 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_destination": { 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_destination", 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.378 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Email address to send alerts to", 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_from_name": { 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_from_name", 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "Ceph", 2024-04-21T08:50:21.379 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Email From: name", 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_host": { 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_host", 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.380 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SMTP server", 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_password": { 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_password", 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.381 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Password to authenticate with", 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_port": { 2024-04-21T08:50:21.382 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_port", 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "465", 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SMTP port", 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_sender": { 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_sender", 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SMTP envelope sender", 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.384 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_ssl": { 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_ssl", 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.385 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "smtp_user": { 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "smtp_user", 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.386 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "User to authenticate as", 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "balancer", 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: "active": { 2024-04-21T08:50:21.387 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "active", 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.388 INFO:teuthology.orchestra.run.smithi094.stdout: "begin_time": { 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "begin_time", 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0000", 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "begin_weekday": { 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "begin_weekday", 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "0", 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "6", 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:50:21.390 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_compat_max_iterations": { 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crush_compat_max_iterations", 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "25", 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "250", 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_compat_metrics": { 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crush_compat_metrics", 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.392 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "crush_compat_step": { 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crush_compat_step", 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0.5", 2024-04-21T08:50:21.393 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "0.001", 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "0.999", 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "aggressiveness of optimization", 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "end_time": { 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "end_time", 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.394 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "2359", 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: "end_weekday": { 2024-04-21T08:50:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "end_weekday", 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "0", 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "6", 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.396 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.397 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.398 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.399 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.400 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.401 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "min_score": { 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "min_score", 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.402 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": { 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mode", 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.403 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "upmap", 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "crush-compat", 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "none", 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "read", 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap", 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap-read" 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Balancer mode", 2024-04-21T08:50:21.404 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "pool_ids": { 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pool_ids", 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.405 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.406 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap_max_deviation": { 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "upmap_max_deviation", 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.407 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "upmap_max_optimizations": { 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "upmap_max_optimizations", 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:21.408 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cephadm", 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "agent_down_multiplier": { 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "agent_down_multiplier", 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "3.0", 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.410 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "agent_refresh_rate": { 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "agent_refresh_rate", 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "20", 2024-04-21T08:50:21.411 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "agent_starting_port": { 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "agent_starting_port", 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.412 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "4721", 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "allow_ptrace": { 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "allow_ptrace", 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.413 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "autotune_interval": { 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "autotune_interval", 2024-04-21T08:50:21.414 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "autotune_memory_target_ratio": { 2024-04-21T08:50:21.415 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0.7", 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.416 INFO:teuthology.orchestra.run.smithi094.stdout: "cephadm_log_destination": { 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cephadm_log_destination", 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "file", 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "file,syslog", 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "syslog" 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.417 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "cgroups_split": { 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cgroups_split", 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.418 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "config_checks_enabled": { 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "config_checks_enabled", 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.419 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "config_dashboard": { 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "config_dashboard", 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.420 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_alertmanager": { 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_alertmanager", 2024-04-21T08:50:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_base": { 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_base", 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Container image name, without the tag", 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_elasticsearch": { 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_elasticsearch", 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "elasticsearch container image", 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_grafana": { 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_grafana", 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_haproxy": { 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_haproxy", 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.426 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "HAproxy container image", 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_jaeger_agent": { 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_jaeger_agent", 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.427 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Jaeger agent container image", 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_jaeger_collector": { 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_jaeger_collector", 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T08:50:21.428 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Jaeger collector container image", 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_jaeger_query": { 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_jaeger_query", 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.429 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T08:50:21.430 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.430 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.430 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.430 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Jaeger query container image", 2024-04-21T08:50:21.430 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.430 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.430 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.430 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_keepalived": { 2024-04-21T08:50:21.431 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_keepalived", 2024-04-21T08:50:21.431 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.431 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.431 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.431 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T08:50:21.431 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.431 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.431 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.432 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Keepalived container image", 2024-04-21T08:50:21.432 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.432 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.432 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.432 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_loki": { 2024-04-21T08:50:21.432 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_loki", 2024-04-21T08:50:21.432 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.432 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Loki container image", 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_node_exporter": { 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_node_exporter", 2024-04-21T08:50:21.433 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.434 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_nvmeof": { 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_nvmeof", 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Nvme-of container image", 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.435 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_prometheus": { 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_prometheus", 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Prometheus container image", 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.436 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_promtail": { 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_promtail", 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Promtail container image", 2024-04-21T08:50:21.437 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_samba": { 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_samba", 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.438 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Samba/SMB container image", 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "container_image_snmp_gateway": { 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_image_snmp_gateway", 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.439 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "SNMP Gateway container image", 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "container_init": { 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "container_init", 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "daemon_cache_timeout": { 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "daemon_cache_timeout", 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "default_cephadm_command_timeout": { 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T08:50:21.442 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "900", 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.443 INFO:teuthology.orchestra.run.smithi094.stdout: "default_registry": { 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default_registry", 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "docker.io", 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "device_cache_timeout": { 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "device_cache_timeout", 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "1800", 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "seconds to cache device inventory", 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "device_enhanced_scan": { 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "device_enhanced_scan", 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T08:50:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "facts_cache_timeout": { 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "facts_cache_timeout", 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.447 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "seconds to cache host facts data", 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "grafana_dashboards_path": { 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "grafana_dashboards_path", 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.448 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "host_check_interval": { 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "host_check_interval", 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.449 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to perform a host check", 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: "hw_monitoring": { 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "hw_monitoring", 2024-04-21T08:50:21.450 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.451 INFO:teuthology.orchestra.run.smithi094.stdout: "inventory_list_all": { 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "inventory_list_all", 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.453 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "log_refresh_metadata": { 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_refresh_metadata", 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.454 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.455 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.456 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.458 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.459 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T08:50:21.460 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "*", 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "max_count_per_host": { 2024-04-21T08:50:21.461 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_count_per_host", 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "max number of daemons per service per host", 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "max_osd_draining_count": { 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_osd_draining_count", 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "migration_current": { 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "migration_current", 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "internal - do not modify", 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.464 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "mode": { 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mode", 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "root", 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.465 INFO:teuthology.orchestra.run.smithi094.stdout: "cephadm-package", 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: "root" 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: "oob_default_addr": { 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "oob_default_addr", 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.466 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "169.254.1.1", 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "prometheus_alerts_path": { 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "prometheus_alerts_path", 2024-04-21T08:50:21.467 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_insecure": { 2024-04-21T08:50:21.468 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_insecure", 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.469 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_password": { 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_password", 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.470 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_url": { 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_url", 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T08:50:21.471 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "registry_username": { 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "registry_username", 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.472 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "secure_monitoring_stack": { 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "secure_monitoring_stack", 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.473 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "service_discovery_port": { 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "service_discovery_port", 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.474 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8765", 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "cephadm service discovery port", 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "ssh_config_file": { 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssh_config_file", 2024-04-21T08:50:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.476 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.476 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.476 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.476 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.476 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.476 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.476 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T08:50:21.476 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.476 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.476 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "use_agent": { 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "use_agent", 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.477 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "use_repo_digest": { 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "use_repo_digest", 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.478 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_on_failed_host_check": { 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_on_failed_host_check", 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.479 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_on_stray_daemons": { 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_on_stray_daemons", 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.480 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_on_stray_hosts": { 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_on_stray_hosts", 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.481 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crash", 2024-04-21T08:50:21.482 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.483 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.484 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.485 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.486 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.487 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "retain_interval": { 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "retain_interval", 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.488 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "31536000", 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_recent_interval": { 2024-04-21T08:50:21.489 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_recent_interval", 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "1209600", 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "dashboard", 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "AUDIT_API_ENABLED": { 2024-04-21T08:50:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T08:50:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "admin", 2024-04-21T08:50:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_URL": { 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_URL", 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "admin", 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "[]", 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "{}", 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T08:50:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T08:50:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T08:50:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T08:50:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8", 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "45", 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "admin", 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T08:50:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "cross_origin_url": { 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cross_origin_url", 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "crt_file": { 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "crt_file", 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "debug": { 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "debug", 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Enable/disable debug options", 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "jwt_token_ttl": { 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "jwt_token_ttl", 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "28800", 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "key_file": { 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "key_file", 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "motd": { 2024-04-21T08:50:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "motd", 2024-04-21T08:50:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "The message of the day", 2024-04-21T08:50:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_resolve_ip_addr": { 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "server_addr": { 2024-04-21T08:50:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_addr", 2024-04-21T08:50:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "::", 2024-04-21T08:50:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "server_port": { 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_port", 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8080", 2024-04-21T08:50:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "ssl": { 2024-04-21T08:50:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssl", 2024-04-21T08:50:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "ssl_server_port": { 2024-04-21T08:50:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssl_server_port", 2024-04-21T08:50:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8443", 2024-04-21T08:50:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_behaviour": { 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_behaviour", 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "redirect", 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect" 2024-04-21T08:50:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_error_status_code": { 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_error_status_code", 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "500", 2024-04-21T08:50:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "400", 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "599", 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "url_prefix": { 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "url_prefix", 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "devicehealth", 2024-04-21T08:50:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "enable_monitoring": { 2024-04-21T08:50:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enable_monitoring", 2024-04-21T08:50:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "monitor device health metrics", 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "mark_out_threshold": { 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mark_out_threshold", 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "2419200", 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "pool_name": { 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pool_name", 2024-04-21T08:50:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "device_health_metrics", 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "retention_period": { 2024-04-21T08:50:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "retention_period", 2024-04-21T08:50:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "15552000", 2024-04-21T08:50:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how long to retain device health metrics", 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "scrape_frequency": { 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "scrape_frequency", 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "86400", 2024-04-21T08:50:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "self_heal": { 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "self_heal", 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "warn_threshold": { 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "warn_threshold", 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "7257600", 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "diskprediction_local", 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "predict_interval": { 2024-04-21T08:50:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "predict_interval", 2024-04-21T08:50:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "86400", 2024-04-21T08:50:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "predictor_model": { 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "predictor_model", 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "prophetstor", 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "600", 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "influx", 2024-04-21T08:50:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": false, 2024-04-21T08:50:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "influxdb python module not found", 2024-04-21T08:50:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "batch_size": { 2024-04-21T08:50:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "batch_size", 2024-04-21T08:50:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5000", 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "database": { 2024-04-21T08:50:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "database", 2024-04-21T08:50:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "ceph", 2024-04-21T08:50:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T08:50:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "hostname": { 2024-04-21T08:50:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "hostname", 2024-04-21T08:50:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "InfluxDB server hostname", 2024-04-21T08:50:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "30", 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "5", 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T08:50:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "password": { 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "password", 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.607 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.607 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "password of InfluxDB server user", 2024-04-21T08:50:21.607 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.607 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.607 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.607 INFO:teuthology.orchestra.run.smithi094.stdout: "port": { 2024-04-21T08:50:21.607 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "port", 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "8086", 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "InfluxDB server port", 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.608 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "ssl": { 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ssl", 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "false", 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:50:21.609 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "threads": { 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "threads", 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "32", 2024-04-21T08:50:21.610 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "username": { 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "username", 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.611 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "username of InfluxDB server user", 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "verify_ssl": { 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "verify_ssl", 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "true", 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.612 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "insights", 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.613 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.614 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.615 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.617 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.618 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.619 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "iostat", 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.620 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.621 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.622 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.623 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.624 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "k8sevents", 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_event_retention_days": { 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "ceph_event_retention_days", 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "7", 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.626 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "config_check_secs": { 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "config_check_secs", 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.627 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "10", 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.628 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.629 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.631 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.632 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.633 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "localpool", 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: "failure_domain": { 2024-04-21T08:50:21.634 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "failure_domain", 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "host", 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "failure domain for any created local pool", 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.635 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.636 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.638 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.639 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.640 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "min_size": { 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "min_size", 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.641 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "default min_size for any created local pool", 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: "num_rep": { 2024-04-21T08:50:21.642 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "num_rep", 2024-04-21T08:50:21.643 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.643 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.643 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.643 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "3", 2024-04-21T08:50:21.643 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.643 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.643 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.643 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "default replica count for any created local pool", 2024-04-21T08:50:21.643 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.643 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "pg_num": { 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pg_num", 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "128", 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.644 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T08:50:21.645 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.645 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.645 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.645 INFO:teuthology.orchestra.run.smithi094.stdout: "prefix": { 2024-04-21T08:50:21.645 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "prefix", 2024-04-21T08:50:21.645 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.645 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.645 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.645 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.646 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.646 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.646 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.646 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "name prefix for any created local pool", 2024-04-21T08:50:21.646 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.646 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.646 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.646 INFO:teuthology.orchestra.run.smithi094.stdout: "subtree": { 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "subtree", 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "rack", 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.647 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mds_autoscaler", 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.648 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.649 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.650 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.651 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.652 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.653 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mirroring", 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.654 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.655 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.656 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.657 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.659 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "nfs", 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.660 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.661 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.662 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.662 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.662 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.662 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.662 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.665 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.665 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.665 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.666 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.666 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.666 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.666 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.666 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.666 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.666 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.666 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.666 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.666 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.667 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.667 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.667 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.667 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.667 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.667 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.667 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.667 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.668 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.668 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.668 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.668 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.668 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.668 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.668 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.668 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.668 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.668 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.669 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.669 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.669 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.669 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.669 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.669 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.669 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.669 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.669 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.670 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.670 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.670 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.670 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.670 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.670 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.671 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "orchestrator", 2024-04-21T08:50:21.671 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.671 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.671 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.671 INFO:teuthology.orchestra.run.smithi094.stdout: "fail_fs": { 2024-04-21T08:50:21.671 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "fail_fs", 2024-04-21T08:50:21.671 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.671 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.671 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.672 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.672 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.672 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.672 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.672 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T08:50:21.672 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.672 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.672 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.673 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.673 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.673 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.673 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.673 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.673 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.673 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.673 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.673 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.673 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.674 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.674 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.674 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.674 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.674 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.674 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.674 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.674 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.674 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.675 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.675 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.675 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.675 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.675 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.675 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.675 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.675 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.675 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.676 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.677 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "orchestrator": { 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "orchestrator", 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.679 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: "cephadm", 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: "rook", 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: "test_orchestrator" 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Orchestrator backend", 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.681 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "osd_perf_query", 2024-04-21T08:50:21.681 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.681 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.681 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.681 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.681 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.682 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.683 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.683 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.683 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.683 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.683 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.683 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.683 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.683 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.683 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.684 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.684 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.684 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.684 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.684 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.684 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.684 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.684 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.684 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.684 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.685 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.686 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.687 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "osd_support", 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.688 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.689 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.690 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.691 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.692 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.693 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "pg_autoscaler", 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.695 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.696 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.697 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.697 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.697 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.697 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.697 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.697 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.697 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.697 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.697 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.698 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.700 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "threshold": { 2024-04-21T08:50:21.702 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "threshold", 2024-04-21T08:50:21.703 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:21.703 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.703 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.703 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "3.0", 2024-04-21T08:50:21.703 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1.0", 2024-04-21T08:50:21.703 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.703 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.703 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "scaling threshold", 2024-04-21T08:50:21.703 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "progress", 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: "allow_pg_recovery_event": { 2024-04-21T08:50:21.704 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "allow_pg_recovery_event", 2024-04-21T08:50:21.705 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.705 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.705 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.705 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.705 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.705 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.705 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.705 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T08:50:21.705 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": { 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enabled", 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.707 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.708 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.709 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.710 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.711 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.711 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.711 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.711 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.711 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.711 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.711 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.711 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.711 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.712 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.712 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.712 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.712 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.712 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.712 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.712 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.712 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.713 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.713 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.713 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.713 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.713 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.713 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.713 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.713 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.713 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.714 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.714 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.714 INFO:teuthology.orchestra.run.smithi094.stdout: "max_completed_events": { 2024-04-21T08:50:21.714 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_completed_events", 2024-04-21T08:50:21.714 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.714 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.714 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.714 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "50", 2024-04-21T08:50:21.714 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.715 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.715 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.715 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "number of past completed events to remember", 2024-04-21T08:50:21.715 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.715 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.715 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.715 INFO:teuthology.orchestra.run.smithi094.stdout: "sleep_interval": { 2024-04-21T08:50:21.715 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "sleep_interval", 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "5", 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "how long the module is going to sleep", 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.716 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "prometheus", 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: "cache": { 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "cache", 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi094.stdout: "exclude_perf_counters": { 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "exclude_perf_counters", 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_stats_pools": { 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rbd_stats_pools", 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.727 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "300", 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "scrape_interval": { 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "scrape_interval", 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "15.0", 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: "server_addr": { 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_addr", 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "::", 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "server_port": { 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_port", 2024-04-21T08:50:21.731 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "9283", 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "stale_cache_strategy": { 2024-04-21T08:50:21.732 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "stale_cache_strategy", 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "log", 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_behaviour": { 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_behaviour", 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "default", 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "default", 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: "error" 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "standby_error_status_code": { 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "standby_error_status_code", 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "500", 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "400", 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "599", 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rbd_support", 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.739 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.742 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "max_concurrent_snap_create": { 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_concurrent_snap_create", 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10", 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.743 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "mirror_snapshot_schedule": { 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.744 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "trash_purge_schedule": { 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "trash_purge_schedule", 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.745 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "restful", 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "enable_auth": { 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enable_auth", 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.747 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "key_file": { 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "key_file", 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.748 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.749 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.750 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.751 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.752 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.753 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.754 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "server_addr": { 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_addr", 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.755 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "server_port": { 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "server_port", 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rgw", 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.762 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rook", 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.767 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.769 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_class": { 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "storage_class", 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "local", 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.770 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "selftest", 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.771 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.772 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.773 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.774 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.775 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.776 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "roption1": { 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "roption1", 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.777 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "roption2": { 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "roption2", 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "xyz", 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.778 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption1": { 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption1", 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.779 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption2": { 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption2", 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.780 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption3": { 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption3", 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "float", 2024-04-21T08:50:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption4": { 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption4", 2024-04-21T08:50:21.782 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption5": { 2024-04-21T08:50:21.783 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption5", 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption6": { 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption6", 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.785 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "rwoption7": { 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rwoption7", 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "1", 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "42", 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.786 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "testkey": { 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "testkey", 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.787 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "testlkey": { 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "testlkey", 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.788 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "testnewline": { 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "testnewline", 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.789 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.790 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snap_schedule", 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "allow_m_granularity": { 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "allow_m_granularity", 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.791 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "dump_on_update": { 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "dump_on_update", 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.792 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "dump database to debug log on update", 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.793 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.794 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.795 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.796 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.797 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.798 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.798 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.798 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.798 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.798 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.798 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.798 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.798 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "stats", 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.799 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.800 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.801 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.803 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.804 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "status", 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.805 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.806 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.807 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.808 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.809 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.810 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.810 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.810 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.810 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.810 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.810 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.810 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.810 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.810 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "telegraf", 2024-04-21T08:50:21.811 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "address": { 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "address", 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.812 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.813 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.813 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.813 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:21.813 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:21.813 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.813 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.813 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.813 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "15", 2024-04-21T08:50:21.813 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.814 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.814 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.814 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.814 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.814 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.814 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.814 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.814 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.816 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.816 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.816 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.816 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.816 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.816 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.816 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.816 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.818 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.818 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.818 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.818 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.818 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.818 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.818 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.818 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.818 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.820 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.820 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.820 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.820 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.820 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.820 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.820 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.820 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.820 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.820 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.821 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.821 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.821 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.821 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "telemetry", 2024-04-21T08:50:21.821 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.821 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.822 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.822 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_basic": { 2024-04-21T08:50:21.822 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_basic", 2024-04-21T08:50:21.822 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.822 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.822 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.822 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.822 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_crash": { 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_crash", 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.823 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.824 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.824 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.824 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.824 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.824 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T08:50:21.824 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.824 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.824 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.824 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_device": { 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_device", 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_ident": { 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_ident", 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T08:50:21.826 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "channel_perf": { 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "channel_perf", 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.827 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.828 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T08:50:21.828 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.828 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.828 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.828 INFO:teuthology.orchestra.run.smithi094.stdout: "contact": { 2024-04-21T08:50:21.828 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "contact", 2024-04-21T08:50:21.828 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "description": { 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "description", 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.830 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: "device_url": { 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "device_url", 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.831 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": { 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "enabled", 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.832 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.833 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "24", 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "8", 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "last_opt_revision": { 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "last_opt_revision", 2024-04-21T08:50:21.834 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "1", 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "leaderboard": { 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "leaderboard", 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.836 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "leaderboard_description": { 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "leaderboard_description", 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.837 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.838 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.839 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.840 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.841 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.842 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.843 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "organization": { 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "organization", 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.844 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: "proxy": { 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "proxy", 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: "url": { 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "url", 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test_orchestrator", 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.848 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.849 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.850 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.850 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.850 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.850 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.850 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.850 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.850 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.851 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.852 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.853 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.854 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "volumes", 2024-04-21T08:50:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.855 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.856 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.857 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.859 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.860 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.861 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.862 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.863 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "max_concurrent_clones": { 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "max_concurrent_clones", 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "4", 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.864 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "periodic_async_work": { 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "periodic_async_work", 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.865 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Periodically check for async work", 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "snapshot_clone_delay": { 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snapshot_clone_delay", 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "0", 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.866 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "snapshot_clone_no_wait": { 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.867 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "True", 2024-04-21T08:50:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T08:50:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.868 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix", 2024-04-21T08:50:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "can_run": true, 2024-04-21T08:50:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "error_string": "", 2024-04-21T08:50:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "module_options": { 2024-04-21T08:50:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "discovery_interval": { 2024-04-21T08:50:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "discovery_interval", 2024-04-21T08:50:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "uint", 2024-04-21T08:50:21.869 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "100", 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "identifier": { 2024-04-21T08:50:21.870 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "identifier", 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "interval": { 2024-04-21T08:50:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "interval", 2024-04-21T08:50:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "secs", 2024-04-21T08:50:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "60", 2024-04-21T08:50:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.872 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "log_level": { 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_level", 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.873 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.874 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster": { 2024-04-21T08:50:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster", 2024-04-21T08:50:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.875 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.876 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_cluster_level": { 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_cluster_level", 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "info", 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [ 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "", 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "critical", 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "debug", 2024-04-21T08:50:21.877 INFO:teuthology.orchestra.run.smithi094.stdout: "error", 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "info", 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "warning" 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "log_to_file": { 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "log_to_file", 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "bool", 2024-04-21T08:50:21.878 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 1, 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "False", 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "zabbix_host": { 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix_host", 2024-04-21T08:50:21.879 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "", 2024-04-21T08:50:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.880 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "zabbix_port": { 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix_port", 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "int", 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "10051", 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.881 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "zabbix_sender": { 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "zabbix_sender", 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "str", 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "level": "advanced", 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "flags": 0, 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "min": "", 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "max": "", 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "enum_allowed": [], 2024-04-21T08:50:21.882 INFO:teuthology.orchestra.run.smithi094.stdout: "desc": "", 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: "long_desc": "", 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: "see_also": [] 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: "services": { 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: "dashboard": "https://172.21.15.94:8443/" 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: "always_on_modules": { 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: "octopus": [ 2024-04-21T08:50:21.883 INFO:teuthology.orchestra.run.smithi094.stdout: "balancer", 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "crash", 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "devicehealth", 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "orchestrator", 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "pg_autoscaler", 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "progress", 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_support", 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "status", 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "telemetry", 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "volumes" 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "pacific": [ 2024-04-21T08:50:21.884 INFO:teuthology.orchestra.run.smithi094.stdout: "balancer", 2024-04-21T08:50:21.885 INFO:teuthology.orchestra.run.smithi094.stdout: "crash", 2024-04-21T08:50:21.885 INFO:teuthology.orchestra.run.smithi094.stdout: "devicehealth", 2024-04-21T08:50:21.885 INFO:teuthology.orchestra.run.smithi094.stdout: "orchestrator", 2024-04-21T08:50:21.885 INFO:teuthology.orchestra.run.smithi094.stdout: "pg_autoscaler", 2024-04-21T08:50:21.885 INFO:teuthology.orchestra.run.smithi094.stdout: "progress", 2024-04-21T08:50:21.885 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_support", 2024-04-21T08:50:21.885 INFO:teuthology.orchestra.run.smithi094.stdout: "status", 2024-04-21T08:50:21.885 INFO:teuthology.orchestra.run.smithi094.stdout: "telemetry", 2024-04-21T08:50:21.885 INFO:teuthology.orchestra.run.smithi094.stdout: "volumes" 2024-04-21T08:50:21.885 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.886 INFO:teuthology.orchestra.run.smithi094.stdout: "quincy": [ 2024-04-21T08:50:21.886 INFO:teuthology.orchestra.run.smithi094.stdout: "balancer", 2024-04-21T08:50:21.886 INFO:teuthology.orchestra.run.smithi094.stdout: "crash", 2024-04-21T08:50:21.886 INFO:teuthology.orchestra.run.smithi094.stdout: "devicehealth", 2024-04-21T08:50:21.886 INFO:teuthology.orchestra.run.smithi094.stdout: "orchestrator", 2024-04-21T08:50:21.886 INFO:teuthology.orchestra.run.smithi094.stdout: "pg_autoscaler", 2024-04-21T08:50:21.886 INFO:teuthology.orchestra.run.smithi094.stdout: "progress", 2024-04-21T08:50:21.886 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_support", 2024-04-21T08:50:21.886 INFO:teuthology.orchestra.run.smithi094.stdout: "status", 2024-04-21T08:50:21.886 INFO:teuthology.orchestra.run.smithi094.stdout: "telemetry", 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "volumes" 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "reef": [ 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "balancer", 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "crash", 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "devicehealth", 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "orchestrator", 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "pg_autoscaler", 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "progress", 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_support", 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "status", 2024-04-21T08:50:21.887 INFO:teuthology.orchestra.run.smithi094.stdout: "telemetry", 2024-04-21T08:50:21.888 INFO:teuthology.orchestra.run.smithi094.stdout: "volumes" 2024-04-21T08:50:21.888 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2024-04-21T08:50:21.888 INFO:teuthology.orchestra.run.smithi094.stdout: "squid": [ 2024-04-21T08:50:21.888 INFO:teuthology.orchestra.run.smithi094.stdout: "balancer", 2024-04-21T08:50:21.888 INFO:teuthology.orchestra.run.smithi094.stdout: "crash", 2024-04-21T08:50:21.888 INFO:teuthology.orchestra.run.smithi094.stdout: "devicehealth", 2024-04-21T08:50:21.888 INFO:teuthology.orchestra.run.smithi094.stdout: "orchestrator", 2024-04-21T08:50:21.888 INFO:teuthology.orchestra.run.smithi094.stdout: "pg_autoscaler", 2024-04-21T08:50:21.888 INFO:teuthology.orchestra.run.smithi094.stdout: "progress", 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: "rbd_support", 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: "status", 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: "telemetry", 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: "volumes" 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: "last_failure_osd_epoch": 94, 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: "active_clients": [ 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "devicehealth", 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-04-21T08:50:21.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.890 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v2", 2024-04-21T08:50:21.890 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:0", 2024-04-21T08:50:21.890 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 1416412742 2024-04-21T08:50:21.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.890 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:21.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.890 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "libcephsqlite", 2024-04-21T08:50:21.890 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v2", 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:0", 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 1115020518 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "rbd_support", 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-04-21T08:50:21.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.892 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v2", 2024-04-21T08:50:21.892 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:0", 2024-04-21T08:50:21.892 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 921689283 2024-04-21T08:50:21.892 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.892 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:21.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.892 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snap_schedule", 2024-04-21T08:50:21.893 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-04-21T08:50:21.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.893 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v2", 2024-04-21T08:50:21.893 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:0", 2024-04-21T08:50:21.893 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 299028252 2024-04-21T08:50:21.893 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.893 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:21.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.893 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "snap_schedule", 2024-04-21T08:50:21.894 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-04-21T08:50:21.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.894 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v2", 2024-04-21T08:50:21.894 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:0", 2024-04-21T08:50:21.894 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 2752938193 2024-04-21T08:50:21.894 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.894 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:21.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:21.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.895 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "volumes", 2024-04-21T08:50:21.895 INFO:teuthology.orchestra.run.smithi094.stdout: "addrvec": [ 2024-04-21T08:50:21.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-21T08:50:21.895 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v2", 2024-04-21T08:50:21.895 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:0", 2024-04-21T08:50:21.895 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 930453207 2024-04-21T08:50:21.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.895 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:21.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:21.896 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-21T08:50:21.896 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:50:21.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:20 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/522484407' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:21.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:20 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3927849834' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T08:50:21.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:20 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/522484407' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:21.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:20 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3927849834' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T08:50:21.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/3927849834 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2d040a3d20 msgr2=0x7f2d040a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2d040a3d20 0x7f2d040a61e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2d181006c0 tx=0x7f2d00009590 comp rx=0 tx=0).stop 2024-04-21T08:50:21.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/3927849834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d18101090 msgr2=0x7f2d181975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d18101090 0x7f2d181975d0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000eab0 tx=0x7f2d1000ef70 comp rx=0 tx=0).stop 2024-04-21T08:50:21.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/3927849834 shutdown_connections 2024-04-21T08:50:21.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f2d040a3d20 0x7f2d040a61e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d18106160 0x7f2d18197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d1819be90 0x7f2d1819c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 --2- 172.21.15.94:0/3927849834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d18101090 0x7f2d181975d0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/3927849834 >> 172.21.15.94:0/3927849834 conn(0x7f2d180771d0 msgr2=0x7f2d180fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/3927849834 shutdown_connections 2024-04-21T08:50:21.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:20.340+0000 7f2d1ee3a640 1 -- 172.21.15.94:0/3927849834 wait complete. 2024-04-21T08:50:21.947 DEBUG:tasks.check_counter:Getting stats from a 2024-04-21T08:50:21.948 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-21T08:50:21.948 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:20 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/522484407' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:21.948 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:20 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3927849834' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T08:50:22.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:21 smithi111 ceph-mon[31493]: pgmap v322: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T08:50:22.200 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:22.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:21 smithi094 ceph-mon[27988]: pgmap v322: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T08:50:22.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:21 smithi173 ceph-mon[31428]: pgmap v322: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T08:50:23.981 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:23 smithi094 ceph-mon[27988]: pgmap v323: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2024-04-21T08:50:24.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:23 smithi111 ceph-mon[31493]: pgmap v323: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 0, 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 2, 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 0, 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 4, 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 0, 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.001289168, 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:50:24.249 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.000252544, 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:50:24.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 6, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 163, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 3360, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 184992, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 5, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 3, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.013592969, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.009407714, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.001978626, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 161, 2024-04-21T08:50:24.251 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.011195934, 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000069539 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 3360, 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 184992 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 221, 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 211, 2024-04-21T08:50:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 1451184, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 129344, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 2, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 1, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.040022499, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.013804601, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.029252770, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.004430806, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 210, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.011585986, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000055171 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:24.253 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 1451184, 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 129344 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "cct": { 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "total_workers": 0, 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "unhealthy_workers": 0 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "mempool": { 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:24.254 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:24.255 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_items": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:24.256 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_anon_bytes": 148994, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_anon_items": 31, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_meta_items": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_bytes": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_items": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_bytes": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_items": 0, 2024-04-21T08:50:24.257 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "pgmap_items": 0, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_co_bytes": 928, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_co_items": 2, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_1_items": 0, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_2_items": 0 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 0, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "max": 104857600, 2024-04-21T08:50:24.258 INFO:teuthology.orchestra.run.smithi094.stdout: "get_started": 0, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "get": 223, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "get_sum": 1420004, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "get_or_fail_success": 223, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "take": 0, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "take_sum": 0, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "put": 223, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "put_sum": 1420004, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "wait": { 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:24.259 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:50:24.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:23 smithi173 ceph-mon[31428]: pgmap v323: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2024-04-21T08:50:25.054 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-21T08:50:25.054 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-21T08:50:25.055 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-21T08:50:25.055 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-21T08:50:25.055 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-21T08:50:25.055 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-21T08:50:25.055 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-21T08:50:25.055 DEBUG:tasks.check_counter:Getting stats from d 2024-04-21T08:50:25.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-21T08:50:25.306 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:26.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:25 smithi094 ceph-mon[27988]: pgmap v324: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:50:26.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:25 smithi111 ceph-mon[31493]: pgmap v324: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:50:26.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:25 smithi173 ceph-mon[31428]: pgmap v324: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T08:50:27.054 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:50:27.054 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:27.054 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 226, 2024-04-21T08:50:27.054 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 216, 2024-04-21T08:50:27.054 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 1452544, 2024-04-21T08:50:27.054 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 132544, 2024-04-21T08:50:27.054 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 4, 2024-04-21T08:50:27.054 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 0, 2024-04-21T08:50:27.054 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.046546622, 2024-04-21T08:50:27.054 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.014384582, 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.035155605, 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.004516491, 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 215, 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.012884789, 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000059929 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 1452544, 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 132544 2024-04-21T08:50:27.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 6, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 167, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 3360, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 185696, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 5, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 3, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.014371641, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.009779859, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.002330287, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 165, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.013202340, 2024-04-21T08:50:27.056 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000080014 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 3360, 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 185696 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 0, 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 2, 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 0, 2024-04-21T08:50:27.057 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 2, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 1, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.000303377, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.000250964, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:27.058 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "cct": { 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "total_workers": 0, 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "unhealthy_workers": 0 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "mempool": { 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:27.060 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_items": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:27.061 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_anon_bytes": 149057, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_anon_items": 32, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_meta_items": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_bytes": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_items": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_bytes": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_items": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:27.062 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "pgmap_items": 0, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_co_bytes": 928, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_co_items": 2, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_1_items": 0, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_2_items": 0 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 0, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "max": 104857600, 2024-04-21T08:50:27.063 INFO:teuthology.orchestra.run.smithi094.stdout: "get_started": 0, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "get": 228, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "get_sum": 1420654, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "get_or_fail_success": 228, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "take": 0, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "take_sum": 0, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "put": 228, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "put_sum": 1420654, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "wait": { 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:27.064 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:50:27.692 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-21T08:50:27.692 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-21T08:50:27.692 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-21T08:50:27.692 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-21T08:50:27.692 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-21T08:50:27.692 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-21T08:50:27.692 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-21T08:50:27.692 DEBUG:tasks.check_counter:Getting stats from g 2024-04-21T08:50:27.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-21T08:50:27.932 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:28.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:27 smithi094 ceph-mon[27988]: pgmap v325: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:28.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:27 smithi111 ceph-mon[31493]: pgmap v325: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:28.262 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:27 smithi173 ceph-mon[31428]: pgmap v325: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:29.625 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 0, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 2, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 0, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 4, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 0, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.001092048, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.000174342, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:29.626 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 2, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 162, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 184720, 2024-04-21T08:50:29.627 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 5, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 3, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.014539022, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.010191109, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.001785655, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 159, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.013249617, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000083330 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.628 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 184720 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 227, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 215, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 1454768, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 131072, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 2, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 1, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.043550140, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.014143344, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.032960456, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.004483082, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 215, 2024-04-21T08:50:29.629 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.012632896, 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000058757 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 1454768, 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 131072 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "cct": { 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "total_workers": 0, 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "unhealthy_workers": 0 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.630 INFO:teuthology.orchestra.run.smithi094.stdout: "mempool": { 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:29.631 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:29.632 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_items": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_anon_bytes": 149017, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_anon_items": 31, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_meta_items": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_bytes": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_items": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:29.633 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_bytes": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_items": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "pgmap_items": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_co_bytes": 928, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_co_items": 2, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_1_items": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_2_items": 0 2024-04-21T08:50:29.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 0, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "max": 104857600, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "get_started": 0, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "get": 225, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "get_sum": 1420264, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "get_or_fail_success": 225, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "take": 0, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "take_sum": 0, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "put": 225, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "put_sum": 1420264, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "wait": { 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:29.635 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:29.636 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:29.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:29.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:29.636 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:50:30.181 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:29 smithi094 ceph-mon[27988]: pgmap v326: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:30.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:29 smithi111 ceph-mon[31493]: pgmap v326: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:30.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:29 smithi173 ceph-mon[31428]: pgmap v326: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:30.335 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-21T08:50:30.335 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-21T08:50:30.335 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-04-21T08:50:30.335 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-04-21T08:50:30.335 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-21T08:50:30.335 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-21T08:50:30.336 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-21T08:50:30.336 DEBUG:tasks.check_counter:Getting stats from j 2024-04-21T08:50:30.336 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-21T08:50:30.575 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:32.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:31 smithi094 ceph-mon[27988]: pgmap v327: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:50:32.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:31 smithi111 ceph-mon[31493]: pgmap v327: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:50:32.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:31 smithi173 ceph-mon[31428]: pgmap v327: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:50:32.356 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 232, 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 220, 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 1456128, 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 134272, 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 4, 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 0, 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.046694002, 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.014824792, 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.034705710, 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.004379152, 2024-04-21T08:50:32.357 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 220, 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.011416308, 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000051892 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 1456128, 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 134272 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.358 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 2, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 166, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 185424, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 5, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 3, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.014286712, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.009681937, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.002160196, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 163, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.012962742, 2024-04-21T08:50:32.359 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000079526 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 185424 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_messages": 0, 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages": 2, 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:50:32.360 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_bytes": 0, 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_created_connections": 2, 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_active_connections": 1, 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_total_time": 0.000293497, 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_recv_time": 0.000241397, 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:32.361 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: "cct": { 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: "total_workers": 0, 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: "unhealthy_workers": 0 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: "mempool": { 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:32.362 INFO:teuthology.orchestra.run.smithi094.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi094.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_items": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_anon_bytes": 148994, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_anon_items": 31, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "buffer_meta_items": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_bytes": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_items": 0, 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_bytes": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_items": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "pgmap_items": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_co_bytes": 928, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_co_items": 2, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_1_items": 0, 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "unittest_2_items": 0 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 0, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "max": 104857600, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "get_started": 0, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "get": 230, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "get_sum": 1420914, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "get_or_fail_success": 230, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "take": 0, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "take_sum": 0, 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi094.stdout: "put": 230, 2024-04-21T08:50:32.368 INFO:teuthology.orchestra.run.smithi094.stdout: "put_sum": 1420914, 2024-04-21T08:50:32.368 INFO:teuthology.orchestra.run.smithi094.stdout: "wait": { 2024-04-21T08:50:32.368 INFO:teuthology.orchestra.run.smithi094.stdout: "avgcount": 0, 2024-04-21T08:50:32.368 INFO:teuthology.orchestra.run.smithi094.stdout: "sum": 0.000000000, 2024-04-21T08:50:32.368 INFO:teuthology.orchestra.run.smithi094.stdout: "avgtime": 0.000000000 2024-04-21T08:50:32.368 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:32.368 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-21T08:50:32.368 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-21T08:50:33.074 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-21T08:50:33.074 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-21T08:50:33.074 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-21T08:50:33.074 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-21T08:50:33.075 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-21T08:50:33.075 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-21T08:50:33.075 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-21T08:50:33.075 DEBUG:tasks.check_counter:Getting stats from b 2024-04-21T08:50:33.075 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-21T08:50:33.324 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:50:34.187 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:33 smithi111 ceph-mon[31493]: pgmap v328: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:34.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:33 smithi094 ceph-mon[27988]: pgmap v328: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:34.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:33 smithi173 ceph-mon[31428]: pgmap v328: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:35.140 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T08:50:35.140 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:35.140 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 6059, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 5286, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1740645, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 10397513, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 33, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 4, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.540294808, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.164992147, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.144298440, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.182734257, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 5287, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.679374487, 2024-04-21T08:50:35.141 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000128499 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1810, 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000380963, 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000210 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 344816, 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 88512 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 3523, 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 3497, 2024-04-21T08:50:35.142 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 2343939, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 6803549, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 167, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 5, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.337273610, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.092133022, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.141432757, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.091070572, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3496, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.626740257, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000179273 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:35.143 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 421, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000068329, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000162 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1604752, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 514304 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 8168, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 9057, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 2639026, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 3279531, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 14, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 7, 2024-04-21T08:50:35.144 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.575451812, 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.202326726, 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.396058425, 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.124257316, 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9052, 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.077670250, 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000119053 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9632, 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002641173, 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000274 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.145 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 42720, 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 140544 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 5392, 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 2.681827093, 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000497371 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:35.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4343, 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.134939452, 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000031070 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 174, 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 174, 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 174, 2024-04-21T08:50:35.147 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.169616151, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000974805 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 0, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 60, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 34, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 7, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 27, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 0, 2024-04-21T08:50:35.148 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 23, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 0, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 2, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 2, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 2, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 241, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 233, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 0, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 0, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-21T08:50:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 0, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 0, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 0, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 0, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 216, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 4, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 234, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 6, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 216, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-21T08:50:35.150 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 139, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 19, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 120, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 77, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 0, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 12, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 1695, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 4390, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 4374, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 4, 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 26 2024-04-21T08:50:35.151 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 6, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 0, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 13, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 2, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 5, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 3, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-21T08:50:35.152 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 19, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-21T08:50:35.153 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 4516, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 4457, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 4457, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 59, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 5, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 16, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 15, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 15, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 1, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 1, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 1, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 16006280, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 16083096, 2024-04-21T08:50:35.154 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 0, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4485, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 313.453749562, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.069889353 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 0 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 27, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 49, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 22, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 20, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 20, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 0, 2024-04-21T08:50:35.155 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 34, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 39, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 5, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 2, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 20, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 18, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 59512, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 231800 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 218, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 1, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 174, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 42, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 2, 2024-04-21T08:50:35.156 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 7, 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.007603320, 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001086188 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 113, 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.023239948, 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000205663 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.157 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1, 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000184948, 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000184948 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.158 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-21T08:50:35.159 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.159 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.159 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.159 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-21T08:50:35.159 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.159 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.159 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.159 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10, 2024-04-21T08:50:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001422178, 2024-04-21T08:50:35.161 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000142217 2024-04-21T08:50:35.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.161 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-21T08:50:35.161 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 8, 2024-04-21T08:50:35.161 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.013603417, 2024-04-21T08:50:35.161 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001700427 2024-04-21T08:50:35.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.161 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-21T08:50:35.161 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10, 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.056146201, 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.005614620 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1, 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000235008, 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000235008 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.162 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.005812890, 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002906445 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002863419, 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001431709 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.163 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.008079919, 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.004039959 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4, 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.020518251, 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.005129562 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.164 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001181902, 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000590951 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.165 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9, 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.022188942, 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002465438 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.166 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3, 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.006535808, 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002178602 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-21T08:50:35.167 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 2, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 7, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 5, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 2, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 27, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 13, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 603, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.168 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:35.169 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:35.170 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 358705, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 144, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 792, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 9, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:35.171 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9816, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 190, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 126852, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 1049, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-21T08:50:35.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 10074, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 11981178, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 37, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 10074, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10074, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 373.741415459, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.037099604 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-21T08:50:35.173 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10074, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 10265 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 10074, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 257, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 9817, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 40, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 38, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 22, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 4488, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 57, 2024-04-21T08:50:35.174 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 5129, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 235, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 69, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-21T08:50:35.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-21T08:50:35.176 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 102, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 18, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 12, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 12, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 171, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 0, 2024-04-21T08:50:35.177 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 16 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 1, 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-21T08:50:35.178 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 3585, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 1, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 4613, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 3, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 17746, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 5057556, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 17746, 2024-04-21T08:50:35.179 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T08:50:35.180 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T08:50:35.180 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 17746, 2024-04-21T08:50:35.181 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 5057556, 2024-04-21T08:50:35.181 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:35.181 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.181 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.181 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.181 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 10074, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 11981521, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 10074, 2024-04-21T08:50:35.182 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 11981521, 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:35.183 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 10074, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 10074, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 10074, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 10074, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:35.184 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 3, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 343, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 3, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 3, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 343, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:35.185 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x55ef017c8500": { 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 4516, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 11888792, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 4516, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 4485, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 11888792, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:35.186 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:35.187 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:35.187 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:35.187 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:35.187 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T08:50:35.802 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-21T08:50:35.803 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-21T08:50:35.803 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=6 2024-04-21T08:50:35.803 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-04-21T08:50:35.803 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-21T08:50:35.803 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-04-21T08:50:35.803 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-04-21T08:50:35.804 DEBUG:tasks.check_counter:Getting stats from e 2024-04-21T08:50:35.804 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-21T08:50:36.047 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:50:36.176 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:35 smithi111 ceph-mon[31493]: pgmap v329: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:50:36.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:35 smithi094 ceph-mon[27988]: pgmap v329: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:50:36.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:35 smithi173 ceph-mon[31428]: pgmap v329: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 774, 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 774, 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 208331, 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 245372, 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 6, 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 2, 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.070856286, 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.029262918, 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.016208129, 2024-04-21T08:50:37.774 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.017491801, 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 773, 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.030867971, 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000039932 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.775 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 701, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 865, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 172296, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 668671, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 7, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 5, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.075725330, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.027721998, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.015385382, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.023988008, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 862, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.037678789, 2024-04-21T08:50:37.776 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000043710 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 446384 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 1008, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 996, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1741136, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 378624, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 3, 2024-04-21T08:50:37.777 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 2, 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.146285413, 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.044363788, 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.062200183, 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.039298997, 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 995, 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.061524164, 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000061833 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.778 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1485776, 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 136944 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3702, 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.115764577, 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000031270 2024-04-21T08:50:37.779 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 0, 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 0, 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.780 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 0, 2024-04-21T08:50:37.781 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 1, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-21T08:50:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 0, 2024-04-21T08:50:37.783 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 0, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 0 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-21T08:50:37.784 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-21T08:50:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 0, 2024-04-21T08:50:37.786 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 0, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 0, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 11, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 0, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 0, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 0, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 0, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 2, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 2, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 0, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 4194304, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 4197328, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 4197187, 2024-04-21T08:50:37.787 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 11 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 3, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 3, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 0, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 1, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 1, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 0, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 0, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 0, 2024-04-21T08:50:37.788 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 0, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 0, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 0, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 0, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 50520, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 215416 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 0, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 0, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 0, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 0, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 0, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.789 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-21T08:50:37.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.791 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.792 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-21T08:50:37.793 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.794 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.794 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.794 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-21T08:50:37.794 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.794 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-21T08:50:37.795 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.796 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.798 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.799 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-21T08:50:37.800 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 1, 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 5, 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 4, 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 4, 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 0, 2024-04-21T08:50:37.801 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 0, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 5199, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:37.802 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:37.803 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:37.804 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 189411, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 52, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9816, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 190, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 8512, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 13, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 2229, 2024-04-21T08:50:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 21, 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 3, 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 2229, 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2229, 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.321976638, 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000593080 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2229, 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 2230 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 2229, 2024-04-21T08:50:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 2229, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 1457, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 770, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 0, 2024-04-21T08:50:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 1, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-21T08:50:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-21T08:50:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 102, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 13, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 5, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 5, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 0, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 2, 2024-04-21T08:50:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 0 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 0, 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-21T08:50:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 0, 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 0, 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 0, 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 2479, 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 1882885, 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 2479, 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2479, 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1882885, 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T08:50:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 2229, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 3029, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2229, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 3029, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 2229, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 2229, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2229, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 2229, 2024-04-21T08:50:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x55a5bdbe46e0": { 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:37.820 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T08:50:38.092 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:37 smithi111 ceph-mon[31493]: pgmap v330: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:38.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:37 smithi094 ceph-mon[27988]: pgmap v330: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:38.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:37 smithi173 ceph-mon[31428]: pgmap v330: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:38.614 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-21T08:50:38.615 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-21T08:50:38.615 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-21T08:50:38.615 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-21T08:50:38.615 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-21T08:50:38.615 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-21T08:50:38.615 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T08:50:38.615 DEBUG:tasks.check_counter:Getting stats from h 2024-04-21T08:50:38.615 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-21T08:50:38.853 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:50:40.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:39 smithi111 ceph-mon[31493]: pgmap v331: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:40.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:39 smithi094 ceph-mon[27988]: pgmap v331: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:40.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:39 smithi173 ceph-mon[31428]: pgmap v331: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:40.565 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T08:50:40.565 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:40.565 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 399, 2024-04-21T08:50:40.565 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 537, 2024-04-21T08:50:40.565 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 107761, 2024-04-21T08:50:40.565 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 253360, 2024-04-21T08:50:40.565 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 9, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 2, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.060768308, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.026657167, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.024372211, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.004364999, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 539, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.042827270, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000079456 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 870, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000213060, 2024-04-21T08:50:40.566 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000244 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 928, 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 976 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 342, 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 331, 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1505195, 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 626722, 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 46, 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 6, 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.066277027, 2024-04-21T08:50:40.567 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.017790602, 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.049899442, 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.006283018, 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 330, 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.025000678, 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000075759 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 55, 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000006206, 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000112 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1494752, 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 143968 2024-04-21T08:50:40.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 39, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 235, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 10435, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 511856, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 15, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 5, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.022940654, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.010392834, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.009831750, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000113113, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 229, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.024195644, 2024-04-21T08:50:40.569 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000105657 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 78, 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000016687, 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000213 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 6464, 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 453824 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-21T08:50:40.570 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 66, 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001720408, 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000026066 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000006892, 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000003446 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:40.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 10, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 7, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 7, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.003410805, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000487257 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 3, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 11, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-21T08:50:40.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 14, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 1, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 13, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 0, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 13, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 1, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 2, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 2, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 2, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 19, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 11, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 3, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 3, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-21T08:50:40.573 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 4, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 234, 2024-04-21T08:50:40.574 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 6, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T08:50:40.575 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 0, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 0, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 0, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 0 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 8, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 5, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 7, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 0, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 0, 2024-04-21T08:50:40.576 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-21T08:50:40.577 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 11, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 0, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 0, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 11, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 0, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 2, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 0, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 0, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 2, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 2, 2024-04-21T08:50:40.578 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 0, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 4194304, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 4197328, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 0, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.018752631, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001875263 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 0 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 17, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 17, 2024-04-21T08:50:40.579 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 0, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 16, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 17, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 1, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 14, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 15, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 1, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 2, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 6, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 4, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 52380, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 223608 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-21T08:50:40.580 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 14, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 0, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 10, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 25, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 1, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002103841, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001051920 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-21T08:50:40.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3, 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001031143, 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000343714 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.582 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000275821, 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000137910 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-21T08:50:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-21T08:50:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.590 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 1, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 5, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 4, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 1, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-21T08:50:40.592 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 6, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 1, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 644, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:40.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:40.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 256283, 2024-04-21T08:50:40.595 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 92, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9816, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 190, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 69872, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 269, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 50, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 13767, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 33, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 50, 2024-04-21T08:50:40.597 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 50, 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.234995902, 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.004699918 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 50, 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 75 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 50, 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 0, 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 50, 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-21T08:50:40.598 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 0, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 10, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 0, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 10, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 8, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 6, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-21T08:50:40.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 20, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-21T08:50:40.600 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 102, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-21T08:50:40.601 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 13, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 8, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 8, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 20, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 0, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 1 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 1, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-21T08:50:40.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 0, 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 0, 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 0, 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:40.603 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 776, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 1535399, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 776, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 776, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1535399, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:40.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 50, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 13767, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 50, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 13767, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.605 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 50, 2024-04-21T08:50:40.606 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 50, 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 50, 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 50, 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T08:50:40.607 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x5578c0c1c500": { 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:40.608 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 11, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 3024, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 11, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 10, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 3024, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:40.609 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:40.610 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:40.610 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T08:50:41.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:40 smithi094 ceph-mon[27988]: pgmap v332: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:50:41.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:40 smithi173 ceph-mon[31428]: pgmap v332: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:50:41.373 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:40 smithi111 ceph-mon[31493]: pgmap v332: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:50:41.401 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-21T08:50:41.402 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-21T08:50:41.402 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-04-21T08:50:41.402 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=8 2024-04-21T08:50:41.402 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-21T08:50:41.402 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-21T08:50:41.402 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T08:50:41.402 DEBUG:tasks.check_counter:Getting stats from k 2024-04-21T08:50:41.402 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-21T08:50:41.639 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:50:43.336 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T08:50:43.336 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:43.336 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 0, 2024-04-21T08:50:43.336 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 2, 2024-04-21T08:50:43.336 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:50:43.336 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 0, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 4, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 0, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.000858909, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.000095348, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:43.337 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 6, 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 171, 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 3360, 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 186400, 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 5, 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2024-04-21T08:50:43.338 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.012425678, 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.007771884, 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.002189299, 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 169, 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.015605986, 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000092343 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 3360, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 186400 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 232, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 222, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1454176, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 136384, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 2, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 1, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.051523217, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.015487299, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.037431635, 2024-04-21T08:50:43.340 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.006279881, 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 221, 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.011590082, 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000052443 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1454176, 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 136384 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.341 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 0, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:43.342 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:43.343 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 148682, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 29, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:50:43.344 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:43.345 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 928, 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 2, 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T08:50:43.346 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T08:50:43.347 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 234, 2024-04-21T08:50:43.349 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 1421434, 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 234, 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 234, 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1421434, 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T08:50:43.350 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T08:50:43.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:43 smithi111 ceph-mon[31493]: pgmap v333: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:43.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:43 smithi094 ceph-mon[27988]: pgmap v333: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:43.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:43 smithi173 ceph-mon[31428]: pgmap v333: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:44.175 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-21T08:50:44.260 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-21T08:50:44.260 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-04-21T08:50:44.260 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-04-21T08:50:44.260 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-21T08:50:44.260 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-21T08:50:44.260 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-21T08:50:44.260 DEBUG:tasks.check_counter:Getting stats from c 2024-04-21T08:50:44.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-21T08:50:44.511 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:50:45.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:45 smithi173 ceph-mon[31428]: pgmap v334: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:50:45.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:45 smithi111 ceph-mon[31493]: pgmap v334: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:50:45.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:45 smithi094 ceph-mon[27988]: pgmap v334: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:50:46.243 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-21T08:50:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 1029, 2024-04-21T08:50:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 1031, 2024-04-21T08:50:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 4520551, 2024-04-21T08:50:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 327232, 2024-04-21T08:50:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-04-21T08:50:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-04-21T08:50:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.122722054, 2024-04-21T08:50:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.035043367, 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.035590223, 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.040430978, 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1029, 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.052112597, 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000050643 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 1128, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 1291, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 3788868, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 808718, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 7, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.094067356, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.035275322, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.022674430, 2024-04-21T08:50:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.028052512, 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1289, 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.068884217, 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000053440 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3360, 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 451856 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 494, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 482, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 5740911, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 215337, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 3, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.071512981, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.023256799, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.041207382, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.013547236, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 481, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.026653771, 2024-04-21T08:50:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000055413 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 1486208, 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 137744 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-21T08:50:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4000, 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.117584957, 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000029396 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-04-21T08:50:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-04-21T08:50:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-21T08:50:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-04-21T08:50:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-04-21T08:50:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-21T08:50:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 28, 2024-04-21T08:50:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 9, 2024-04-21T08:50:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 19, 2024-04-21T08:50:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-21T08:50:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 16, 2024-04-21T08:50:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 2, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-04-21T08:50:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 4, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 234, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 6, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-21T08:50:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T08:50:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-21T08:50:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-21T08:50:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T08:50:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T08:50:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-21T08:50:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-21T08:50:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T08:50:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-04-21T08:50:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T08:50:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T08:50:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T08:50:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-04-21T08:50:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-04-21T08:50:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-04-21T08:50:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-04-21T08:50:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-21T08:50:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-21T08:50:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-21T08:50:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-21T08:50:46.257 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-04-21T08:50:46.257 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-04-21T08:50:46.257 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-21T08:50:46.257 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-21T08:50:46.257 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-21T08:50:46.257 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-04-21T08:50:46.257 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-04-21T08:50:46.257 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-21T08:50:46.257 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-21T08:50:46.257 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-21T08:50:46.258 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 4516, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 11, 2024-04-21T08:50:46.259 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 16006280, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 16083096, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 16082960, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 4516 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 19, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 22, 2024-04-21T08:50:46.260 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 3, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 12, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 12, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 0, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 28, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 28, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 0, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 53332, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 215416 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-04-21T08:50:46.261 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.262 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.263 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.265 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-21T08:50:46.266 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.266 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.266 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.266 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-21T08:50:46.266 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-21T08:50:46.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.268 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.269 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.270 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.272 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 2, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 7, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 5, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 5, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 3576, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 189706, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 60, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-21T08:50:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 9816, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 190, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 90148, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 645, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 2395, 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 26, 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 4, 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 2395, 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2395, 2024-04-21T08:50:46.278 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 3.645915490, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001522302 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2395, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2396 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 2395, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 2395, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 1532, 2024-04-21T08:50:46.279 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 860, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 2, 2024-04-21T08:50:46.280 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-21T08:50:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-21T08:50:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 102, 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 13, 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 5, 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 5, 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.283 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-21T08:50:46.284 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 2647, 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 13796234, 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:46.285 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 2647, 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2647, 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 13796234, 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-21T08:50:46.286 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 2395, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 11888802, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2395, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 11888802, 2024-04-21T08:50:46.287 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:46.288 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:46.289 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:46.289 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 2395, 2024-04-21T08:50:46.289 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 2395, 2024-04-21T08:50:46.289 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2395, 2024-04-21T08:50:46.289 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 2395, 2024-04-21T08:50:46.289 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:46.289 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.289 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.289 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.289 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:46.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.290 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-21T08:50:46.290 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:46.290 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-21T08:50:46.290 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:46.290 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:46.290 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:46.290 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:46.290 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:46.290 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55e9c8bba6e0": { 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:46.291 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:46.292 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-21T08:50:47.087 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-21T08:50:47.088 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-21T08:50:47.088 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-21T08:50:47.088 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-04-21T08:50:47.088 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-21T08:50:47.088 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-21T08:50:47.088 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T08:50:47.088 DEBUG:tasks.check_counter:Getting stats from f 2024-04-21T08:50:47.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-21T08:50:47.328 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:50:47.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:47 smithi111 ceph-mon[31493]: pgmap v335: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:47.450 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:47 smithi173 ceph-mon[31428]: pgmap v335: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:47.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:47 smithi094 ceph-mon[27988]: pgmap v335: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:49.028 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 2200, 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 2200, 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 627116, 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 698783, 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.191866058, 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.063931500, 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.047955560, 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.064712942, 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2199, 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.085950021, 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000039085 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 252, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 240, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 1486876, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 138922, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 3, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.047750671, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.014942132, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.034885417, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.005801468, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 239, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.014529567, 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000060793 2024-04-21T08:50:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 1486320, 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 138224 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 3, 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 170, 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 588, 2024-04-21T08:50:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 453694, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.013482559, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.006903188, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.004740334, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000026143, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 167, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.014022696, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000083968 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 453376 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-21T08:50:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 3740, 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.101095373, 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000027030 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-04-21T08:50:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 1, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-04-21T08:50:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-21T08:50:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T08:50:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-04-21T08:50:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-21T08:50:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2024-04-21T08:50:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 5, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 4195815, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 4195674, 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-21T08:50:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 5 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 3, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 3, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 0, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 1, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 1, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 0, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 0, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 0, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 0, 2024-04-21T08:50:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 50592, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 215416 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-21T08:50:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-21T08:50:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-21T08:50:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-21T08:50:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-21T08:50:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-21T08:50:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-21T08:50:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-21T08:50:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 2, 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 1, 2024-04-21T08:50:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 2, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 2992, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 181313, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 47, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 9816, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 190, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 8512, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 13, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-21T08:50:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 2199, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 21, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 3, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 2199, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2199, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1.458966152, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000663468 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2199, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2200 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 2199, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 2199, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 1419, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 778, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2024-04-21T08:50:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 1, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-21T08:50:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-21T08:50:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 102, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 13, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 4, 2024-04-21T08:50:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 4, 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-04-21T08:50:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-21T08:50:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-21T08:50:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T08:50:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-21T08:50:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-04-21T08:50:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-04-21T08:50:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-04-21T08:50:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-21T08:50:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-21T08:50:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 2451, 2024-04-21T08:50:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 1878118, 2024-04-21T08:50:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 2451, 2024-04-21T08:50:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-21T08:50:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-21T08:50:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2451, 2024-04-21T08:50:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1878118, 2024-04-21T08:50:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 2199, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 1516, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2199, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1516, 2024-04-21T08:50:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 2199, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 2199, 2024-04-21T08:50:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2199, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 2199, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55d14c1766e0": { 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-21T08:50:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:49.078 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-21T08:50:49.261 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:49 smithi173 ceph-mon[31428]: pgmap v336: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:49.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:49 smithi111 ceph-mon[31493]: pgmap v336: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:49.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:49 smithi094 ceph-mon[27988]: pgmap v336: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:49.792 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-21T08:50:49.792 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-21T08:50:49.792 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-21T08:50:49.792 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-21T08:50:49.792 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-21T08:50:49.792 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-21T08:50:49.792 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T08:50:49.793 DEBUG:tasks.check_counter:Getting stats from i 2024-04-21T08:50:49.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-21T08:50:50.038 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:50:51.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:51 smithi111 ceph-mon[31493]: pgmap v337: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:50:51.494 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:51 smithi173 ceph-mon[31428]: pgmap v337: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:50:51.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:51 smithi094 ceph-mon[27988]: pgmap v337: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T08:50:51.752 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-21T08:50:51.752 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:51.752 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 30, 2024-04-21T08:50:51.752 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 164, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 7022, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 405273, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 16, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.016407213, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.005681207, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.004510252, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000427918, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 161, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.025163476, 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000156294 2024-04-21T08:50:51.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 6, 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000588, 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000098 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 322576 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 427, 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 602, 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 110332, 2024-04-21T08:50:51.754 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 724112, 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 42, 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 8, 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.068306072, 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.024617282, 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.035767591, 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.004443952, 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 601, 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.041988649, 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000069864 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:51.755 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 923, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000283027, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000306 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 5376, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 134704 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 302, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 301, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 1502186, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 204919, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 10, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.051479808, 2024-04-21T08:50:51.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.015682234, 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.038212672, 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.005931334, 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 300, 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.021165868, 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000070552 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 34, 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000006900, 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000202 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 1498800, 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 147696 2024-04-21T08:50:51.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 47, 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002130050, 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000045320 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:51.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000073646, 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000036823 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 4, 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 3, 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-21T08:50:51.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 3, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000589077, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000196359 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 1, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 11, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 14, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 1, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 13, 2024-04-21T08:50:51.760 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 12, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 2, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 11, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 9, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 2, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-21T08:50:51.761 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 4, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 234, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 6, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-21T08:50:51.762 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-04-21T08:50:51.763 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 8, 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 5, 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 6, 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-21T08:50:51.764 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-04-21T08:50:51.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 5, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 5, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 2, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-21T08:50:51.766 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 4195815, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 0, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.006405611, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001601402 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 0 2024-04-21T08:50:51.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 17, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 17, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 0, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 16, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 17, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 1, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 14, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 14, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 0, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 3, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 3, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 52468, 2024-04-21T08:50:51.768 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 223608 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 4, 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 4, 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 12, 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-04-21T08:50:51.769 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000236859, 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000236859 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-21T08:50:51.770 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000352218, 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000176109 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-21T08:50:51.771 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-21T08:50:51.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.773 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.773 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-21T08:50:51.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.773 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.774 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.777 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.778 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.779 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 2, 2024-04-21T08:50:51.780 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 1, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 2, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 2, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 587, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 255527, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 93, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 88, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 1, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 9816, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 190, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:51.784 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 69120, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 263, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 43, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 12128, 2024-04-21T08:50:51.785 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 30, 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 43, 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 43, 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.210995137, 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.004906863 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 43, 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 67 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 43, 2024-04-21T08:50:51.786 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 0, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 43, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 0, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 10, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 0, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 4, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 8, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 6, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-21T08:50:51.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 20, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-21T08:50:51.788 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-21T08:50:51.789 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 102, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 13, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 9, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 9, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 17, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 0, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 2 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 1, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-21T08:50:51.790 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-21T08:50:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 755, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 1533637, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 755, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 755, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1533637, 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:51.792 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 43, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 12128, 2024-04-21T08:50:51.793 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 43, 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 12128, 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:51.794 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 43, 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 43, 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 43, 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 43, 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-21T08:50:51.795 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-21T08:50:51.796 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x559a3c07a500": { 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 5, 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 1511, 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:51.797 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 5, 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 4, 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1511, 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:51.798 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-21T08:50:52.546 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-21T08:50:52.546 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-21T08:50:52.546 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-04-21T08:50:52.546 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=8 2024-04-21T08:50:52.546 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-21T08:50:52.546 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-21T08:50:52.546 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T08:50:52.546 DEBUG:tasks.check_counter:Getting stats from l 2024-04-21T08:50:52.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-21T08:50:52.785 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:50:53.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:53 smithi111 ceph-mon[31493]: pgmap v338: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:53.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:53 smithi094 ceph-mon[27988]: pgmap v338: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:53.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:53 smithi173 ceph-mon[31428]: pgmap v338: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:54.390 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-21T08:50:54.390 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:50:54.390 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 239, 2024-04-21T08:50:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 227, 2024-04-21T08:50:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 1458032, 2024-04-21T08:50:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 138752, 2024-04-21T08:50:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-04-21T08:50:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 0, 2024-04-21T08:50:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.049387876, 2024-04-21T08:50:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.015111084, 2024-04-21T08:50:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.034479871, 2024-04-21T08:50:54.392 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.006165999, 2024-04-21T08:50:54.392 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:54.392 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 227, 2024-04-21T08:50:54.392 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.013530856, 2024-04-21T08:50:54.392 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000059607 2024-04-21T08:50:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 1458032, 2024-04-21T08:50:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 138752 2024-04-21T08:50:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:50:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 2, 2024-04-21T08:50:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 173, 2024-04-21T08:50:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:50:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 186656, 2024-04-21T08:50:54.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2024-04-21T08:50:54.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2024-04-21T08:50:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.012519378, 2024-04-21T08:50:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.008545374, 2024-04-21T08:50:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.001517773, 2024-04-21T08:50:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 170, 2024-04-21T08:50:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.015804180, 2024-04-21T08:50:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000092965 2024-04-21T08:50:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:50:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 186656 2024-04-21T08:50:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:50:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 0, 2024-04-21T08:50:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 2, 2024-04-21T08:50:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:50:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 0, 2024-04-21T08:50:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 2, 2024-04-21T08:50:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 1, 2024-04-21T08:50:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.000255234, 2024-04-21T08:50:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:50:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.000201637, 2024-04-21T08:50:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:50:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:50:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 0, 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:50:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:50:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:50:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:50:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:50:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:50:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:50:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:50:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:50:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-21T08:50:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:50:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-21T08:50:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:50:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:50:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:50:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:50:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:50:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:50:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:50:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 148732, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 31, 2024-04-21T08:50:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:50:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 928, 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 2, 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-21T08:50:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 237, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 1421824, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 237, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 237, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1421824, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-21T08:50:54.412 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-21T08:50:55.084 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-21T08:50:55.084 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-21T08:50:55.084 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-21T08:50:55.084 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-21T08:50:55.084 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-21T08:50:55.084 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-21T08:50:55.084 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-21T08:50:55.084 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-21T08:50:55.096 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T08:50:55.096 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:50:55.097 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:50:55.117 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:50:55.117 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:50:55.176 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:50:55.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.323+0000 7f04cfa31640 1 -- 172.21.15.94:0/1448743980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8107360 msgr2=0x7f04c806ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.323+0000 7f04cfa31640 1 --2- 172.21.15.94:0/1448743980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8107360 0x7f04c806ab80 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f04b8009900 tx=0x7f04b802bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:55.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 -- 172.21.15.94:0/1448743980 shutdown_connections 2024-04-21T08:50:55.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 --2- 172.21.15.94:0/1448743980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04c8104a00 0x7f04c8104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 --2- 172.21.15.94:0/1448743980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8107360 0x7f04c806ab80 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 --2- 172.21.15.94:0/1448743980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04c8106ef0 0x7f04c806a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 -- 172.21.15.94:0/1448743980 >> 172.21.15.94:0/1448743980 conn(0x7f04c8077e30 msgr2=0x7f04c8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:55.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 -- 172.21.15.94:0/1448743980 shutdown_connections 2024-04-21T08:50:55.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 -- 172.21.15.94:0/1448743980 wait complete. 2024-04-21T08:50:55.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 Processor -- start 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 -- start start 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04c8104a00 0x7f04c819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8106ef0 0x7f04c819f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04c81a3690 0x7f04c81a3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f04c81187d0 con 0x7f04c8106ef0 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f04c8118650 con 0x7f04c81a3690 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.324+0000 7f04cfa31640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f04c8118950 con 0x7f04c8104a00 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04ccfcc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8106ef0 0x7f04c819f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04ccfcc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8106ef0 0x7f04c819f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40786/0 (socket says 172.21.15.94:40786) 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04ccfcc640 1 -- 172.21.15.94:0/2593172979 learned_addr learned my addr 172.21.15.94:0/2593172979 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04cd7cd640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04c8104a00 0x7f04c819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04cdfce640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04c81a3690 0x7f04c81a3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04ccfcc640 1 -- 172.21.15.94:0/2593172979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04c8104a00 msgr2=0x7f04c819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04ccfcc640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04c8104a00 0x7f04c819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04ccfcc640 1 -- 172.21.15.94:0/2593172979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04c81a3690 msgr2=0x7f04c81a3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04ccfcc640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04c81a3690 0x7f04c81a3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04ccfcc640 1 -- 172.21.15.94:0/2593172979 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c81a4180 con 0x7f04c8106ef0 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04cdfce640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04c81a3690 0x7f04c81a3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04ccfcc640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8106ef0 0x7f04c819f310 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f04b8005940 tx=0x7f04b802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04be7fc640 1 -- 172.21.15.94:0/2593172979 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8043020 con 0x7f04c8106ef0 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04be7fc640 1 -- 172.21.15.94:0/2593172979 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04b802cbe0 con 0x7f04c8106ef0 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04cfa31640 1 -- 172.21.15.94:0/2593172979 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c81a4470 con 0x7f04c8106ef0 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04be7fc640 1 -- 172.21.15.94:0/2593172979 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8003bf0 con 0x7f04c8106ef0 2024-04-21T08:50:55.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.325+0000 7f04cfa31640 1 -- 172.21.15.94:0/2593172979 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c81abc60 con 0x7f04c8106ef0 2024-04-21T08:50:55.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.326+0000 7f04cfa31640 1 -- 172.21.15.94:0/2593172979 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c8069230 con 0x7f04c8106ef0 2024-04-21T08:50:55.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.327+0000 7f04be7fc640 1 -- 172.21.15.94:0/2593172979 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f04b8046020 con 0x7f04c8106ef0 2024-04-21T08:50:55.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.328+0000 7f04be7fc640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f04b40a3d20 0x7f04b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.328+0000 7f04cd7cd640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f04b40a3d20 0x7f04b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.328+0000 7f04be7fc640 1 -- 172.21.15.94:0/2593172979 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f04b80f33a0 con 0x7f04c8106ef0 2024-04-21T08:50:55.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.329+0000 7f04cd7cd640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f04b40a3d20 0x7f04b40a61e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f04c4005fd0 tx=0x7f04c400b390 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:55.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.332+0000 7f04be7fc640 1 -- 172.21.15.94:0/2593172979 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04b804d020 con 0x7f04c8106ef0 2024-04-21T08:50:55.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:55 smithi111 ceph-mon[31493]: pgmap v339: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:50:55.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:55 smithi094 ceph-mon[27988]: pgmap v339: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:50:55.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.510+0000 7f04cfa31640 1 -- 172.21.15.94:0/2593172979 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04c81073c0 con 0x7f04c8106ef0 2024-04-21T08:50:55.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:55 smithi173 ceph-mon[31428]: pgmap v339: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-21T08:50:55.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.512+0000 7f04be7fc640 1 -- 172.21.15.94:0/2593172979 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+25452 (secure 0 0 0) 0x7f04b80bf720 con 0x7f04c8106ef0 2024-04-21T08:50:55.512 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:50:55.512 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":102,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:50:11.852327+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:50:55.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 -- 172.21.15.94:0/2593172979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f04b40a3d20 msgr2=0x7f04b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f04b40a3d20 0x7f04b40a61e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f04c4005fd0 tx=0x7f04c400b390 comp rx=0 tx=0).stop 2024-04-21T08:50:55.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 -- 172.21.15.94:0/2593172979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8106ef0 msgr2=0x7f04c819f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8106ef0 0x7f04c819f310 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f04b8005940 tx=0x7f04b802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 -- 172.21.15.94:0/2593172979 shutdown_connections 2024-04-21T08:50:55.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f04b40a3d20 0x7f04b40a61e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04c81a3690 0x7f04c81a3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8106ef0 0x7f04c819f310 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 --2- 172.21.15.94:0/2593172979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04c8104a00 0x7f04c819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 -- 172.21.15.94:0/2593172979 >> 172.21.15.94:0/2593172979 conn(0x7f04c8077e30 msgr2=0x7f04c80fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:55.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 -- 172.21.15.94:0/2593172979 shutdown_connections 2024-04-21T08:50:55.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.514+0000 7f04cfa31640 1 -- 172.21.15.94:0/2593172979 wait complete. 2024-04-21T08:50:55.525 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:32:55.571794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:37:28.427740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:37:29.734210+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:37:33.577891+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:50:55.526 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:50:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.627+0000 7f5796a70640 1 -- 172.21.15.94:0/4081164813 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f579006a570 msgr2=0x7f579006a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.627+0000 7f5796a70640 1 --2- 172.21.15.94:0/4081164813 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f579006a570 0x7f579006a9f0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f5780009930 tx=0x7f578002bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.627+0000 7f5796a70640 1 -- 172.21.15.94:0/4081164813 shutdown_connections 2024-04-21T08:50:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.627+0000 7f5796a70640 1 --2- 172.21.15.94:0/4081164813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f579006af30 0x7f579010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.627+0000 7f5796a70640 1 --2- 172.21.15.94:0/4081164813 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f579006a570 0x7f579006a9f0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.627+0000 7f5796a70640 1 --2- 172.21.15.94:0/4081164813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5790106f90 0x7f5790107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.627+0000 7f5796a70640 1 -- 172.21.15.94:0/4081164813 >> 172.21.15.94:0/4081164813 conn(0x7f5790077e30 msgr2=0x7f5790078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.627+0000 7f5796a70640 1 -- 172.21.15.94:0/4081164813 shutdown_connections 2024-04-21T08:50:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.627+0000 7f5796a70640 1 -- 172.21.15.94:0/4081164813 wait complete. 2024-04-21T08:50:55.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5796a70640 1 Processor -- start 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5796a70640 1 -- start start 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5796a70640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f579006a570 0x7f5790198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5796a70640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f579006af30 0x7f5790198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5796a70640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5790106f90 0x7f579019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5796a70640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5790112210 con 0x7f579006af30 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5796a70640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5790112090 con 0x7f5790106f90 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5796a70640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5790112390 con 0x7f579006a570 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5787fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f579006af30 0x7f5790198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5787fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f579006af30 0x7f5790198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40802/0 (socket says 172.21.15.94:40802) 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5787fff640 1 -- 172.21.15.94:0/2509678832 learned_addr learned my addr 172.21.15.94:0/2509678832 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f579500d640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5790106f90 0x7f579019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f579480c640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f579006a570 0x7f5790198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5787fff640 1 -- 172.21.15.94:0/2509678832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f579006a570 msgr2=0x7f5790198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5787fff640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f579006a570 0x7f5790198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5787fff640 1 -- 172.21.15.94:0/2509678832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5790106f90 msgr2=0x7f579019cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5787fff640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5790106f90 0x7f579019cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f5787fff640 1 -- 172.21.15.94:0/2509678832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f579019d6f0 con 0x7f579006af30 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.628+0000 7f579500d640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5790106f90 0x7f579019cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.629+0000 7f5787fff640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f579006af30 0x7f5790198c90 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f5780009230 tx=0x7f57800045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.629+0000 7f5785ffb640 1 -- 172.21.15.94:0/2509678832 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780043020 con 0x7f579006af30 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.629+0000 7f5796a70640 1 -- 172.21.15.94:0/2509678832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f579019d980 con 0x7f579006af30 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.629+0000 7f5785ffb640 1 -- 172.21.15.94:0/2509678832 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f578002cb70 con 0x7f579006af30 2024-04-21T08:50:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.629+0000 7f5785ffb640 1 -- 172.21.15.94:0/2509678832 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780003b50 con 0x7f579006af30 2024-04-21T08:50:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.629+0000 7f5796a70640 1 -- 172.21.15.94:0/2509678832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57901a5270 con 0x7f579006af30 2024-04-21T08:50:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.630+0000 7f5796a70640 1 -- 172.21.15.94:0/2509678832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5790199a50 con 0x7f579006af30 2024-04-21T08:50:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.631+0000 7f5785ffb640 1 -- 172.21.15.94:0/2509678832 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5780046020 con 0x7f579006af30 2024-04-21T08:50:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.632+0000 7f5785ffb640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57780a3d20 0x7f57780a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.632+0000 7f579480c640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57780a3d20 0x7f57780a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.632+0000 7f5785ffb640 1 -- 172.21.15.94:0/2509678832 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f57800f31b0 con 0x7f579006af30 2024-04-21T08:50:55.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.633+0000 7f579480c640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57780a3d20 0x7f57780a61e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f578c005fd0 tx=0x7f578c005850 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.636+0000 7f5785ffb640 1 -- 172.21.15.94:0/2509678832 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f578004d020 con 0x7f579006af30 2024-04-21T08:50:55.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.816+0000 7f5796a70640 1 -- 172.21.15.94:0/2509678832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f579010a500 con 0x7f579006af30 2024-04-21T08:50:55.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.816+0000 7f5785ffb640 1 -- 172.21.15.94:0/2509678832 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7f57800bf530 con 0x7f579006af30 2024-04-21T08:50:55.816 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-21T08:50:55.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 -- 172.21.15.94:0/2509678832 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57780a3d20 msgr2=0x7f57780a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57780a3d20 0x7f57780a61e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f578c005fd0 tx=0x7f578c005850 comp rx=0 tx=0).stop 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 -- 172.21.15.94:0/2509678832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f579006af30 msgr2=0x7f5790198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f579006af30 0x7f5790198c90 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f5780009230 tx=0x7f57800045e0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 -- 172.21.15.94:0/2509678832 shutdown_connections 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57780a3d20 0x7f57780a61e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5790106f90 0x7f579019cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f579006af30 0x7f5790198c90 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 --2- 172.21.15.94:0/2509678832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f579006a570 0x7f5790198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 -- 172.21.15.94:0/2509678832 >> 172.21.15.94:0/2509678832 conn(0x7f5790077e30 msgr2=0x7f57900feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 -- 172.21.15.94:0/2509678832 shutdown_connections 2024-04-21T08:50:55.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.818+0000 7f5796a70640 1 -- 172.21.15.94:0/2509678832 wait complete. 2024-04-21T08:50:55.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:50:55.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.930+0000 7f225f57e640 1 -- 172.21.15.94:0/2135936976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2258101090 msgr2=0x7f2258101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.930+0000 7f225f57e640 1 --2- 172.21.15.94:0/2135936976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2258101090 0x7f2258101470 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f2254009900 tx=0x7f225402bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:55.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 -- 172.21.15.94:0/2135936976 shutdown_connections 2024-04-21T08:50:55.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 --2- 172.21.15.94:0/2135936976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2258106160 0x7f225810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 --2- 172.21.15.94:0/2135936976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2258101a40 0x7f2258105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 --2- 172.21.15.94:0/2135936976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2258101090 0x7f2258101470 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 -- 172.21.15.94:0/2135936976 >> 172.21.15.94:0/2135936976 conn(0x7f2258077f50 msgr2=0x7f2258078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:55.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 -- 172.21.15.94:0/2135936976 shutdown_connections 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 -- 172.21.15.94:0/2135936976 wait complete. 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 Processor -- start 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 -- start start 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2258101090 0x7f225819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2258106160 0x7f225819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22581985f0 0x7f2258198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f22581110a0 con 0x7f22581985f0 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2258110f20 con 0x7f2258101090 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225f57e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2258111220 con 0x7f2258106160 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.931+0000 7f225db1b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22581985f0 0x7f2258198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225db1b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22581985f0 0x7f2258198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40820/0 (socket says 172.21.15.94:40820) 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225db1b640 1 -- 172.21.15.94:0/3065649537 learned_addr learned my addr 172.21.15.94:0/3065649537 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:55.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225cb19640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2258106160 0x7f225819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225d31a640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2258101090 0x7f225819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225db1b640 1 -- 172.21.15.94:0/3065649537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2258106160 msgr2=0x7f225819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225db1b640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2258106160 0x7f225819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225db1b640 1 -- 172.21.15.94:0/3065649537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2258101090 msgr2=0x7f225819e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225db1b640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2258101090 0x7f225819e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225db1b640 1 -- 172.21.15.94:0/3065649537 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2258199110 con 0x7f22581985f0 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225cb19640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2258106160 0x7f225819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225db1b640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22581985f0 0x7f2258198aa0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f2250009fd0 tx=0x7f225000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f224a7fc640 1 -- 172.21.15.94:0/3065649537 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250017070 con 0x7f22581985f0 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225f57e640 1 -- 172.21.15.94:0/3065649537 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2258199400 con 0x7f22581985f0 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f224a7fc640 1 -- 172.21.15.94:0/3065649537 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2250003e20 con 0x7f22581985f0 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f224a7fc640 1 -- 172.21.15.94:0/3065649537 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250010460 con 0x7f22581985f0 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.932+0000 7f225f57e640 1 -- 172.21.15.94:0/3065649537 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22581a5600 con 0x7f22581985f0 2024-04-21T08:50:55.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.933+0000 7f225f57e640 1 -- 172.21.15.94:0/3065649537 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2258068d70 con 0x7f22581985f0 2024-04-21T08:50:55.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.934+0000 7f224a7fc640 1 -- 172.21.15.94:0/3065649537 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f2250003970 con 0x7f22581985f0 2024-04-21T08:50:55.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.935+0000 7f224a7fc640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22440a3d20 0x7f22440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:55.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.935+0000 7f224a7fc640 1 -- 172.21.15.94:0/3065649537 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f22500d09d0 con 0x7f22581985f0 2024-04-21T08:50:55.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.935+0000 7f225d31a640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22440a3d20 0x7f22440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:55.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.935+0000 7f225d31a640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22440a3d20 0x7f22440a61e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2254000c00 tx=0x7f2254004040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:55.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:55.938+0000 7f224a7fc640 1 -- 172.21.15.94:0/3065649537 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f225009cc40 con 0x7f22581985f0 2024-04-21T08:50:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.116+0000 7f225f57e640 1 -- 172.21.15.94:0/3065649537 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2258101a40 con 0x7f22581985f0 2024-04-21T08:50:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.117+0000 7f224a7fc640 1 -- 172.21.15.94:0/3065649537 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7f225009c540 con 0x7f22581985f0 2024-04-21T08:50:56.117 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:50:56.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 -- 172.21.15.94:0/3065649537 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22440a3d20 msgr2=0x7f22440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22440a3d20 0x7f22440a61e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2254000c00 tx=0x7f2254004040 comp rx=0 tx=0).stop 2024-04-21T08:50:56.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 -- 172.21.15.94:0/3065649537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22581985f0 msgr2=0x7f2258198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22581985f0 0x7f2258198aa0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f2250009fd0 tx=0x7f225000b260 comp rx=0 tx=0).stop 2024-04-21T08:50:56.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 -- 172.21.15.94:0/3065649537 shutdown_connections 2024-04-21T08:50:56.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f22440a3d20 0x7f22440a61e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2258106160 0x7f225819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22581985f0 0x7f2258198aa0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 --2- 172.21.15.94:0/3065649537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2258101090 0x7f225819e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 -- 172.21.15.94:0/3065649537 >> 172.21.15.94:0/3065649537 conn(0x7f2258077f50 msgr2=0x7f2258069500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:56.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 -- 172.21.15.94:0/3065649537 shutdown_connections 2024-04-21T08:50:56.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.119+0000 7f225f57e640 1 -- 172.21.15.94:0/3065649537 wait complete. 2024-04-21T08:50:56.130 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:50:56.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.233+0000 7f89da8c1640 1 -- 172.21.15.94:0/3169432297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d41031d0 msgr2=0x7f89d41035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.233+0000 7f89da8c1640 1 --2- 172.21.15.94:0/3169432297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d41031d0 0x7f89d41035b0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009900 tx=0x7f89c402bd60 comp rx=0 tx=0).stop 2024-04-21T08:50:56.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.233+0000 7f89da8c1640 1 -- 172.21.15.94:0/3169432297 shutdown_connections 2024-04-21T08:50:56.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.233+0000 7f89da8c1640 1 --2- 172.21.15.94:0/3169432297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89d4103af0 0x7f89d410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.233+0000 7f89da8c1640 1 --2- 172.21.15.94:0/3169432297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89d410c510 0x7f89d410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.233+0000 7f89da8c1640 1 --2- 172.21.15.94:0/3169432297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d41031d0 0x7f89d41035b0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.233+0000 7f89da8c1640 1 -- 172.21.15.94:0/3169432297 >> 172.21.15.94:0/3169432297 conn(0x7f89d4077f50 msgr2=0x7f89d4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:56.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.233+0000 7f89da8c1640 1 -- 172.21.15.94:0/3169432297 shutdown_connections 2024-04-21T08:50:56.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.233+0000 7f89da8c1640 1 -- 172.21.15.94:0/3169432297 wait complete. 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.234+0000 7f89da8c1640 1 Processor -- start 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.234+0000 7f89da8c1640 1 -- start start 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.234+0000 7f89da8c1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89d41031d0 0x7f89d4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.234+0000 7f89da8c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4103af0 0x7f89d4198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.234+0000 7f89da8c1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89d410c510 0x7f89d419cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.234+0000 7f89da8c1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4112180 con 0x7f89d4103af0 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.234+0000 7f89da8c1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4112000 con 0x7f89d41031d0 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.234+0000 7f89da8c1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4112300 con 0x7f89d410c510 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4103af0 0x7f89d4198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4103af0 0x7f89d4198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40848/0 (socket says 172.21.15.94:40848) 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d37fe640 1 -- 172.21.15.94:0/3089839426 learned_addr learned my addr 172.21.15.94:0/3089839426 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d3fff640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89d41031d0 0x7f89d4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d8e5e640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89d410c510 0x7f89d419cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d37fe640 1 -- 172.21.15.94:0/3089839426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89d410c510 msgr2=0x7f89d419cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d37fe640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89d410c510 0x7f89d419cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d37fe640 1 -- 172.21.15.94:0/3089839426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89d41031d0 msgr2=0x7f89d4198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d37fe640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89d41031d0 0x7f89d4198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d37fe640 1 -- 172.21.15.94:0/3089839426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d419d710 con 0x7f89d4103af0 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d8e5e640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89d410c510 0x7f89d419cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d37fe640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4103af0 0x7f89d4198c60 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f89c800c890 tx=0x7f89c800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d17fa640 1 -- 172.21.15.94:0/3089839426 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8004220 con 0x7f89d4103af0 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89da8c1640 1 -- 172.21.15.94:0/3089839426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d419da00 con 0x7f89d4103af0 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d17fa640 1 -- 172.21.15.94:0/3089839426 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89c80043c0 con 0x7f89d4103af0 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89d17fa640 1 -- 172.21.15.94:0/3089839426 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8004a00 con 0x7f89d4103af0 2024-04-21T08:50:56.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.235+0000 7f89da8c1640 1 -- 172.21.15.94:0/3089839426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d41a5290 con 0x7f89d4103af0 2024-04-21T08:50:56.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.236+0000 7f89da8c1640 1 -- 172.21.15.94:0/3089839426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d4068d70 con 0x7f89d4103af0 2024-04-21T08:50:56.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.237+0000 7f89d17fa640 1 -- 172.21.15.94:0/3089839426 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f89c80079b0 con 0x7f89d4103af0 2024-04-21T08:50:56.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.238+0000 7f89d17fa640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89bc0a3d20 0x7f89bc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.238+0000 7f89d3fff640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89bc0a3d20 0x7f89bc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.238+0000 7f89d17fa640 1 -- 172.21.15.94:0/3089839426 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f89c80d0980 con 0x7f89d4103af0 2024-04-21T08:50:56.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.238+0000 7f89d3fff640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89bc0a3d20 0x7f89bc0a61e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009230 tx=0x7f89c4005fd0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:56.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.242+0000 7f89d17fa640 1 -- 172.21.15.94:0/3089839426 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f89c8014030 con 0x7f89d4103af0 2024-04-21T08:50:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.418+0000 7f89da8c1640 1 -- 172.21.15.94:0/3089839426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f89d4107100 con 0x7f89d4103af0 2024-04-21T08:50:56.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.419+0000 7f89d17fa640 1 -- 172.21.15.94:0/3089839426 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7f89c809cbf0 con 0x7f89d4103af0 2024-04-21T08:50:56.419 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-21T08:50:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 -- 172.21.15.94:0/3089839426 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89bc0a3d20 msgr2=0x7f89bc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89bc0a3d20 0x7f89bc0a61e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009230 tx=0x7f89c4005fd0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 -- 172.21.15.94:0/3089839426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4103af0 msgr2=0x7f89d4198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4103af0 0x7f89d4198c60 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f89c800c890 tx=0x7f89c800cd50 comp rx=0 tx=0).stop 2024-04-21T08:50:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 -- 172.21.15.94:0/3089839426 shutdown_connections 2024-04-21T08:50:56.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f89bc0a3d20 0x7f89bc0a61e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89d410c510 0x7f89d419cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89d4103af0 0x7f89d4198c60 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 --2- 172.21.15.94:0/3089839426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89d41031d0 0x7f89d4198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 -- 172.21.15.94:0/3089839426 >> 172.21.15.94:0/3089839426 conn(0x7f89d4077f50 msgr2=0x7f89d40fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 -- 172.21.15.94:0/3089839426 shutdown_connections 2024-04-21T08:50:56.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.421+0000 7f89da8c1640 1 -- 172.21.15.94:0/3089839426 wait complete. 2024-04-21T08:50:56.430 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:56 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2593172979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:56.430 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:56 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2509678832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:56.431 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:56 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3065649537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:56.432 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:50:56.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:56 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2593172979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:56.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:56 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2509678832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:56.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:56 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3065649537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:56.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:56 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2593172979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:56.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:56 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2509678832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:56.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:56 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3065649537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:56.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.532+0000 7f576d18f640 1 -- 172.21.15.94:0/307640911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5768106f30 msgr2=0x7f5768107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.532+0000 7f576d18f640 1 --2- 172.21.15.94:0/307640911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5768106f30 0x7f5768107310 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f5758009930 tx=0x7f575802bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:56.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.532+0000 7f576d18f640 1 -- 172.21.15.94:0/307640911 shutdown_connections 2024-04-21T08:50:56.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.532+0000 7f576d18f640 1 --2- 172.21.15.94:0/307640911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5768100450 0x7f57681008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.532+0000 7f576d18f640 1 --2- 172.21.15.94:0/307640911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5768100e10 0x7f576810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.532+0000 7f576d18f640 1 --2- 172.21.15.94:0/307640911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5768106f30 0x7f5768107310 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.532+0000 7f576d18f640 1 -- 172.21.15.94:0/307640911 >> 172.21.15.94:0/307640911 conn(0x7f5768077410 msgr2=0x7f57680fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:56.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.532+0000 7f576d18f640 1 -- 172.21.15.94:0/307640911 shutdown_connections 2024-04-21T08:50:56.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.532+0000 7f576d18f640 1 -- 172.21.15.94:0/307640911 wait complete. 2024-04-21T08:50:56.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.533+0000 7f576d18f640 1 Processor -- start 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.533+0000 7f576d18f640 1 -- start start 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.533+0000 7f576d18f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5768100450 0x7f576819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.533+0000 7f576d18f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5768100e10 0x7f576819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.533+0000 7f576d18f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5768106f30 0x7f5768198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.533+0000 7f576d18f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5768112290 con 0x7f5768100450 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.533+0000 7f576d18f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5768112110 con 0x7f5768100e10 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.533+0000 7f576d18f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5768112410 con 0x7f5768106f30 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.533+0000 7f5766d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5768100450 0x7f576819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f5766d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5768100450 0x7f576819e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40866/0 (socket says 172.21.15.94:40866) 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f5766d9d640 1 -- 172.21.15.94:0/2502501131 learned_addr learned my addr 172.21.15.94:0/2502501131 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f576759e640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5768106f30 0x7f5768198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f576659c640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5768100e10 0x7f576819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f5766d9d640 1 -- 172.21.15.94:0/2502501131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5768106f30 msgr2=0x7f5768198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f5766d9d640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5768106f30 0x7f5768198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f5766d9d640 1 -- 172.21.15.94:0/2502501131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5768100e10 msgr2=0x7f576819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f5766d9d640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5768100e10 0x7f576819eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f5766d9d640 1 -- 172.21.15.94:0/2502501131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5768199030 con 0x7f5768100450 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f576659c640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5768100e10 0x7f576819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f5766d9d640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5768100450 0x7f576819e5c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f5758009230 tx=0x7f575802c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f574ffff640 1 -- 172.21.15.94:0/2502501131 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758043020 con 0x7f5768100450 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f576d18f640 1 -- 172.21.15.94:0/2502501131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57681992c0 con 0x7f5768100450 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f574ffff640 1 -- 172.21.15.94:0/2502501131 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f575802caf0 con 0x7f5768100450 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f574ffff640 1 -- 172.21.15.94:0/2502501131 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758003af0 con 0x7f5768100450 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.534+0000 7f576d18f640 1 -- 172.21.15.94:0/2502501131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57681a5540 con 0x7f5768100450 2024-04-21T08:50:56.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.535+0000 7f576d18f640 1 -- 172.21.15.94:0/2502501131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5768199630 con 0x7f5768100450 2024-04-21T08:50:56.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.536+0000 7f574ffff640 1 -- 172.21.15.94:0/2502501131 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f5758046020 con 0x7f5768100450 2024-04-21T08:50:56.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.537+0000 7f574ffff640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57500a3d20 0x7f57500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.537+0000 7f576659c640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57500a3d20 0x7f57500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.537+0000 7f574ffff640 1 -- 172.21.15.94:0/2502501131 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f57580f2c10 con 0x7f5768100450 2024-04-21T08:50:56.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.537+0000 7f576659c640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57500a3d20 0x7f57500a61e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5768199f70 tx=0x7f575c005f90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:56.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.541+0000 7f574ffff640 1 -- 172.21.15.94:0/2502501131 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f575804d020 con 0x7f5768100450 2024-04-21T08:50:56.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.716+0000 7f576d18f640 1 -- 172.21.15.94:0/2502501131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5768199c50 con 0x7f5768100450 2024-04-21T08:50:56.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.717+0000 7f574ffff640 1 -- 172.21.15.94:0/2502501131 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) ==== 103+0+11 (secure 0 0 0) 0x7f57580bef90 con 0x7f5768100450 2024-04-21T08:50:56.717 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 -- 172.21.15.94:0/2502501131 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57500a3d20 msgr2=0x7f57500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57500a3d20 0x7f57500a61e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5768199f70 tx=0x7f575c005f90 comp rx=0 tx=0).stop 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 -- 172.21.15.94:0/2502501131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5768100450 msgr2=0x7f576819e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5768100450 0x7f576819e5c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f5758009230 tx=0x7f575802c710 comp rx=0 tx=0).stop 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 -- 172.21.15.94:0/2502501131 shutdown_connections 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f57500a3d20 0x7f57500a61e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5768106f30 0x7f5768198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5768100e10 0x7f576819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 --2- 172.21.15.94:0/2502501131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5768100450 0x7f576819e5c0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 -- 172.21.15.94:0/2502501131 >> 172.21.15.94:0/2502501131 conn(0x7f5768077410 msgr2=0x7f5768105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:56.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 -- 172.21.15.94:0/2502501131 shutdown_connections 2024-04-21T08:50:56.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.719+0000 7f576d18f640 1 -- 172.21.15.94:0/2502501131 wait complete. 2024-04-21T08:50:56.730 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:50:56.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.830+0000 7f8e8d123640 1 -- 172.21.15.94:0/2508444533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e88106170 msgr2=0x7f8e8810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.830+0000 7f8e8d123640 1 --2- 172.21.15.94:0/2508444533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e88106170 0x7f8e8810dd40 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009900 tx=0x7f8e7802bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.830+0000 7f8e8d123640 1 -- 172.21.15.94:0/2508444533 shutdown_connections 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.830+0000 7f8e8d123640 1 --2- 172.21.15.94:0/2508444533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e88101a50 0x7f8e88105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.830+0000 7f8e8d123640 1 --2- 172.21.15.94:0/2508444533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e88106170 0x7f8e8810dd40 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.830+0000 7f8e8d123640 1 --2- 172.21.15.94:0/2508444533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e881010a0 0x7f8e88101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.830+0000 7f8e8d123640 1 -- 172.21.15.94:0/2508444533 >> 172.21.15.94:0/2508444533 conn(0x7f8e880771d0 msgr2=0x7f8e880775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.830+0000 7f8e8d123640 1 -- 172.21.15.94:0/2508444533 shutdown_connections 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.830+0000 7f8e8d123640 1 -- 172.21.15.94:0/2508444533 wait complete. 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.830+0000 7f8e8d123640 1 Processor -- start 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8d123640 1 -- start start 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8d123640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e881010a0 0x7f8e8819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8d123640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e88106170 0x7f8e8819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8d123640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e88198590 0x7f8e88198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8d123640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8e881110c0 con 0x7f8e881010a0 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8d123640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8e88110f40 con 0x7f8e88106170 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8d123640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8e88111240 con 0x7f8e88198590 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e86d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e881010a0 0x7f8e8819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e86d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e881010a0 0x7f8e8819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40880/0 (socket says 172.21.15.94:40880) 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e86d9d640 1 -- 172.21.15.94:0/4267823422 learned_addr learned my addr 172.21.15.94:0/4267823422 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8759e640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e88198590 0x7f8e88198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8659c640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e88106170 0x7f8e8819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e86d9d640 1 -- 172.21.15.94:0/4267823422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e88198590 msgr2=0x7f8e88198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e86d9d640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e88198590 0x7f8e88198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e86d9d640 1 -- 172.21.15.94:0/4267823422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e88106170 msgr2=0x7f8e8819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e86d9d640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e88106170 0x7f8e8819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e86d9d640 1 -- 172.21.15.94:0/4267823422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e881990b0 con 0x7f8e881010a0 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8759e640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e88198590 0x7f8e88198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e8659c640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e88106170 0x7f8e8819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e86d9d640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e881010a0 0x7f8e8819e3e0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000cce0 tx=0x7f8e8000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.831+0000 7f8e6ffff640 1 -- 172.21.15.94:0/4267823422 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80004b20 con 0x7f8e881010a0 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.832+0000 7f8e6ffff640 1 -- 172.21.15.94:0/4267823422 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e80004cc0 con 0x7f8e881010a0 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.832+0000 7f8e8d123640 1 -- 172.21.15.94:0/4267823422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e881993a0 con 0x7f8e881010a0 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.832+0000 7f8e6ffff640 1 -- 172.21.15.94:0/4267823422 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8000f6b0 con 0x7f8e881010a0 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.832+0000 7f8e8d123640 1 -- 172.21.15.94:0/4267823422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e881a55f0 con 0x7f8e881010a0 2024-04-21T08:50:56.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.833+0000 7f8e8d123640 1 -- 172.21.15.94:0/4267823422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e880691c0 con 0x7f8e881010a0 2024-04-21T08:50:56.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.834+0000 7f8e6ffff640 1 -- 172.21.15.94:0/4267823422 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8e8000f850 con 0x7f8e881010a0 2024-04-21T08:50:56.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.834+0000 7f8e6ffff640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8e700a3d20 0x7f8e700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:56.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.834+0000 7f8e8659c640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8e700a3d20 0x7f8e700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:56.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.835+0000 7f8e6ffff640 1 -- 172.21.15.94:0/4267823422 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f8e800d0d90 con 0x7f8e881010a0 2024-04-21T08:50:56.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.835+0000 7f8e8659c640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8e700a3d20 0x7f8e700a61e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c005fd0 tx=0x7f8e7c0094d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:56.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:56.839+0000 7f8e6ffff640 1 -- 172.21.15.94:0/4267823422 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e80014030 con 0x7f8e881010a0 2024-04-21T08:50:57.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.032+0000 7f8e8d123640 1 -- 172.21.15.94:0/4267823422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8e88199ac0 con 0x7f8e881010a0 2024-04-21T08:50:57.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.033+0000 7f8e6ffff640 1 -- 172.21.15.94:0/4267823422 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v102) ==== 81+0+942 (secure 0 0 0) 0x7f8e8009d000 con 0x7f8e881010a0 2024-04-21T08:50:57.033 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/1188403796 2024-04-22T08:39:57.898744+0000 2024-04-21T08:50:57.033 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/441167573 2024-04-22T08:30:29.483746+0000 2024-04-21T08:50:57.033 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/2502723264 2024-04-22T08:30:29.483746+0000 2024-04-21T08:50:57.033 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/613544885 2024-04-22T08:30:29.483746+0000 2024-04-21T08:50:57.033 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/3394521972 2024-04-22T08:39:57.898744+0000 2024-04-21T08:50:57.033 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/3877440809 2024-04-22T08:30:29.483746+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3977681577 2024-04-22T08:30:03.511107+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/4286642695 2024-04-22T08:39:57.898744+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3965359185 2024-04-22T08:30:03.511107+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/694152474 2024-04-22T08:39:57.898744+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/1761971189 2024-04-22T08:30:03.511107+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/2629131251 2024-04-22T08:30:03.511107+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/3877440809 2024-04-22T08:30:29.483746+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/909646532 2024-04-22T08:39:57.898744+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/3394521972 2024-04-22T08:39:57.898744+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/2629131251 2024-04-22T08:30:03.511107+0000 2024-04-21T08:50:57.034 INFO:teuthology.orchestra.run.smithi094.stderr:listed 16 entries 2024-04-21T08:50:57.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 -- 172.21.15.94:0/4267823422 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8e700a3d20 msgr2=0x7f8e700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8e700a3d20 0x7f8e700a61e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c005fd0 tx=0x7f8e7c0094d0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 -- 172.21.15.94:0/4267823422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e881010a0 msgr2=0x7f8e8819e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e881010a0 0x7f8e8819e3e0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000cce0 tx=0x7f8e8000a620 comp rx=0 tx=0).stop 2024-04-21T08:50:57.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 -- 172.21.15.94:0/4267823422 shutdown_connections 2024-04-21T08:50:57.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f8e700a3d20 0x7f8e700a61e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8e88198590 0x7f8e88198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e88106170 0x7f8e8819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 --2- 172.21.15.94:0/4267823422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e881010a0 0x7f8e8819e3e0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 -- 172.21.15.94:0/4267823422 >> 172.21.15.94:0/4267823422 conn(0x7f8e880771d0 msgr2=0x7f8e88104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:57.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 -- 172.21.15.94:0/4267823422 shutdown_connections 2024-04-21T08:50:57.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.035+0000 7f8e8d123640 1 -- 172.21.15.94:0/4267823422 wait complete. 2024-04-21T08:50:57.046 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-21T08:50:57.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:50:57.047 DEBUG:teuthology.orchestra.run.smithi094:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-21T08:50:57.083 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:50:57.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:50:57.084 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-21T08:50:57.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:50:57.139 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-04-21T08:50:57.195 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-21T08:50:57.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:50:57.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T08:50:57.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:50:57.260 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2024-04-21T08:50:57.321 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:57 smithi094 ceph-mon[27988]: pgmap v340: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:57.321 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:57 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3089839426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:57.321 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:57 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2502501131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:57.321 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:57 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/4267823422' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:50:57.348 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T08:50:57.349 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-21T08:50:57.349 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:50:57.407 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=311 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/b99249c2025ca746201f8c02ad2c26fbe574483f84cd34a04c3896cdb1ded182/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/3LD76PCLMM5XSFBAEUUEMT56FY:/var/lib/containers/storage/overlay/l/KOSKSQL4F5XA5OZDJUE3G356L4,upperdir=/var/lib/containers/storage/overlay/b99249c2025ca746201f8c02ad2c26fbe574483f84cd34a04c3896cdb1ded182/diff,workdir=/var/lib/containers/storage/overlay/b99249c2025ca746201f8c02ad2c26fbe574483f84cd34a04c3896cdb1ded182/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b70cc6eb0a815af975db75765bc60a678949b9deab84dc0e45c77fca41413945/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3LD76PCLMM5XSFBAEUUEMT56FY:/var/lib/containers/storage/overlay/l/KOSKSQL4F5XA5OZDJUE3G356L4,upperdir=/var/lib/containers/storage/overlay/b70cc6eb0a815af975db75765bc60a678949b9deab84dc0e45c77fca41413945/diff,workdir=/var/lib/containers/storage/overlay/b70cc6eb0a815af975db75765bc60a678949b9deab84dc0e45c77fca41413945/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/13f2f7fa62b402c1c2c2781f3f1dc51a47d88a05e3c3018ea8a2c8d3cf68bf32/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/3LD76PCLMM5XSFBAEUUEMT56FY:/var/lib/containers/storage/overlay/l/KOSKSQL4F5XA5OZDJUE3G356L4,upperdir=/var/lib/containers/storage/overlay/13f2f7fa62b402c1c2c2781f3f1dc51a47d88a05e3c3018ea8a2c8d3cf68bf32/diff,workdir=/var/lib/containers/storage/overlay/13f2f7fa62b402c1c2c2781f3f1dc51a47d88a05e3c3018ea8a2c8d3cf68bf32/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/cbb788570ad7dc629f5b1466bcb47bbc17ddc00bc35d5765ab9e2892e3a2c657/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/3LD76PCLMM5XSFBAEUUEMT56FY:/var/lib/containers/storage/overlay/l/KOSKSQL4F5XA5OZDJUE3G356L4,upperdir=/var/lib/containers/storage/overlay/cbb788570ad7dc629f5b1466bcb47bbc17ddc00bc35d5765ab9e2892e3a2c657/diff,workdir=/var/lib/containers/storage/overlay/cbb788570ad7dc629f5b1466bcb47bbc17ddc00bc35d5765ab9e2892e3a2c657/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f3444ba9f1e92ced913d22b295637daf7ab546c7ca35e9c405ced206e35203f4/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/3LD76PCLMM5XSFBAEUUEMT56FY:/var/lib/containers/storage/overlay/l/KOSKSQL4F5XA5OZDJUE3G356L4,upperdir=/var/lib/containers/storage/overlay/f3444ba9f1e92ced913d22b295637daf7ab546c7ca35e9c405ced206e35203f4/diff,workdir=/var/lib/containers/storage/overlay/f3444ba9f1e92ced913d22b295637daf7ab546c7ca35e9c405ced206e35203f4/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/3e296cfa4e759f81fb4fc4a4714b8bd70bb4d218df1811ae3d874ca6f1fa50f4/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/3LD76PCLMM5XSFBAEUUEMT56FY:/var/lib/containers/storage/overlay/l/KOSKSQL4F5XA5OZDJUE3G356L4,upperdir=/var/lib/containers/storage/overlay/3e296cfa4e759f81fb4fc4a4714b8bd70bb4d218df1811ae3d874ca6f1fa50f4/diff,workdir=/var/lib/containers/storage/overlay/3e296cfa4e759f81fb4fc4a4714b8bd70bb4d218df1811ae3d874ca6f1fa50f4/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/01ef8418978330e6307f645403cb38bc723931ab480fc2ff19034f10e5b324d1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3LD76PCLMM5XSFBAEUUEMT56FY:/var/lib/containers/storage/overlay/l/KOSKSQL4F5XA5OZDJUE3G356L4,upperdir=/var/lib/containers/storage/overlay/01ef8418978330e6307f645403cb38bc723931ab480fc2ff19034f10e5b324d1/diff,workdir=/var/lib/containers/storage/overlay/01ef8418978330e6307f645403cb38bc723931ab480fc2ff19034f10e5b324d1/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/2ebe427004a60ed3c6cca82362a798d17b35838a8ff3e330445f71d219bbdea2/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3LD76PCLMM5XSFBAEUUEMT56FY:/var/lib/containers/storage/overlay/l/KOSKSQL4F5XA5OZDJUE3G356L4,upperdir=/var/lib/containers/storage/overlay/2ebe427004a60ed3c6cca82362a798d17b35838a8ff3e330445f71d219bbdea2/diff,workdir=/var/lib/containers/storage/overlay/2ebe427004a60ed3c6cca82362a798d17b35838a8ff3e330445f71d219bbdea2/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4842c6ca69add00dd13f36c8b2b9743d6677fd96109c9e70efb429e2a2d1e379/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3LD76PCLMM5XSFBAEUUEMT56FY:/var/lib/containers/storage/overlay/l/KOSKSQL4F5XA5OZDJUE3G356L4,upperdir=/var/lib/containers/storage/overlay/4842c6ca69add00dd13f36c8b2b9743d6677fd96109c9e70efb429e2a2d1e379/diff,workdir=/var/lib/containers/storage/overlay/4842c6ca69add00dd13f36c8b2b9743d6677fd96109c9e70efb429e2a2d1e379/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/34b10c40db371f4ebd02a2e4ea5f0933296724621c50435e8ae7f1a97db8a6db/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3LD76PCLMM5XSFBAEUUEMT56FY:/var/lib/containers/storage/overlay/l/KOSKSQL4F5XA5OZDJUE3G356L4,upperdir=/var/lib/containers/storage/overlay/34b10c40db371f4ebd02a2e4ea5f0933296724621c50435e8ae7f1a97db8a6db/diff,workdir=/var/lib/containers/storage/overlay/34b10c40db371f4ebd02a2e4ea5f0933296724621c50435e8ae7f1a97db8a6db/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-21T08:50:57.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:50:57.408 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-04-21T08:50:57.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:57 smithi111 ceph-mon[31493]: pgmap v340: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:57.437 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:57 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3089839426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:57.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:57 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2502501131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:57.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:57 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/4267823422' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:50:57.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:50:57.463 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2024-04-21T08:50:57.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:57 smithi173 ceph-mon[31428]: pgmap v340: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:57.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:57 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3089839426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:57.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:57 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2502501131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:57.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:57 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/4267823422' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:50:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:Cannot find device "ceph-brx" 2024-04-21T08:50:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:50:57.530 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-21T08:50:57.543 INFO:tasks.cephadm:Teardown begin 2024-04-21T08:50:57.543 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-21T08:50:57.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:50:57.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.668+0000 7f436b5fb640 1 -- 172.21.15.94:0/505505495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f436410c5a0 msgr2=0x7f436410e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.668+0000 7f436b5fb640 1 --2- 172.21.15.94:0/505505495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f436410c5a0 0x7f436410e9d0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f435c009ca0 tx=0x7f435c02c180 comp rx=0 tx=0).stop 2024-04-21T08:50:57.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.668+0000 7f436b5fb640 1 -- 172.21.15.94:0/505505495 shutdown_connections 2024-04-21T08:50:57.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.668+0000 7f436b5fb640 1 --2- 172.21.15.94:0/505505495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4364103af0 0x7f436410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.668+0000 7f436b5fb640 1 --2- 172.21.15.94:0/505505495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f436410c5a0 0x7f436410e9d0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.668+0000 7f436b5fb640 1 --2- 172.21.15.94:0/505505495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43641031d0 0x7f43641035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.668+0000 7f436b5fb640 1 -- 172.21.15.94:0/505505495 >> 172.21.15.94:0/505505495 conn(0x7f43640771d0 msgr2=0x7f43640775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f436b5fb640 1 -- 172.21.15.94:0/505505495 shutdown_connections 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f436b5fb640 1 -- 172.21.15.94:0/505505495 wait complete. 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f436b5fb640 1 Processor -- start 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f436b5fb640 1 -- start start 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f436b5fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43641031d0 0x7f4364198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f436b5fb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4364103af0 0x7f4364198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f436b5fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436410c5a0 0x7f436419cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f436b5fb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f43641121a0 con 0x7f43641031d0 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f436b5fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4364112020 con 0x7f436410c5a0 2024-04-21T08:50:57.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f436b5fb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4364112320 con 0x7f4364103af0 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f4369397640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43641031d0 0x7f4364198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f4369397640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43641031d0 0x7f4364198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40896/0 (socket says 172.21.15.94:40896) 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.669+0000 7f4369397640 1 -- 172.21.15.94:0/1380419986 learned_addr learned my addr 172.21.15.94:0/1380419986 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f4369b98640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436410c5a0 0x7f436419cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f4368b96640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4364103af0 0x7f4364198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f4369397640 1 -- 172.21.15.94:0/1380419986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4364103af0 msgr2=0x7f4364198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f4369397640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4364103af0 0x7f4364198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f4369397640 1 -- 172.21.15.94:0/1380419986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436410c5a0 msgr2=0x7f436419cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f4369397640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436410c5a0 0x7f436419cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f4369397640 1 -- 172.21.15.94:0/1380419986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436419d620 con 0x7f43641031d0 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f4369397640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43641031d0 0x7f4364198680 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f436000c890 tx=0x7f436000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f435a7fc640 1 -- 172.21.15.94:0/1380419986 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360004220 con 0x7f43641031d0 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f435a7fc640 1 -- 172.21.15.94:0/1380419986 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43600043c0 con 0x7f43641031d0 2024-04-21T08:50:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f436b5fb640 1 -- 172.21.15.94:0/1380419986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436419d910 con 0x7f43641031d0 2024-04-21T08:50:57.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f435a7fc640 1 -- 172.21.15.94:0/1380419986 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360004a00 con 0x7f43641031d0 2024-04-21T08:50:57.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.670+0000 7f436b5fb640 1 -- 172.21.15.94:0/1380419986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43641a51a0 con 0x7f43641031d0 2024-04-21T08:50:57.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.671+0000 7f436b5fb640 1 -- 172.21.15.94:0/1380419986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4364069230 con 0x7f43641031d0 2024-04-21T08:50:57.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.672+0000 7f435a7fc640 1 -- 172.21.15.94:0/1380419986 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f43600079b0 con 0x7f43641031d0 2024-04-21T08:50:57.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.673+0000 7f435a7fc640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f43500a3da0 0x7f43500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:57.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.673+0000 7f4368b96640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f43500a3da0 0x7f43500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:57.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.673+0000 7f435a7fc640 1 -- 172.21.15.94:0/1380419986 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f43600d09b0 con 0x7f43641031d0 2024-04-21T08:50:57.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.673+0000 7f4368b96640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f43500a3da0 0x7f43500a6260 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f436410e2b0 tx=0x7f435400c4a0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:57.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.677+0000 7f435a7fc640 1 -- 172.21.15.94:0/1380419986 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4360014030 con 0x7f43641031d0 2024-04-21T08:50:57.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.896+0000 7f436b5fb640 1 -- 172.21.15.94:0/1380419986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4364107100 con 0x7f43641031d0 2024-04-21T08:50:57.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.897+0000 7f435a7fc640 1 -- 172.21.15.94:0/1380419986 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f436009cc20 con 0x7f43641031d0 2024-04-21T08:50:57.897 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:50:57.897 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:50:57.898 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:50:57.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.898+0000 7f436b5fb640 1 -- 172.21.15.94:0/1380419986 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f43500a3da0 msgr2=0x7f43500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.898+0000 7f436b5fb640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f43500a3da0 0x7f43500a6260 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f436410e2b0 tx=0x7f435400c4a0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.898+0000 7f436b5fb640 1 -- 172.21.15.94:0/1380419986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43641031d0 msgr2=0x7f4364198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.898+0000 7f436b5fb640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43641031d0 0x7f4364198680 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f436000c890 tx=0x7f436000cd50 comp rx=0 tx=0).stop 2024-04-21T08:50:57.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.899+0000 7f436b5fb640 1 -- 172.21.15.94:0/1380419986 shutdown_connections 2024-04-21T08:50:57.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.899+0000 7f436b5fb640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f43500a3da0 0x7f43500a6260 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.899+0000 7f436b5fb640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4364103af0 0x7f4364198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.899+0000 7f436b5fb640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436410c5a0 0x7f436419cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.899+0000 7f436b5fb640 1 --2- 172.21.15.94:0/1380419986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43641031d0 0x7f4364198680 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.899+0000 7f436b5fb640 1 -- 172.21.15.94:0/1380419986 >> 172.21.15.94:0/1380419986 conn(0x7f43640771d0 msgr2=0x7f43640fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:57.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.899+0000 7f436b5fb640 1 -- 172.21.15.94:0/1380419986 shutdown_connections 2024-04-21T08:50:57.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:57.899+0000 7f436b5fb640 1 -- 172.21.15.94:0/1380419986 wait complete. 2024-04-21T08:50:57.910 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:50:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.008+0000 7f9d14b98640 1 -- 172.21.15.94:0/1291768772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d10107260 msgr2=0x7f9d1010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.008+0000 7f9d14b98640 1 --2- 172.21.15.94:0/1291768772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d10107260 0x7f9d1010ee30 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009ca0 tx=0x7f9d0802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.008+0000 7f9d14b98640 1 -- 172.21.15.94:0/1291768772 shutdown_connections 2024-04-21T08:50:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.008+0000 7f9d14b98640 1 --2- 172.21.15.94:0/1291768772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10102b40 0x7f9d10106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.008+0000 7f9d14b98640 1 --2- 172.21.15.94:0/1291768772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d10107260 0x7f9d1010ee30 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.008+0000 7f9d14b98640 1 --2- 172.21.15.94:0/1291768772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d10102190 0x7f9d10102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.008+0000 7f9d14b98640 1 -- 172.21.15.94:0/1291768772 >> 172.21.15.94:0/1291768772 conn(0x7f9d100793e0 msgr2=0x7f9d100797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.008+0000 7f9d14b98640 1 -- 172.21.15.94:0/1291768772 shutdown_connections 2024-04-21T08:50:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.008+0000 7f9d14b98640 1 -- 172.21.15.94:0/1291768772 wait complete. 2024-04-21T08:50:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.009+0000 7f9d14b98640 1 Processor -- start 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.009+0000 7f9d14b98640 1 -- start start 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.009+0000 7f9d14b98640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10102190 0x7f9d101986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.009+0000 7f9d14b98640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d10107260 0x7f9d10198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.009+0000 7f9d14b98640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d1019cfb0 0x7f9d1019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.009+0000 7f9d14b98640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9d101121d0 con 0x7f9d10107260 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.009+0000 7f9d14b98640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9d10112050 con 0x7f9d1019cfb0 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.009+0000 7f9d14b98640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9d10112350 con 0x7f9d10102190 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.009+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d10107260 0x7f9d10198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d10107260 0x7f9d10198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40920/0 (socket says 172.21.15.94:40920) 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d07fff640 1 -- 172.21.15.94:0/372181867 learned_addr learned my addr 172.21.15.94:0/372181867 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d0e59c640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10102190 0x7f9d101986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d0ed9d640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d1019cfb0 0x7f9d1019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d07fff640 1 -- 172.21.15.94:0/372181867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10102190 msgr2=0x7f9d101986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d07fff640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10102190 0x7f9d101986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d07fff640 1 -- 172.21.15.94:0/372181867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d1019cfb0 msgr2=0x7f9d1019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d07fff640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d1019cfb0 0x7f9d1019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d07fff640 1 -- 172.21.15.94:0/372181867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1019daa0 con 0x7f9d10107260 2024-04-21T08:50:58.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d0ed9d640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d1019cfb0 0x7f9d1019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:58.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d07fff640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d10107260 0x7f9d10198c30 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800d740 tx=0x7f9cf800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d077fe640 1 -- 172.21.15.94:0/372181867 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8017070 con 0x7f9d10107260 2024-04-21T08:50:58.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d077fe640 1 -- 172.21.15.94:0/372181867 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cf800de50 con 0x7f9d10107260 2024-04-21T08:50:58.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d14b98640 1 -- 172.21.15.94:0/372181867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1019dd90 con 0x7f9d10107260 2024-04-21T08:50:58.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d077fe640 1 -- 172.21.15.94:0/372181867 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8004ce0 con 0x7f9d10107260 2024-04-21T08:50:58.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.010+0000 7f9d14b98640 1 -- 172.21.15.94:0/372181867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d101a5530 con 0x7f9d10107260 2024-04-21T08:50:58.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.011+0000 7f9d14b98640 1 -- 172.21.15.94:0/372181867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d10069230 con 0x7f9d10107260 2024-04-21T08:50:58.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.012+0000 7f9d077fe640 1 -- 172.21.15.94:0/372181867 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f9cf8004e80 con 0x7f9d10107260 2024-04-21T08:50:58.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.013+0000 7f9d077fe640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9cfc0a3d20 0x7f9cfc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.013+0000 7f9d0e59c640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9cfc0a3d20 0x7f9cfc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.013+0000 7f9d077fe640 1 -- 172.21.15.94:0/372181867 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f9cf80d0560 con 0x7f9d10107260 2024-04-21T08:50:58.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.013+0000 7f9d0e59c640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9cfc0a3d20 0x7f9cfc0a61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000bbe0 tx=0x7f9d0000b390 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.017+0000 7f9d077fe640 1 -- 172.21.15.94:0/372181867 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9cf809c7d0 con 0x7f9d10107260 2024-04-21T08:50:58.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.253+0000 7f9d14b98640 1 -- 172.21.15.94:0/372181867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d10102b40 con 0x7f9d10107260 2024-04-21T08:50:58.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.253+0000 7f9d077fe640 1 -- 172.21.15.94:0/372181867 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11885 (secure 0 0 0) 0x7f9cf809c0d0 con 0x7f9d10107260 2024-04-21T08:50:58.254 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:50:58.254 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1080139605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":1080139605},{"type":"v1","addr":"172.21.15.94:6835","nonce":1080139605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/4105566219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":4105566219},{"type":"v1","addr":"172.21.15.94:6837","nonce":4105566219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/4167052698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":4167052698},{"type":"v1","addr":"172.21.15.94:6839","nonce":4167052698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/2997402360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":2997402360},{"type":"v1","addr":"172.21.15.94:6841","nonce":2997402360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2495824461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2495824461},{"type":"v1","addr":"172.21.15.173:6833","nonce":2495824461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4021460222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4021460222},{"type":"v1","addr":"172.21.15.111:6833","nonce":4021460222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:37:30.753966+0000","modified":"2024-04-21T08:39:56.419394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24454,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24416":{"gid":24416,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3967081988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3967081988},{"type":"v1","addr":"172.21.15.111:6835","nonce":3967081988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3922099450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3922099450},{"type":"v1","addr":"172.21.15.173:6835","nonce":3922099450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1466520447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1466520447},{"type":"v1","addr":"172.21.15.173:6837","nonce":1466520447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/550365815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":550365815},{"type":"v1","addr":"172.21.15.111:6837","nonce":550365815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.173:6839/2987518473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2987518473},{"type":"v1","addr":"172.21.15.173:6839","nonce":2987518473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3878872423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3878872423},{"type":"v1","addr":"172.21.15.111:6839","nonce":3878872423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24454,24488,24446]},"id":1}]} 2024-04-21T08:50:58.254 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-21T08:50:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.255+0000 7f9d14b98640 1 -- 172.21.15.94:0/372181867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9cfc0a3d20 msgr2=0x7f9cfc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.255+0000 7f9d14b98640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9cfc0a3d20 0x7f9cfc0a61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000bbe0 tx=0x7f9d0000b390 comp rx=0 tx=0).stop 2024-04-21T08:50:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.255+0000 7f9d14b98640 1 -- 172.21.15.94:0/372181867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d10107260 msgr2=0x7f9d10198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.255+0000 7f9d14b98640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d10107260 0x7f9d10198c30 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800d740 tx=0x7f9cf800dc00 comp rx=0 tx=0).stop 2024-04-21T08:50:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.255+0000 7f9d14b98640 1 -- 172.21.15.94:0/372181867 shutdown_connections 2024-04-21T08:50:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.255+0000 7f9d14b98640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f9cfc0a3d20 0x7f9cfc0a61e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.255+0000 7f9d14b98640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d1019cfb0 0x7f9d1019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.255+0000 7f9d14b98640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d10107260 0x7f9d10198c30 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.255+0000 7f9d14b98640 1 --2- 172.21.15.94:0/372181867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d10102190 0x7f9d101986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.255+0000 7f9d14b98640 1 -- 172.21.15.94:0/372181867 >> 172.21.15.94:0/372181867 conn(0x7f9d100793e0 msgr2=0x7f9d100ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.256+0000 7f9d14b98640 1 -- 172.21.15.94:0/372181867 shutdown_connections 2024-04-21T08:50:58.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.256+0000 7f9d14b98640 1 -- 172.21.15.94:0/372181867 wait complete. 2024-04-21T08:50:58.267 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:50:58.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.365+0000 7fe9383c7640 1 -- 172.21.15.94:0/4017233694 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93010c5a0 msgr2=0x7fe93010e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.365+0000 7fe9383c7640 1 --2- 172.21.15.94:0/4017233694 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93010c5a0 0x7fe93010e9d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fe928009ca0 tx=0x7fe92802c180 comp rx=0 tx=0).stop 2024-04-21T08:50:58.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 -- 172.21.15.94:0/4017233694 shutdown_connections 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 --2- 172.21.15.94:0/4017233694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe930103af0 0x7fe93010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 --2- 172.21.15.94:0/4017233694 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93010c5a0 0x7fe93010e9d0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 --2- 172.21.15.94:0/4017233694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9301031d0 0x7fe9301035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 -- 172.21.15.94:0/4017233694 >> 172.21.15.94:0/4017233694 conn(0x7fe9300771d0 msgr2=0x7fe9300775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 -- 172.21.15.94:0/4017233694 shutdown_connections 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 -- 172.21.15.94:0/4017233694 wait complete. 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 Processor -- start 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 -- start start 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9301031d0 0x7fe9301986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930103af0 0x7fe930198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93010c5a0 0x7fe93019cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe9301121a0 con 0x7fe93010c5a0 2024-04-21T08:50:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe930112020 con 0x7fe930103af0 2024-04-21T08:50:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.366+0000 7fe9383c7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe930112320 con 0x7fe9301031d0 2024-04-21T08:50:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936964640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93010c5a0 0x7fe93019cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936163640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9301031d0 0x7fe9301986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936964640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93010c5a0 0x7fe93019cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40952/0 (socket says 172.21.15.94:40952) 2024-04-21T08:50:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936964640 1 -- 172.21.15.94:0/378822292 learned_addr learned my addr 172.21.15.94:0/378822292 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936163640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9301031d0 0x7fe9301986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.94:38602/0 (socket says 172.21.15.94:38602) 2024-04-21T08:50:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe935962640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930103af0 0x7fe930198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936964640 1 -- 172.21.15.94:0/378822292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9301031d0 msgr2=0x7fe9301986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936964640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9301031d0 0x7fe9301986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936964640 1 -- 172.21.15.94:0/378822292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930103af0 msgr2=0x7fe930198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936964640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930103af0 0x7fe930198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936964640 1 -- 172.21.15.94:0/378822292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93019d690 con 0x7fe93010c5a0 2024-04-21T08:50:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe935962640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930103af0 0x7fe930198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe936964640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93010c5a0 0x7fe93019cf60 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fe92802c790 tx=0x7fe928003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe9277fe640 1 -- 172.21.15.94:0/378822292 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928039040 con 0x7fe93010c5a0 2024-04-21T08:50:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe9383c7640 1 -- 172.21.15.94:0/378822292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe93019d920 con 0x7fe93010c5a0 2024-04-21T08:50:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe9277fe640 1 -- 172.21.15.94:0/378822292 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe928044070 con 0x7fe93010c5a0 2024-04-21T08:50:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe9383c7640 1 -- 172.21.15.94:0/378822292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9301a5210 con 0x7fe93010c5a0 2024-04-21T08:50:58.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.367+0000 7fe9277fe640 1 -- 172.21.15.94:0/378822292 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9280327c0 con 0x7fe93010c5a0 2024-04-21T08:50:58.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.368+0000 7fe9383c7640 1 -- 172.21.15.94:0/378822292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe930069230 con 0x7fe93010c5a0 2024-04-21T08:50:58.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.369+0000 7fe9277fe640 1 -- 172.21.15.94:0/378822292 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fe928004040 con 0x7fe93010c5a0 2024-04-21T08:50:58.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.370+0000 7fe9277fe640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe91c0a3d20 0x7fe91c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.370+0000 7fe936163640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe91c0a3d20 0x7fe91c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.370+0000 7fe9277fe640 1 -- 172.21.15.94:0/378822292 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7fe9280f2e10 con 0x7fe93010c5a0 2024-04-21T08:50:58.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.370+0000 7fe936163640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe91c0a3d20 0x7fe91c0a61e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009bd0 tx=0x7fe92c005f90 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.374+0000 7fe9277fe640 1 -- 172.21.15.94:0/378822292 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe92804e020 con 0x7fe93010c5a0 2024-04-21T08:50:58.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:58 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1380419986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:58.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1380419986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:58.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.555+0000 7fe9383c7640 1 -- 172.21.15.94:0/378822292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe930107690 con 0x7fe93010c5a0 2024-04-21T08:50:58.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.556+0000 7fe9277fe640 1 -- 172.21.15.94:0/378822292 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+25452 (secure 0 0 0) 0x7fe9280bf190 con 0x7fe93010c5a0 2024-04-21T08:50:58.557 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-21T08:50:58.557 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":102,"fsid":"2518ed94-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:29:35.091752+0000","modified":"2024-04-21T08:50:11.852327+0000","last_up_change":"2024-04-21T08:36:42.255301+0000","last_in_change":"2024-04-21T08:36:22.534917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:32:55.571794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:37:28.427740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:37:29.734210+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:37:33.577891+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec26f622-360a-48df-bc52-c85842709d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6803","nonce":676652749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6805","nonce":676652749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6809","nonce":676652749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":676652749},{"type":"v1","addr":"172.21.15.94:6807","nonce":676652749}]},"public_addr":"172.21.15.94:6803/676652749","cluster_addr":"172.21.15.94:6805/676652749","heartbeat_back_addr":"172.21.15.94:6809/676652749","heartbeat_front_addr":"172.21.15.94:6807/676652749","state":["exists","up"]},{"osd":1,"uuid":"188e7b8f-e006-4b64-8136-9e71097e107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6801","nonce":2507021892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6803","nonce":2507021892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6807","nonce":2507021892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2507021892},{"type":"v1","addr":"172.21.15.111:6805","nonce":2507021892}]},"public_addr":"172.21.15.111:6801/2507021892","cluster_addr":"172.21.15.111:6803/2507021892","heartbeat_back_addr":"172.21.15.111:6807/2507021892","heartbeat_front_addr":"172.21.15.111:6805/2507021892","state":["exists","up"]},{"osd":2,"uuid":"0f3dff21-6860-4c3b-9c15-3f120e0f3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6801","nonce":4262691909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6803","nonce":4262691909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6807","nonce":4262691909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":4262691909},{"type":"v1","addr":"172.21.15.173:6805","nonce":4262691909}]},"public_addr":"172.21.15.173:6801/4262691909","cluster_addr":"172.21.15.173:6803/4262691909","heartbeat_back_addr":"172.21.15.173:6807/4262691909","heartbeat_front_addr":"172.21.15.173:6805/4262691909","state":["exists","up"]},{"osd":3,"uuid":"b30afbd0-4fe9-432a-ac54-7db53d055e37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6811","nonce":66052007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6813","nonce":66052007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6817","nonce":66052007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":66052007},{"type":"v1","addr":"172.21.15.94:6815","nonce":66052007}]},"public_addr":"172.21.15.94:6811/66052007","cluster_addr":"172.21.15.94:6813/66052007","heartbeat_back_addr":"172.21.15.94:6817/66052007","heartbeat_front_addr":"172.21.15.94:6815/66052007","state":["exists","up"]},{"osd":4,"uuid":"125dfc8c-d9ca-4ea4-a06f-605e4c76c919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6809","nonce":4084294051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6811","nonce":4084294051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6815","nonce":4084294051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4084294051},{"type":"v1","addr":"172.21.15.111:6813","nonce":4084294051}]},"public_addr":"172.21.15.111:6809/4084294051","cluster_addr":"172.21.15.111:6811/4084294051","heartbeat_back_addr":"172.21.15.111:6815/4084294051","heartbeat_front_addr":"172.21.15.111:6813/4084294051","state":["exists","up"]},{"osd":5,"uuid":"01402189-deb7-49ad-bda8-0c35038a3348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6809","nonce":2394839325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6811","nonce":2394839325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6815","nonce":2394839325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2394839325},{"type":"v1","addr":"172.21.15.173:6813","nonce":2394839325}]},"public_addr":"172.21.15.173:6809/2394839325","cluster_addr":"172.21.15.173:6811/2394839325","heartbeat_back_addr":"172.21.15.173:6815/2394839325","heartbeat_front_addr":"172.21.15.173:6813/2394839325","state":["exists","up"]},{"osd":6,"uuid":"7a09d9c1-e8fb-4ff8-a349-8d5aba865099","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6819","nonce":234088342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6821","nonce":234088342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6825","nonce":234088342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":234088342},{"type":"v1","addr":"172.21.15.94:6823","nonce":234088342}]},"public_addr":"172.21.15.94:6819/234088342","cluster_addr":"172.21.15.94:6821/234088342","heartbeat_back_addr":"172.21.15.94:6825/234088342","heartbeat_front_addr":"172.21.15.94:6823/234088342","state":["exists","up"]},{"osd":7,"uuid":"ed806162-485b-4c84-9687-78fcc1f8cc59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6817","nonce":659250531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6819","nonce":659250531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6823","nonce":659250531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":659250531},{"type":"v1","addr":"172.21.15.111:6821","nonce":659250531}]},"public_addr":"172.21.15.111:6817/659250531","cluster_addr":"172.21.15.111:6819/659250531","heartbeat_back_addr":"172.21.15.111:6823/659250531","heartbeat_front_addr":"172.21.15.111:6821/659250531","state":["exists","up"]},{"osd":8,"uuid":"0b61b68b-048f-4904-923a-df40dda39925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6817","nonce":1973310944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6819","nonce":1973310944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6823","nonce":1973310944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":1973310944},{"type":"v1","addr":"172.21.15.173:6821","nonce":1973310944}]},"public_addr":"172.21.15.173:6817/1973310944","cluster_addr":"172.21.15.173:6819/1973310944","heartbeat_back_addr":"172.21.15.173:6823/1973310944","heartbeat_front_addr":"172.21.15.173:6821/1973310944","state":["exists","up"]},{"osd":9,"uuid":"5f384696-fd22-4f83-b99c-e55623ce934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6827","nonce":3583446749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6829","nonce":3583446749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6833","nonce":3583446749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3583446749},{"type":"v1","addr":"172.21.15.94:6831","nonce":3583446749}]},"public_addr":"172.21.15.94:6827/3583446749","cluster_addr":"172.21.15.94:6829/3583446749","heartbeat_back_addr":"172.21.15.94:6833/3583446749","heartbeat_front_addr":"172.21.15.94:6831/3583446749","state":["exists","up"]},{"osd":10,"uuid":"7f6aaa6b-83b7-4f4e-8626-6bde0352caf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6825","nonce":176065870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6827","nonce":176065870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6831","nonce":176065870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":176065870},{"type":"v1","addr":"172.21.15.111:6829","nonce":176065870}]},"public_addr":"172.21.15.111:6825/176065870","cluster_addr":"172.21.15.111:6827/176065870","heartbeat_back_addr":"172.21.15.111:6831/176065870","heartbeat_front_addr":"172.21.15.111:6829/176065870","state":["exists","up"]},{"osd":11,"uuid":"e7529c3d-ab5c-4963-9b8a-40aa40bf97dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6825","nonce":1600242692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6827","nonce":1600242692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6831","nonce":1600242692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1600242692},{"type":"v1","addr":"172.21.15.173:6829","nonce":1600242692}]},"public_addr":"172.21.15.173:6825/1600242692","cluster_addr":"172.21.15.173:6827/1600242692","heartbeat_back_addr":"172.21.15.173:6831/1600242692","heartbeat_front_addr":"172.21.15.173:6829/1600242692","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:05.898733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:29.457490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:32:53.186204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:17.887098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:33:42.275159+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:07.063532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:32.758352+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:34:57.643987+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:22.932109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:35:49.394002+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:14.631239+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:41.024913+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8},{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/1188403796":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/441167573":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/2502723264":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/613544885":"2024-04-22T08:30:29.483746+0000","172.21.15.94:6800/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/3977681577":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/4286642695":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/3965359185":"2024-04-22T08:30:03.511107+0000","172.21.15.94:0/694152474":"2024-04-22T08:39:57.898744+0000","172.21.15.94:0/1761971189":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/2629131251":"2024-04-22T08:30:03.511107+0000","172.21.15.94:6800/3877440809":"2024-04-22T08:30:29.483746+0000","172.21.15.94:0/909646532":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/3394521972":"2024-04-22T08:39:57.898744+0000","172.21.15.94:6801/2629131251":"2024-04-22T08:30:03.511107+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:50:58.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.558+0000 7fe9383c7640 1 -- 172.21.15.94:0/378822292 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe91c0a3d20 msgr2=0x7fe91c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.558+0000 7fe9383c7640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe91c0a3d20 0x7fe91c0a61e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009bd0 tx=0x7fe92c005f90 comp rx=0 tx=0).stop 2024-04-21T08:50:58.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.558+0000 7fe9383c7640 1 -- 172.21.15.94:0/378822292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93010c5a0 msgr2=0x7fe93019cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.558+0000 7fe9383c7640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93010c5a0 0x7fe93019cf60 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fe92802c790 tx=0x7fe928003ba0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.558+0000 7fe9383c7640 1 -- 172.21.15.94:0/378822292 shutdown_connections 2024-04-21T08:50:58.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.558+0000 7fe9383c7640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fe91c0a3d20 0x7fe91c0a61e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.558+0000 7fe9383c7640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93010c5a0 0x7fe93019cf60 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.558+0000 7fe9383c7640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930103af0 0x7fe930198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.558+0000 7fe9383c7640 1 --2- 172.21.15.94:0/378822292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9301031d0 0x7fe9301986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.558+0000 7fe9383c7640 1 -- 172.21.15.94:0/378822292 >> 172.21.15.94:0/378822292 conn(0x7fe9300771d0 msgr2=0x7fe9300fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.559+0000 7fe9383c7640 1 -- 172.21.15.94:0/378822292 shutdown_connections 2024-04-21T08:50:58.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.559+0000 7fe9383c7640 1 -- 172.21.15.94:0/378822292 wait complete. 2024-04-21T08:50:58.570 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-21T08:50:58.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.669+0000 7f58aabc7640 1 -- 172.21.15.94:0/393603581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58a41031d0 msgr2=0x7f58a41035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.669+0000 7f58aabc7640 1 --2- 172.21.15.94:0/393603581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58a41031d0 0x7f58a41035b0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f5898009900 tx=0x7f589802bd60 comp rx=0 tx=0).stop 2024-04-21T08:50:58.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 -- 172.21.15.94:0/393603581 shutdown_connections 2024-04-21T08:50:58.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 --2- 172.21.15.94:0/393603581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a410c510 0x7f58a410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 --2- 172.21.15.94:0/393603581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a4103af0 0x7f58a410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 --2- 172.21.15.94:0/393603581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58a41031d0 0x7f58a41035b0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 -- 172.21.15.94:0/393603581 >> 172.21.15.94:0/393603581 conn(0x7f58a4077f50 msgr2=0x7f58a4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 -- 172.21.15.94:0/393603581 shutdown_connections 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 -- 172.21.15.94:0/393603581 wait complete. 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 Processor -- start 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 -- start start 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a41031d0 0x7f58a419e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a4103af0 0x7f58a419e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58a410c510 0x7f58a4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f58a4112220 con 0x7f58a410c510 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f58a41120a0 con 0x7f58a41031d0 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58aabc7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f58a41123a0 con 0x7f58a4103af0 2024-04-21T08:50:58.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.670+0000 7f58a9164640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58a410c510 0x7f58a4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a9164640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58a410c510 0x7f58a4198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40970/0 (socket says 172.21.15.94:40970) 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a9164640 1 -- 172.21.15.94:0/2276653398 learned_addr learned my addr 172.21.15.94:0/2276653398 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a3fff640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a4103af0 0x7f58a419e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a8963640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a41031d0 0x7f58a419e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a9164640 1 -- 172.21.15.94:0/2276653398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a4103af0 msgr2=0x7f58a419e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a9164640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a4103af0 0x7f58a419e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a9164640 1 -- 172.21.15.94:0/2276653398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a41031d0 msgr2=0x7f58a419e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a9164640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a41031d0 0x7f58a419e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a9164640 1 -- 172.21.15.94:0/2276653398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a4198d80 con 0x7f58a410c510 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a9164640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58a410c510 0x7f58a4198600 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f589c00eaa0 tx=0x7f589c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a1ffb640 1 -- 172.21.15.94:0/2276653398 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c004d20 con 0x7f58a410c510 2024-04-21T08:50:58.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58aabc7640 1 -- 172.21.15.94:0/2276653398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a4199070 con 0x7f58a410c510 2024-04-21T08:50:58.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a1ffb640 1 -- 172.21.15.94:0/2276653398 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f589c005020 con 0x7f58a410c510 2024-04-21T08:50:58.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58a1ffb640 1 -- 172.21.15.94:0/2276653398 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c0116b0 con 0x7f58a410c510 2024-04-21T08:50:58.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.671+0000 7f58aabc7640 1 -- 172.21.15.94:0/2276653398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a41a5310 con 0x7f58a410c510 2024-04-21T08:50:58.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.672+0000 7f58aabc7640 1 -- 172.21.15.94:0/2276653398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a4068d70 con 0x7f58a410c510 2024-04-21T08:50:58.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.673+0000 7f58a1ffb640 1 -- 172.21.15.94:0/2276653398 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f589c0051d0 con 0x7f58a410c510 2024-04-21T08:50:58.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.674+0000 7f58a1ffb640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58940a3d20 0x7f58940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.674+0000 7f58a8963640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58940a3d20 0x7f58940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.674+0000 7f58a1ffb640 1 -- 172.21.15.94:0/2276653398 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f589c0d0e50 con 0x7f58a410c510 2024-04-21T08:50:58.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.674+0000 7f58a8963640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58940a3d20 0x7f58940a61e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5898000c00 tx=0x7f5898004040 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.678+0000 7f58a1ffb640 1 -- 172.21.15.94:0/2276653398 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f589c016030 con 0x7f58a410c510 2024-04-21T08:50:58.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1380419986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:58.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.893+0000 7f58aabc7640 1 -- 172.21.15.94:0/2276653398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f58a4107690 con 0x7f58a410c510 2024-04-21T08:50:58.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.895+0000 7f58a1ffb640 1 -- 172.21.15.94:0/2276653398 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) ==== 161+0+0 (secure 0 0 0) 0x7f589c09d0c0 con 0x7f58a410c510 2024-04-21T08:50:58.895 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-21T08:50:58.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 -- 172.21.15.94:0/2276653398 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58940a3d20 msgr2=0x7f58940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58940a3d20 0x7f58940a61e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5898000c00 tx=0x7f5898004040 comp rx=0 tx=0).stop 2024-04-21T08:50:58.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 -- 172.21.15.94:0/2276653398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58a410c510 msgr2=0x7f58a4198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58a410c510 0x7f58a4198600 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f589c00eaa0 tx=0x7f589c00eea0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 -- 172.21.15.94:0/2276653398 shutdown_connections 2024-04-21T08:50:58.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f58940a3d20 0x7f58940a61e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a4103af0 0x7f58a419e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58a410c510 0x7f58a4198600 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 --2- 172.21.15.94:0/2276653398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a41031d0 0x7f58a419e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 -- 172.21.15.94:0/2276653398 >> 172.21.15.94:0/2276653398 conn(0x7f58a4077f50 msgr2=0x7f58a40fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 -- 172.21.15.94:0/2276653398 shutdown_connections 2024-04-21T08:50:58.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:58.898+0000 7f58aabc7640 1 -- 172.21.15.94:0/2276653398 wait complete. 2024-04-21T08:50:58.909 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-21T08:50:59.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.011+0000 7f1aafc8c640 1 -- 172.21.15.94:0/1159225646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1aa8106680 msgr2=0x7f1aa8106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.011+0000 7f1aafc8c640 1 --2- 172.21.15.94:0/1159225646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1aa8106680 0x7f1aa8106a60 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009ca0 tx=0x7f1aa402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.011+0000 7f1aafc8c640 1 -- 172.21.15.94:0/1159225646 shutdown_connections 2024-04-21T08:50:59.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.011+0000 7f1aafc8c640 1 --2- 172.21.15.94:0/1159225646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8100f80 0x7f1aa810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.011+0000 7f1aafc8c640 1 --2- 172.21.15.94:0/1159225646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aa81005c0 0x7f1aa8100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.011+0000 7f1aafc8c640 1 --2- 172.21.15.94:0/1159225646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1aa8106680 0x7f1aa8106a60 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.011+0000 7f1aafc8c640 1 -- 172.21.15.94:0/1159225646 >> 172.21.15.94:0/1159225646 conn(0x7f1aa8077410 msgr2=0x7f1aa80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.011+0000 7f1aafc8c640 1 -- 172.21.15.94:0/1159225646 shutdown_connections 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.011+0000 7f1aafc8c640 1 -- 172.21.15.94:0/1159225646 wait complete. 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aafc8c640 1 Processor -- start 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aafc8c640 1 -- start start 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aafc8c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1aa81005c0 0x7f1aa8198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aafc8c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aa8100f80 0x7f1aa8198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aafc8c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8106680 0x7f1aa819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aafc8c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa81110a0 con 0x7f1aa81005c0 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aafc8c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa8110f20 con 0x7f1aa8100f80 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aafc8c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa8111220 con 0x7f1aa8106680 2024-04-21T08:50:59.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aada28640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1aa81005c0 0x7f1aa8198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aada28640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1aa81005c0 0x7f1aa8198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40990/0 (socket says 172.21.15.94:40990) 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aada28640 1 -- 172.21.15.94:0/3072870839 learned_addr learned my addr 172.21.15.94:0/3072870839 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.012+0000 7f1aad227640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aa8100f80 0x7f1aa8198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1aada28640 1 -- 172.21.15.94:0/3072870839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8106680 msgr2=0x7f1aa819ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1aae229640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8106680 0x7f1aa819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1aada28640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8106680 0x7f1aa819ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1aada28640 1 -- 172.21.15.94:0/3072870839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aa8100f80 msgr2=0x7f1aa8198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1aada28640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aa8100f80 0x7f1aa8198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1aada28640 1 -- 172.21.15.94:0/3072870839 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa819d600 con 0x7f1aa81005c0 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1aae229640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8106680 0x7f1aa819ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1aada28640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1aa81005c0 0x7f1aa8198660 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4000f80 tx=0x7f1aa4035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1a9effd640 1 -- 172.21.15.94:0/3072870839 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4039040 con 0x7f1aa81005c0 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1a9effd640 1 -- 172.21.15.94:0/3072870839 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1aa4044070 con 0x7f1aa81005c0 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1aafc8c640 1 -- 172.21.15.94:0/3072870839 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa819d890 con 0x7f1aa81005c0 2024-04-21T08:50:59.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1a9effd640 1 -- 172.21.15.94:0/3072870839 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40042a0 con 0x7f1aa81005c0 2024-04-21T08:50:59.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.013+0000 7f1aafc8c640 1 -- 172.21.15.94:0/3072870839 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa81a5180 con 0x7f1aa81005c0 2024-04-21T08:50:59.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.014+0000 7f1aafc8c640 1 -- 172.21.15.94:0/3072870839 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa80691c0 con 0x7f1aa81005c0 2024-04-21T08:50:59.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.015+0000 7f1a9effd640 1 -- 172.21.15.94:0/3072870839 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f1aa4004440 con 0x7f1aa81005c0 2024-04-21T08:50:59.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.016+0000 7f1a9effd640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1a940a3d20 0x7f1a940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.016+0000 7f1aad227640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1a940a3d20 0x7f1a940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.016+0000 7f1a9effd640 1 -- 172.21.15.94:0/3072870839 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9769+0+0 (secure 0 0 0) 0x7f1aa40f36f0 con 0x7f1aa81005c0 2024-04-21T08:50:59.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.016+0000 7f1aad227640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1a940a3d20 0x7f1a940a61e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8105ca0 tx=0x7f1a9800a2d0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:59.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.019+0000 7f1a9effd640 1 -- 172.21.15.94:0/3072870839 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1aa40bf9c0 con 0x7f1aa81005c0 2024-04-21T08:50:59.194 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a[27984]: 2024-04-21T08:50:58.894+0000 7f5c6308a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:50:59.195 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b[61988]: ignoring --setuser ceph since I am not root 2024-04-21T08:50:59.195 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b[61988]: ignoring --setgroup ceph since I am not root 2024-04-21T08:50:59.195 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-mds[61992]: -- 172.21.15.111:0/779288206 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555e7001fa40 con 0x555e6fff2c00 2024-04-21T08:50:59.195 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b[61988]: starting mds.b at 2024-04-21T08:50:59.195 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e[61372]: ignoring --setuser ceph since I am not root 2024-04-21T08:50:59.195 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e[61372]: ignoring --setgroup ceph since I am not root 2024-04-21T08:50:59.195 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-mds[61376]: -- 172.21.15.111:0/317235049 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571c1c2da40 con 0x5571c1c00c00 2024-04-21T08:50:59.195 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e[61372]: starting mds.e at 2024-04-21T08:50:59.196 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h[61684]: ignoring --setuser ceph since I am not root 2024-04-21T08:50:59.196 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h[61684]: ignoring --setgroup ceph since I am not root 2024-04-21T08:50:59.196 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-mds[61688]: -- 172.21.15.111:0/2321243346 <== mon.0 v2:172.21.15.94:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c742935a40 con 0x55c742908c00 2024-04-21T08:50:59.196 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:50:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h[61684]: starting mds.h at 2024-04-21T08:50:59.196 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c[62571]: ignoring --setuser ceph since I am not root 2024-04-21T08:50:59.196 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c[62571]: ignoring --setgroup ceph since I am not root 2024-04-21T08:50:59.196 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-mds[62575]: -- 172.21.15.173:0/4192751400 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d400813a40 con 0x55d4007e6800 2024-04-21T08:50:59.197 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c[62571]: starting mds.c at 2024-04-21T08:50:59.197 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i[62891]: ignoring --setuser ceph since I am not root 2024-04-21T08:50:59.197 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i[62891]: ignoring --setgroup ceph since I am not root 2024-04-21T08:50:59.197 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-mds[62895]: -- 172.21.15.173:0/1402672851 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558120439a40 con 0x55812040d000 2024-04-21T08:50:59.198 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i[62891]: starting mds.i at 2024-04-21T08:50:59.198 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f[62263]: ignoring --setuser ceph since I am not root 2024-04-21T08:50:59.198 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f[62263]: ignoring --setgroup ceph since I am not root 2024-04-21T08:50:59.198 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-mds[62267]: -- 172.21.15.173:0/1887108364 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562922121a40 con 0x5629220f4c00 2024-04-21T08:50:59.198 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:50:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f[62263]: starting mds.f at 2024-04-21T08:50:59.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.236+0000 7f1aafc8c640 1 -- 172.21.15.94:0/3072870839 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1aa81a5350 con 0x7f1aa81005c0 2024-04-21T08:50:59.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:59 smithi094 ceph-mon[27988]: pgmap v341: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:59.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:59 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/372181867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:59.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:59 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/378822292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:59.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:59 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2276653398' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:50:59.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:59 smithi094 ceph-mon[27988]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T08:50:59.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:59 smithi094 ceph-mon[27988]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:50:59.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:59 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/2276653398' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T08:50:59.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:59 smithi094 ceph-mon[27988]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T08:50:59.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:50:59 smithi094 ceph-mon[27988]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-21T08:50:59.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:59 smithi173 ceph-mon[31428]: pgmap v341: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:59.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:59 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/372181867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:59.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:59 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/378822292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:59.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:59 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2276653398' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:50:59.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:59 smithi173 ceph-mon[31428]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T08:50:59.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:59 smithi173 ceph-mon[31428]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:50:59.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:59 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/2276653398' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T08:50:59.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:59 smithi173 ceph-mon[31428]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T08:50:59.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:50:59 smithi173 ceph-mon[31428]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-21T08:50:59.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:59 smithi111 ceph-mon[31493]: pgmap v341: 129 pgs: 129 active+clean; 4.6 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:50:59.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:59 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/372181867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:59.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:59 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/378822292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:59.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:59 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2276653398' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:50:59.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:59 smithi111 ceph-mon[31493]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T08:50:59.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:59 smithi111 ceph-mon[31493]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:50:59.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:59 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/2276653398' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T08:50:59.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:59 smithi111 ceph-mon[31493]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T08:50:59.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:50:59 smithi111 ceph-mon[31493]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-21T08:50:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.897+0000 7f1a9effd640 1 -- 172.21.15.94:0/3072870839 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) ==== 104+0+0 (secure 0 0 0) 0x7f1aa40bf2c0 con 0x7f1aa81005c0 2024-04-21T08:50:59.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 -- 172.21.15.94:0/3072870839 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1a940a3d20 msgr2=0x7f1a940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1a940a3d20 0x7f1a940a61e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8105ca0 tx=0x7f1a9800a2d0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 -- 172.21.15.94:0/3072870839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1aa81005c0 msgr2=0x7f1aa8198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1aa81005c0 0x7f1aa8198660 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4000f80 tx=0x7f1aa4035d60 comp rx=0 tx=0).stop 2024-04-21T08:50:59.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 -- 172.21.15.94:0/3072870839 shutdown_connections 2024-04-21T08:50:59.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f1a940a3d20 0x7f1a940a61e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8106680 0x7f1aa819ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aa8100f80 0x7f1aa8198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 --2- 172.21.15.94:0/3072870839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1aa81005c0 0x7f1aa8198660 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 -- 172.21.15.94:0/3072870839 >> 172.21.15.94:0/3072870839 conn(0x7f1aa8077410 msgr2=0x7f1aa80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:59.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 -- 172.21.15.94:0/3072870839 shutdown_connections 2024-04-21T08:50:59.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:50:59.900+0000 7f1aafc8c640 1 -- 172.21.15.94:0/3072870839 wait complete. 2024-04-21T08:50:59.916 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-21T08:51:00.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.026+0000 7fcb25b32640 1 -- 172.21.15.94:0/4257931006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb20106150 msgr2=0x7fcb2010dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.026+0000 7fcb25b32640 1 --2- 172.21.15.94:0/4257931006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb20106150 0x7fcb2010dd20 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fcb18009ca0 tx=0x7fcb1802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.026+0000 7fcb25b32640 1 -- 172.21.15.94:0/4257931006 shutdown_connections 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.026+0000 7fcb25b32640 1 --2- 172.21.15.94:0/4257931006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb20106150 0x7fcb2010dd20 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.026+0000 7fcb25b32640 1 --2- 172.21.15.94:0/4257931006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20101a30 0x7fcb20105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.026+0000 7fcb25b32640 1 --2- 172.21.15.94:0/4257931006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb20101080 0x7fcb20101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.026+0000 7fcb25b32640 1 -- 172.21.15.94:0/4257931006 >> 172.21.15.94:0/4257931006 conn(0x7fcb20078000 msgr2=0x7fcb20078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.026+0000 7fcb25b32640 1 -- 172.21.15.94:0/4257931006 shutdown_connections 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.026+0000 7fcb25b32640 1 -- 172.21.15.94:0/4257931006 wait complete. 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb25b32640 1 Processor -- start 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb25b32640 1 -- start start 2024-04-21T08:51:00.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb25b32640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20101080 0x7fcb20199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb25b32640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb20106150 0x7fcb20199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb25b32640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb20199fd0 0x7fcb201a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb25b32640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcb201110c0 con 0x7fcb20199fd0 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb25b32640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcb20110f40 con 0x7fcb20101080 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb25b32640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb20111240 con 0x7fcb20106150 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb20199fd0 0x7fcb201a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb20199fd0 0x7fcb201a4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40998/0 (socket says 172.21.15.94:40998) 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1ffff640 1 -- 172.21.15.94:0/727507607 learned_addr learned my addr 172.21.15.94:0/727507607 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1f7fe640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20101080 0x7fcb20199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb17fff640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb20106150 0x7fcb20199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1ffff640 1 -- 172.21.15.94:0/727507607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb20106150 msgr2=0x7fcb20199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1ffff640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb20106150 0x7fcb20199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1ffff640 1 -- 172.21.15.94:0/727507607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20101080 msgr2=0x7fcb20199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1ffff640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20101080 0x7fcb20199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1ffff640 1 -- 172.21.15.94:0/727507607 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb201a4d50 con 0x7fcb20199fd0 2024-04-21T08:51:00.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1f7fe640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20101080 0x7fcb20199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:00.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.027+0000 7fcb1ffff640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb20199fd0 0x7fcb201a4580 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fcb18005e30 tx=0x7fcb18034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:00.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.028+0000 7fcb1d7fa640 1 -- 172.21.15.94:0/727507607 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb18039040 con 0x7fcb20199fd0 2024-04-21T08:51:00.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.028+0000 7fcb25b32640 1 -- 172.21.15.94:0/727507607 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb201a4fe0 con 0x7fcb20199fd0 2024-04-21T08:51:00.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.028+0000 7fcb1d7fa640 1 -- 172.21.15.94:0/727507607 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb18044070 con 0x7fcb20199fd0 2024-04-21T08:51:00.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.028+0000 7fcb1d7fa640 1 -- 172.21.15.94:0/727507607 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb18003e60 con 0x7fcb20199fd0 2024-04-21T08:51:00.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.028+0000 7fcb25b32640 1 -- 172.21.15.94:0/727507607 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb201a54f0 con 0x7fcb20199fd0 2024-04-21T08:51:00.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.029+0000 7fcb25b32640 1 -- 172.21.15.94:0/727507607 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb20068d70 con 0x7fcb20199fd0 2024-04-21T08:51:00.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.030+0000 7fcb1d7fa640 1 -- 172.21.15.94:0/727507607 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fcb180324c0 con 0x7fcb20199fd0 2024-04-21T08:51:00.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.030+0000 7fcb1d7fa640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb0c0a3d20 0x7fcb0c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.030+0000 7fcb1f7fe640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb0c0a3d20 0x7fcb0c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.030+0000 7fcb1d7fa640 1 -- 172.21.15.94:0/727507607 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9769+0+0 (secure 0 0 0) 0x7fcb180f2f30 con 0x7fcb20199fd0 2024-04-21T08:51:00.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.031+0000 7fcb1f7fe640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb0c0a3d20 0x7fcb0c0a61e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcb100058c0 tx=0x7fcb10005850 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:00.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.039+0000 7fcb1d7fa640 1 -- 172.21.15.94:0/727507607 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb180bf200 con 0x7fcb20199fd0 2024-04-21T08:51:00.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:00.227+0000 7fcb25b32640 1 -- 172.21.15.94:0/727507607 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcb20109380 con 0x7fcb20199fd0 2024-04-21T08:51:00.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3072870839' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: Cluster is now healthy 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/3072870839' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.173:6834/1758118101,v1:172.21.15.173:6835/1758118101] up:boot 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.111:6834/856727688,v1:172.21.15.111:6835/856727688] up:boot 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.173:6836/2678480920,v1:172.21.15.173:6838/2678480920] up:boot 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.173:6837/2510447250,v1:172.21.15.173:6839/2510447250] up:boot 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.111:6836/124918574,v1:172.21.15.111:6839/124918574] up:boot 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: mds.? [v2:172.21.15.111:6837/1932717055,v1:172.21.15.111:6838/1932717055] up:boot 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: fsmap 12 up:standby 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:51:00.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:51:00.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:51:00.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:51:00.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:51:00.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:00 smithi094 ceph-mon[27988]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:51:00.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3072870839' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:51:00.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T08:51:00.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T08:51:00.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: Cluster is now healthy 2024-04-21T08:51:00.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/3072870839' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.173:6834/1758118101,v1:172.21.15.173:6835/1758118101] up:boot 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.111:6834/856727688,v1:172.21.15.111:6835/856727688] up:boot 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.173:6836/2678480920,v1:172.21.15.173:6838/2678480920] up:boot 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.173:6837/2510447250,v1:172.21.15.173:6839/2510447250] up:boot 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.111:6836/124918574,v1:172.21.15.111:6839/124918574] up:boot 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: mds.? [v2:172.21.15.111:6837/1932717055,v1:172.21.15.111:6838/1932717055] up:boot 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: fsmap 12 up:standby 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:51:00.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:00 smithi173 ceph-mon[31428]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:51:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3072870839' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:51:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T08:51:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T08:51:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: Cluster is now healthy 2024-04-21T08:51:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/3072870839' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:51:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.173:6834/1758118101,v1:172.21.15.173:6835/1758118101] up:boot 2024-04-21T08:51:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.111:6834/856727688,v1:172.21.15.111:6835/856727688] up:boot 2024-04-21T08:51:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.173:6836/2678480920,v1:172.21.15.173:6838/2678480920] up:boot 2024-04-21T08:51:00.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.173:6837/2510447250,v1:172.21.15.173:6839/2510447250] up:boot 2024-04-21T08:51:00.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.111:6836/124918574,v1:172.21.15.111:6839/124918574] up:boot 2024-04-21T08:51:00.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: mds.? [v2:172.21.15.111:6837/1932717055,v1:172.21.15.111:6838/1932717055] up:boot 2024-04-21T08:51:00.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: fsmap 12 up:standby 2024-04-21T08:51:00.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:51:00.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:51:00.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:51:00.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:51:00.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:51:00.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:00 smithi111 ceph-mon[31493]: from='mgr.15453 172.21.15.94:0/3430865897' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:51:01.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.202+0000 7fcb1d7fa640 1 -- 172.21.15.94:0/727507607 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) ==== 181+0+0 (secure 0 0 0) 0x7fcb180beb00 con 0x7fcb20199fd0 2024-04-21T08:51:01.203 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.205+0000 7fcb25b32640 1 -- 172.21.15.94:0/727507607 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb0c0a3d20 msgr2=0x7fcb0c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.205+0000 7fcb25b32640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb0c0a3d20 0x7fcb0c0a61e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcb100058c0 tx=0x7fcb10005850 comp rx=0 tx=0).stop 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.205+0000 7fcb25b32640 1 -- 172.21.15.94:0/727507607 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb20199fd0 msgr2=0x7fcb201a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.205+0000 7fcb25b32640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb20199fd0 0x7fcb201a4580 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fcb18005e30 tx=0x7fcb18034760 comp rx=0 tx=0).stop 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.205+0000 7fcb25b32640 1 -- 172.21.15.94:0/727507607 shutdown_connections 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.205+0000 7fcb25b32640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fcb0c0a3d20 0x7fcb0c0a61e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.206+0000 7fcb25b32640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb20106150 0x7fcb20199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.206+0000 7fcb25b32640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb20199fd0 0x7fcb201a4580 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.206+0000 7fcb25b32640 1 --2- 172.21.15.94:0/727507607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20101080 0x7fcb20199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.206+0000 7fcb25b32640 1 -- 172.21.15.94:0/727507607 >> 172.21.15.94:0/727507607 conn(0x7fcb20078000 msgr2=0x7fcb200fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:01.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.206+0000 7fcb25b32640 1 -- 172.21.15.94:0/727507607 shutdown_connections 2024-04-21T08:51:01.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.206+0000 7fcb25b32640 1 -- 172.21.15.94:0/727507607 wait complete. 2024-04-21T08:51:01.218 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-21T08:51:01.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.321+0000 7f8235d52640 1 -- 172.21.15.94:0/900137630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8230101090 msgr2=0x7f8230101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.321+0000 7f8235d52640 1 --2- 172.21.15.94:0/900137630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8230101090 0x7f8230101470 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f8220009900 tx=0x7f822002bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:01.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.321+0000 7f8235d52640 1 -- 172.21.15.94:0/900137630 shutdown_connections 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.321+0000 7f8235d52640 1 --2- 172.21.15.94:0/900137630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8230106160 0x7f823010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.321+0000 7f8235d52640 1 --2- 172.21.15.94:0/900137630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8230101a40 0x7f8230105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.321+0000 7f8235d52640 1 --2- 172.21.15.94:0/900137630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8230101090 0x7f8230101470 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.321+0000 7f8235d52640 1 -- 172.21.15.94:0/900137630 >> 172.21.15.94:0/900137630 conn(0x7f8230077f50 msgr2=0x7f8230078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.321+0000 7f8235d52640 1 -- 172.21.15.94:0/900137630 shutdown_connections 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.321+0000 7f8235d52640 1 -- 172.21.15.94:0/900137630 wait complete. 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.321+0000 7f8235d52640 1 Processor -- start 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8235d52640 1 -- start start 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8235d52640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8230101090 0x7f82301986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8235d52640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8230106160 0x7f8230198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8235d52640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f823019cf80 0x7f823019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8235d52640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f82301121e0 con 0x7f8230106160 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8235d52640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8230112060 con 0x7f823019cf80 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8235d52640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8230112360 con 0x7f8230101090 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8227fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8230106160 0x7f8230198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8227fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8230106160 0x7f8230198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41010/0 (socket says 172.21.15.94:41010) 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8227fff640 1 -- 172.21.15.94:0/659570765 learned_addr learned my addr 172.21.15.94:0/659570765 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f822ffff640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f823019cf80 0x7f823019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f822f7fe640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8230101090 0x7f82301986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8227fff640 1 -- 172.21.15.94:0/659570765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8230101090 msgr2=0x7f82301986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8227fff640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8230101090 0x7f82301986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8227fff640 1 -- 172.21.15.94:0/659570765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f823019cf80 msgr2=0x7f823019d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8227fff640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f823019cf80 0x7f823019d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8227fff640 1 -- 172.21.15.94:0/659570765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823019da70 con 0x7f8230106160 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8227fff640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8230106160 0x7f8230198c00 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f821800c970 tx=0x7f821800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:01.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f822d7fa640 1 -- 172.21.15.94:0/659570765 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218014070 con 0x7f8230106160 2024-04-21T08:51:01.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f822d7fa640 1 -- 172.21.15.94:0/659570765 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8218004440 con 0x7f8230106160 2024-04-21T08:51:01.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8235d52640 1 -- 172.21.15.94:0/659570765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823019dd60 con 0x7f8230106160 2024-04-21T08:51:01.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f822d7fa640 1 -- 172.21.15.94:0/659570765 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218003e50 con 0x7f8230106160 2024-04-21T08:51:01.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.322+0000 7f8235d52640 1 -- 172.21.15.94:0/659570765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82301a5550 con 0x7f8230106160 2024-04-21T08:51:01.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.324+0000 7f8235d52640 1 -- 172.21.15.94:0/659570765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8230068d70 con 0x7f8230106160 2024-04-21T08:51:01.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.325+0000 7f822d7fa640 1 -- 172.21.15.94:0/659570765 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7f8218017070 con 0x7f8230106160 2024-04-21T08:51:01.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.325+0000 7f822d7fa640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f821c0a3ec0 0x7f821c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.325+0000 7f822f7fe640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f821c0a3ec0 0x7f821c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.326+0000 7f822d7fa640 1 -- 172.21.15.94:0/659570765 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9178+0+0 (secure 0 0 0) 0x7f82180d06d0 con 0x7f8230106160 2024-04-21T08:51:01.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.326+0000 7f822f7fe640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f821c0a3ec0 0x7f821c0a6380 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8220036040 tx=0x7f8220000bc0 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:01.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.330+0000 7f822d7fa640 1 -- 172.21.15.94:0/659570765 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f821801e030 con 0x7f8230106160 2024-04-21T08:51:01.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:01.506+0000 7f8235d52640 1 -- 172.21.15.94:0/659570765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8230101a40 con 0x7f8230106160 2024-04-21T08:51:01.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:01 smithi094 ceph-mon[27988]: pgmap v343: 129 pgs: 129 active+clean; 4.6 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-21T08:51:01.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:01 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/727507607' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:51:01.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:01 smithi173 ceph-mon[31428]: pgmap v343: 129 pgs: 129 active+clean; 4.6 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-21T08:51:01.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:01 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/727507607' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:51:01.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:01 smithi111 ceph-mon[31493]: pgmap v343: 129 pgs: 129 active+clean; 4.6 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-21T08:51:01.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:01 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/727507607' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:51:02.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.207+0000 7f822d7fa640 1 -- 172.21.15.94:0/659570765 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) ==== 169+0+0 (secure 0 0 0) 0x7f821800de20 con 0x7f8230106160 2024-04-21T08:51:02.208 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-04-21T08:51:02.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 -- 172.21.15.94:0/659570765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f821c0a3ec0 msgr2=0x7f821c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f821c0a3ec0 0x7f821c0a6380 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8220036040 tx=0x7f8220000bc0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 -- 172.21.15.94:0/659570765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8230106160 msgr2=0x7f8230198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8230106160 0x7f8230198c00 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f821800c970 tx=0x7f821800ce30 comp rx=0 tx=0).stop 2024-04-21T08:51:02.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 -- 172.21.15.94:0/659570765 shutdown_connections 2024-04-21T08:51:02.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7f821c0a3ec0 0x7f821c0a6380 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f823019cf80 0x7f823019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8230106160 0x7f8230198c00 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 --2- 172.21.15.94:0/659570765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8230101090 0x7f82301986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 -- 172.21.15.94:0/659570765 >> 172.21.15.94:0/659570765 conn(0x7f8230077f50 msgr2=0x7f82300fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 -- 172.21.15.94:0/659570765 shutdown_connections 2024-04-21T08:51:02.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.211+0000 7f8235d52640 1 -- 172.21.15.94:0/659570765 wait complete. 2024-04-21T08:51:02.225 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-21T08:51:02.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.338+0000 7fa2a848c640 1 -- 172.21.15.94:0/2248114721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2a0107270 msgr2=0x7fa2a010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.338+0000 7fa2a848c640 1 --2- 172.21.15.94:0/2248114721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2a0107270 0x7fa2a010ee40 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009900 tx=0x7fa29c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.338+0000 7fa2a848c640 1 -- 172.21.15.94:0/2248114721 shutdown_connections 2024-04-21T08:51:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.338+0000 7fa2a848c640 1 --2- 172.21.15.94:0/2248114721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2a0107270 0x7fa2a010ee40 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.338+0000 7fa2a848c640 1 --2- 172.21.15.94:0/2248114721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a0102b50 0x7fa2a0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.338+0000 7fa2a848c640 1 --2- 172.21.15.94:0/2248114721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a01021a0 0x7fa2a0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.338+0000 7fa2a848c640 1 -- 172.21.15.94:0/2248114721 >> 172.21.15.94:0/2248114721 conn(0x7fa2a00fda40 msgr2=0x7fa2a00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.338+0000 7fa2a848c640 1 -- 172.21.15.94:0/2248114721 shutdown_connections 2024-04-21T08:51:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.338+0000 7fa2a848c640 1 -- 172.21.15.94:0/2248114721 wait complete. 2024-04-21T08:51:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a848c640 1 Processor -- start 2024-04-21T08:51:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a848c640 1 -- start start 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a848c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2a01021a0 0x7fa2a0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a848c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a0107270 0x7fa2a0198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a848c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a019cfb0 0x7fa2a019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a848c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a01121c0 con 0x7fa2a01021a0 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a848c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a0112040 con 0x7fa2a0107270 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a848c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a0112340 con 0x7fa2a019cfb0 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a6228640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2a01021a0 0x7fa2a0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a6228640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2a01021a0 0x7fa2a0198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41032/0 (socket says 172.21.15.94:41032) 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a6228640 1 -- 172.21.15.94:0/1677063812 learned_addr learned my addr 172.21.15.94:0/1677063812 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a5a27640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a0107270 0x7fa2a0198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.339+0000 7fa2a6a29640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a019cfb0 0x7fa2a019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2a6228640 1 -- 172.21.15.94:0/1677063812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a019cfb0 msgr2=0x7fa2a019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2a6228640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a019cfb0 0x7fa2a019d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2a6228640 1 -- 172.21.15.94:0/1677063812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a0107270 msgr2=0x7fa2a0198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2a6228640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a0107270 0x7fa2a0198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2a6228640 1 -- 172.21.15.94:0/1677063812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a019daa0 con 0x7fa2a01021a0 2024-04-21T08:51:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2a6228640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2a01021a0 0x7fa2a0198740 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fa29800cce0 tx=0x7fa29800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2977fe640 1 -- 172.21.15.94:0/1677063812 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa298004b20 con 0x7fa2a01021a0 2024-04-21T08:51:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2977fe640 1 -- 172.21.15.94:0/1677063812 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa298004cc0 con 0x7fa2a01021a0 2024-04-21T08:51:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2a848c640 1 -- 172.21.15.94:0/1677063812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2a019dd90 con 0x7fa2a01021a0 2024-04-21T08:51:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2977fe640 1 -- 172.21.15.94:0/1677063812 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29800f6b0 con 0x7fa2a01021a0 2024-04-21T08:51:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.340+0000 7fa2a848c640 1 -- 172.21.15.94:0/1677063812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2a01a5580 con 0x7fa2a01021a0 2024-04-21T08:51:02.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.341+0000 7fa2a848c640 1 -- 172.21.15.94:0/1677063812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2a00691c0 con 0x7fa2a01021a0 2024-04-21T08:51:02.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.342+0000 7fa2977fe640 1 -- 172.21.15.94:0/1677063812 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 25) ==== 138777+0+0 (secure 0 0 0) 0x7fa29800f850 con 0x7fa2a01021a0 2024-04-21T08:51:02.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.343+0000 7fa2977fe640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa28c0a3d20 0x7fa28c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.343+0000 7fa2a5a27640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa28c0a3d20 0x7fa28c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.343+0000 7fa2977fe640 1 -- 172.21.15.94:0/1677063812 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8561+0+0 (secure 0 0 0) 0x7fa2980d08e0 con 0x7fa2a01021a0 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.343+0000 7fa2a5a27640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa28c0a3d20 0x7fa28c0a61e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00fdf60 tx=0x7fa29000b390 comp rx=0 tx=0).ready entity=mgr.15453 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.348+0000 7fa2977fe640 1 -- 172.21.15.94:0/1677063812 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa298014030 con 0x7fa2a01021a0 2024-04-21T08:51:02.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:02 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/727507607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:51:02.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:02 smithi094 ceph-mon[27988]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T08:51:02.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:02 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/659570765' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:51:02.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:02 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/727507607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:51:02.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:02 smithi173 ceph-mon[31428]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T08:51:02.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:02 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/659570765' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:51:02.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:02.526+0000 7fa2a848c640 1 -- 172.21.15.94:0/1677063812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa2a0199a40 con 0x7fa2a01021a0 2024-04-21T08:51:02.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:02 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/727507607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:51:02.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:02 smithi111 ceph-mon[31493]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T08:51:02.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:02 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/659570765' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:51:03.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.216+0000 7fa2977fe640 1 -- 172.21.15.94:0/1677063812 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v106) ==== 178+0+0 (secure 0 0 0) 0x7fa29809cf50 con 0x7fa2a01021a0 2024-04-21T08:51:03.217 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data_ec' removed 2024-04-21T08:51:03.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 -- 172.21.15.94:0/1677063812 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa28c0a3d20 msgr2=0x7fa28c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa28c0a3d20 0x7fa28c0a61e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00fdf60 tx=0x7fa29000b390 comp rx=0 tx=0).stop 2024-04-21T08:51:03.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 -- 172.21.15.94:0/1677063812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2a01021a0 msgr2=0x7fa2a0198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2a01021a0 0x7fa2a0198740 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fa29800cce0 tx=0x7fa29800a620 comp rx=0 tx=0).stop 2024-04-21T08:51:03.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 -- 172.21.15.94:0/1677063812 shutdown_connections 2024-04-21T08:51:03.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.94:6800/4147145528,v1:172.21.15.94:6801/4147145528] conn(0x7fa28c0a3d20 0x7fa28c0a61e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a019cfb0 0x7fa2a019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a0107270 0x7fa2a0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 --2- 172.21.15.94:0/1677063812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2a01021a0 0x7fa2a0198740 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 -- 172.21.15.94:0/1677063812 >> 172.21.15.94:0/1677063812 conn(0x7fa2a00fda40 msgr2=0x7fa2a00fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:03.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 -- 172.21.15.94:0/1677063812 shutdown_connections 2024-04-21T08:51:03.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-21T08:51:03.220+0000 7fa2a848c640 1 -- 172.21.15.94:0/1677063812 wait complete. 2024-04-21T08:51:03.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:03.264 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:03.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:03.327 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-21T08:51:03.328 DEBUG:teuthology.orchestra.run.smithi094:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:51:03.344 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:51:03.360 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:51:03.384 INFO:tasks.cephadm:Stopping all daemons... 2024-04-21T08:51:03.384 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-21T08:51:03.384 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.a 2024-04-21T08:51:03.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:03 smithi094 ceph-mon[27988]: pgmap v345: 97 pgs: 97 active+clean; 577 KiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:03 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/659570765' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:51:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:03 smithi094 ceph-mon[27988]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T08:51:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:03 smithi094 ceph-mon[27988]: from='client.? 172.21.15.94:0/1677063812' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:51:03.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:03 smithi094 systemd[1]: Stopping Ceph mon.a for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:03.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:03 smithi173 ceph-mon[31428]: pgmap v345: 97 pgs: 97 active+clean; 577 KiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:03.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:03 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/659570765' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:51:03.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:03 smithi173 ceph-mon[31428]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T08:51:03.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:03 smithi173 ceph-mon[31428]: from='client.? 172.21.15.94:0/1677063812' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:51:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:03 smithi111 ceph-mon[31493]: pgmap v345: 97 pgs: 97 active+clean; 577 KiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:03 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/659570765' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:51:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:03 smithi111 ceph-mon[31493]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T08:51:03.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:03 smithi111 ceph-mon[31493]: from='client.? 172.21.15.94:0/1677063812' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:51:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a[27984]: 2024-04-21T08:51:03.705+0000 7f5c68895700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:04.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a[27984]: 2024-04-21T08:51:03.705+0000 7f5c68895700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-21T08:51:04.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:03 smithi094 podman[98179]: 2024-04-21 08:51:03.758429888 +0000 UTC m=+0.158740809 container died 6dd38797f9a6c816fc29822477c71bc45161baa2b0b7f439ff07d335e6b57fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-21T08:51:04.442 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:04 smithi094 podman[98179]: 2024-04-21 08:51:04.149087732 +0000 UTC m=+0.549398650 container cleanup 6dd38797f9a6c816fc29822477c71bc45161baa2b0b7f439ff07d335e6b57fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416) 2024-04-21T08:51:04.442 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 21 08:51:04 smithi094 bash[98179]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-a 2024-04-21T08:51:04.594 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.a.service' 2024-04-21T08:51:04.640 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:04.640 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-21T08:51:04.640 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-21T08:51:04.640 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.b 2024-04-21T08:51:04.937 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:04 smithi111 systemd[1]: Stopping Ceph mon.b for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:05.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:04 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-b[31489]: 2024-04-21T08:51:04.943+0000 7fe92fd7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:05.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:04 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-b[31489]: 2024-04-21T08:51:04.943+0000 7fe92fd7f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-21T08:51:05.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:05 smithi111 podman[67400]: 2024-04-21 08:51:05.17956054 +0000 UTC m=+0.343942231 container died 5317e573d84226ee4fb002c3a6e0eafebf91ed027aff7065a1e2b16eb203a0dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-b, RELEASE=wip-lusov-quiescer-a630765, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-21T08:51:05.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:05 smithi111 podman[67400]: 2024-04-21 08:51:05.594800692 +0000 UTC m=+0.759182375 container cleanup 5317e573d84226ee4fb002c3a6e0eafebf91ed027aff7065a1e2b16eb203a0dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-b, RELEASE=wip-lusov-quiescer-a630765, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:51:05.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 08:51:05 smithi111 bash[67400]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-b 2024-04-21T08:51:06.045 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.b.service' 2024-04-21T08:51:06.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:06.089 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-21T08:51:06.089 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-21T08:51:06.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.c 2024-04-21T08:51:06.435 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:06 smithi173 systemd[1]: Stopping Ceph mon.c for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:06.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:06 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-c[31423]: 2024-04-21T08:51:06.435+0000 7f3c72ce3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:06.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:06 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-c[31423]: 2024-04-21T08:51:06.435+0000 7f3c72ce3700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-21T08:51:06.761 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:06 smithi173 podman[68367]: 2024-04-21 08:51:06.466368952 +0000 UTC m=+0.173423755 container died 55d1db549a7082bd5f8c42e65899f9d87220db5348417914ef33e13d4b96b14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-21T08:51:07.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:06 smithi173 podman[68367]: 2024-04-21 08:51:06.861418984 +0000 UTC m=+0.568473787 container cleanup 55d1db549a7082bd5f8c42e65899f9d87220db5348417914ef33e13d4b96b14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-21T08:51:07.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:06 smithi173 bash[68367]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-c 2024-04-21T08:51:07.331 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:07 smithi173 podman[68381]: 2024-04-21 08:51:07.080352095 +0000 UTC m=+0.608566381 container remove 55d1db549a7082bd5f8c42e65899f9d87220db5348417914ef33e13d4b96b14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mon-c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T08:51:07.331 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:07 smithi173 systemd[1]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-21T08:51:07.331 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:07 smithi173 systemd[1]: Stopped Ceph mon.c for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:51:07.331 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 21 08:51:07 smithi173 systemd[1]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.c.service: Consumed 12.020s CPU time. 2024-04-21T08:51:07.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mon.c.service' 2024-04-21T08:51:07.378 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:07.378 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-21T08:51:07.378 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-21T08:51:07.378 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.x 2024-04-21T08:51:07.686 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:51:07 smithi094 systemd[1]: Stopping Ceph mgr.x for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:07.995 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:51:07 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x[28222]: 2024-04-21T08:51:07.686+0000 7f5d25afb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:07.995 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:51:07 smithi094 podman[98388]: 2024-04-21 08:51:07.717789846 +0000 UTC m=+0.148882768 container died a11d562ff4903c19ec9c4cb69bbe11bced32060f9d63e9073399fa1e8e13dfa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:51:08.262 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:51:08 smithi094 podman[98388]: 2024-04-21 08:51:08.096502945 +0000 UTC m=+0.527595865 container cleanup a11d562ff4903c19ec9c4cb69bbe11bced32060f9d63e9073399fa1e8e13dfa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, ceph=True) 2024-04-21T08:51:08.263 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 21 08:51:08 smithi094 bash[98388]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-x 2024-04-21T08:51:08.512 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.x.service' 2024-04-21T08:51:08.546 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:08.547 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-21T08:51:08.547 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-21T08:51:08.547 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.y 2024-04-21T08:51:08.936 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:51:08 smithi111 systemd[1]: Stopping Ceph mgr.y for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:09.187 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:51:08 smithi111 podman[67522]: 2024-04-21 08:51:08.936956667 +0000 UTC m=+0.193707120 container died bfe91950e324280a2936b8d3b6490adde9c8c43fa8f24b96fa2885809fbb9b15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-21T08:51:09.600 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:51:09 smithi111 podman[67522]: 2024-04-21 08:51:09.333626161 +0000 UTC m=+0.590376614 container cleanup bfe91950e324280a2936b8d3b6490adde9c8c43fa8f24b96fa2885809fbb9b15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-21T08:51:09.600 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 08:51:09 smithi111 bash[67522]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-y 2024-04-21T08:51:09.753 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.y.service' 2024-04-21T08:51:09.786 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:09.786 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-21T08:51:09.786 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-21T08:51:09.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.z 2024-04-21T08:51:10.128 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:51:09 smithi173 systemd[1]: Stopping Ceph mgr.z for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:10.434 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:51:10 smithi173 podman[68490]: 2024-04-21 08:51:10.12925926 +0000 UTC m=+0.150911868 container died d3f8281e890fa13cd845924d4f8bb26f2844e5587ee1415195190727327988a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:51:10.761 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:51:10 smithi173 podman[68490]: 2024-04-21 08:51:10.517840795 +0000 UTC m=+0.539493412 container cleanup d3f8281e890fa13cd845924d4f8bb26f2844e5587ee1415195190727327988a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=) 2024-04-21T08:51:10.762 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 21 08:51:10 smithi173 bash[68490]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mgr-z 2024-04-21T08:51:10.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mgr.z.service' 2024-04-21T08:51:11.020 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:11.020 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-21T08:51:11.020 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-21T08:51:11.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.0 2024-04-21T08:51:11.334 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:11 smithi094 systemd[1]: Stopping Ceph osd.0 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:11.760 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:11 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0[42388]: 2024-04-21T08:51:11.334+0000 7fa22e974700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:11.760 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:11 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0[42388]: 2024-04-21T08:51:11.334+0000 7fa22e974700 -1 osd.0 106 *** Got signal Terminated *** 2024-04-21T08:51:11.761 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:11 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0[42388]: 2024-04-21T08:51:11.334+0000 7fa22e974700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:51:17.010 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:16 smithi094 podman[98507]: 2024-04-21 08:51:16.749196352 +0000 UTC m=+5.527094715 container died 40033951fba99b33f84a80ba456d21bf6507052b0fc5d9abe74ca0df1eedd768 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-21T08:51:17.394 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:17 smithi094 podman[98507]: 2024-04-21 08:51:17.152008716 +0000 UTC m=+5.929907078 container cleanup 40033951fba99b33f84a80ba456d21bf6507052b0fc5d9abe74ca0df1eedd768 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:51:17.394 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:17 smithi094 bash[98507]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0 2024-04-21T08:51:17.716 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:17 smithi094 podman[98524]: 2024-04-21 08:51:17.395502449 +0000 UTC m=+0.639965656 container remove 40033951fba99b33f84a80ba456d21bf6507052b0fc5d9abe74ca0df1eedd768 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-04-21T08:51:18.006 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:17 smithi094 podman[98614]: 2024-04-21 08:51:17.620197568 +0000 UTC m=+0.018114433 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:51:18.006 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:17 smithi094 podman[98614]: 2024-04-21 08:51:17.729644794 +0000 UTC m=+0.127561641 container create 2637409761f8bc8ec1f1d6692c9f76b57306620e122a41e3d30cb6d6b86639c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-21T08:51:18.007 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:18 smithi094 podman[98614]: 2024-04-21 08:51:18.00743613 +0000 UTC m=+0.405352977 container init 2637409761f8bc8ec1f1d6692c9f76b57306620e122a41e3d30cb6d6b86639c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T08:51:18.260 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:18 smithi094 podman[98614]: 2024-04-21 08:51:18.01072104 +0000 UTC m=+0.408637892 container start 2637409761f8bc8ec1f1d6692c9f76b57306620e122a41e3d30cb6d6b86639c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-04-21T08:51:18.260 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:18 smithi094 podman[98614]: 2024-04-21 08:51:18.124045233 +0000 UTC m=+0.521962091 container attach 2637409761f8bc8ec1f1d6692c9f76b57306620e122a41e3d30cb6d6b86639c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-21T08:51:18.651 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 21 08:51:18 smithi094 podman[98614]: 2024-04-21 08:51:18.296506149 +0000 UTC m=+0.694422998 container died 2637409761f8bc8ec1f1d6692c9f76b57306620e122a41e3d30cb6d6b86639c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-21T08:51:18.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.0.service' 2024-04-21T08:51:18.993 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:18.993 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-21T08:51:18.993 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-21T08:51:18.993 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.1 2024-04-21T08:51:19.281 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:19 smithi111 systemd[1]: Stopping Ceph osd.1 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:19.687 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:19 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1[36106]: 2024-04-21T08:51:19.304+0000 7f90cfec6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:19.688 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:19 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1[36106]: 2024-04-21T08:51:19.304+0000 7f90cfec6700 -1 osd.1 106 *** Got signal Terminated *** 2024-04-21T08:51:19.688 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:19 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1[36106]: 2024-04-21T08:51:19.304+0000 7f90cfec6700 -1 osd.1 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:51:24.930 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:24 smithi111 podman[67645]: 2024-04-21 08:51:24.654572226 +0000 UTC m=+5.478521308 container died 7cf8d4b0fe344475a18e8a023136de5840835b501c68f5f79e6df6db7619ec3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:51:25.188 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:25 smithi111 podman[67645]: 2024-04-21 08:51:25.021781605 +0000 UTC m=+5.845730726 container cleanup 7cf8d4b0fe344475a18e8a023136de5840835b501c68f5f79e6df6db7619ec3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:51:25.188 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:25 smithi111 bash[67645]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1 2024-04-21T08:51:25.687 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:25 smithi111 podman[67672]: 2024-04-21 08:51:25.340338478 +0000 UTC m=+0.679750371 container remove 7cf8d4b0fe344475a18e8a023136de5840835b501c68f5f79e6df6db7619ec3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-21T08:51:25.938 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:25 smithi111 podman[67769]: 2024-04-21 08:51:25.591308367 +0000 UTC m=+0.018011823 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:51:25.938 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:25 smithi111 podman[67769]: 2024-04-21 08:51:25.733029385 +0000 UTC m=+0.159732829 container create d08d5a53a46ae21774776e34384f7bb2a59d29379ef05739b2200b629ee68e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:51:26.298 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:25 smithi111 podman[67769]: 2024-04-21 08:51:25.994440233 +0000 UTC m=+0.421143675 container init d08d5a53a46ae21774776e34384f7bb2a59d29379ef05739b2200b629ee68e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:51:26.298 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:25 smithi111 podman[67769]: 2024-04-21 08:51:25.997575143 +0000 UTC m=+0.424278592 container start d08d5a53a46ae21774776e34384f7bb2a59d29379ef05739b2200b629ee68e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2) 2024-04-21T08:51:26.298 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:26 smithi111 podman[67769]: 2024-04-21 08:51:26.077821471 +0000 UTC m=+0.504524937 container attach d08d5a53a46ae21774776e34384f7bb2a59d29379ef05739b2200b629ee68e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-04-21T08:51:26.646 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 08:51:26 smithi111 podman[67769]: 2024-04-21 08:51:26.299656952 +0000 UTC m=+0.726360413 container died d08d5a53a46ae21774776e34384f7bb2a59d29379ef05739b2200b629ee68e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416) 2024-04-21T08:51:26.979 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.1.service' 2024-04-21T08:51:27.018 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:27.018 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-21T08:51:27.018 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-21T08:51:27.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.2 2024-04-21T08:51:27.331 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:27 smithi173 systemd[1]: Stopping Ceph osd.2 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:27.705 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:27 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2[36601]: 2024-04-21T08:51:27.330+0000 7fd3c79a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:27.705 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:27 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2[36601]: 2024-04-21T08:51:27.331+0000 7fd3c79a0700 -1 osd.2 106 *** Got signal Terminated *** 2024-04-21T08:51:27.705 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:27 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2[36601]: 2024-04-21T08:51:27.331+0000 7fd3c79a0700 -1 osd.2 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:51:32.974 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:32 smithi173 podman[68613]: 2024-04-21 08:51:32.668457698 +0000 UTC m=+5.448780351 container died fa1404fd60fafa81b8dcd6407547fa1e78cf860e20365c0d0981c0da6aa8c2fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-21T08:51:33.261 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:33 smithi173 podman[68613]: 2024-04-21 08:51:33.074206424 +0000 UTC m=+5.854529075 container cleanup fa1404fd60fafa81b8dcd6407547fa1e78cf860e20365c0d0981c0da6aa8c2fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-21T08:51:33.261 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:33 smithi173 bash[68613]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2 2024-04-21T08:51:33.689 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:33 smithi173 podman[68642]: 2024-04-21 08:51:33.33516076 +0000 UTC m=+0.663670280 container remove fa1404fd60fafa81b8dcd6407547fa1e78cf860e20365c0d0981c0da6aa8c2fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-21T08:51:33.980 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:33 smithi173 podman[68734]: 2024-04-21 08:51:33.593674688 +0000 UTC m=+0.018183938 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:51:33.980 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:33 smithi173 podman[68734]: 2024-04-21 08:51:33.736331503 +0000 UTC m=+0.160840781 container create 3be4422c941a9e42db904541c9cd30cb45b337f8cd8dd94ff31d6ac615b0c3ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-21T08:51:33.981 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:33 smithi173 podman[68734]: 2024-04-21 08:51:33.981369415 +0000 UTC m=+0.405878661 container init 3be4422c941a9e42db904541c9cd30cb45b337f8cd8dd94ff31d6ac615b0c3ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:51:34.261 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:33 smithi173 podman[68734]: 2024-04-21 08:51:33.985054963 +0000 UTC m=+0.409564209 container start 3be4422c941a9e42db904541c9cd30cb45b337f8cd8dd94ff31d6ac615b0c3ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:51:34.261 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:34 smithi173 podman[68734]: 2024-04-21 08:51:34.089653231 +0000 UTC m=+0.514162490 container attach 3be4422c941a9e42db904541c9cd30cb45b337f8cd8dd94ff31d6ac615b0c3ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-21T08:51:34.643 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 21 08:51:34 smithi173 podman[68734]: 2024-04-21 08:51:34.275995049 +0000 UTC m=+0.700504299 container died 3be4422c941a9e42db904541c9cd30cb45b337f8cd8dd94ff31d6ac615b0c3ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:51:34.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.2.service' 2024-04-21T08:51:35.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:35.004 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-21T08:51:35.004 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-21T08:51:35.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.3 2024-04-21T08:51:35.313 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:35 smithi094 systemd[1]: Stopping Ceph osd.3 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:35.571 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:35 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3[49208]: 2024-04-21T08:51:35.313+0000 7f2498004700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:35.571 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:35 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3[49208]: 2024-04-21T08:51:35.313+0000 7f2498004700 -1 osd.3 106 *** Got signal Terminated *** 2024-04-21T08:51:35.571 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:35 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3[49208]: 2024-04-21T08:51:35.313+0000 7f2498004700 -1 osd.3 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:51:38.260 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:38 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3[49208]: 2024-04-21T08:51:38.051+0000 7f24945f8700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.861405+0000 front 2024-04-21T08:51:11.861425+0000 (oldest deadline 2024-04-21T08:51:37.761521+0000) 2024-04-21T08:51:38.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:37 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:37.839+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:39.094 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:38 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:38.830+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:39.187 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:38 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:38.809+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:39.510 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:39 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3[49208]: 2024-04-21T08:51:39.094+0000 7f24945f8700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.861405+0000 front 2024-04-21T08:51:11.861425+0000 (oldest deadline 2024-04-21T08:51:37.761521+0000) 2024-04-21T08:51:40.011 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:39 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:39.546+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:40.067 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:39 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:39.782+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:40.187 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:39 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:39.773+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:40.510 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:40 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3[49208]: 2024-04-21T08:51:40.068+0000 7f24945f8700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.861405+0000 front 2024-04-21T08:51:11.861425+0000 (oldest deadline 2024-04-21T08:51:37.761521+0000) 2024-04-21T08:51:40.798 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:40.515+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:40.947 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:40 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:40.749+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:40.948 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:40 smithi094 podman[98722]: 2024-04-21 08:51:40.659354706 +0000 UTC m=+5.455285781 container died 0f8409ef8e0181d40c313369602630e2223639c0562d0227bec27db5340a3be8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T08:51:41.187 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:40 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:40.791+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:41.260 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:41 smithi094 podman[98722]: 2024-04-21 08:51:41.039328688 +0000 UTC m=+5.835259772 container cleanup 0f8409ef8e0181d40c313369602630e2223639c0562d0227bec27db5340a3be8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-21T08:51:41.260 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:41 smithi094 bash[98722]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3 2024-04-21T08:51:41.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:40.799+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:41.656 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:41 smithi094 podman[98751]: 2024-04-21 08:51:41.323857062 +0000 UTC m=+0.659325326 container remove 0f8409ef8e0181d40c313369602630e2223639c0562d0227bec27db5340a3be8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-21T08:51:41.760 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:41.467+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:41.943 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:41 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:41.759+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:41.943 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:41 smithi094 podman[98841]: 2024-04-21 08:51:41.560696599 +0000 UTC m=+0.017774356 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:51:41.943 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:41 smithi094 podman[98841]: 2024-04-21 08:51:41.691404406 +0000 UTC m=+0.148482150 container create 103f3d32562825c8dfa0b471b8b5be3ecdb4e5ceee8d45f87877a6d44c68ebf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-21T08:51:41.944 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:41 smithi094 podman[98841]: 2024-04-21 08:51:41.94448011 +0000 UTC m=+0.401557854 container init 103f3d32562825c8dfa0b471b8b5be3ecdb4e5ceee8d45f87877a6d44c68ebf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-21T08:51:42.187 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:41 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:41.743+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:42.238 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:41 smithi094 podman[98841]: 2024-04-21 08:51:41.947719047 +0000 UTC m=+0.404796793 container start 103f3d32562825c8dfa0b471b8b5be3ecdb4e5ceee8d45f87877a6d44c68ebf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:51:42.238 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:42 smithi094 podman[98841]: 2024-04-21 08:51:42.035966039 +0000 UTC m=+0.493043782 container attach 103f3d32562825c8dfa0b471b8b5be3ecdb4e5ceee8d45f87877a6d44c68ebf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-21T08:51:42.238 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 21 08:51:42 smithi094 podman[98841]: 2024-04-21 08:51:42.238593705 +0000 UTC m=+0.695671449 container died 103f3d32562825c8dfa0b471b8b5be3ecdb4e5ceee8d45f87877a6d44c68ebf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:51:42.260 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:41.809+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:42.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:42.468+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:42.914 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:42 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:42.803+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:42.939 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.3.service' 2024-04-21T08:51:42.978 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:42.978 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-21T08:51:42.978 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-21T08:51:42.978 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.4 2024-04-21T08:51:43.188 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:43 smithi111 systemd[1]: Stopping Ceph osd.4 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:43.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:42 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:42.779+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:43.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:42.826+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:43.687 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:43 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4[43045]: 2024-04-21T08:51:43.284+0000 7fb62450b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:43.688 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:43 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4[43045]: 2024-04-21T08:51:43.284+0000 7fb62450b700 -1 osd.4 106 *** Got signal Terminated *** 2024-04-21T08:51:43.688 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:43 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4[43045]: 2024-04-21T08:51:43.284+0000 7fb62450b700 -1 osd.4 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:51:43.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:43.466+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:44.187 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:43 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:43.787+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:44.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:43 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:43.788+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:44.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:43.864+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:44.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:44 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:44.424+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:45.187 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:44 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:44.746+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:45.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:44 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:44.793+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:45.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:44 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:44.895+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:45.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:45 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:45.394+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:46.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:45 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:45.876+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:46.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:45 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:45.775+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:46.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:45 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:45.775+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:46.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:45 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:45.825+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:46.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:45 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:45.926+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:46.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:46 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:46.377+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:46.938 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:46 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4[43045]: 2024-04-21T08:51:46.623+0000 7fb620aff700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.852810+0000 front 2024-04-21T08:51:22.852838+0000 (oldest deadline 2024-04-21T08:51:45.753012+0000) 2024-04-21T08:51:46.939 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:46 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:46.847+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:46.939 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:46 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:46.804+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:46.939 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:46 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:46.804+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:47.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:46 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:46.809+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:47.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:46 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:46.878+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:47.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:47 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:47.373+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:47.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:47 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:47.343+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:47.937 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:47 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4[43045]: 2024-04-21T08:51:47.579+0000 7fb620aff700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.852810+0000 front 2024-04-21T08:51:22.852838+0000 (oldest deadline 2024-04-21T08:51:45.753012+0000) 2024-04-21T08:51:47.938 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:47 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:47.823+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:47.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:47 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:47.756+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:47.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:47 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:47.756+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:48.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:47 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:47.790+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:48.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:47 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:47.790+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:48.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:47 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:47.906+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:48.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:48 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:48.012+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:48.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:48 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:48.413+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:48.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:48 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:48.369+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:48.935 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:48 smithi111 podman[67876]: 2024-04-21 08:51:48.661809847 +0000 UTC m=+5.492266119 container died ae63335f21041112d4b5cfcfe7d24ddab3acd291c2c27915f7f5c8ee0c6a97e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-21T08:51:48.935 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:48 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:48.776+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:48.936 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:48 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:48.756+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:48.936 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:48 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:48.756+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:49.188 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:49 smithi111 podman[67876]: 2024-04-21 08:51:49.034871497 +0000 UTC m=+5.865327762 container cleanup ae63335f21041112d4b5cfcfe7d24ddab3acd291c2c27915f7f5c8ee0c6a97e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:51:49.188 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:49 smithi111 bash[67876]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4 2024-04-21T08:51:49.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:48 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:48.800+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:49.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:48 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:48.800+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:49.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:48 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:48.905+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:49.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:48 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:48.905+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:23.311059+0000 front 2024-04-21T08:51:23.311157+0000 (oldest deadline 2024-04-21T08:51:47.911658+0000) 2024-04-21T08:51:49.262 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:48 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:48.992+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:49.618 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:49 smithi111 podman[67893]: 2024-04-21 08:51:49.278244736 +0000 UTC m=+0.611609113 container remove ae63335f21041112d4b5cfcfe7d24ddab3acd291c2c27915f7f5c8ee0c6a97e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-21T08:51:49.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:49 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:49.417+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:49.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:49 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:49.403+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:49.888 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:49 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:49.796+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:49.889 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:49 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:49.789+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:49.889 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:49 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:49.789+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:49.889 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:49 smithi111 podman[67991]: 2024-04-21 08:51:49.52274686 +0000 UTC m=+0.019037690 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:51:49.890 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:49 smithi111 podman[67991]: 2024-04-21 08:51:49.654199712 +0000 UTC m=+0.150490514 container create 64bccae1967a9f9997219506607b5c92f1d9d77ed907151a8f7e4711ace227d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-04-21T08:51:50.188 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:49 smithi111 podman[67991]: 2024-04-21 08:51:49.890061038 +0000 UTC m=+0.386351848 container init 64bccae1967a9f9997219506607b5c92f1d9d77ed907151a8f7e4711ace227d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416) 2024-04-21T08:51:50.188 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:49 smithi111 podman[67991]: 2024-04-21 08:51:49.893507185 +0000 UTC m=+0.389797991 container start 64bccae1967a9f9997219506607b5c92f1d9d77ed907151a8f7e4711ace227d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-a630765, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-21T08:51:50.188 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:49 smithi111 podman[67991]: 2024-04-21 08:51:49.981883614 +0000 UTC m=+0.478174437 container attach 64bccae1967a9f9997219506607b5c92f1d9d77ed907151a8f7e4711ace227d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-21T08:51:50.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:49 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:49.842+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:50.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:49 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:49.842+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:50.260 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:49 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:49.978+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:50.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:49 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:49.869+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:50.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:49 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:49.869+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:23.311059+0000 front 2024-04-21T08:51:23.311157+0000 (oldest deadline 2024-04-21T08:51:47.911658+0000) 2024-04-21T08:51:50.475 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 08:51:50 smithi111 podman[67991]: 2024-04-21 08:51:50.194801897 +0000 UTC m=+0.691092710 container died 64bccae1967a9f9997219506607b5c92f1d9d77ed907151a8f7e4711ace227d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-21T08:51:50.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:50 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:50.371+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:50.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:50 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:50.422+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:50.762 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.4.service' 2024-04-21T08:51:50.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:50.799 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-21T08:51:50.799 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-21T08:51:50.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.5 2024-04-21T08:51:51.122 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:50 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:50.989+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:51.122 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:50 smithi173 systemd[1]: Stopping Ceph osd.5 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:51.122 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:50 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:50.858+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:51.122 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:50 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:50.858+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:23.311059+0000 front 2024-04-21T08:51:23.311157+0000 (oldest deadline 2024-04-21T08:51:47.911658+0000) 2024-04-21T08:51:51.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:50 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:50.834+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:51.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:50 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:50.787+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:51.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:50 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:50.787+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:51.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:50 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:50.817+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:51.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:50 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:50.817+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:51.439 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:51.122+0000 7f818ee55700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:51.439 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:51.122+0000 7f818ee55700 -1 osd.5 106 *** Got signal Terminated *** 2024-04-21T08:51:51.439 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:51.122+0000 7f818ee55700 -1 osd.5 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:51:51.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:51 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:51.330+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:51.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:51.439+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:52.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:51 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:51.858+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:52.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:51 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:51.806+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:52.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:51 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:51.806+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:52.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:51 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:51.816+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:52.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:51 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:51.816+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:52.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:51.884+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:52.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:51.884+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:23.311059+0000 front 2024-04-21T08:51:23.311157+0000 (oldest deadline 2024-04-21T08:51:47.911658+0000) 2024-04-21T08:51:52.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:51 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:51.973+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:52.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:52 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:52.347+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:52.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:52 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:52.461+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:53.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:52 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:52.901+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:53.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:52 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:52.769+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:53.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:52 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:52.769+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:53.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:52 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:52.826+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:53.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:52 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:52.826+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:53.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:53 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:53.011+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:53.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:52 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:52.915+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:53.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:52 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:52.915+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:23.311059+0000 front 2024-04-21T08:51:23.311157+0000 (oldest deadline 2024-04-21T08:51:47.911658+0000) 2024-04-21T08:51:53.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:53.387+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:53.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:53 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:53.457+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:54.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:53 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:53.855+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:54.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:53 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:53.783+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:54.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:53 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:53.783+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:54.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:53 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:53.783+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:51:54.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:53.802+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:54.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:53.802+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:54.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:53.802+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:51:54.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:53 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:53.962+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:54.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:53 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:53.879+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:54.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:53 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:53.879+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:23.311059+0000 front 2024-04-21T08:51:23.311157+0000 (oldest deadline 2024-04-21T08:51:47.911658+0000) 2024-04-21T08:51:54.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:54 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:54.393+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:54.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:54 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:54.393+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.800757+0000 front 2024-04-21T08:51:27.800876+0000 (oldest deadline 2024-04-21T08:51:53.700841+0000) 2024-04-21T08:51:54.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:54 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:54.502+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:54.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:54 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:54.502+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:51:55.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:54 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:54.866+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:55.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:54 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:54.866+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:51:55.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:54 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:54.746+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:55.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:54 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:54.746+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:55.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:54 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:54.746+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:51:55.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:54 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:54.792+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:55.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:54 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:54.792+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:55.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:54 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:54.792+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:51:55.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:54 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:54.885+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:55.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:54 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:54.885+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:23.311059+0000 front 2024-04-21T08:51:23.311157+0000 (oldest deadline 2024-04-21T08:51:47.911658+0000) 2024-04-21T08:51:55.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:54 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:54.972+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:55.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:54 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:54.972+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:51:55.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:55 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:55.345+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:55.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:55 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:55.345+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.800757+0000 front 2024-04-21T08:51:27.800876+0000 (oldest deadline 2024-04-21T08:51:53.700841+0000) 2024-04-21T08:51:55.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:55.483+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:55.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:55.483+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:51:56.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:55 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:55.855+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:56.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:55 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:55.855+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:51:56.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:55 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:55.707+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:56.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:55 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:55.707+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:56.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:55 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:55.707+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:51:56.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:55 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:55.769+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:56.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:55 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:55.769+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:56.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:55 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:55.769+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:51:56.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:55.929+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:15.710057+0000 front 2024-04-21T08:51:15.709917+0000 (oldest deadline 2024-04-21T08:51:39.810078+0000) 2024-04-21T08:51:56.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5[43562]: 2024-04-21T08:51:55.929+0000 7f8187441700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:23.311059+0000 front 2024-04-21T08:51:23.311157+0000 (oldest deadline 2024-04-21T08:51:47.911658+0000) 2024-04-21T08:51:56.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:55.940+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:56.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:55 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:55.940+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:51:56.740 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:56.310+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:56.740 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:56.310+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.800757+0000 front 2024-04-21T08:51:27.800876+0000 (oldest deadline 2024-04-21T08:51:53.700841+0000) 2024-04-21T08:51:56.881 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:56 smithi173 podman[68838]: 2024-04-21 08:51:56.564856727 +0000 UTC m=+5.575552897 container died 5bea7cbcf3f49d3fa892b40c72b5d790427456e43f299b2925a53025f6ef25a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:51:56.881 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:56 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:56.529+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:56.881 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:56 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:56.529+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:51:57.010 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:56.739+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:57.010 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:56.739+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:57.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:56 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:56.739+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:51:57.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:56.839+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:57.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:56.839+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:51:57.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:56.722+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:57.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:56.722+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:57.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:56 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:56.722+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:51:57.208 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:56 smithi173 podman[68838]: 2024-04-21 08:51:56.97320202 +0000 UTC m=+5.983898192 container cleanup 5bea7cbcf3f49d3fa892b40c72b5d790427456e43f299b2925a53025f6ef25a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T08:51:57.208 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:56 smithi173 bash[68838]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5 2024-04-21T08:51:57.208 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:56 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:56.971+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:57.208 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:56 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:56.971+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:51:57.511 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:57 smithi173 podman[68855]: 2024-04-21 08:51:57.20844536 +0000 UTC m=+0.637924487 container remove 5bea7cbcf3f49d3fa892b40c72b5d790427456e43f299b2925a53025f6ef25a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-21T08:51:57.747 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:57.326+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:57.747 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:57.326+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.800757+0000 front 2024-04-21T08:51:27.800876+0000 (oldest deadline 2024-04-21T08:51:53.700841+0000) 2024-04-21T08:51:57.803 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:57 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:57.528+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:57.803 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:57 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:57.528+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:51:57.804 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:57 smithi173 podman[68946]: 2024-04-21 08:51:57.443474326 +0000 UTC m=+0.018142977 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:51:57.804 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:57 smithi173 podman[68946]: 2024-04-21 08:51:57.568109985 +0000 UTC m=+0.142778676 container create 4d874c469c336d09e3e39bf427ea3b995261eefe647bd3ed4cb679354e8e4dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-21T08:51:58.010 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:57.747+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:58.010 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:57.747+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:58.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:57.747+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:51:58.094 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:57 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:57.990+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:58.095 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:57 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:57.990+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:51:58.095 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:57 smithi173 podman[68946]: 2024-04-21 08:51:57.804879558 +0000 UTC m=+0.379548204 container init 4d874c469c336d09e3e39bf427ea3b995261eefe647bd3ed4cb679354e8e4dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-21T08:51:58.095 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:57 smithi173 podman[68946]: 2024-04-21 08:51:57.807957356 +0000 UTC m=+0.382626010 container start 4d874c469c336d09e3e39bf427ea3b995261eefe647bd3ed4cb679354e8e4dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-04-21T08:51:58.095 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:57 smithi173 podman[68946]: 2024-04-21 08:51:57.912506988 +0000 UTC m=+0.487175641 container attach 4d874c469c336d09e3e39bf427ea3b995261eefe647bd3ed4cb679354e8e4dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-21T08:51:58.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:57 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:57.838+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:58.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:57 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:57.838+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:51:58.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:57 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:57.767+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:58.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:57 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:57.767+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:58.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:57 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:57.767+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:51:58.511 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 21 08:51:58 smithi173 podman[68946]: 2024-04-21 08:51:58.095666716 +0000 UTC m=+0.670335364 container died 4d874c469c336d09e3e39bf427ea3b995261eefe647bd3ed4cb679354e8e4dbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T08:51:58.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:58.361+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:58.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:58.361+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.800757+0000 front 2024-04-21T08:51:27.800876+0000 (oldest deadline 2024-04-21T08:51:53.700841+0000) 2024-04-21T08:51:58.876 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:58.539+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:58.876 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:58.539+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:51:58.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.5.service' 2024-04-21T08:51:58.949 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:51:58.949 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-21T08:51:58.949 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-21T08:51:58.950 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.6 2024-04-21T08:51:59.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:58.839+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:59.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:58.839+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:51:59.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:58.775+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:59.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:58.775+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:59.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:58 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:58.775+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:51:59.258 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:58 smithi094 systemd[1]: Stopping Ceph osd.6 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:51:59.259 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:58.764+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:51:59.259 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:58.764+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:51:59.259 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:58 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:58.764+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:51:59.260 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:58.967+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:51:59.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:58 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:58.967+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:51:59.510 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:59.258+0000 7f1bd548a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:51:59.511 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:59.258+0000 7f1bd548a700 -1 osd.6 106 *** Got signal Terminated *** 2024-04-21T08:51:59.511 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:59.258+0000 7f1bd548a700 -1 osd.6 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:51:59.511 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:59.411+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:51:59.511 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:51:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:51:59.411+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.800757+0000 front 2024-04-21T08:51:27.800876+0000 (oldest deadline 2024-04-21T08:51:53.700841+0000) 2024-04-21T08:51:59.935 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:59 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:59.544+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:51:59.936 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:51:59 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:51:59.544+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:51:59.937 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:59 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:59.819+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:51:59.937 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:51:59 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:51:59.819+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:51:59.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:59 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:59.809+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:51:59.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:59 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:59.809+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:51:59.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:51:59 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:51:59.809+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:00.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:59.772+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:00.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:59.772+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:00.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:51:59 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:51:59.772+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:00.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:59 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:59.936+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:00.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:51:59 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:51:59.936+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:00.745 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:00.408+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:52:00.745 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:00.408+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.800757+0000 front 2024-04-21T08:51:27.800876+0000 (oldest deadline 2024-04-21T08:51:53.700841+0000) 2024-04-21T08:52:00.943 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:00 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:00.549+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:00.944 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:00 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:00.549+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:01.010 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:00.745+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:01.010 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:00.745+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:01.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:00 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:00.745+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:01.191 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:00 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:00.855+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:01.191 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:00 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:00.855+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:01.191 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:00 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:00.778+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:01.191 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:00 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:00.778+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:01.191 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:00 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:00.778+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:01.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:00 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:00.944+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:01.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:00 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:00.944+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:01.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:00 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:00.944+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:01.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:01 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:01.430+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:52:01.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:01 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:01.430+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.800757+0000 front 2024-04-21T08:51:27.800876+0000 (oldest deadline 2024-04-21T08:51:53.700841+0000) 2024-04-21T08:52:01.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:01 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:01.430+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.401787+0000 front 2024-04-21T08:51:35.401668+0000 (oldest deadline 2024-04-21T08:52:01.301996+0000) 2024-04-21T08:52:01.902 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:01 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:01.511+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:01.902 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:01 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:01.511+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:02.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:01 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:01.874+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:02.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:01 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:01.874+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:02.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:01 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:01.745+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:02.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:01 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:01.745+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:02.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:01 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:01.745+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:02.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:01 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:01.774+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:02.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:01 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:01.774+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:02.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:01 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:01.774+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:02.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:01 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:01.902+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:02.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:01 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:01.902+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:02.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:01 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:01.902+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:02.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:02.392+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:52:02.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:02.392+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.800757+0000 front 2024-04-21T08:51:27.800876+0000 (oldest deadline 2024-04-21T08:51:53.700841+0000) 2024-04-21T08:52:02.760 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:02.392+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.401787+0000 front 2024-04-21T08:51:35.401668+0000 (oldest deadline 2024-04-21T08:52:01.301996+0000) 2024-04-21T08:52:02.872 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:02 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:02.558+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:02.872 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:02 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:02.558+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:02.872 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:02 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:02.558+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:03.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:02 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:02.865+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:03.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:02 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:02.865+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:03.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:02 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:02.865+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.109499+0000 front 2024-04-21T08:51:38.109413+0000 (oldest deadline 2024-04-21T08:52:02.809418+0000) 2024-04-21T08:52:03.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:02 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:02.733+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:03.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:02 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:02.733+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:03.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:02 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:02.733+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:03.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:02.761+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:03.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:02.761+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:03.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:02.761+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:03.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:02 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:02.761+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:03.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:02 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:02.872+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:03.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:02 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:02.872+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:03.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:02 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:02.872+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:03.750 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:03.383+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.599815+0000 front 2024-04-21T08:51:22.599730+0000 (oldest deadline 2024-04-21T08:51:47.299760+0000) 2024-04-21T08:52:03.750 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:03.383+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.800757+0000 front 2024-04-21T08:51:27.800876+0000 (oldest deadline 2024-04-21T08:51:53.700841+0000) 2024-04-21T08:52:03.750 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6[56190]: 2024-04-21T08:52:03.383+0000 7f1bcda76700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.401787+0000 front 2024-04-21T08:51:35.401668+0000 (oldest deadline 2024-04-21T08:52:01.301996+0000) 2024-04-21T08:52:03.878 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:03 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:03.544+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:03.878 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:03 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:03.544+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:03.878 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:03 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:03.544+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:04.010 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:03.750+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:04.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:03.750+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:04.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:03.750+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:04.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:03 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:03.750+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:04.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:03 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:03.856+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:04.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:03 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:03.857+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:04.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:03 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:03.857+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.109499+0000 front 2024-04-21T08:51:38.109413+0000 (oldest deadline 2024-04-21T08:52:02.809418+0000) 2024-04-21T08:52:04.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:03 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:03.773+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:04.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:03 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:03.773+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:04.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:03 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:03.773+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:04.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:03 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:03.878+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:04.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:03 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:03.878+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:04.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:03 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:03.878+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:04.896 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:04 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:04.543+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:04.896 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:04 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:04.543+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:04.896 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:04 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:04.543+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:05.093 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:04 smithi094 podman[98948]: 2024-04-21 08:52:04.808833851 +0000 UTC m=+5.667348350 container died ed77bb2cbccbd8db43d7ff733f5b0f83ec8fb47a72192190f1f5aa2f393fabd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:52:05.093 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:04 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:04.784+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:05.093 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:04 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:04.784+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:05.093 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:04 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:04.784+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:05.093 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:04 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:04.784+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:05.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:04 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:04.827+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:05.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:04 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:04.827+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:05.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:04 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:04.827+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.109499+0000 front 2024-04-21T08:51:38.109413+0000 (oldest deadline 2024-04-21T08:52:02.809418+0000) 2024-04-21T08:52:05.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:04 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:04.800+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:05.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:04 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:04.800+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:05.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:04 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:04.800+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:05.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:04 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:04.800+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:05.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:04 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:04.896+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:05.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:04 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:04.896+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:05.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:04 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:04.896+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:05.434 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:05 smithi094 podman[98948]: 2024-04-21 08:52:05.192674138 +0000 UTC m=+6.051188647 container cleanup ed77bb2cbccbd8db43d7ff733f5b0f83ec8fb47a72192190f1f5aa2f393fabd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:52:05.434 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:05 smithi094 bash[98948]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6 2024-04-21T08:52:05.753 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:05 smithi094 podman[98965]: 2024-04-21 08:52:05.435536317 +0000 UTC m=+0.625638329 container remove ed77bb2cbccbd8db43d7ff733f5b0f83ec8fb47a72192190f1f5aa2f393fabd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-21T08:52:05.863 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:05 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:05.533+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:05.863 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:05 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:05.533+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:05.863 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:05 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:05.533+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:06.010 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:05 smithi094 podman[99056]: 2024-04-21 08:52:05.657796112 +0000 UTC m=+0.018254880 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:52:06.010 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:05 smithi094 podman[99056]: 2024-04-21 08:52:05.779874163 +0000 UTC m=+0.140332920 container create c3f10799f98a1062e6f442fea168ce5d7c274342cb137b6e33f1c22ba206621c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-04-21T08:52:06.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:05 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:05.818+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:06.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:05 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:05.818+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:06.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:05 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:05.818+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:06.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:05 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:05.818+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:06.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:05 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:05.847+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:06.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:05 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:05.847+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:06.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:05 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:05.847+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.109499+0000 front 2024-04-21T08:51:38.109413+0000 (oldest deadline 2024-04-21T08:52:02.809418+0000) 2024-04-21T08:52:06.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:05 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:05.783+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:06.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:05 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:05.783+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:06.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:05 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:05.783+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:06.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:05 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:05.783+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:06.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:05 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:05.863+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:06.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:05 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:05.863+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:06.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:05 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:05.863+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:06.339 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:06 smithi094 podman[99056]: 2024-04-21 08:52:06.047161683 +0000 UTC m=+0.407620438 container init c3f10799f98a1062e6f442fea168ce5d7c274342cb137b6e33f1c22ba206621c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:52:06.339 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:06 smithi094 podman[99056]: 2024-04-21 08:52:06.050373614 +0000 UTC m=+0.410832375 container start c3f10799f98a1062e6f442fea168ce5d7c274342cb137b6e33f1c22ba206621c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-21T08:52:06.339 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:06 smithi094 podman[99056]: 2024-04-21 08:52:06.147111404 +0000 UTC m=+0.507570172 container attach c3f10799f98a1062e6f442fea168ce5d7c274342cb137b6e33f1c22ba206621c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0) 2024-04-21T08:52:06.649 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 21 08:52:06 smithi094 podman[99056]: 2024-04-21 08:52:06.339568855 +0000 UTC m=+0.700027604 container died c3f10799f98a1062e6f442fea168ce5d7c274342cb137b6e33f1c22ba206621c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-21T08:52:06.760 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:06 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:06.503+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:06.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:06 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:06.503+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:06.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:06 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:06.503+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:06.966 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:06 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:06.848+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:06.966 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:06 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:06.848+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:06.966 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:06 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:06.848+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:06.966 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:06 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:06.848+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:06.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.6.service' 2024-04-21T08:52:07.025 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:52:07.026 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-21T08:52:07.026 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-21T08:52:07.026 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.7 2024-04-21T08:52:07.051 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:06 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:06.830+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:07.051 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:06 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:06.830+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:07.051 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:06 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:06.830+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.109499+0000 front 2024-04-21T08:51:38.109413+0000 (oldest deadline 2024-04-21T08:52:02.809418+0000) 2024-04-21T08:52:07.052 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:06 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:06.763+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:07.052 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:06 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:06.763+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:07.052 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:06 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:06.763+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:07.052 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:06 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:06.763+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:07.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:06 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:06.825+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:07.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:06 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:06.825+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:07.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:06 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:06.825+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:07.337 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:07 smithi111 systemd[1]: Stopping Ceph osd.7 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:52:07.688 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:07.338+0000 7fc42d83a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:52:07.688 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:07.338+0000 7fc42d83a700 -1 osd.7 106 *** Got signal Terminated *** 2024-04-21T08:52:07.688 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:07.338+0000 7fc42d83a700 -1 osd.7 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:52:07.837 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:07 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:07.513+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:07.838 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:07 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:07.513+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:07.838 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:07 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:07.513+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:08.181 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:07 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:07.846+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:08.181 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:07 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:07.846+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:08.181 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:07 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:07.846+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:08.181 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:07 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:07.846+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:08.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:07.879+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:08.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:07.879+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:08.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:07.879+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.109499+0000 front 2024-04-21T08:51:38.109413+0000 (oldest deadline 2024-04-21T08:52:02.809418+0000) 2024-04-21T08:52:08.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:07.760+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:08.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:07.760+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:08.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:07.760+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:08.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:07.760+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:08.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:07 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:07.838+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:08.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:07 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:07.838+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:08.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:07 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:07.838+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:08.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:08 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:08.471+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:08.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:08 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:08.471+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:08.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:08 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:08.471+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:09.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:08 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:08.846+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:09.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:08 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:08.846+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:09.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:08 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:08.846+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.109499+0000 front 2024-04-21T08:51:38.109413+0000 (oldest deadline 2024-04-21T08:52:02.809418+0000) 2024-04-21T08:52:09.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:08 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:08.756+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:09.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:08 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:08.756+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:09.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:08 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:08.756+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:09.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:08 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:08.756+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:09.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:08.865+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:09.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:08.865+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:09.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:08.865+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:09.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:08 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:08.865+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:09.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:08 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:08.887+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:09.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:08 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:08.887+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:09.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:08 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:08.887+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:09.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:09 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:09.480+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:09.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:09 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:09.480+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:09.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:09 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:09.480+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:09.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:09 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:09.480+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:10.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:09 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:09.841+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:10.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:09 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:09.841+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:10.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:09 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:09.841+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.109499+0000 front 2024-04-21T08:51:38.109413+0000 (oldest deadline 2024-04-21T08:52:02.809418+0000) 2024-04-21T08:52:10.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:09 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:09.773+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:10.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:09 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:09.773+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:10.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:09 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:09.773+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:10.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:09 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:09.773+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:10.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:09 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:09.822+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:10.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:09 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:09.822+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:10.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:09 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:09.822+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:10.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:09 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:09.822+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:10.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:09 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:09.873+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:10.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:09 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:09.873+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:10.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:09 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:09.873+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:10.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:10 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:10.496+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:10.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:10 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:10.496+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:10.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:10 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:10.496+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:10.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:10 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:10.496+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:11.187 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:10 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:10.818+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:11.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:10 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:10.818+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:11.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:10 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:10.818+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.109499+0000 front 2024-04-21T08:51:38.109413+0000 (oldest deadline 2024-04-21T08:52:02.809418+0000) 2024-04-21T08:52:11.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:10 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:10.808+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:11.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:10 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:10.808+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:11.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:10 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:10.808+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:11.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:10 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:10.808+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:11.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:10 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:10.845+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:11.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:10 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:10.845+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:11.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:10 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:10.845+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:11.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:10 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:10.845+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:11.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:10 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:10.873+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:11.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:10 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:10.873+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:11.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:10 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:10.873+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:11.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:11 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:11.507+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:11.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:11 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:11.507+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:11.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:11 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:11.507+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:11.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:11 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:11.507+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:12.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:11 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:11.819+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.306123+0000 front 2024-04-21T08:51:21.306149+0000 (oldest deadline 2024-04-21T08:51:45.406269+0000) 2024-04-21T08:52:12.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:11 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:11.819+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.707330+0000 front 2024-04-21T08:51:30.707415+0000 (oldest deadline 2024-04-21T08:51:54.107918+0000) 2024-04-21T08:52:12.188 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:11 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7[50044]: 2024-04-21T08:52:11.819+0000 7fc429e2e700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.109499+0000 front 2024-04-21T08:51:38.109413+0000 (oldest deadline 2024-04-21T08:52:02.809418+0000) 2024-04-21T08:52:12.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:11 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:11.805+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:12.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:11 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:11.805+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:12.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:11 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:11.805+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:12.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:11 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:11.805+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:12.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:11 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:11.805+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:12.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:11 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:11.820+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:12.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:11 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:11.820+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:12.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:11 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:11.820+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:12.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:11 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:11.820+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:12.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:11 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:11.903+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:12.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:11 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:11.903+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:12.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:11 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:11.903+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:12.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:11 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:11.903+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:45.931806+0000 front 2024-04-21T08:51:45.931746+0000 (oldest deadline 2024-04-21T08:52:11.831484+0000) 2024-04-21T08:52:12.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:12 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:12.482+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:12.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:12 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:12.482+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:12.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:12 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:12.482+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:12.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:12 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:12.482+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:12.937 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:12 smithi111 podman[68100]: 2024-04-21 08:52:12.681466247 +0000 UTC m=+5.449669778 container died 5a7782f6eb6fe7f13eec5ec05b8428455eca5230f89eec097973e4c1297cfbee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-21T08:52:12.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:12 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:12.790+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:12.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:12 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:12.790+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:12.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:12 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:12.790+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:12.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:12 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:12.790+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:12.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:12 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:12.790+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:13.250 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:13 smithi111 podman[68100]: 2024-04-21 08:52:13.029917947 +0000 UTC m=+5.798121477 container cleanup 5a7782f6eb6fe7f13eec5ec05b8428455eca5230f89eec097973e4c1297cfbee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:52:13.250 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:13 smithi111 bash[68100]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7 2024-04-21T08:52:13.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:12 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:12.834+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:13.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:12 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:12.834+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:13.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:12 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:12.834+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:13.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:12 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:12.834+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:13.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:12 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:12.834+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:13.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:12 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:12.913+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:13.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:12 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:12.913+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:13.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:12 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:12.913+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:13.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:12 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:12.913+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:45.931806+0000 front 2024-04-21T08:51:45.931746+0000 (oldest deadline 2024-04-21T08:52:11.831484+0000) 2024-04-21T08:52:13.631 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:13 smithi111 podman[68117]: 2024-04-21 08:52:13.251353498 +0000 UTC m=+0.563761557 container remove 5a7782f6eb6fe7f13eec5ec05b8428455eca5230f89eec097973e4c1297cfbee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-21T08:52:13.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:13 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:13.466+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:13.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:13 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:13.466+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:13.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:13 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:13.466+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:13.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:13 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:13.466+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:13.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:13 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:13.754+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:13.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:13 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:13.754+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:13.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:13 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:13.754+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:13.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:13 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:13.754+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:13.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:13 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:13.754+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:13.939 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:13 smithi111 podman[68207]: 2024-04-21 08:52:13.632111204 +0000 UTC m=+0.097124396 container create fc5f994a4342be4c742b0e9b562c212bde9bcc15906ec903626eb648a3120cfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-21T08:52:13.939 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:13 smithi111 podman[68207]: 2024-04-21 08:52:13.552840078 +0000 UTC m=+0.017853251 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:52:13.939 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:13 smithi111 podman[68207]: 2024-04-21 08:52:13.876482279 +0000 UTC m=+0.341495429 container init fc5f994a4342be4c742b0e9b562c212bde9bcc15906ec903626eb648a3120cfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD) 2024-04-21T08:52:13.939 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:13 smithi111 podman[68207]: 2024-04-21 08:52:13.879548589 +0000 UTC m=+0.344561745 container start fc5f994a4342be4c742b0e9b562c212bde9bcc15906ec903626eb648a3120cfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:52:14.196 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:13 smithi111 podman[68207]: 2024-04-21 08:52:13.959637781 +0000 UTC m=+0.424650949 container attach fc5f994a4342be4c742b0e9b562c212bde9bcc15906ec903626eb648a3120cfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-21T08:52:14.196 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 08:52:14 smithi111 podman[68207]: 2024-04-21 08:52:14.171829793 +0000 UTC m=+0.636842956 container died fc5f994a4342be4c742b0e9b562c212bde9bcc15906ec903626eb648a3120cfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-21T08:52:14.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:13 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:13.867+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:14.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:13 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:13.867+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:14.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:13 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:13.867+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:14.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:13 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:13.867+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:14.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:13 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:13.867+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:14.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:13 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:13.898+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:14.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:13 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:13.898+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:14.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:13 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:13.898+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:14.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:13 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:13.898+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:45.931806+0000 front 2024-04-21T08:51:45.931746+0000 (oldest deadline 2024-04-21T08:52:11.831484+0000) 2024-04-21T08:52:14.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:14 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:14.467+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:14.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:14 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:14.467+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:14.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:14 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:14.467+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:14.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:14 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:14.467+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:14.805 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:14 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:14.765+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:14.805 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:14 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:14.765+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:14.805 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:14 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:14.765+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:14.805 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:14 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:14.765+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:14.805 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:14 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:14.765+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:14.825 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.7.service' 2024-04-21T08:52:14.902 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:52:14.906 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-21T08:52:14.906 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-21T08:52:14.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.8 2024-04-21T08:52:15.213 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:14 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:14.919+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:15.213 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:14 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:14.919+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:15.213 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:14 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:14.919+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:15.213 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:14 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:14.919+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:45.931806+0000 front 2024-04-21T08:51:45.931746+0000 (oldest deadline 2024-04-21T08:52:11.831484+0000) 2024-04-21T08:52:15.213 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:14 smithi173 systemd[1]: Stopping Ceph osd.8 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:52:15.213 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:15.212+0000 7fd08f1b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:52:15.213 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:15.212+0000 7fd08f1b8700 -1 osd.8 106 *** Got signal Terminated *** 2024-04-21T08:52:15.213 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:15.212+0000 7fd08f1b8700 -1 osd.8 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:52:15.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:14 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:14.817+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:15.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:14 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:14.817+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:15.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:14 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:14.817+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:15.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:14 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:14.817+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:15.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:14 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:14.817+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:15.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:15.483+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:15.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:15.483+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:15.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:15.483+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:15.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:15.483+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:16.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:15 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:15.719+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:16.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:15 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:15.719+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:16.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:15 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:15.719+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:16.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:15 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:15.719+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:16.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:15 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:15.719+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:16.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:15 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:15.818+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:16.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:15 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:15.818+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:16.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:15 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:15.818+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:16.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:15 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:15.818+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:16.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:15 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:15.818+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:16.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:15.930+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:16.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:15.930+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:16.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:15.930+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:16.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:15 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:15.930+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:45.931806+0000 front 2024-04-21T08:51:45.931746+0000 (oldest deadline 2024-04-21T08:52:11.831484+0000) 2024-04-21T08:52:16.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:16 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:16.449+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:16.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:16 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:16.449+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:16.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:16 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:16.449+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:16.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:16 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:16.449+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:16.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:16 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:16.671+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:16.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:16 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:16.671+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:16.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:16 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:16.671+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:16.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:16 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:16.671+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:16.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:16 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:16.671+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:16.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:16 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:16.671+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:17.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:16 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:16.819+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:17.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:16 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:16.819+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:17.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:16 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:16.819+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:17.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:16 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:16.819+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:17.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:16 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:16.819+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:17.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:16 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:16.970+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:17.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:16 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:16.970+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:17.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:16 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:16.970+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:17.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:16 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:16.970+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:45.931806+0000 front 2024-04-21T08:51:45.931746+0000 (oldest deadline 2024-04-21T08:52:11.831484+0000) 2024-04-21T08:52:17.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:17 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:17.431+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:17.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:17 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:17.431+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:17.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:17 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:17.431+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:17.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:17 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:17.431+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:17.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:17 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:17.431+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:17.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:17 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:17.681+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:17.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:17 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:17.681+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:17.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:17 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:17.681+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:17.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:17 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:17.681+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:17.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:17 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:17.681+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:17.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:17 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:17.681+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:18.010 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:17 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:17.851+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:18.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:17 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:17.851+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:18.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:17 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:17.851+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:18.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:17 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:17.851+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:18.011 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:17 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:17.851+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:18.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:17 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:17.963+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:18.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:17 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:17.963+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:18.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:17 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:17.963+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:18.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:17 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:17.963+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:45.931806+0000 front 2024-04-21T08:51:45.931746+0000 (oldest deadline 2024-04-21T08:52:11.831484+0000) 2024-04-21T08:52:18.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:18 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:18.409+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:18.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:18 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:18.409+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:18.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:18 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:18.409+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:18.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:18 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:18.409+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:18.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:18 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:18.409+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:19.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:18 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:18.704+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:19.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:18 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:18.704+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:19.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:18 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:18.704+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:19.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:18 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:18.704+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:19.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:18 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:18.704+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:19.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:18 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:18.705+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:19.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:18 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:18.836+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:19.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:18 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:18.836+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:19.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:18 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:18.836+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:19.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:18 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:18.836+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:19.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:18 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:18.836+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:19.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:19.003+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:19.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:19.003+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:19.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:19.003+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:19.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:19.003+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:45.931806+0000 front 2024-04-21T08:51:45.931746+0000 (oldest deadline 2024-04-21T08:52:11.831484+0000) 2024-04-21T08:52:19.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:19.003+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:54.732300+0000 front 2024-04-21T08:51:54.732361+0000 (oldest deadline 2024-04-21T08:52:18.832534+0000) 2024-04-21T08:52:19.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:19.381+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:19.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:19.381+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:19.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:19.381+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:19.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:19.381+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:19.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:19.381+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:20.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:19 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:19.691+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:20.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:19 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:19.691+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:20.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:19 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:19.691+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:20.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:19 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:19.691+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:20.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:19 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:19.691+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:20.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:19 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:19.691+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:20.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:19 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:19.830+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:20.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:19 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:19.830+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:20.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:19 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:19.830+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:20.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:19 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:19.830+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:20.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:19 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:19.830+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:20.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:19 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:19.830+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.370460+0000 front 2024-04-21T08:51:55.370295+0000 (oldest deadline 2024-04-21T08:52:18.870131+0000) 2024-04-21T08:52:20.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:19.988+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.527568+0000 front 2024-04-21T08:51:21.527588+0000 (oldest deadline 2024-04-21T08:51:47.427558+0000) 2024-04-21T08:52:20.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:19.988+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.928413+0000 front 2024-04-21T08:51:30.928394+0000 (oldest deadline 2024-04-21T08:51:54.428630+0000) 2024-04-21T08:52:20.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:19.988+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.529713+0000 front 2024-04-21T08:51:38.529695+0000 (oldest deadline 2024-04-21T08:52:00.829766+0000) 2024-04-21T08:52:20.261 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:19.988+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:45.931806+0000 front 2024-04-21T08:51:45.931746+0000 (oldest deadline 2024-04-21T08:52:11.831484+0000) 2024-04-21T08:52:20.262 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8[50096]: 2024-04-21T08:52:19.988+0000 7fd08b7ac700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:54.732300+0000 front 2024-04-21T08:51:54.732361+0000 (oldest deadline 2024-04-21T08:52:18.832534+0000) 2024-04-21T08:52:20.761 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:20 smithi173 podman[69052]: 2024-04-21 08:52:20.554982124 +0000 UTC m=+5.458447742 container died a941438091063d8bd18f041631a0d34bf0a46e06775d37f7ffb675f327ddc89f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:52:20.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:20 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:20.388+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:20.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:20 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:20.388+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:20.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:20 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:20.388+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:20.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:20 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:20.388+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:20.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:20 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:20.388+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:21.113 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:20 smithi173 podman[69052]: 2024-04-21 08:52:20.89621938 +0000 UTC m=+5.799685004 container cleanup a941438091063d8bd18f041631a0d34bf0a46e06775d37f7ffb675f327ddc89f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:52:21.113 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:20 smithi173 bash[69052]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8 2024-04-21T08:52:21.187 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:20 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:20.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:21.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:20 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:20.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:21.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:20 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:20.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:21.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:20 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:20.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:21.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:20 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:20.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:21.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:20 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:20.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:21.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:20 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:20.833+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:21.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:20 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:20.833+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:21.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:20 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:20.833+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:21.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:20 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:20.833+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:21.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:20 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:20.833+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:21.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:20 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:20.833+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.370460+0000 front 2024-04-21T08:51:55.370295+0000 (oldest deadline 2024-04-21T08:52:18.870131+0000) 2024-04-21T08:52:21.409 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:21 smithi173 podman[69070]: 2024-04-21 08:52:21.114463181 +0000 UTC m=+0.554524285 container remove a941438091063d8bd18f041631a0d34bf0a46e06775d37f7ffb675f327ddc89f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-21T08:52:21.761 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:21 smithi173 podman[69162]: 2024-04-21 08:52:21.42462268 +0000 UTC m=+0.018489000 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:52:21.761 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:21 smithi173 podman[69162]: 2024-04-21 08:52:21.532045056 +0000 UTC m=+0.125911352 container create 1dfaaa5cd8cbc67bcca8be399096bbab4cf386dd92289d2e5bbfa32633369985 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T08:52:21.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:21 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:21.409+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:21.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:21 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:21.409+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:21.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:21 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:21.409+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:21.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:21 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:21.409+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:21.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:21 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:21.409+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:21.939 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:21 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:21.686+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:21.939 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:21 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:21.686+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:21.939 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:21 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:21.686+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:21.939 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:21 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:21.686+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:21.939 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:21 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:21.686+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:21.939 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:21 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:21.686+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:22.083 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:21 smithi173 podman[69162]: 2024-04-21 08:52:21.789216475 +0000 UTC m=+0.383082788 container init 1dfaaa5cd8cbc67bcca8be399096bbab4cf386dd92289d2e5bbfa32633369985 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-21T08:52:22.083 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:21 smithi173 podman[69162]: 2024-04-21 08:52:21.792489665 +0000 UTC m=+0.386355964 container start 1dfaaa5cd8cbc67bcca8be399096bbab4cf386dd92289d2e5bbfa32633369985 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T08:52:22.084 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:21 smithi173 podman[69162]: 2024-04-21 08:52:21.851048327 +0000 UTC m=+0.444914651 container attach 1dfaaa5cd8cbc67bcca8be399096bbab4cf386dd92289d2e5bbfa32633369985 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-04-21T08:52:22.084 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 21 08:52:22 smithi173 podman[69162]: 2024-04-21 08:52:22.084296802 +0000 UTC m=+0.678163119 container died 1dfaaa5cd8cbc67bcca8be399096bbab4cf386dd92289d2e5bbfa32633369985 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-21T08:52:22.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:21 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:21.872+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:22.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:21 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:21.872+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:22.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:21 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:21.872+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:22.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:21 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:21.872+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:22.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:21 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:21.872+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:22.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:21 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:21.872+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.370460+0000 front 2024-04-21T08:51:55.370295+0000 (oldest deadline 2024-04-21T08:52:18.870131+0000) 2024-04-21T08:52:22.629 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:22 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:22.424+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:22.629 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:22 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:22.424+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:22.629 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:22 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:22.424+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:22.629 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:22 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:22.424+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:22.629 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:22 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:22.424+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:22.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.8.service' 2024-04-21T08:52:22.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:52:22.688 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-21T08:52:22.689 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-21T08:52:22.689 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.9 2024-04-21T08:52:22.994 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:22 smithi094 systemd[1]: Stopping Ceph osd.9 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:52:22.995 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:22.888+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:22.995 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:22.888+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:22.995 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:22.888+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:22.995 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:22.888+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:22.995 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:22.888+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:22.995 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:22.888+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.370460+0000 front 2024-04-21T08:51:55.370295+0000 (oldest deadline 2024-04-21T08:52:18.870131+0000) 2024-04-21T08:52:22.995 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:22.994+0000 7f8101626700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:52:22.995 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:22.994+0000 7f8101626700 -1 osd.9 106 *** Got signal Terminated *** 2024-04-21T08:52:22.995 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:22 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:22.994+0000 7f8101626700 -1 osd.9 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:52:23.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:22 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:22.727+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:23.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:22 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:22.727+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:23.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:22 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:22.727+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:23.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:22 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:22.727+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:23.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:22 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:22.727+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:23.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:22 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:22.727+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:23.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:23 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:23.449+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:23.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:23 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:23.449+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:23.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:23 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:23.449+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:23.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:23 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:23.449+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:23.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:23 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:23.449+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:24.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:23 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:23.688+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:24.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:23 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:23.688+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:24.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:23 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:23.688+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:24.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:23 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:23.688+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:24.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:23 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:23.688+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:24.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:23 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:23.688+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:24.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:23 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:23.905+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:24.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:23 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:23.905+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:24.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:23 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:23.905+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:24.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:23 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:23.905+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:24.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:23 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:23.905+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:24.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:23 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:23.905+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.370460+0000 front 2024-04-21T08:51:55.370295+0000 (oldest deadline 2024-04-21T08:52:18.870131+0000) 2024-04-21T08:52:24.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:24 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:24.446+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:24.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:24 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:24.446+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:24.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:24 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:24.446+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:24.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:24 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:24.446+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:24.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:24 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:24.446+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:25.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:24 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:24.735+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:25.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:24 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:24.735+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:25.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:24 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:24.735+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:25.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:24 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:24.735+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:25.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:24 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:24.735+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:25.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:24 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:24.735+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:25.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:24 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:24.861+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:25.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:24 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:24.861+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:25.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:24 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:24.861+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:25.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:24 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:24.861+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:25.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:24 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:24.861+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:25.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:24 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:24.861+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.370460+0000 front 2024-04-21T08:51:55.370295+0000 (oldest deadline 2024-04-21T08:52:18.870131+0000) 2024-04-21T08:52:25.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:25 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:25.416+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:25.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:25 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:25.416+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:25.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:25 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:25.416+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:25.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:25 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:25.416+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:25.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:25 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:25.416+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:25.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:25 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:25.416+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:26.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:25 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:25.705+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:26.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:25 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:25.705+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:26.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:25 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:25.705+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:26.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:25 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:25.705+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:26.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:25 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:25.705+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:26.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:25 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:25.705+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:26.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:25.877+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:26.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:25.877+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:26.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:25.877+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:26.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:25.877+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:26.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:25.877+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:26.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:25 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:25.877+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.370460+0000 front 2024-04-21T08:51:55.370295+0000 (oldest deadline 2024-04-21T08:52:18.870131+0000) 2024-04-21T08:52:26.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:26 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:26.418+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:26.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:26 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:26.418+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:26.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:26 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:26.418+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:26.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:26 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:26.418+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:26.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:26 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:26.418+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:26.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:26 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:26.418+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:26.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:26 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:26.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:26.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:26 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:26.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:26.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:26 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:26.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:26.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:26 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:26.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:26.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:26 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:26.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:26.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:26 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:26.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:27.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:26 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:26.841+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:27.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:26 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:26.841+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:27.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:26 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:26.841+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:27.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:26 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:26.841+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:27.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:26 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:26.841+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:27.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:26 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:26.841+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.370460+0000 front 2024-04-21T08:51:55.370295+0000 (oldest deadline 2024-04-21T08:52:18.870131+0000) 2024-04-21T08:52:27.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:27 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:27.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:27.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:27 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:27.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:27.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:27 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:27.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:27.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:27 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:27.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:27.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:27 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:27.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:27.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:27 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:27.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:27.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:27 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:27.669+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:27.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:27 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:27.669+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:27.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:27 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:27.669+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:27.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:27 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:27.669+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:27.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:27 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:27.669+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:27.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:27 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:27.669+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:27.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:27 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:27.669+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:02.335442+0000 front 2024-04-21T08:52:02.335482+0000 (oldest deadline 2024-04-21T08:52:27.035239+0000) 2024-04-21T08:52:28.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:27 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:27.863+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:11.763596+0000 front 2024-04-21T08:51:11.763724+0000 (oldest deadline 2024-04-21T08:51:37.663741+0000) 2024-04-21T08:52:28.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:27 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:27.863+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:21.565630+0000 front 2024-04-21T08:51:21.565837+0000 (oldest deadline 2024-04-21T08:51:47.465594+0000) 2024-04-21T08:52:28.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:27 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:27.863+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:27.466048+0000 front 2024-04-21T08:51:27.465971+0000 (oldest deadline 2024-04-21T08:51:53.366072+0000) 2024-04-21T08:52:28.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:27 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:27.863+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:35.666804+0000 front 2024-04-21T08:51:35.666823+0000 (oldest deadline 2024-04-21T08:52:02.667369+0000) 2024-04-21T08:52:28.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:27 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:27.863+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.268836+0000 front 2024-04-21T08:51:47.269027+0000 (oldest deadline 2024-04-21T08:52:11.968648+0000) 2024-04-21T08:52:28.261 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:27 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9[63064]: 2024-04-21T08:52:27.863+0000 7f80f9c12700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.370460+0000 front 2024-04-21T08:51:55.370295+0000 (oldest deadline 2024-04-21T08:52:18.870131+0000) 2024-04-21T08:52:28.627 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:28 smithi094 podman[99173]: 2024-04-21 08:52:28.354505383 +0000 UTC m=+5.464572301 container died 11109b4ec88a4e9562d18d89050753644932bad8ad4e41dc25228600daa68a43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-21T08:52:28.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:28 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:28.458+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:28.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:28 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:28.458+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:28.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:28 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:28.458+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:28.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:28 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:28.458+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:28.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:28 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:28.458+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:28.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:28 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:28.458+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:28.927 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:28 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:28.642+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:28.927 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:28 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:28.642+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:28.927 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:28 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:28.642+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:28.927 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:28 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:28.642+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:28.928 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:28 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:28.642+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:28.928 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:28 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:28.642+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:28.928 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:28 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:28.642+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:02.335442+0000 front 2024-04-21T08:52:02.335482+0000 (oldest deadline 2024-04-21T08:52:27.035239+0000) 2024-04-21T08:52:29.001 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:28 smithi094 podman[99173]: 2024-04-21 08:52:28.728007737 +0000 UTC m=+5.838074664 container cleanup 11109b4ec88a4e9562d18d89050753644932bad8ad4e41dc25228600daa68a43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-21T08:52:29.001 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:28 smithi094 bash[99173]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9 2024-04-21T08:52:29.260 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:29 smithi094 podman[99193]: 2024-04-21 08:52:29.002365296 +0000 UTC m=+0.639151291 container remove 11109b4ec88a4e9562d18d89050753644932bad8ad4e41dc25228600daa68a43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:52:29.573 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:29 smithi094 podman[99286]: 2024-04-21 08:52:29.202028256 +0000 UTC m=+0.017890434 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:52:29.573 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:29 smithi094 podman[99286]: 2024-04-21 08:52:29.324152415 +0000 UTC m=+0.140014634 container create 816d3465389cb9b44eef07d65c969e083cec35608a1808663ffc47174c3dd526 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T08:52:29.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:29 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:29.419+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:29.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:29 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:29.419+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:29.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:29 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:29.419+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:29.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:29 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:29.419+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:29.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:29 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:29.419+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:29.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:29 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:29.419+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:29.869 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:29 smithi094 podman[99286]: 2024-04-21 08:52:29.574100496 +0000 UTC m=+0.389962677 container init 816d3465389cb9b44eef07d65c969e083cec35608a1808663ffc47174c3dd526 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:52:29.869 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:29 smithi094 podman[99286]: 2024-04-21 08:52:29.577354712 +0000 UTC m=+0.393216899 container start 816d3465389cb9b44eef07d65c969e083cec35608a1808663ffc47174c3dd526 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:52:29.869 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:29 smithi094 podman[99286]: 2024-04-21 08:52:29.657147613 +0000 UTC m=+0.473009808 container attach 816d3465389cb9b44eef07d65c969e083cec35608a1808663ffc47174c3dd526 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=) 2024-04-21T08:52:29.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:29 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:29.683+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:29.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:29 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:29.683+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:29.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:29 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:29.683+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:29.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:29 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:29.683+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:29.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:29 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:29.683+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:29.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:29 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:29.683+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:29.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:29 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:29.683+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:02.335442+0000 front 2024-04-21T08:52:02.335482+0000 (oldest deadline 2024-04-21T08:52:27.035239+0000) 2024-04-21T08:52:30.159 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 21 08:52:29 smithi094 podman[99286]: 2024-04-21 08:52:29.870306455 +0000 UTC m=+0.686168623 container died 816d3465389cb9b44eef07d65c969e083cec35608a1808663ffc47174c3dd526 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-21T08:52:30.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.9.service' 2024-04-21T08:52:30.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:52:30.474 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-21T08:52:30.474 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-21T08:52:30.474 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.10 2024-04-21T08:52:30.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:30 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:30.396+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:30.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:30 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:30.396+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:30.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:30 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:30.396+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:30.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:30 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:30.396+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:30.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:30 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:30.396+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:30.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:30 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:30.396+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 systemd[1]: Stopping Ceph osd.10 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:30.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:30.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:30.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:30.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:30.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:30.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:30.695+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:02.335442+0000 front 2024-04-21T08:52:02.335482+0000 (oldest deadline 2024-04-21T08:52:27.035239+0000) 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:30.798+0000 7fba19366700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:30.798+0000 7fba19366700 -1 osd.10 106 *** Got signal Terminated *** 2024-04-21T08:52:30.799 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:30 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:30.799+0000 7fba19366700 -1 osd.10 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:52:31.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:31 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:31.436+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:31.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:31 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:31.436+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:31.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:31 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:31.436+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:31.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:31 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:31.436+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:31.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:31 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:31.436+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:31.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:31 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:31.436+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:31.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:31 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:31.657+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:31.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:31 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:31.657+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:31.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:31 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:31.657+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:31.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:31 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:31.657+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:31.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:31 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:31.657+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:31.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:31 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:31.657+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:31.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:31 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:31.657+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:02.335442+0000 front 2024-04-21T08:52:02.335482+0000 (oldest deadline 2024-04-21T08:52:27.035239+0000) 2024-04-21T08:52:32.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:32 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:32.450+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:32.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:32 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:32.450+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:32.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:32 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:32.450+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:32.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:32 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:32.450+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:32.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:32 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:32.450+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:32.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:32 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:32.450+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:32.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:32 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:32.675+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:32.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:32 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:32.675+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:32.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:32 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:32.675+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:32.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:32 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:32.675+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:32.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:32 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:32.675+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:32.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:32 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:32.675+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:32.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:32 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:32.675+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:02.335442+0000 front 2024-04-21T08:52:02.335482+0000 (oldest deadline 2024-04-21T08:52:27.035239+0000) 2024-04-21T08:52:33.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:33 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:33.477+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:33.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:33 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:33.477+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:33.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:33 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:33.477+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:33.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:33 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:33.477+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:33.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:33 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:33.477+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:33.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:33 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:33.477+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:33.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:33 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:33.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:33.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:33 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:33.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:33.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:33 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:33.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:33.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:33 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:33.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:33.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:33 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:33.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:33.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:33 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:33.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:33.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:33 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:33.680+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:02.335442+0000 front 2024-04-21T08:52:02.335482+0000 (oldest deadline 2024-04-21T08:52:27.035239+0000) 2024-04-21T08:52:34.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:34 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:34.439+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:34.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:34 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:34.439+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:34.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:34 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:34.439+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:34.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:34 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:34.439+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:34.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:34 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:34.439+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:34.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:34 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:34.439+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:34.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:34 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:34.439+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T08:52:10.708520+0000 front 2024-04-21T08:52:10.708469+0000 (oldest deadline 2024-04-21T08:52:33.608384+0000) 2024-04-21T08:52:35.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:34 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:34.718+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:35.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:34 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:34.718+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:35.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:34 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:34.718+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:35.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:34 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:34.718+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:35.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:34 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:34.718+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:35.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:34 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:34.718+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:35.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:34 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:34.718+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:02.335442+0000 front 2024-04-21T08:52:02.335482+0000 (oldest deadline 2024-04-21T08:52:27.035239+0000) 2024-04-21T08:52:35.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:35 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:35.426+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:35.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:35 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:35.426+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:35.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:35 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:35.426+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:35.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:35 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:35.426+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:35.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:35 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:35.426+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:35.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:35 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:35.426+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:35.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:35 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:35.426+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T08:52:10.708520+0000 front 2024-04-21T08:52:10.708469+0000 (oldest deadline 2024-04-21T08:52:33.608384+0000) 2024-04-21T08:52:36.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:35 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:35.698+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:14.027659+0000 front 2024-04-21T08:51:14.027562+0000 (oldest deadline 2024-04-21T08:51:38.727732+0000) 2024-04-21T08:52:36.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:35 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:35.698+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T08:51:22.828813+0000 front 2024-04-21T08:51:22.828875+0000 (oldest deadline 2024-04-21T08:51:45.128971+0000) 2024-04-21T08:52:36.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:35 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:35.698+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.429906+0000 front 2024-04-21T08:51:30.429924+0000 (oldest deadline 2024-04-21T08:51:53.329670+0000) 2024-04-21T08:52:36.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:35 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:35.699+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:39.230908+0000 front 2024-04-21T08:51:39.230804+0000 (oldest deadline 2024-04-21T08:52:04.530673+0000) 2024-04-21T08:52:36.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:35 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:35.699+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:46.231516+0000 front 2024-04-21T08:51:46.231559+0000 (oldest deadline 2024-04-21T08:52:11.432366+0000) 2024-04-21T08:52:36.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:35 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:35.699+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:51.432944+0000 front 2024-04-21T08:51:51.432964+0000 (oldest deadline 2024-04-21T08:52:16.633219+0000) 2024-04-21T08:52:36.188 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:35 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10[56755]: 2024-04-21T08:52:35.699+0000 7fba11952700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:02.335442+0000 front 2024-04-21T08:52:02.335482+0000 (oldest deadline 2024-04-21T08:52:27.035239+0000) 2024-04-21T08:52:36.532 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:36 smithi111 podman[68323]: 2024-04-21 08:52:36.245294816 +0000 UTC m=+5.572210148 container died e7a41661a14a937a272d94178530bca68e985ad7e3bff5ff2bf3ced17ad96135 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:52:36.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:36 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:36.377+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:36.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:36 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:36.377+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:36.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:36 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:36.377+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:36.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:36 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:36.377+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:36.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:36 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:36.377+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:36.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:36 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:36.377+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:36.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:36 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:36.377+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T08:52:10.708520+0000 front 2024-04-21T08:52:10.708469+0000 (oldest deadline 2024-04-21T08:52:33.608384+0000) 2024-04-21T08:52:36.857 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:36 smithi111 podman[68323]: 2024-04-21 08:52:36.624240231 +0000 UTC m=+5.951155562 container cleanup e7a41661a14a937a272d94178530bca68e985ad7e3bff5ff2bf3ced17ad96135 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:52:36.857 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:36 smithi111 bash[68323]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10 2024-04-21T08:52:37.167 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:36 smithi111 podman[68342]: 2024-04-21 08:52:36.858499863 +0000 UTC m=+0.608267558 container remove e7a41661a14a937a272d94178530bca68e985ad7e3bff5ff2bf3ced17ad96135 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-21T08:52:37.437 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:37 smithi111 podman[68430]: 2024-04-21 08:52:37.167986838 +0000 UTC m=+0.101204347 container create 20cb2ca419751b7def783b699717c212dd003c361437b17d3a9f20c6ab6dabcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:52:37.438 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:37 smithi111 podman[68430]: 2024-04-21 08:52:37.08440081 +0000 UTC m=+0.017618296 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:52:37.438 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:37 smithi111 podman[68430]: 2024-04-21 08:52:37.370424793 +0000 UTC m=+0.303642260 container init 20cb2ca419751b7def783b699717c212dd003c361437b17d3a9f20c6ab6dabcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-21T08:52:37.438 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:37 smithi111 podman[68430]: 2024-04-21 08:52:37.373485175 +0000 UTC m=+0.306702643 container start 20cb2ca419751b7def783b699717c212dd003c361437b17d3a9f20c6ab6dabcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-21T08:52:37.714 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:37 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:37.385+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:37.714 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:37 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:37.385+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:37.714 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:37 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:37.385+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:37.715 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:37 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:37.385+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:37.715 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:37 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:37.385+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:37.715 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:37 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:37.385+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:37.715 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:37 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:37.385+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T08:52:10.708520+0000 front 2024-04-21T08:52:10.708469+0000 (oldest deadline 2024-04-21T08:52:33.608384+0000) 2024-04-21T08:52:37.922 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:37 smithi111 podman[68430]: 2024-04-21 08:52:37.445075407 +0000 UTC m=+0.378292875 container attach 20cb2ca419751b7def783b699717c212dd003c361437b17d3a9f20c6ab6dabcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-21T08:52:37.922 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 08:52:37 smithi111 podman[68456]: 2024-04-21 08:52:37.690505106 +0000 UTC m=+0.018315407 container died 20cb2ca419751b7def783b699717c212dd003c361437b17d3a9f20c6ab6dabcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-21T08:52:38.204 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.10.service' 2024-04-21T08:52:38.244 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:52:38.244 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-21T08:52:38.244 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-21T08:52:38.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.11 2024-04-21T08:52:38.573 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 systemd[1]: Stopping Ceph osd.11 for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:52:38.573 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:38.380+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:38.573 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:38.380+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:38.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:38.380+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:38.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:38.380+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:38.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:38.380+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:38.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:38.380+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:38.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:38.380+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T08:52:10.708520+0000 front 2024-04-21T08:52:10.708469+0000 (oldest deadline 2024-04-21T08:52:33.608384+0000) 2024-04-21T08:52:38.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:38.573+0000 7fa40a126700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:52:38.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:38.573+0000 7fa40a126700 -1 osd.11 106 *** Got signal Terminated *** 2024-04-21T08:52:38.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:38 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:38.573+0000 7fa40a126700 -1 osd.11 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:52:39.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:39 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:39.390+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:39.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:39 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:39.390+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:39.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:39 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:39.390+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:39.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:39 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:39.390+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:39.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:39 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:39.390+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:39.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:39 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:39.390+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:39.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:39 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:39.390+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T08:52:10.708520+0000 front 2024-04-21T08:52:10.708469+0000 (oldest deadline 2024-04-21T08:52:33.608384+0000) 2024-04-21T08:52:40.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:40.395+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:40.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:40.395+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:40.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:40.395+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:40.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:40.395+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:40.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:40.395+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:40.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:40.395+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:40.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:40 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:40.395+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T08:52:10.708520+0000 front 2024-04-21T08:52:10.708469+0000 (oldest deadline 2024-04-21T08:52:33.608384+0000) 2024-04-21T08:52:41.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:41.429+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:41.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:41.429+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:41.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:41.429+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:41.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:41.429+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:41.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:41.429+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:41.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:41.429+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:41.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:41 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:41.429+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T08:52:10.708520+0000 front 2024-04-21T08:52:10.708469+0000 (oldest deadline 2024-04-21T08:52:33.608384+0000) 2024-04-21T08:52:42.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:42.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:42.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:42.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:42.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:42.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:42.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:42.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:42.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:42.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:42.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:42.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:42.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:42 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:42.438+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T08:52:10.708520+0000 front 2024-04-21T08:52:10.708469+0000 (oldest deadline 2024-04-21T08:52:33.608384+0000) 2024-04-21T08:52:43.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:43.437+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-21T08:51:12.697441+0000 front 2024-04-21T08:51:12.697555+0000 (oldest deadline 2024-04-21T08:51:38.597502+0000) 2024-04-21T08:52:43.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:43.437+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-21T08:51:30.101169+0000 front 2024-04-21T08:51:30.101043+0000 (oldest deadline 2024-04-21T08:51:54.201175+0000) 2024-04-21T08:52:43.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:43.437+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-21T08:51:38.302195+0000 front 2024-04-21T08:51:38.302313+0000 (oldest deadline 2024-04-21T08:52:02.402110+0000) 2024-04-21T08:52:43.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:43.437+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T08:51:47.604017+0000 front 2024-04-21T08:51:47.603907+0000 (oldest deadline 2024-04-21T08:52:08.703845+0000) 2024-04-21T08:52:43.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:43.437+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-21T08:51:55.705521+0000 front 2024-04-21T08:51:55.705487+0000 (oldest deadline 2024-04-21T08:52:16.805635+0000) 2024-04-21T08:52:43.761 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:43.437+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-21T08:52:03.107809+0000 front 2024-04-21T08:52:03.107909+0000 (oldest deadline 2024-04-21T08:52:24.807636+0000) 2024-04-21T08:52:43.762 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:43 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11[57142]: 2024-04-21T08:52:43.437+0000 7fa40671a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T08:52:10.708520+0000 front 2024-04-21T08:52:10.708469+0000 (oldest deadline 2024-04-21T08:52:33.608384+0000) 2024-04-21T08:52:44.339 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:44 smithi173 podman[69282]: 2024-04-21 08:52:44.080802007 +0000 UTC m=+5.636184424 container died b826371eeeb6a5b5b4108f41395d5183334c3804bd9ec386a42e476adbdf36f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-04-21T08:52:44.339 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:44 smithi173 podman[69282]: 2024-04-21 08:52:44.340232078 +0000 UTC m=+5.895614485 container cleanup b826371eeeb6a5b5b4108f41395d5183334c3804bd9ec386a42e476adbdf36f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-04-21T08:52:44.606 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:44 smithi173 bash[69282]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11 2024-04-21T08:52:44.962 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:44 smithi173 podman[69299]: 2024-04-21 08:52:44.607961871 +0000 UTC m=+0.521558896 container remove b826371eeeb6a5b5b4108f41395d5183334c3804bd9ec386a42e476adbdf36f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-21T08:52:45.235 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:44 smithi173 podman[69394]: 2024-04-21 08:52:44.866409692 +0000 UTC m=+0.017721308 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:52:45.235 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:44 smithi173 podman[69394]: 2024-04-21 08:52:44.975316846 +0000 UTC m=+0.126628498 container create 67f8081c2b342cf6a1814c00a043ebb4413c8a6ee14cbc3545e26b2c59962f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-21T08:52:45.235 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:45 smithi173 podman[69394]: 2024-04-21 08:52:45.161251614 +0000 UTC m=+0.312563228 container init 67f8081c2b342cf6a1814c00a043ebb4413c8a6ee14cbc3545e26b2c59962f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-21T08:52:45.235 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:45 smithi173 podman[69394]: 2024-04-21 08:52:45.16466533 +0000 UTC m=+0.315976937 container start 67f8081c2b342cf6a1814c00a043ebb4413c8a6ee14cbc3545e26b2c59962f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-21T08:52:45.235 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:45 smithi173 podman[69394]: 2024-04-21 08:52:45.235964834 +0000 UTC m=+0.387276451 container attach 67f8081c2b342cf6a1814c00a043ebb4413c8a6ee14cbc3545e26b2c59962f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:52:45.511 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:45 smithi173 podman[69394]: 2024-04-21 08:52:45.454616672 +0000 UTC m=+0.605928284 container died 67f8081c2b342cf6a1814c00a043ebb4413c8a6ee14cbc3545e26b2c59962f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-04-21T08:52:45.991 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 21 08:52:45 smithi173 podman[69420]: 2024-04-21 08:52:45.972711245 +0000 UTC m=+0.511065182 container remove 67f8081c2b342cf6a1814c00a043ebb4413c8a6ee14cbc3545e26b2c59962f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:52:45.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@osd.11.service' 2024-04-21T08:52:46.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:52:46.033 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-21T08:52:46.033 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-21T08:52:46.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.a 2024-04-21T08:52:46.387 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:52:46 smithi094 systemd[1]: Stopping Ceph mds.a for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:52:46.760 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:52:46 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-a[69463]: 2024-04-21T08:52:46.388+0000 7faa78050700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:52:46.761 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:52:46 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-a[69463]: 2024-04-21T08:52:46.388+0000 7faa78050700 -1 mds.a *** got signal Terminated *** 2024-04-21T08:52:47.862 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:52:47 smithi094 podman[99415]: 2024-04-21 08:52:47.59620327 +0000 UTC m=+1.369670500 container died a445d8ba8f2d7ebde8efeceb722063104d91a272d220033adb711f9340a88511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-21T08:52:48.246 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:52:47 smithi094 podman[99415]: 2024-04-21 08:52:47.970986554 +0000 UTC m=+1.744453790 container cleanup a445d8ba8f2d7ebde8efeceb722063104d91a272d220033adb711f9340a88511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:52:48.246 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 21 08:52:47 smithi094 bash[99415]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-a 2024-04-21T08:52:48.397 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.a.service' 2024-04-21T08:52:48.431 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:52:48.431 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-21T08:52:48.432 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-21T08:52:48.432 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.d 2024-04-21T08:52:48.760 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:52:48 smithi094 systemd[1]: Stopping Ceph mds.d for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:52:48.760 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:52:48 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-d[68506]: 2024-04-21T08:52:48.741+0000 7f34d5193700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:52:48.760 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:52:48 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-d[68506]: 2024-04-21T08:52:48.741+0000 7f34d5193700 -1 mds.d *** got signal Terminated *** 2024-04-21T08:52:52.260 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:52:52 smithi094 podman[99535]: 2024-04-21 08:52:51.999899318 +0000 UTC m=+3.353523969 container died a9482c9c542f18a93cbb731de240e387928ecc804a66153a638a17f0805fc25b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-21T08:52:52.703 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:52:52 smithi094 podman[99535]: 2024-04-21 08:52:52.349154043 +0000 UTC m=+3.702778703 container cleanup a9482c9c542f18a93cbb731de240e387928ecc804a66153a638a17f0805fc25b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-21T08:52:52.703 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:52:52 smithi094 bash[99535]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-d 2024-04-21T08:52:52.703 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 21 08:52:52 smithi094 podman[99550]: 2024-04-21 08:52:52.500215512 +0000 UTC m=+0.492651822 container remove a9482c9c542f18a93cbb731de240e387928ecc804a66153a638a17f0805fc25b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:52:52.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.d.service' 2024-04-21T08:52:52.751 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:52:52.751 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-21T08:52:52.751 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-21T08:52:52.751 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.g 2024-04-21T08:52:53.010 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:52:52 smithi094 systemd[1]: Stopping Ceph mds.g for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:52:53.510 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:52:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-g[69100]: 2024-04-21T08:52:53.102+0000 7fb2eee7b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:52:53.511 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:52:53 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-g[69100]: 2024-04-21T08:52:53.102+0000 7fb2eee7b700 -1 mds.g *** got signal Terminated *** 2024-04-21T08:52:57.010 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:52:56 smithi094 podman[99656]: 2024-04-21 08:52:56.751944512 +0000 UTC m=+3.773688160 container died b8094153e360ba799837f82256ceb499e7753ef39deadc50cacb3064ae425e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-21T08:52:57.394 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:52:57 smithi094 podman[99656]: 2024-04-21 08:52:57.136299308 +0000 UTC m=+4.158042955 container cleanup b8094153e360ba799837f82256ceb499e7753ef39deadc50cacb3064ae425e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:52:57.394 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 21 08:52:57 smithi094 bash[99656]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-g 2024-04-21T08:52:57.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.g.service' 2024-04-21T08:52:57.555 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:52:57.555 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-21T08:52:57.555 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-21T08:52:57.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.j 2024-04-21T08:52:57.760 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:52:57 smithi094 systemd[1]: Stopping Ceph mds.j for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:52:58.260 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:52:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-j[68150]: 2024-04-21T08:52:57.872+0000 7f997fd70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:52:58.260 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:52:57 smithi094 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-j[68150]: 2024-04-21T08:52:57.872+0000 7f997fd70700 -1 mds.j *** got signal Terminated *** 2024-04-21T08:53:01.364 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:53:01 smithi094 podman[99777]: 2024-04-21 08:53:01.068784734 +0000 UTC m=+3.299250407 container died fc928bce4bb99a605fb4b2cb3617b38b6e2a93e71d234c9bb08b1f4fa01067b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2) 2024-04-21T08:53:01.656 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:53:01 smithi094 podman[99777]: 2024-04-21 08:53:01.464498844 +0000 UTC m=+3.694964515 container cleanup fc928bce4bb99a605fb4b2cb3617b38b6e2a93e71d234c9bb08b1f4fa01067b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0) 2024-04-21T08:53:01.656 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 21 08:53:01 smithi094 bash[99777]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-j 2024-04-21T08:53:01.882 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.j.service' 2024-04-21T08:53:01.912 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:53:01.913 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-21T08:53:01.913 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-21T08:53:01.913 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.b 2024-04-21T08:53:02.348 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:53:01 smithi111 systemd[1]: Stopping Ceph mds.b for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:53:02.605 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:53:02 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b[61988]: 2024-04-21T08:53:02.348+0000 7f24ea6dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:53:02.605 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:53:02 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b[61988]: 2024-04-21T08:53:02.348+0000 7f24ea6dc700 -1 mds.b *** got signal Terminated *** 2024-04-21T08:53:06.688 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:53:06 smithi111 podman[68536]: 2024-04-21 08:53:06.355318343 +0000 UTC m=+4.252351122 container died e1ed5b9cf8df733d6c79e456121e8acd6e7f64c9c6da986e8e591be4c2efe962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:53:06.991 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:53:06 smithi111 podman[68536]: 2024-04-21 08:53:06.790275871 +0000 UTC m=+4.687308650 container cleanup e1ed5b9cf8df733d6c79e456121e8acd6e7f64c9c6da986e8e591be4c2efe962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-21T08:53:06.991 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 08:53:06 smithi111 bash[68536]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-b 2024-04-21T08:53:07.200 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.b.service' 2024-04-21T08:53:07.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:53:07.239 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-21T08:53:07.239 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-21T08:53:07.239 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.e 2024-04-21T08:53:07.578 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:53:07 smithi111 systemd[1]: Stopping Ceph mds.e for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:53:07.938 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:53:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e[61372]: 2024-04-21T08:53:07.576+0000 7faecc52a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:53:07.938 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:53:07 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e[61372]: 2024-04-21T08:53:07.576+0000 7faecc52a700 -1 mds.e *** got signal Terminated *** 2024-04-21T08:53:11.345 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:53:10 smithi111 podman[68668]: 2024-04-21 08:53:10.969104116 +0000 UTC m=+3.515260653 container died 53f3a3b8425cfa5fc05f3ede1c778bddfddac59f48e7066126f58e40e1f3ebc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-21T08:53:11.604 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:53:11 smithi111 podman[68668]: 2024-04-21 08:53:11.4460528 +0000 UTC m=+3.992209336 container cleanup 53f3a3b8425cfa5fc05f3ede1c778bddfddac59f48e7066126f58e40e1f3ebc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-21T08:53:11.604 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 08:53:11 smithi111 bash[68668]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-e 2024-04-21T08:53:11.810 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.e.service' 2024-04-21T08:53:11.842 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:53:11.842 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-21T08:53:11.842 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-21T08:53:11.843 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.h 2024-04-21T08:53:12.188 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:53:11 smithi111 systemd[1]: Stopping Ceph mds.h for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:53:12.687 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:53:12 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h[61684]: 2024-04-21T08:53:12.207+0000 7fa21895a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:53:12.688 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:53:12 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h[61684]: 2024-04-21T08:53:12.207+0000 7fa21895a700 -1 mds.h *** got signal Terminated *** 2024-04-21T08:53:15.249 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:53:14 smithi111 podman[68788]: 2024-04-21 08:53:14.975735931 +0000 UTC m=+2.920818097 container died e79d7014fd23efce0643c28bd344793f6aa97bffcd9c6cc46ae75a509deb6605 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-21T08:53:15.607 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:53:15 smithi111 podman[68788]: 2024-04-21 08:53:15.349405314 +0000 UTC m=+3.294487485 container cleanup e79d7014fd23efce0643c28bd344793f6aa97bffcd9c6cc46ae75a509deb6605 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:53:15.608 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 08:53:15 smithi111 bash[68788]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-h 2024-04-21T08:53:15.779 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.h.service' 2024-04-21T08:53:15.810 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:53:15.811 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-21T08:53:15.811 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-21T08:53:15.811 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.k 2024-04-21T08:53:16.135 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:53:15 smithi111 systemd[1]: Stopping Ceph mds.k for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:53:16.437 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:53:16 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-k[61053]: 2024-04-21T08:53:16.135+0000 7f03991b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:53:16.438 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:53:16 smithi111 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-k[61053]: 2024-04-21T08:53:16.135+0000 7f03991b9700 -1 mds.k *** got signal Terminated *** 2024-04-21T08:53:18.544 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:53:18 smithi111 podman[68908]: 2024-04-21 08:53:18.249985866 +0000 UTC m=+2.224313059 container died 85af0acb4c4a92af54a3cca7cd68e45205c643c2be86d053f3c8f7489a15c46d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-k, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True) 2024-04-21T08:53:18.853 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:53:18 smithi111 podman[68908]: 2024-04-21 08:53:18.645334763 +0000 UTC m=+2.619661998 container cleanup 85af0acb4c4a92af54a3cca7cd68e45205c643c2be86d053f3c8f7489a15c46d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-21T08:53:18.853 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 08:53:18 smithi111 bash[68908]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-k 2024-04-21T08:53:18.999 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.k.service' 2024-04-21T08:53:19.029 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:53:19.030 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-21T08:53:19.030 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-21T08:53:19.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.c 2024-04-21T08:53:19.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:53:19 smithi173 systemd[1]: Stopping Ceph mds.c for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:53:19.587 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:53:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c[62571]: 2024-04-21T08:53:19.321+0000 7fd9fd09d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:53:19.587 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:53:19 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c[62571]: 2024-04-21T08:53:19.321+0000 7fd9fd09d700 -1 mds.c *** got signal Terminated *** 2024-04-21T08:53:23.261 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:53:22 smithi173 podman[69498]: 2024-04-21 08:53:22.973277921 +0000 UTC m=+3.752145304 container died a35cf0a58fae7f0822f92a165df06a9ffd3ae50dc3d39a43261b65a435971473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:53:23.713 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:53:23 smithi173 podman[69498]: 2024-04-21 08:53:23.438508109 +0000 UTC m=+4.217375489 container cleanup a35cf0a58fae7f0822f92a165df06a9ffd3ae50dc3d39a43261b65a435971473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:53:23.714 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 21 08:53:23 smithi173 bash[69498]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-c 2024-04-21T08:53:23.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.c.service' 2024-04-21T08:53:23.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:53:23.904 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-21T08:53:23.904 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-21T08:53:23.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.f 2024-04-21T08:53:24.261 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:53:23 smithi173 systemd[1]: Stopping Ceph mds.f for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:53:24.761 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:53:24 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f[62263]: 2024-04-21T08:53:24.366+0000 7fe709825700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:53:24.761 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:53:24 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f[62263]: 2024-04-21T08:53:24.366+0000 7fe709825700 -1 mds.f *** got signal Terminated *** 2024-04-21T08:53:27.261 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:53:26 smithi173 podman[69630]: 2024-04-21 08:53:26.960612154 +0000 UTC m=+2.829000902 container died da02ff2d0f1c78a0e80c22c77fb04adca65d8c33896fd7d001802142cdb303c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-21T08:53:27.584 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:53:27 smithi173 podman[69630]: 2024-04-21 08:53:27.3841445 +0000 UTC m=+3.252533302 container cleanup da02ff2d0f1c78a0e80c22c77fb04adca65d8c33896fd7d001802142cdb303c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-21T08:53:27.584 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:53:27 smithi173 bash[69630]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f 2024-04-21T08:53:27.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.f.service' 2024-04-21T08:53:27.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:53:27 smithi173 podman[69645]: 2024-04-21 08:53:27.585272699 +0000 UTC m=+0.615786306 container remove da02ff2d0f1c78a0e80c22c77fb04adca65d8c33896fd7d001802142cdb303c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-21T08:53:27.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:53:27 smithi173 systemd[1]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.f.service: Deactivated successfully. 2024-04-21T08:53:27.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:53:27 smithi173 systemd[1]: Stopped Ceph mds.f for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:53:27.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 21 08:53:27 smithi173 systemd[1]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.f.service: Consumed 2.195s CPU time. 2024-04-21T08:53:27.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:53:27.847 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-21T08:53:27.847 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-21T08:53:27.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.i 2024-04-21T08:53:28.179 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:53:27 smithi173 systemd[1]: Stopping Ceph mds.i for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:53:28.511 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:53:28 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i[62891]: 2024-04-21T08:53:28.179+0000 7f101f547700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:53:28.511 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:53:28 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i[62891]: 2024-04-21T08:53:28.179+0000 7f101f547700 -1 mds.i *** got signal Terminated *** 2024-04-21T08:53:31.255 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:53:30 smithi173 podman[69754]: 2024-04-21 08:53:30.976307478 +0000 UTC m=+2.909005162 container died 63bb39d51f32adcf0f761bdbded302d32215fd8d9a4768aa917939bfbef7c096 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True) 2024-04-21T08:53:31.511 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:53:31 smithi173 podman[69754]: 2024-04-21 08:53:31.354764355 +0000 UTC m=+3.287462091 container cleanup 63bb39d51f32adcf0f761bdbded302d32215fd8d9a4768aa917939bfbef7c096 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:53:31.511 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 21 08:53:31 smithi173 bash[69754]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-i 2024-04-21T08:53:31.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.i.service' 2024-04-21T08:53:31.812 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:53:31.812 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-21T08:53:31.812 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-21T08:53:31.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.l 2024-04-21T08:53:32.261 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:53:31 smithi173 systemd[1]: Stopping Ceph mds.l for 2518ed94-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:53:32.261 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:53:32 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-l[61954]: 2024-04-21T08:53:32.116+0000 7f28861ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:53:32.261 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:53:32 smithi173 ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-l[61954]: 2024-04-21T08:53:32.116+0000 7f28861ca700 -1 mds.l *** got signal Terminated *** 2024-04-21T08:53:35.761 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:53:35 smithi173 podman[69876]: 2024-04-21 08:53:35.493482622 +0000 UTC m=+3.467929657 container died ed554a819eb2b7acfa1c2245be2d476bc2d95071b4c6786b92485e3fd2a12243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-21T08:53:36.299 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:53:36 smithi173 podman[69876]: 2024-04-21 08:53:36.107815135 +0000 UTC m=+4.082262166 container cleanup ed554a819eb2b7acfa1c2245be2d476bc2d95071b4c6786b92485e3fd2a12243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416) 2024-04-21T08:53:36.299 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:53:36 smithi173 bash[69876]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-l 2024-04-21T08:53:36.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.l.service' 2024-04-21T08:53:36.554 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:53:36 smithi173 podman[69888]: 2024-04-21 08:53:36.300293429 +0000 UTC m=+0.801685868 container remove ed554a819eb2b7acfa1c2245be2d476bc2d95071b4c6786b92485e3fd2a12243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-2518ed94-ffb9-11ee-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:53:36.554 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:53:36 smithi173 systemd[1]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.l.service: Deactivated successfully. 2024-04-21T08:53:36.554 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:53:36 smithi173 systemd[1]: Stopped Ceph mds.l for 2518ed94-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:53:36.554 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 21 08:53:36 smithi173 systemd[1]: ceph-2518ed94-ffb9-11ee-bc93-c7b262605968@mds.l.service: Consumed 1.110s CPU time. 2024-04-21T08:53:36.564 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:53:36.565 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-21T08:53:36.565 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T08:53:36.752 INFO:teuthology.orchestra.run.smithi094.stdout:Deleting cluster with fsid: 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:53:39.129 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T08:53:39.318 INFO:teuthology.orchestra.run.smithi111.stdout:Deleting cluster with fsid: 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:53:41.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T08:53:41.975 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:53:44.276 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:53:44.307 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:53:44.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:53:44.363 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-21T08:53:44.366 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512/remote/smithi094/crash 2024-04-21T08:53:44.367 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/crash -- . 2024-04-21T08:53:44.402 INFO:teuthology.orchestra.run.smithi094.stderr:tar: /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T08:53:44.402 INFO:teuthology.orchestra.run.smithi094.stderr:tar: Error is not recoverable: exiting now 2024-04-21T08:53:44.404 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512/remote/smithi111/crash 2024-04-21T08:53:44.405 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/crash -- . 2024-04-21T08:53:44.449 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T08:53:44.449 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now 2024-04-21T08:53:44.451 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512/remote/smithi173/crash 2024-04-21T08:53:44.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/crash -- . 2024-04-21T08:53:44.489 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T08:53:44.489 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-04-21T08:53:44.490 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-21T08:53:44.490 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-21T08:53:44.524 INFO:tasks.cephadm:Compressing logs... 2024-04-21T08:53:44.524 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T08:53:44.566 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T08:53:44.569 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T08:53:44.589 INFO:teuthology.orchestra.run.smithi094.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.log 2024-04-21T08:53:44.590 INFO:teuthology.orchestra.run.smithi094.stderr:: No such file or directory 2024-04-21T08:53:44.590 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T08:53:44.590 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.g.log 2024-04-21T08:53:44.590 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.j.log 2024-04-21T08:53:44.594 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.3.log 2024-04-21T08:53:44.594 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.0.log 2024-04-21T08:53:44.595 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mgr.x.log 2024-04-21T08:53:44.595 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mon.a.log 2024-04-21T08:53:44.595 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.6.log 2024-04-21T08:53:44.595 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mon.a.log: 87.3% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T08:53:44.595 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T08:53:44.595 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.6.log: 90.2% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T08:53:44.595 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.a.log 2024-04-21T08:53:44.595 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log: 94.6% 84.3% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T08:53:44.595 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-21T08:53:44.595 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T08:53:44.599 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.d.log 2024-04-21T08:53:44.599 INFO:teuthology.orchestra.run.smithi094.stderr: 94.6%/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.a.log: -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-21T08:53:44.599 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.9.log 2024-04-21T08:53:44.606 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-21T08:53:44.606 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T08:53:44.606 INFO:teuthology.orchestra.run.smithi094.stderr: 94.7% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-21T08:53:44.606 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-21T08:53:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.log 2024-04-21T08:53:44.784 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-21T08:53:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T08:53:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.7.log 2024-04-21T08:53:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.log: /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.b.log 2024-04-21T08:53:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.4.log 2024-04-21T08:53:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.1.log 2024-04-21T08:53:44.785 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mgr.y.log 2024-04-21T08:53:44.785 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.h.log 2024-04-21T08:53:44.785 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mgr.y.log: 87.4%gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.10.log 2024-04-21T08:53:44.785 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T08:53:44.785 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T08:53:44.785 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.h.log: /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.10.log: 90.4% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T08:53:44.785 INFO:teuthology.orchestra.run.smithi111.stderr: 91.9% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-21T08:53:44.791 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cephadm.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T08:53:44.791 INFO:teuthology.orchestra.run.smithi094.stderr: 91.1% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T08:53:44.791 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-21T08:53:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.log 2024-04-21T08:53:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-04-21T08:53:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.2.log 2024-04-21T08:53:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.f.log 2024-04-21T08:53:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.log: /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T08:53:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.5.log 2024-04-21T08:53:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.c.log 2024-04-21T08:53:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mon.c.log 2024-04-21T08:53:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mgr.z.log 2024-04-21T08:53:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.i.log 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mgr.z.log: 87.4% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.i.log: 90.4% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.11.log 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log: 84.1% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.8.log 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.11.log: 91.9% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.l.log 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 94.6% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr: 91.0% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T08:53:44.799 INFO:teuthology.orchestra.run.smithi173.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T08:53:44.800 INFO:teuthology.orchestra.run.smithi173.stderr: 92.9% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-21T08:53:44.800 INFO:teuthology.orchestra.run.smithi173.stderr: 94.6% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-21T08:53:44.801 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.e.log 2024-04-21T08:53:44.801 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T08:53:44.801 INFO:teuthology.orchestra.run.smithi111.stderr: 84.1% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T08:53:44.801 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mon.b.log 2024-04-21T08:53:44.801 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.k.log 2024-04-21T08:53:44.801 INFO:teuthology.orchestra.run.smithi111.stderr: 91.0% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-volume.log.gz/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mon.b.log: 2024-04-21T08:53:44.801 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T08:53:44.802 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.k.log: 94.5% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-21T08:53:44.802 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cephadm.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T08:53:44.802 INFO:teuthology.orchestra.run.smithi111.stderr: 92.8% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-21T08:53:44.802 INFO:teuthology.orchestra.run.smithi111.stderr: 94.4% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-21T08:53:44.845 INFO:teuthology.orchestra.run.smithi094.stderr: 90.3% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-21T08:53:44.876 INFO:teuthology.orchestra.run.smithi173.stderr: 96.4% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-21T08:53:44.994 INFO:teuthology.orchestra.run.smithi173.stderr: 92.0% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-21T08:53:45.040 INFO:teuthology.orchestra.run.smithi111.stderr: 92.0% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-21T08:53:45.544 INFO:teuthology.orchestra.run.smithi094.stderr: 90.6% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-21T08:53:46.496 INFO:teuthology.orchestra.run.smithi111.stderr: 94.8% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-21T08:53:49.512 INFO:teuthology.orchestra.run.smithi173.stderr: 92.4% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-21T08:53:49.829 INFO:teuthology.orchestra.run.smithi173.stderr: 92.6% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-21T08:53:49.937 INFO:teuthology.orchestra.run.smithi111.stderr: 92.6% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-21T08:53:50.534 INFO:teuthology.orchestra.run.smithi111.stderr: 92.6% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-21T08:53:51.104 INFO:teuthology.orchestra.run.smithi094.stderr: 92.7% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-21T08:53:51.769 INFO:teuthology.orchestra.run.smithi094.stderr: 92.6% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-21T08:53:51.799 INFO:teuthology.orchestra.run.smithi173.stderr: 92.5% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-21T08:53:52.344 INFO:teuthology.orchestra.run.smithi094.stderr: 92.5% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-21T08:53:52.711 INFO:teuthology.orchestra.run.smithi111.stderr: 92.5% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-21T08:53:52.958 INFO:teuthology.orchestra.run.smithi111.stderr: 92.5% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-21T08:53:52.960 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-21T08:53:52.961 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m8.377s 2024-04-21T08:53:52.961 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m29.028s 2024-04-21T08:53:52.961 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m0.985s 2024-04-21T08:53:54.052 INFO:teuthology.orchestra.run.smithi094.stderr: 92.4% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-21T08:53:54.054 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-04-21T08:53:54.054 INFO:teuthology.orchestra.run.smithi094.stderr:real 0m9.474s 2024-04-21T08:53:54.054 INFO:teuthology.orchestra.run.smithi094.stderr:user 0m31.018s 2024-04-21T08:53:54.054 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m1.103s 2024-04-21T08:53:54.621 INFO:teuthology.orchestra.run.smithi173.stderr: 92.4% -- replaced with /var/log/ceph/2518ed94-ffb9-11ee-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-21T08:53:54.622 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-04-21T08:53:54.623 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m10.040s 2024-04-21T08:53:54.623 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m27.227s 2024-04-21T08:53:54.623 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.954s 2024-04-21T08:53:54.623 INFO:tasks.cephadm:Archiving logs... 2024-04-21T08:53:54.623 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512/remote/smithi094/log 2024-04-21T08:53:54.624 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T08:53:59.673 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512/remote/smithi111/log 2024-04-21T08:53:59.674 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T08:54:04.262 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512/remote/smithi173/log 2024-04-21T08:54:04.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T08:54:08.966 INFO:tasks.cephadm:Removing cluster... 2024-04-21T08:54:08.966 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 --force 2024-04-21T08:54:09.161 INFO:teuthology.orchestra.run.smithi094.stdout:Deleting cluster with fsid: 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:54:09.491 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 --force 2024-04-21T08:54:09.681 INFO:teuthology.orchestra.run.smithi111.stdout:Deleting cluster with fsid: 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:54:10.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2518ed94-ffb9-11ee-bc93-c7b262605968 --force 2024-04-21T08:54:10.209 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 2518ed94-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:54:10.507 INFO:tasks.cephadm:Removing cephadm ... 2024-04-21T08:54:10.508 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T08:54:10.524 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T08:54:10.539 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T08:54:10.555 INFO:tasks.cephadm:Teardown complete 2024-04-21T08:54:10.555 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-21T08:54:10.569 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-21T08:54:10.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T08:54:10.572 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T08:54:10.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T08:54:10.673 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-21T08:54:10.685 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-21T08:54:10.685 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:54:10.687 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:54:10.689 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:54:10.699 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:54:10.701 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:54:10.702 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:54:10.702 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-04-21T08:54:10.703 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 25 -25us[ -25us] +/- 77ms 2024-04-21T08:54:10.703 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv02.front.sepia.ceph.com 3 7 377 29 -406us[ -400us] +/- 61ms 2024-04-21T08:54:10.703 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +548us[ +548us] +/- 82ms 2024-04-21T08:54:10.703 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:54:10.704 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:54:10.706 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:54:10.706 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-21T08:54:10.706 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 -13us[ -13us] +/- 76ms 2024-04-21T08:54:10.706 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -404us[ -408us] +/- 61ms 2024-04-21T08:54:10.707 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +568us[ +564us] +/- 82ms 2024-04-21T08:54:10.707 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:54:10.708 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:54:10.709 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-04-21T08:54:10.709 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 -13us[ -13us] +/- 76ms 2024-04-21T08:54:10.709 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -381us[ -375us] +/- 61ms 2024-04-21T08:54:10.709 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +558us[ +558us] +/- 82ms 2024-04-21T08:54:10.709 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:54:10.709 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-21T08:54:10.724 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-21T08:54:10.725 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-21T08:54:10.751 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T08:54:10.827 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T08:54:10.890 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T08:54:10.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:54:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:54:10.982 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:54:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:54:11.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:54:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:54:11.035 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-21T08:54:11.047 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-21T08:54:11.075 INFO:teuthology.task.internal:Duration was 1914.517230 seconds 2024-04-21T08:54:11.076 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-21T08:54:11.117 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-21T08:54:11.117 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T08:54:11.120 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T08:54:11.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T08:54:11.153 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:54:11.156 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:54:11.159 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:54:11.474 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-21T08:54:11.474 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-04-21T08:54:11.474 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T08:54:11.494 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-04-21T08:54:11.494 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T08:54:11.543 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-04-21T08:54:11.543 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T08:54:11.567 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-21T08:54:11.567 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T08:54:11.570 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T08:54:11.586 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T08:54:11.632 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-21T08:54:11.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T08:54:11.635 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T08:54:11.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T08:54:12.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-21T08:54:12.022 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-21T08:54:12.022 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T08:54:12.046 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T08:54:12.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T08:54:12.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-21T08:54:12.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T08:54:12.110 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T08:54:12.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T08:54:12.130 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-04-21T08:54:12.132 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-04-21T08:54:12.155 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-04-21T08:54:12.179 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T08:54:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:54:12.195 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T08:54:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:54:12.209 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T08:54:12.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:54:12.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-21T08:54:12.251 INFO:teuthology.task.internal:Transferring archived files... 2024-04-21T08:54:12.252 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512/remote/smithi094 2024-04-21T08:54:12.252 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T08:54:12.301 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512/remote/smithi111 2024-04-21T08:54:12.302 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T08:54:12.338 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512/remote/smithi173 2024-04-21T08:54:12.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T08:54:12.373 INFO:teuthology.task.internal:Removing archive directory... 2024-04-21T08:54:12.374 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T08:54:12.376 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T08:54:12.378 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T08:54:12.422 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-21T08:54:12.438 INFO:teuthology.task.internal:Not uploading archives. 2024-04-21T08:54:12.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-21T08:54:12.461 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-21T08:54:12.461 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T08:54:12.463 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T08:54:12.465 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T08:54:12.475 INFO:teuthology.orchestra.run.smithi094.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 08:54 /home/ubuntu/cephtest 2024-04-21T08:54:12.477 INFO:teuthology.orchestra.run.smithi111.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 08:54 /home/ubuntu/cephtest 2024-04-21T08:54:12.479 INFO:teuthology.orchestra.run.smithi173.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 08:54 /home/ubuntu/cephtest 2024-04-21T08:54:12.481 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-21T08:54:12.500 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-21T08:54:12.586 INFO:teuthology.nuke:Checking targets against current locks 2024-04-21T08:54:12.610 DEBUG:teuthology.nuke:shortname: smithi094 2024-04-21T08:54:12.611 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T08:54:12.637 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:16:23.096186', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNIFBuKaEXxjOb8QzHl0379u67RH2hUXtlQSyz+IhlA2fUAl9brqO3uP4Db7DxWwQxcl0KQ23aNOT2nUfs9Zs3U='} 2024-04-21T08:54:12.641 DEBUG:teuthology.nuke:shortname: smithi111 2024-04-21T08:54:12.642 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T08:54:12.664 DEBUG:teuthology.nuke:shortname: smithi173 2024-04-21T08:54:12.665 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T08:54:12.670 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:16:23.094181', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlL+im8GNPnIN8QL3FOltbdTy6qnnD72uwVmDgWrJ/hzNgIck0YuGjsYJDDbmih146ixgWH//ZAH7V+oDaFMDY='} 2024-04-21T08:54:12.695 INFO:teuthology.orchestra.console.smithi094:Power off 2024-04-21T08:54:12.695 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T08:54:12.709 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:16:23.097246', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB7GJmLqkmdcWAPzZ4FhfC/FlnOrpfoB1ofRRCSGeX0eL47SV5maah1HDYDCVJiX6TnNmpa9w8JLe11slWsC+3g='} 2024-04-21T08:54:12.721 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2024-04-21T08:54:12.721 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T08:54:12.745 INFO:teuthology.orchestra.console.smithi111:Power off 2024-04-21T08:54:12.745 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T08:54:12.755 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-04-21T08:54:12.768 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2024-04-21T08:54:12.768 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T08:54:12.783 INFO:teuthology.orchestra.console.smithi173:Power off 2024-04-21T08:54:12.783 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T08:54:12.791 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-04-21T08:54:12.804 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-04-21T08:54:12.804 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T08:54:12.825 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-21T08:54:16.756 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T08:54:16.792 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T08:54:16.826 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T08:54:16.867 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-04-21T08:54:16.903 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-04-21T08:54:16.935 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-21T08:54:20.870 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T08:54:20.904 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T08:54:20.936 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T08:54:20.981 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2024-04-21T08:54:21.014 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2024-04-21T08:54:21.046 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-04-21T08:54:21.083 INFO:teuthology.orchestra.console.smithi094:Power off completed 2024-04-21T08:54:21.115 INFO:teuthology.orchestra.console.smithi111:Power off completed 2024-04-21T08:54:21.148 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-04-21T08:54:21.254 INFO:teuthology.run:Summary data: description: fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iozone}} duration: 1914.517230272293 failure_reason: 'error during scrub thrashing: ' flavor: default owner: scheduled_leonidus@teuthology status: fail success: false 2024-04-21T08:54:21.254 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T08:54:21.339 INFO:teuthology.run:FAIL